What is this?
DMI is a decoupled, asynchronous observability stack that gives real-time access to a running LLM's internal state — residual streams, attention patterns, MLP outputs, KV-cache slices, logits — during inference or training, with minimal overhead.
DMI is a decoupled, asynchronous observability stack that gives real-time access to a running LLM's internal state — residual streams, attention patterns, MLP outputs, KV-cache slices, logits — during inference or training, with minimal overhead.Is this a standalone app or something I integrate into my own code?It is a Python toolkit you integrate into an existing model: you drop a HookPoint primitive into a PyTorch model (or use the thin HF/vLLM wrappers or the Megatron-LM training integration), plus an optional visual configurator and CLI for building capture configs.
It is a Python toolkit you integrate into an existing model: you drop a HookPoint primitive into a PyTorch model (or use the thin HF/vLLM wrappers or the Megatron-LM training integration), plus an optional visual configurator and CLI for building capture configs.How do I get it running?DMI currently only installs from source. You set up a separate environment and checkout for each backend you want (core, HuggingFace, vLLM, or Megatron-LM) and install only the integration you need.
DMI currently only installs from source. You set up a separate environment and checkout for each backend you want (core, HuggingFace, vLLM, or Megatron-LM) and install only the integration you need.Which inference or training backends does DMI plug into?It integrates with HuggingFace Transformers and vLLM for inference and with Megatron-LM for training; SGLang support is planned but not yet available.
It integrates with HuggingFace Transformers and vLLM for inference and with Megatron-LM for training; SGLang support is planned but not yet available.Is this ready for production use?No — it's explicitly labeled a research preview with APIs that may still change, and SGLang support is still on the roadmap.
No — it's explicitly labeled a research preview with APIs that may still change, and SGLang support is still on the roadmap.Who is this built for?Researchers and engineers debugging hallucinations in production, studying interpretability or activation steering, building speculative-decoding drafts, mining distillation data, or monitoring attention collapse during long generations.
Researchers and engineers debugging hallucinations in production, studying interpretability or activation steering, building speculative-decoding drafts, mining distillation data, or monitoring attention collapse during long generations.What models does DMI support?20+ dense and MoE model families including Qwen3, Llama, Gemma, Mistral, GPT-OSS, Phi, and Granite, with early support for newer architectures like Llama 4, DeepSeek V4 Flash, and GLM-5.2.
20+ dense and MoE model families including Qwen3, Llama, Gemma, Mistral, GPT-OSS, Phi, and Granite, with early support for newer architectures like Llama 4, DeepSeek V4 Flash, and GLM-5.2.What is Ring2 and why does it matter?Ring2 is DMI's GPU↔CPU co-designed staging mechanism: a GPU-side payload ring keeps captured tensors isolated from the KV-cache memory pool, while an on-host meta ring drains them asynchronously to avoid slowing inference.
Ring2 is DMI's GPU↔CPU co-designed staging mechanism: a GPU-side payload ring keeps captured tensors isolated from the KV-cache memory pool, while an on-host meta ring drains them asynchronously to avoid slowing inference.How does DMI compare to vLLM's own hidden-state extraction?On Qwen3-4B prefill, DMI captures about 13x more tensor data per token while keeping extraction overhead 15-17x lower than vLLM's native hidden-state saving at batch sizes 16-32.
On Qwen3-4B prefill, DMI captures about 13x more tensor data per token while keeping extraction overhead 15-17x lower than vLLM's native hidden-state saving at batch sizes 16-32.What is the DMI-configurator?It's an optional web UI (dmi ui) that turns a model plus a few clicks into a validated capture configuration and estimates the cost before you run anything; it needs the optional [ui] extra and, for resolving bare model directories or Hugging Face IDs, the [hf] extra.
It's an optional web UI (dmi ui) that turns a model plus a few clicks into a validated capture configuration and estimates the cost before you run anything; it needs the optional [ui] extra and, for resolving bare model directories or Hugging Face IDs, the [hf] extra.What are similar projects?Model-Optimizer, jeeves, minimind, MiniCPM, halo, StartLux-Decision, nano-vllm, lectures
Model-Optimizer, jeeves, minimind, MiniCPM, halo, StartLux-Decision, nano-vllm, lecturesWhich topics is it in?hallucination-detection, interpretability, knowledge-distillation, profiler, distributed-training, speculative-decoding, attention-mechanism, moe, jupyter-notebook, transformers, llm-inference, vllm
hallucination-detection, interpretability, knowledge-distillation, profiler, distributed-training, speculative-decoding, attention-mechanism, moe, jupyter-notebook, transformers, llm-inference, vllm