NVIDIA's Nemotron LLM yields 6.82 accepted tokens per step in speculative decoding without a separate draft model.

What Happened

NVIDIA has introduced Nemotron-Labs-Diffusion, a novel large language model architecture designed to drastically optimize inference speed through an advanced approach to speculative decoding. Unlike traditional speculative decoding setups that rely on a separate, smaller draft model to guess upcoming tokens, Nemotron-Labs-Diffusion utilizes a unified "tri-mode" architecture. This allows the model to handle both the drafting and verification phases natively, achieving an impressive 6.82 accepted tokens per step—more than double the 2.75 average of the current state-of-the-art, Eagle3.

Technical Details

Standard speculative decoding mitigates memory bandwidth bottlenecks during autoregressive generation by using a lightweight auxiliary model to generate candidate tokens, which the larger target model then verifies in parallel. However, maintaining two separate model weights in VRAM introduces memory overhead and complex KV-cache orchestration. Nemotron-Labs-Diffusion circumvents this by integrating the drafting mechanism directly into the primary model. By operating in three distinct modes, the model shares internal representations and its KV-cache. This deep integration allows the drafting mechanism to leverage the rich contextual embeddings of the primary model, resulting in highly accurate token predictions and the remarkable 6.82 acceptance rate per step without the secondary model tax.

Why It Matters

For inference engineers, memory bandwidth and VRAM capacity are the primary bottlenecks in scaling LLM deployments. Eliminating the auxiliary draft model frees up critical VRAM, allowing for larger batch sizes or longer context windows on the same hardware footprint. Furthermore, jumping from 2.75 to 6.82 accepted tokens per step translates to a massive reduction in latency for end-users, particularly in generation-heavy tasks like coding assistants or long-form content creation. It simplifies the deployment stack by removing the need to train, tune, and align a separate draft model for every target model update.

What to Watch Next

Keep an eye on how quickly NVIDIA integrates this tri-mode architecture into production inference engines like TensorRT-LLM, and whether community-driven frameworks like vLLM or TGI can support this pattern natively. Additionally, watch for other foundational model builders adopting self-speculative or multi-mode architectures to remain competitive on inference efficiency.