NVIDIA has released Nemotron-Labs-Diffusion, a tri-mode language model that combines autoregressive and diffusion capabilities into a single checkpoint, achieving up to 4x higher throughput in self-speculation mode on GB200 GPUs. The model is now available as open weights on Hugging Face.
TL;DR
- NVIDIA's new tri-mode model combines autoregressive and diffusion capabilities, eliminating the need for a separate draft model in speculative decoding.
- The 8B instruct variant of Nemotron-Labs-Diffusion achieves 4x higher throughput on NVIDIA's SPEED-Bench when deployed with the SGLang inference framework on a GB200 GPU.
- The model is available as open weights on Hugging Face under a commercial license, with inference support via vLLM and SGLang.
What happened
NVIDIA's research team has introduced Nemotron-Labs-Diffusion, a family of language models trained simultaneously on autoregressive (AR) and masked-diffusion objectives. The model is designed to switch between three generation modes at inference time without any weight changes.
The 8B instruct variant of the model decodes six times more tokens per forward pass than a comparable AR model and hits four times higher throughput on NVIDIA's SPEED-Bench in self-speculation mode when deployed with the SGLang inference framework on a GB200 GPU.
The model is available as open weights on Hugging Face under the NVIDIA Nemotron Open Model License, which permits commercial use. Inference support is available via vLLM and SGLang.
Why it matters
Nemotron-Labs-Diffusion addresses a structural engineering problem in large language models, where autoregressive models generate text sequentially, leaving GPUs idle at low concurrency. Diffusion models, while faster, have historically required more training data to match autoregressive quality.
By combining both paradigms, Nemotron-Labs-Diffusion offers a single model that can adapt to different workloads, switching between modes with a configuration change rather than an architectural one. This makes it an attractive option for new deployments choosing an inference stack.
However, the model requires trust_remote_code=True on load, as custom modeling code is bundled with the Hugging Face checkpoint. Additionally, the throughput figures have not been independently replicated on diverse hardware, and the theoretical quality guarantee that underpins traditional speculative decoding has not yet been established for this architecture.
Key facts
- Nemotron-Labs-Diffusion is a tri-mode language model that combines autoregressive and diffusion capabilities.
- The 8B instruct variant achieves 4x higher throughput on NVIDIA's SPEED-Bench in self-speculation mode on a GB200 GPU.
- The model is available as open weights on Hugging Face under a commercial license.
- Inference support is available via vLLM and SGLang.
- The model requires trust_remote_code=True on load and CUDA-capable GPU hardware.
- The model family scales across three sizes: 3B, 8B, and 14B, with base, instruction-tuned, and vision-language variants at each scale.
- The model is initialized from Mistral AI's Ministral3 base model and trained for roughly one trillion tokens on a purely autoregressive objective, followed by an additional 300 billion tokens on a joint objective.
- The model achieves 6.82 accepted tokens per draft step in self-speculation mode, compared to 2.75 for Eagle3 and 4.24 for Qwen3-9B-MTP.
Context
Large language models have traditionally generated text autoregressively, one token at a time. However, this approach can leave GPUs idle at low concurrency, limiting throughput. Diffusion models, which generate text in parallel, have been advancing rapidly as an alternative, but have historically required more training data to match autoregressive quality.
Speculative decoding, a technique formalized in 2022, works by running a small 'draft' model to propose candidate tokens and a large 'target' model to verify them in parallel. This can significantly improve throughput, but requires maintaining a separate trained component that must stay in GPU memory alongside the main model.
NVIDIA's Nemotron-Labs-Diffusion eliminates the need for a separate draft model by collapsing the draft and verify roles into a single set of weights. This offers a more efficient and adaptable solution for text generation, particularly at low concurrency.