license: cc-by-nc-4.0
language:
- en
library_name: pytorch
tags:
- diffusion-language-modeling
- text-generation
SDLLM-EsoLM-1.7B-Base
Model Overview
Description:
SDLLM-EsoLM-1.7B-Base is a 1.7B parameter language model combining autoregressive and masked diffusion language modeling objectives. This model is evaluated as part of the study "Scaling Beyond Masked Diffusion Language Models," which investigates diffusion-based approaches to language modeling.
This model is for research and development only.
License/Terms of Use:
SDLLM-EsoLM-1.7B-Base is released under the CC BY-NC 4.0 license. By using this model, you are agreeing to the terms and conditions of the license. This model is intended for research purposes only and may not be used for commercial applications.
Deployment Geography:
Global
Use Case:
SDLLM-EsoLM-1.7B-Base is intended for use by researchers studying diffusion-based language modeling approaches.
Release Date:
Hugging Face 09/02/2026 via https://huggingface.co/nvidia/SDLLM-EsoLM-1.7B-Base
References(s):
Sampling and likelihood evaluation
This repository includes a self-contained SDLLM inference and evaluation package.
Sampling
python sample.py nvidia/SDLLM-EsoLM-1.7B-Base --prompt "In a quiet village, the old clock tower rang at midnight, and" --num-samples 1
Use python sample.py --help for sampler overrides. With no prompt the
script performs unconditional generation.
Release defaults are: AR uses Gumbel-max ancestral sampling (temperature 1);
MDLM uses ancestral masked diffusion; EsoLM uses its first-hitting ancestral
sampler; and Duo uses its uniform-state posterior sampler. All diffusion
models use 1,024 reverse steps, top_p=1.0 (no nucleus filtering), greedy
final denoising, and token-sampling temperature 1.0. By default sampling
returns the complete 2,048-token continuation; use --max-new-tokens only to
crop displayed text.
Sampling uses compiled FlexAttention by default. The first run for a new canvas
shape includes compilation warm-up, after which it is substantially faster.
For MDLM, top_p=1.0 (pure ancestral sampling) is the default. In practice,
--top-p 0.9 (nucleus sampling) can give substantially better generation
quality, at the cost of additional sampling compute.
Sampling diagnostics default to --verbose minimal, which prints the prompt
and canvas lengths and a progress bar. Use --verbose full for model-loading
and effective-parameter details, or --verbose none to suppress diagnostics.
Likelihood evaluation
pip install -r requirements.txt
accelerate launch -m eval.lm_eval --model dLLM \
--model_args model_path=nvidia/SDLLM-EsoLM-1.7B-Base \
--tasks arc_easy,boolq,openbookqa,piqa,race,social_iqa
The evaluation code is adapted from the experiment harness and uses the conditional likelihood estimators appropriate to each model family.
Model Architecture:
Architecture Type: Transformer
Network Architecture: Decoder-only transformer
Model Size: 1.7B parameters
Training Objective: Joint autoregressive and masked diffusion language modeling
Computational Load (Internal Only: For NVIDIA Models Only)
Cumulative Compute:
- 2.14 × 10²² FLOPs
Estimated Energy and Emissions for Model Training:
- Estimated Energy: 50176 kWh
- Estimated Emissions: 20.59 tCO2e
Input(s):
Input Type(s): Text
Input Format(s): String
Input Parameters: One-Dimensional (1D): Sequences
Other Properties Related to Input: Context length: 2048. Supported languages include English.
Output(s)
Output Type(s): Text
Output Format(s): String
Output Parameters: One-Dimensional (1D): Sequences
Other Properties Related to Output: Context length: 2048. Supported languages include English.
Our AI models are designed and/or optimized to run on NVIDIA GPU-accelerated systems. By leveraging NVIDIA’s hardware (e.g. GPU cores) and software frameworks (e.g., CUDA libraries), the model achieves faster training and inference times compared to CPU-only solutions.
Software Integration:
Runtime Engine(s):
- Not Applicable (N/A) – custom PyTorch research code (no production runtime)
Supported Hardware Microarchitecture Compatibility:
- NVIDIA Ampere
- NVIDIA Hopper
- NVIDIA Blackwell
Preferred/Supported Operating System(s):
- Linux
The integration of foundation and fine-tuned models into AI systems requires additional testing using use-case-specific data to ensure safe and effective deployment. Following the V-model methodology, iterative testing and validation at both unit and system levels are essential to mitigate risks, meet technical and functional requirements, and ensure compliance with safety and ethical standards before deployment.
Model Version(s):
1.0 - Initial release.
Training and Evaluation Datasets:
Dataset Overview
Total Size: 2.1T tokens
Total Number of Datasets: 1
Training Dataset:
Data Modality:
- Text
Text Training Data Size:
- 1 Billion to 10 Trillion Tokens
Data Collection Method by dataset:
- Hybrid: Automated, Human
Labeling Method by dataset:
- Not Applicable
Properties: 2.1T tokens from Phase 1 and Phase 2 mixtures of the Nemotron Nano 2 dataset. Text data primarily in English.
Evaluation Results
| Benchmark | Score |
|---|---|
| ARC-Easy | 46.0 |
| BoolQ | 53.4 |
| OBQA | 29.6 |
| PIQA | 55.6 |
| RACE | 26.1 |
| SIQA | 36.1 |
Ethical Considerations:
NVIDIA believes Trustworthy AI is a shared responsibility and we have established policies and practices to enable development for a wide array of AI applications. When downloaded or used in accordance with our terms of service, developers should work with their internal model team to ensure this model meets requirements for the relevant industry and use case and addresses unforeseen product misuse.
Please report model quality, risk, security vulnerabilities or NVIDIA AI Concerns here.