--- license: cc-by-nc-4.0 language: - en library_name: pytorch tags: - diffusion-language-modeling - text-generation --- # SDLLM-EsoLM-1.7B-Base ## Model Overview ### Description: SDLLM-EsoLM-1.7B-Base is a 1.7B parameter language model combining autoregressive and masked diffusion language modeling objectives. This model is evaluated as part of the study "Scaling Beyond Masked Diffusion Language Models," which investigates diffusion-based approaches to language modeling. This model is for research and development only. ### License/Terms of Use: SDLLM-EsoLM-1.7B-Base is released under the [CC BY-NC 4.0](https://creativecommons.org/licenses/by-nc/4.0/) license. By using this model, you are agreeing to the terms and conditions of the license. This model is intended for research purposes only and may not be used for commercial applications.
### Deployment Geography: Global
### Use Case: SDLLM-EsoLM-1.7B-Base is intended for use by researchers studying diffusion-based language modeling approaches.
### Release Date: Hugging Face 09/02/2026 via https://huggingface.co/nvidia/SDLLM-EsoLM-1.7B-Base
## References(s): * [Scaling Beyond Masked Diffusion Language Models](https://arxiv.org/abs/2602.15014)
* [Nemotron Nano 2 Dataset](https://arxiv.org/abs/2508.14444)
## Sampling and likelihood evaluation This repository includes a self-contained SDLLM inference and evaluation package. ## Sampling ```bash python sample.py nvidia/SDLLM-EsoLM-1.7B-Base --prompt "In a quiet village, the old clock tower rang at midnight, and" --num-samples 1 ``` Use `python sample.py --help` for sampler overrides. With no prompt the script performs unconditional generation. Release defaults are: AR uses Gumbel-max ancestral sampling (temperature 1); MDLM uses ancestral masked diffusion; EsoLM uses its first-hitting ancestral sampler; and Duo uses its uniform-state posterior sampler. All diffusion models use 1,024 reverse steps, `top_p=1.0` (no nucleus filtering), greedy final denoising, and token-sampling temperature 1.0. By default sampling returns the complete 2,048-token continuation; use `--max-new-tokens` only to crop displayed text. Sampling uses compiled FlexAttention by default. The first run for a new canvas shape includes compilation warm-up, after which it is substantially faster. For MDLM, `top_p=1.0` (pure ancestral sampling) is the default. In practice, `--top-p 0.9` (nucleus sampling) can give substantially better generation quality, at the cost of additional sampling compute. Sampling diagnostics default to `--verbose minimal`, which prints the prompt and canvas lengths and a progress bar. Use `--verbose full` for model-loading and effective-parameter details, or `--verbose none` to suppress diagnostics. ## Likelihood evaluation ```bash pip install -r requirements.txt accelerate launch -m eval.lm_eval --model dLLM \ --model_args model_path=nvidia/SDLLM-EsoLM-1.7B-Base \ --tasks arc_easy,boolq,openbookqa,piqa,race,social_iqa ``` The evaluation code is adapted from the experiment harness and uses the conditional likelihood estimators appropriate to each model family. ## Model Architecture: **Architecture Type:** Transformer
**Network Architecture:** Decoder-only transformer
**Model Size:** 1.7B parameters
**Training Objective:** Joint autoregressive and masked diffusion language modeling
## Computational Load (Internal Only: For NVIDIA Models Only) **Cumulative Compute:** - 2.14 × 10²² FLOPs **Estimated Energy and Emissions for Model Training:** - Estimated Energy: 50176 kWh - Estimated Emissions: 20.59 tCO2e ## Input(s): **Input Type(s):** Text
**Input Format(s):** String
**Input Parameters:** One-Dimensional (1D): Sequences
**Other Properties Related to Input:** Context length: 2048. Supported languages include English.
## Output(s) **Output Type(s):** Text
**Output Format(s):** String
**Output Parameters:** One-Dimensional (1D): Sequences
**Other Properties Related to Output:** Context length: 2048. Supported languages include English. Our AI models are designed and/or optimized to run on NVIDIA GPU-accelerated systems. By leveraging NVIDIA’s hardware (e.g. GPU cores) and software frameworks (e.g., CUDA libraries), the model achieves faster training and inference times compared to CPU-only solutions. ## Software Integration: **Runtime Engine(s):** * Not Applicable (N/A) – custom PyTorch research code (no production runtime)
**Supported Hardware Microarchitecture Compatibility:** * NVIDIA Ampere
* NVIDIA Hopper
* NVIDIA Blackwell
**Preferred/Supported Operating System(s):** * Linux
The integration of foundation and fine-tuned models into AI systems requires additional testing using use-case-specific data to ensure safe and effective deployment. Following the V-model methodology, iterative testing and validation at both unit and system levels are essential to mitigate risks, meet technical and functional requirements, and ensure compliance with safety and ethical standards before deployment. ## Model Version(s): 1.0 - Initial release.
## Training and Evaluation Datasets: ### Dataset Overview **Total Size:** 2.1T tokens
**Total Number of Datasets:** 1
### Training Dataset: **Data Modality:** * Text
**Text Training Data Size:** * 1 Billion to 10 Trillion Tokens
**Data Collection Method by dataset:** * Hybrid: Automated, Human
**Labeling Method by dataset:** * Not Applicable
**Properties:** 2.1T tokens from Phase 1 and Phase 2 mixtures of the [Nemotron Nano 2 dataset](https://arxiv.org/abs/2508.14444). Text data primarily in English.
### Evaluation Results | Benchmark | Score | |-----------|-------| | ARC-Easy | 46.0 | | BoolQ | 53.4 | | OBQA | 29.6 | | PIQA | 55.6 | | RACE | 26.1 | | SIQA | 36.1 | ## Ethical Considerations: NVIDIA believes Trustworthy AI is a shared responsibility and we have established policies and practices to enable development for a wide array of AI applications. When downloaded or used in accordance with our terms of service, developers should work with their internal model team to ensure this model meets requirements for the relevant industry and use case and addresses unforeseen product misuse.
Please report model quality, risk, security vulnerabilities or NVIDIA AI Concerns [here](https://www.nvidia.com/en-us/support/submit-security-vulnerability/).