---
license: apache-2.0
base_model: Dream-org/Dream-v0-Instruct-7B
tags:
  - fp8
  - quantized
  - llmcompressor
  - tevunahai
  - professional-grade
  - diffusion-lm
  - dream
  - dllm
---

# Dream-v0-Instruct-7B-FP8

## TevunahAi Professional Quantization

**🏆 First FP8 quantized Dream model for native PyTorch/transformers inference.**

This is an FP8 quantized version of [Dream-v0-Instruct-7B](https://huggingface.co/Dream-org/Dream-v0-Instruct-7B), 
a diffusion-based large language model from HKU NLP Group.

### What is Dream?

Dream 7B is a **Diffusion Large Language Model (dLLM)** - unlike traditional autoregressive models 
(GPT, LLaMA, Claude) that generate text left-to-right one token at a time, Dream uses 
**parallel denoising** to refine the entire sequence simultaneously.

Key advantages:
- 🔄 **Bidirectional context modeling** - considers full context in both directions
- 🎯 **Flexible text generation order** - not constrained to left-to-right
- 🧠 **Superior planning abilities** - excels at tasks requiring multi-step reasoning
- ⚡ **Adjustable quality-speed tradeoff** - control inference steps for your needs

### Quantization Details

| Property | Value |
|----------|-------|
| Original Model | Dream-v0-Instruct-7B |
| Quantization | FP8 Dynamic (Weight-only) |
| Method | llmcompressor FP8_DYNAMIC |
| Calibration | Data-free |
| Hardware | Dual Xeon Max 9480 + RTX 5000 Ada |
| Quantization Time | 1.7 minutes |

### Memory Comparison

| Precision | Size | VRAM Required |
|-----------|------|---------------|
| BF16 | ~14 GB | ~16 GB |
| **FP8** | **~8.7 GB** | **~10 GB** |

### Usage

```python
import torch
from transformers import AutoModel, AutoTokenizer

model_path = "TevunahAi/Dream-v0-Instruct-7B-FP8"

model = AutoModel.from_pretrained(
    model_path,
    torch_dtype=torch.bfloat16,
    trust_remote_code=True,
    device_map="auto"
)
tokenizer = AutoTokenizer.from_pretrained(model_path, trust_remote_code=True)

messages = [
    {"role": "user", "content": "Explain quantum computing in simple terms."}
]

inputs = tokenizer.apply_chat_template(
    messages,
    return_tensors="pt",
    return_dict=True,
    add_generation_prompt=True
)

input_ids = inputs.input_ids.to(model.device)
attention_mask = inputs.attention_mask.to(model.device)

# Dream uses diffusion_generate, not generate!
output = model.diffusion_generate(
    input_ids,
    attention_mask=attention_mask,
    max_new_tokens=256,
    steps=256,
    temperature=0.3,
    top_p=0.95,
    alg="entropy",
    alg_temp=0.,
)

# Decode and clean up response
response = tokenizer.decode(output[0][input_ids.shape[1]:].tolist())
response = response.split("<|endoftext|>")[0].strip()
print(response)
```

### Generation Parameters

| Parameter | Description | Recommended |
|-----------|-------------|-------------|
| `steps` | Number of diffusion steps (quality vs speed) | 128-512 |
| `max_new_tokens` | Maximum tokens to generate | 256-512 |
| `temperature` | Randomness (lower = more deterministic) | 0.2-0.5 |
| `top_p` | Nucleus sampling threshold | 0.9-0.95 |
| `alg` | Decoding algorithm | "entropy" |
| `alg_temp` | Algorithm temperature | 0.0 |

**Tips:**
- More `steps` = higher quality but slower
- For math/code: use lower temperature (0.1-0.2)
- For creative tasks: use higher temperature (0.5-0.7)

### Important Notes

1. ⚠️ **Use `diffusion_generate()`** not `generate()` - Dream is a diffusion model!
2. Requires `trust_remote_code=True` for custom model code
3. Stop token cleanup: split response on `<|endoftext|>`
4. Context length: 2048 tokens

### Verified Working

```
Input: "What is 2+2? Answer briefly."
Output: "4"
✓ Correct!
```

### Credits

- **Original Model**: [Dream-org / HKU NLP Group](https://huggingface.co/Dream-org) - Pioneering diffusion-based language models
- **Quantization**: [TevunahAi](https://tevunah.ai) - Professional AI model quantization services
- **Method**: [llmcompressor](https://github.com/vllm-project/llm-compressor) by vLLM Project

### Citation

If you use Dream, please cite the original paper:

```bibtex
@article{dream2025,
  title={Dream 7B: Diffusion Large Language Models},
  author={Ye, Jiacheng and Xie, Zhihui and others},
  journal={arXiv preprint},
  year={2025}
}
```

### License

Apache 2.0 (same as original Dream model)

---