README.md

---
license: mit
license_link: https://huggingface.co/deepreinforce-ai/Ornith-1.0-35B/blob/main/LICENSE
base_model: deepreinforce-ai/Ornith-1.0-35B
base_model_relation: quantized
pipeline_tag: text-generation
library_name: transformers
tags:
  - fp8
  - vllm
  - agentic-coding
  - moe
---

# Ornith-1.0-35B-FP8

Block-wise **FP8 (E4M3)** quantization of [`deepreinforce-ai/Ornith-1.0-35B`](https://huggingface.co/deepreinforce-ai/Ornith-1.0-35B) — DeepReinforce's self-scaffolding agentic-coding MoE (Qwen3.5-35B-A3B hybrid). Served size ~35 GB; fits a single 80–96 GB GPU with full context.

Quantized and serve-validated by [protoLabs](https://protolabs.studio) on RTX PRO 6000 Blackwell (vLLM 0.22.1).

## Why this exists

The upstream FP8 release was reported broken. This is a clean, serve-validated FP8 built to the **Qwen official block-wise format** (served natively on vLLM's fused-MoE Triton path), with the **entire linear-attention / SSM path and the MoE router kept in bf16** — FP8-quantizing those corrupts the hybrid SSM and is the most likely failure mode for a naive FP8 of this architecture.

## Quantization recipe

- **Scheme:** block-wise `[128, 128]` FP8 E4M3, dynamic per-token activations (`quant_method: fp8`, `activation_scheme: dynamic`).
- **Quantized:** all expert FFNs (gate/up/down), shared-expert FFNs, full-attention projections (q/k/v/o).
- **Kept bf16 (`modules_to_not_convert`):** `lm_head`, `embed_tokens`, MoE router gates (`mlp.gate`, `shared_expert_gate`), **all `linear_attn.*` (SSM: `in_proj_*`, `out_proj`, `conv1d`, `A_log`, `dt_bias`)**, all norms, and the vision tower.
- Streaming tensor-by-tensor quantizer (peak RAM ≈ 2× largest tensor) — no full-model load.

## Validation (protoLabs harness, thinking-on, single trial)

Smoke: coherent across coding / reasoning / long-form / multilingual / tool-calling — no gibberish, no reasoning leak.

| Metric | bf16 source | This FP8 |
|---|---|---|
| custom coding (one-shot) | 1.00 (10/10) | 0.975 (9/10) |
| function-call | 93% (50/54) | 89% (48/54) |
| decode tok/s (1× RTX PRO 6000) | 208 | 207 |

Deltas are within single-trial run-to-run variance (temp 0.6–0.7); throughput is identical to the source.

## Serving (vLLM)

```bash
vllm serve protoLabsAI/Ornith-1.0-35B-FP8 \
  --served-model-name ornith-35b \
  --max-model-len 262144 \
  --reasoning-parser qwen3 \
  --enable-auto-tool-choice --tool-call-parser qwen3_xml \
  --gpu-memory-utilization 0.90 \
  --trust-remote-code
```

**Context:** full **256K** (262144). **Vision:** the base is multimodal (Qwen-VL-style image + video tokens) and the vision tower is preserved in bf16 — the recipe above keeps it enabled. For **text-only** serving (smaller footprint), add `--language-model-only`. Verified serving with vision on at 256K on RTX PRO 6000 (Blackwell, sm120).

Ornith is a reasoning model: the assistant turn opens with a `<think>…</think>` block surfaced as `reasoning_content`; tool calls are emitted as standard `tool_calls`. Recommended sampling: `temperature=0.6, top_p=0.95, top_k=20`.

## License & attribution

MIT, inheriting [Ornith-1.0](https://huggingface.co/deepreinforce-ai/Ornith-1.0-35B). All credit for the base model to the **DeepReinforce** team ([blog](https://deep-reinforce.com/ornith_1_0.html)); this repo only adds the FP8 quantization.

```bibtex
@misc{ornith-35b, title={{Ornith-1.0-35B}: Agentic Coding, Open to All},
  url={https://deep-reinforce.com/ornith_1_0.html}, author={{DeepReinforce Team}}, year={2026}}
```

## The Ornith quant family

- [`Ornith-1.0-9B-NVFP4`](https://huggingface.co/protoLabsAI/Ornith-1.0-9B-NVFP4) — calibrated
  W4A4 for vLLM, MTP sidecar in-box; 10.4 GB, gate-verified parity, ~1.5x bf16+MTP.
- [`Ornith-1.0-9B-MTP-GGUF`](https://huggingface.co/protoLabsAI/Ornith-1.0-9B-MTP-GGUF) — llama.cpp
  builds incl. the NVFP4+MTP rung (306 tok/s on Blackwell).
- [`Ornith-1.0-9B-MTP`](https://huggingface.co/protoLabsAI/Ornith-1.0-9B-MTP) — the MTP draft head.
- 35B-A3B NVFP4 is next in the pipeline (MoE is NVFP4's best case).

Benchmark rows: [`protoLabsAI/lab-benchmarks`](https://huggingface.co/datasets/protoLabsAI/lab-benchmarks) ·
[protolabs.studio/lab](https://protolabs.studio/lab). Different quant? Community discussion — ~48h.