--- library_name: transformers license: mit license_link: https://huggingface.co/deepreinforce-ai/Ornith-1.0-35B/blob/main/LICENSE pipeline_tag: text-generation base_model: - deepreinforce-ai/Ornith-1.0-35B - shisa-ai/Ornith-1.0-35B-FP8-BLOCK - Qwen/Qwen3.6-35B-A3B tags: - fp8 - fp8-block - compressed-tensors - llm-compressor - qwen3.5 - qwen3.6 - moe - mtp - speculative-decoding - vllm - code - reasoning --- # Ornith-1.0-35B-FP8-BLOCK-MTP This is the official MTP-enabled derivative of [`shisa-ai/Ornith-1.0-35B-FP8-BLOCK`](https://huggingface.co/shisa-ai/Ornith-1.0-35B-FP8-BLOCK). It keeps the original FP8_BLOCK / `compressed-tensors` Ornith base weights and adds a BF16 Qwen3.6 MTP head in `model-mtp.safetensors`. This artifact was built internally as a zero-training Qwen3.6 MTP graft, but the public upload name is: [`shisa-ai/Ornith-1.0-35B-FP8-BLOCK-MTP`](https://huggingface.co/shisa-ai/Ornith-1.0-35B-FP8-BLOCK-MTP). Based on our matched local vLLM tests, this is the recommended Ornith 35B MTP checkpoint for throughput. The best measured row used `num_speculative_tokens=3` and reached `246.522` output tok/s on our validation workload, a `+21.4%` improvement over the no-spec baseline. The MTP recipe was adapted from [`protoLabsAI/Ornith-1.0-9B-MTP`](https://huggingface.co/protoLabsAI/Ornith-1.0-9B-MTP), which demonstrated grafting a same-family Qwen MTP head into an Ornith checkpoint and optionally KL-distilling only the MTP head. Thanks to [`protoLabsAI`](https://huggingface.co/protoLabsAI) for the MTP graft/distillation technique and base recipe. ## What Changed - Base checkpoint: [`shisa-ai/Ornith-1.0-35B-FP8-BLOCK`](https://huggingface.co/shisa-ai/Ornith-1.0-35B-FP8-BLOCK) - Donor MTP checkpoint: `Qwen/Qwen3.6-35B-A3B` - Added shard: `model-mtp.safetensors` - Added tensors: `19` top-level `mtp.*` tensors - MTP dtype: BF16 - Base weights: unchanged FP8_BLOCK / `compressed-tensors` - Training: none; this is a direct MTP tensor graft No private training corpus is needed for this checkpoint because it is not distilled. The graft copies the donor `mtp.*` tensors into the Ornith FP8_BLOCK checkpoint and updates the safetensors index. ## Important Result Summary This is the strongest Ornith 35B MTP checkpoint from our local serving tests. It also beat the one-epoch Qwen3.6 KL-distilled derivative in matched vLLM serving, even though that distilled derivative had a slightly better offline KL proxy. Recommended starting point: - Use `num_speculative_tokens=3` when optimizing for output throughput. - Use `num_speculative_tokens=1` or no speculative decoding if your workload is more sensitive to inter-token latency or acceptance stability. - Re-benchmark on your own traffic before making it a production default. ## Local Benchmark Methodology Hardware: - One `NVIDIA RTX PRO 6000 Blackwell Workstation Edition` - Single-GPU serving on GPU0 only Runtime: - vLLM `0.23.0` - FlashInfer attention backend - FP8 KV cache - `compressed-tensors` quantization - No LMCache for the benchmark rows below - `MAX_MODEL_LEN=32768` - `MAX_NUM_SEQS=16` - `MAX_BATCHED_TOKENS=32768` - `MAX_CUDAGRAPH_CAPTURE_SIZE=16` - `GPU_MEMORY_UTIL=0.95` Workload: - Private custom validation benchmark derived from local code/agentic prompts - `64` requests - `63,327` total input tokens - `16,384` generated tokens (`256` per request) - `max_concurrency=1` - `request_rate=inf` - `temperature=0.6`, `top_p=0.95`, `top_k=20` - `ignore_eos` The private benchmark data is not uploaded. The benchmark used vLLM's `bench serve --dataset-name custom` path. To reproduce the command shape with your own non-private data, use a JSONL custom dataset with `prompt` and `output_tokens` fields and run a command like: ```bash CUDA_VISIBLE_DEVICES=0 vllm bench serve \ --backend vllm \ --base-url http://127.0.0.1:8000 \ --endpoint /v1/completions \ --model ornith-35b-fp8-block-mtp \ --tokenizer shisa-ai/Ornith-1.0-35B-FP8-BLOCK-MTP \ --dataset-name custom \ --dataset-path /path/to/custom-prompts.jsonl \ --skip-chat-template \ --disable-shuffle \ --no-oversample \ --num-prompts 64 \ --custom-output-len -1 \ --max-concurrency 1 \ --request-rate inf \ --temperature 0.6 \ --top-p 0.95 \ --top-k 20 \ --ignore-eos ``` ## Local Results Matched c=1 validation-prompt serving results: | Variant | MTP tokens | Output tok/s | Delta vs baseline | Median TTFT ms | Median TPOT ms | Median ITL ms | Acceptance | | --- | ---: | ---: | ---: | ---: | ---: | ---: | ---: | | Baseline no-spec | 0 | 203.142 | - | 51.722 | 4.647 | 4.648 | - | | Official MTP, Qwen3.6 graft | 1 | 221.032 | +8.8% | 60.311 | 4.209 | 7.713 | 85.82% | | Official MTP, Qwen3.6 graft | 3 | 246.522 | +21.4% | 65.482 | 3.571 | 10.739 | 66.98% | Comparison against the companion KL-distilled artifact: | Variant | MTP tokens | Output tok/s | Delta vs baseline | Median TTFT ms | Median TPOT ms | Median ITL ms | Acceptance | | --- | ---: | ---: | ---: | ---: | ---: | ---: | ---: | | Official MTP, Qwen3.6 graft | 3 | 246.522 | +21.4% | 65.482 | 3.571 | 10.739 | 66.98% | | Qwen3.6 KL-distill | 3 | 237.581 | +17.0% | 66.581 | 3.625 | 10.861 | 67.11% | The distill checkpoint is planned as [`shisa-ai/Ornith-1.0-35B-FP8-BLOCK-MTP-qwen36-distill`](https://huggingface.co/shisa-ai/Ornith-1.0-35B-FP8-BLOCK-MTP-qwen36-distill). It improved the offline proxy, but it did not beat this official MTP checkpoint in matched serving throughput. Offline proxy on the same validation split: - Official MTP, Qwen3.6 graft: distribution-overlap acceptance proxy `0.841`, mean KL `0.3162` - Qwen3.6 KL-distill: distribution-overlap acceptance proxy `0.849`, mean KL `0.2967` ### Internal Alternatives Tested We also tested a Qwen3.5 donor graft, but it is not planned for upload. The Qwen3.5 rows were from an earlier MTP1-only ShareGPT sweep, so they are useful as donor-selection context rather than a direct replacement for the validation table above. | Variant | c | Output tok/s | Delta vs baseline | Median TPOT ms | Acceptance | | --- | ---: | ---: | ---: | ---: | ---: | | Baseline no-spec | 1 | 200.340 | - | 4.733 | - | | Qwen3.6 MTP1 graft | 1 | 211.736 | +5.7% | 4.296 | 80.63% | | Qwen3.5 MTP1 graft | 1 | 213.091 | +6.4% | 4.402 | 77.90% | | Baseline no-spec | 4 | 500.689 | - | 6.955 | - | | Qwen3.6 MTP1 graft | 4 | 528.654 | +5.6% | 6.443 | 81.42% | | Qwen3.5 MTP1 graft | 4 | 531.581 | +6.1% | 6.554 | 76.65% | The Qwen3.5 graft was marginally faster in those MTP1 rows, but it had lower acceptance. The later Qwen3.6 validation sweep at MTP3 produced the best retained throughput result, so Qwen3.6 is the donor used for the official upload. ## vLLM Usage MTP serving requires a vLLM build that supports Qwen3.5 MoE MTP checkpoints. The local runs used vLLM `0.23.0`. ```bash vllm serve shisa-ai/Ornith-1.0-35B-FP8-BLOCK-MTP \ --served-model-name ornith-35b-fp8-block-mtp \ --trust-remote-code \ --quantization compressed-tensors \ --language-model-only \ --max-model-len 32768 \ --gpu-memory-utilization 0.95 \ --max-num-seqs 16 \ --max-num-batched-tokens 32768 \ --max-cudagraph-capture-size 16 \ --attention-backend flashinfer \ --kv-cache-dtype fp8 \ --generation-config vllm \ --enable-prefix-caching \ --enable-auto-tool-choice \ --tool-call-parser qwen3_xml \ --reasoning-parser qwen3 \ --speculative-config '{"method":"mtp","num_speculative_tokens":3}' ``` `num_speculative_tokens=3` gave the best local output throughput for this checkpoint, but it also reduced acceptance to about `67%` and increased inter-token latency relative to no-spec serving. Re-benchmark on your hardware and workload before using it as a default. ## Quantization Summary The base checkpoint is unchanged from [`shisa-ai/Ornith-1.0-35B-FP8-BLOCK`](https://huggingface.co/shisa-ai/Ornith-1.0-35B-FP8-BLOCK): - Source model: [`deepreinforce-ai/Ornith-1.0-35B`](https://huggingface.co/deepreinforce-ai/Ornith-1.0-35B) - Quantization tool: `llm-compressor` model-free PTQ - Quantization format: `compressed-tensors` - Scheme: `FP8_BLOCK` - Calibration data: none; this is data-free/model-free PTQ - Weight quantization: static FP8, symmetric, block strategy, `128x128` blocks - Activation quantization: dynamic FP8, symmetric, group strategy, group size `128` - Target modules: `Linear` - `compressed-tensors` metadata version recorded in `config.json`: `0.15.1.a20260406` The quantization ignore list includes `re:^mtp.*`, so the grafted MTP head remains BF16. ## License and Attribution The source Ornith model is MIT licensed. This derivative keeps the source license metadata and links to the upstream license file. Attribution: - Source model: [`deepreinforce-ai/Ornith-1.0-35B`](https://huggingface.co/deepreinforce-ai/Ornith-1.0-35B) - FP8_BLOCK base: [`shisa-ai/Ornith-1.0-35B-FP8-BLOCK`](https://huggingface.co/shisa-ai/Ornith-1.0-35B-FP8-BLOCK) - MTP donor: [`Qwen/Qwen3.6-35B-A3B`](https://huggingface.co/Qwen/Qwen3.6-35B-A3B) - MTP graft/distill recipe inspiration: [`protoLabsAI/Ornith-1.0-9B-MTP`](https://huggingface.co/protoLabsAI/Ornith-1.0-9B-MTP) If you use the source model, cite the original Ornith release: ```bibtex @misc{ornith-35b, title = {{Ornith-1.0-35B}: Agentic Coding, Open to All}, url = {https://deep-reinforce.com/ornith_1_0.html}, author = {{DeepReinforce Team}}, year = {2026} } ```