Instructions to use shisa-ai/Ornith-1.0-35B-FP8-BLOCK-MTP with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use shisa-ai/Ornith-1.0-35B-FP8-BLOCK-MTP with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="shisa-ai/Ornith-1.0-35B-FP8-BLOCK-MTP")
messages = [
    {
        "role": "user",
        "content": [
            {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"},
            {"type": "text", "text": "What animal is on the candy?"}
        ]
    },
]
pipe(text=messages)

# Load model directly
from transformers import AutoProcessor, AutoModelForMultimodalLM

processor = AutoProcessor.from_pretrained("shisa-ai/Ornith-1.0-35B-FP8-BLOCK-MTP")
model = AutoModelForMultimodalLM.from_pretrained("shisa-ai/Ornith-1.0-35B-FP8-BLOCK-MTP")
messages = [
    {
        "role": "user",
        "content": [
            {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"},
            {"type": "text", "text": "What animal is on the candy?"}
        ]
    },
]
inputs = processor.apply_chat_template(
	messages,
	add_generation_prompt=True,
	tokenize=True,
	return_dict=True,
	return_tensors="pt",
).to(model.device)

outputs = model.generate(**inputs, max_new_tokens=40)
print(processor.decode(outputs[0][inputs["input_ids"].shape[-1]:]))

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use shisa-ai/Ornith-1.0-35B-FP8-BLOCK-MTP with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "shisa-ai/Ornith-1.0-35B-FP8-BLOCK-MTP"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "shisa-ai/Ornith-1.0-35B-FP8-BLOCK-MTP",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker

docker model run hf.co/shisa-ai/Ornith-1.0-35B-FP8-BLOCK-MTP

SGLang

How to use shisa-ai/Ornith-1.0-35B-FP8-BLOCK-MTP with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "shisa-ai/Ornith-1.0-35B-FP8-BLOCK-MTP" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "shisa-ai/Ornith-1.0-35B-FP8-BLOCK-MTP",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "shisa-ai/Ornith-1.0-35B-FP8-BLOCK-MTP" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "shisa-ai/Ornith-1.0-35B-FP8-BLOCK-MTP",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Docker Model Runner
How to use shisa-ai/Ornith-1.0-35B-FP8-BLOCK-MTP with Docker Model Runner:
```
docker model run hf.co/shisa-ai/Ornith-1.0-35B-FP8-BLOCK-MTP
```

Ornith-1.0-35B-FP8-BLOCK-MTP

This is the official MTP-enabled derivative of shisa-ai/Ornith-1.0-35B-FP8-BLOCK. It keeps the original FP8_BLOCK / compressed-tensors Ornith base weights and adds a BF16 Qwen3.6 MTP head in model-mtp.safetensors.

This artifact was built internally as a zero-training Qwen3.6 MTP graft, but the public upload name is: shisa-ai/Ornith-1.0-35B-FP8-BLOCK-MTP. Based on our matched local vLLM tests, this is the recommended Ornith 35B MTP checkpoint for throughput. The best measured row used num_speculative_tokens=3 and reached 246.522 output tok/s on our validation workload, a +21.4% improvement over the no-spec baseline.

The MTP recipe was adapted from protoLabsAI/Ornith-1.0-9B-MTP, which demonstrated grafting a same-family Qwen MTP head into an Ornith checkpoint and optionally KL-distilling only the MTP head. Thanks to protoLabsAI for the MTP graft/distillation technique and base recipe.

What Changed

Base checkpoint: shisa-ai/Ornith-1.0-35B-FP8-BLOCK
Donor MTP checkpoint: Qwen/Qwen3.6-35B-A3B
Added shard: model-mtp.safetensors
Added tensors: 19 top-level mtp.* tensors
MTP dtype: BF16
Base weights: unchanged FP8_BLOCK / compressed-tensors
Training: none; this is a direct MTP tensor graft

No private training corpus is needed for this checkpoint because it is not distilled. The graft copies the donor mtp.* tensors into the Ornith FP8_BLOCK checkpoint and updates the safetensors index.

Important Result Summary

This is the strongest Ornith 35B MTP checkpoint from our local serving tests. It also beat the one-epoch Qwen3.6 KL-distilled derivative in matched vLLM serving, even though that distilled derivative had a slightly better offline KL proxy.

Recommended starting point:

Use num_speculative_tokens=3 when optimizing for output throughput.
Use num_speculative_tokens=1 or no speculative decoding if your workload is more sensitive to inter-token latency or acceptance stability.
Re-benchmark on your own traffic before making it a production default.

Local Benchmark Methodology

Hardware:

One NVIDIA RTX PRO 6000 Blackwell Workstation Edition
Single-GPU serving on GPU0 only

Runtime:

vLLM 0.23.0
FlashInfer attention backend
FP8 KV cache
compressed-tensors quantization
No LMCache for the benchmark rows below
MAX_MODEL_LEN=32768
MAX_NUM_SEQS=16
MAX_BATCHED_TOKENS=32768
MAX_CUDAGRAPH_CAPTURE_SIZE=16
GPU_MEMORY_UTIL=0.95

Workload:

Private custom validation benchmark derived from local code/agentic prompts
64 requests
63,327 total input tokens
16,384 generated tokens (256 per request)
max_concurrency=1
request_rate=inf
temperature=0.6, top_p=0.95, top_k=20
ignore_eos

The private benchmark data is not uploaded. The benchmark used vLLM's bench serve --dataset-name custom path. To reproduce the command shape with your own non-private data, use a JSONL custom dataset with prompt and output_tokens fields and run a command like:

CUDA_VISIBLE_DEVICES=0 vllm bench serve \
  --backend vllm \
  --base-url http://127.0.0.1:8000 \
  --endpoint /v1/completions \
  --model ornith-35b-fp8-block-mtp \
  --tokenizer shisa-ai/Ornith-1.0-35B-FP8-BLOCK-MTP \
  --dataset-name custom \
  --dataset-path /path/to/custom-prompts.jsonl \
  --skip-chat-template \
  --disable-shuffle \
  --no-oversample \
  --num-prompts 64 \
  --custom-output-len -1 \
  --max-concurrency 1 \
  --request-rate inf \
  --temperature 0.6 \
  --top-p 0.95 \
  --top-k 20 \
  --ignore-eos

Local Results

Matched c=1 validation-prompt serving results:

Variant	MTP tokens	Output tok/s	Delta vs baseline	Median TTFT ms	Median TPOT ms	Median ITL ms	Acceptance
Baseline no-spec	0	203.142	-	51.722	4.647	4.648	-
Official MTP, Qwen3.6 graft	1	221.032	+8.8%	60.311	4.209	7.713	85.82%
Official MTP, Qwen3.6 graft	3	246.522	+21.4%	65.482	3.571	10.739	66.98%

Comparison against the companion KL-distilled artifact:

Variant	MTP tokens	Output tok/s	Delta vs baseline	Median TTFT ms	Median TPOT ms	Median ITL ms	Acceptance
Official MTP, Qwen3.6 graft	3	246.522	+21.4%	65.482	3.571	10.739	66.98%
Qwen3.6 KL-distill	3	237.581	+17.0%	66.581	3.625	10.861	67.11%

The distill checkpoint is planned as shisa-ai/Ornith-1.0-35B-FP8-BLOCK-MTP-qwen36-distill. It improved the offline proxy, but it did not beat this official MTP checkpoint in matched serving throughput.

Offline proxy on the same validation split:

Official MTP, Qwen3.6 graft: distribution-overlap acceptance proxy 0.841, mean KL 0.3162
Qwen3.6 KL-distill: distribution-overlap acceptance proxy 0.849, mean KL 0.2967

Internal Alternatives Tested

We also tested a Qwen3.5 donor graft, but it is not planned for upload. The Qwen3.5 rows were from an earlier MTP1-only ShareGPT sweep, so they are useful as donor-selection context rather than a direct replacement for the validation table above.

Variant	c	Output tok/s	Delta vs baseline	Median TPOT ms	Acceptance
Baseline no-spec	1	200.340	-	4.733	-
Qwen3.6 MTP1 graft	1	211.736	+5.7%	4.296	80.63%
Qwen3.5 MTP1 graft	1	213.091	+6.4%	4.402	77.90%
Baseline no-spec	4	500.689	-	6.955	-
Qwen3.6 MTP1 graft	4	528.654	+5.6%	6.443	81.42%
Qwen3.5 MTP1 graft	4	531.581	+6.1%	6.554	76.65%

The Qwen3.5 graft was marginally faster in those MTP1 rows, but it had lower acceptance. The later Qwen3.6 validation sweep at MTP3 produced the best retained throughput result, so Qwen3.6 is the donor used for the official upload.

vLLM Usage

MTP serving requires a vLLM build that supports Qwen3.5 MoE MTP checkpoints. The local runs used vLLM 0.23.0.

vllm serve shisa-ai/Ornith-1.0-35B-FP8-BLOCK-MTP \
  --served-model-name ornith-35b-fp8-block-mtp \
  --trust-remote-code \
  --quantization compressed-tensors \
  --language-model-only \
  --max-model-len 32768 \
  --gpu-memory-utilization 0.95 \
  --max-num-seqs 16 \
  --max-num-batched-tokens 32768 \
  --max-cudagraph-capture-size 16 \
  --attention-backend flashinfer \
  --kv-cache-dtype fp8 \
  --generation-config vllm \
  --enable-prefix-caching \
  --enable-auto-tool-choice \
  --tool-call-parser qwen3_xml \
  --reasoning-parser qwen3 \
  --speculative-config '{"method":"mtp","num_speculative_tokens":3}'

num_speculative_tokens=3 gave the best local output throughput for this checkpoint, but it also reduced acceptance to about 67% and increased inter-token latency relative to no-spec serving. Re-benchmark on your hardware and workload before using it as a default.

Quantization Summary

The base checkpoint is unchanged from shisa-ai/Ornith-1.0-35B-FP8-BLOCK:

Source model: deepreinforce-ai/Ornith-1.0-35B
Quantization tool: llm-compressor model-free PTQ
Quantization format: compressed-tensors
Scheme: FP8_BLOCK
Calibration data: none; this is data-free/model-free PTQ
Weight quantization: static FP8, symmetric, block strategy, 128x128 blocks
Activation quantization: dynamic FP8, symmetric, group strategy, group size 128
Target modules: Linear
compressed-tensors metadata version recorded in config.json: 0.15.1.a20260406

The quantization ignore list includes re:^mtp.*, so the grafted MTP head remains BF16.

License and Attribution

The source Ornith model is MIT licensed. This derivative keeps the source license metadata and links to the upstream license file.

Attribution:

Source model: deepreinforce-ai/Ornith-1.0-35B
FP8_BLOCK base: shisa-ai/Ornith-1.0-35B-FP8-BLOCK
MTP donor: Qwen/Qwen3.6-35B-A3B
MTP graft/distill recipe inspiration: protoLabsAI/Ornith-1.0-9B-MTP

If you use the source model, cite the original Ornith release:

@misc{ornith-35b,
    title = {{Ornith-1.0-35B}: Agentic Coding, Open to All},
    url = {https://deep-reinforce.com/ornith_1_0.html},
    author = {{DeepReinforce Team}},
    year = {2026}
}

Downloads last month: 1,195

Safetensors

Model size

665k params

Tensor type

BF16

F8_E4M3

Model tree for shisa-ai/Ornith-1.0-35B-FP8-BLOCK-MTP

Base model

Qwen/Qwen3.6-35B-A3B

Quantized

(627)

this model

Quantizations

2 models