originalGeek/Orthrus-Qwen3-8B-GGUF

Orthrus-Qwen3-8B GGUF (Q4_K_M)

Q4_K_M GGUF build of chiennv/Orthrus-Qwen3-8B for the Orthrus llama.cpp reference fork:

https://github.com/remesis/orthrus_llamacpp

Orthrus is not just a Qwen3 weight layout. The important runtime feature is shared-KV self-drafting: the autoregressive path owns the persistent target KV cache, the diffusion view drafts against that same cache, transient diffusion cells are removed before verification, and normal AR verification commits or rolls back the candidate path.

This GGUF uses the dedicated orthrus architecture and requires a llama.cpp build with Orthrus support for --spec-type draft-orthrus. It is not expected to run correctly on upstream llama.cpp until equivalent support exists there.

Files

File	Notes
Orthrus-Qwen3-8B-Q4_K_M.gguf	Quantized 8B Orthrus GGUF.
chat_template.jinja	Official template from the source model folder. The same template is embedded in the GGUF.

Run

Clone and build the reference fork:

git clone https://github.com/remesis/orthrus_llamacpp
cd orthrus_llamacpp
cmake -S . -B build-cuda -G "Visual Studio 17 2022" -A x64 -DGGML_CUDA=ON -DLLAMA_CURL=OFF
cmake --build build-cuda --config Release --target llama-server

Download this GGUF and run the tuned local config:

.\build-cuda\bin\Release\llama-server.exe `
  -m models\Orthrus-Qwen3-8B\Orthrus-Qwen3-8B-Q4_K_M.gguf `
  --host 127.0.0.1 `
  --port 8081 `
  -c 40960 `
  -np 1 `
  -b 1024 `
  -ub 512 `
  -ngl 999 `
  -fa on `
  --ui `
  --jinja `
  --chat-template-file models\Orthrus-Qwen3-8B\chat_template.jinja `
  --spec-type draft-orthrus `
  --spec-draft-n-max 15

Notes:

• Use --spec-type none for a plain autoregressive baseline.
• --spec-draft-n-max 15 was the best average local setting in the benchmark below.
• Auto draft length resolves to block_size - 1 (31) and can be better or worse depending on prompt shape.
• Passing --chat-template-file is optional when the template is embedded, but it makes the intended template explicit.

Local Validation

The 8B Q4_K_M GGUF was checked for:

Check	Result
general.architecture	orthrus
orthrus.diffusion.block_size	32
tokenizer.ggml.mask_token_id	151669
Tensor count	615
Diffusion tensors	216 _diff tensors present
Separate output head	output.weight retained
Chat template	Embedded template matches chat_template.jinja

Runtime checks:

• llama-server loaded the 8B Q4_K_M GGUF with CUDA offload.
• draft-orthrus startup logged shared_kv=yes, block_size=32, and mask_id=151669.
• Temp-0 decoded output matched AR-only output in controlled server tests.
• Quantization retained the Orthrus diffusion projection tensors.

Official Runtime Cross-Check

The Orthrus team's own Transformers implementation was also benchmarked locally to confirm the intended shared-KV diffusion path and to sanity-check whether native Orthrus drafting is worthwhile before publishing this GGUF.

Environment:

• RTX 5090
• PyTorch 2.11.0+cu128
• Transformers 5.12.0
• BF16 source checkpoint
• attn_implementation="sdpa"
• temp-0 generation

Raw completion results:

Case	Official HF AR	Official HF Orthrus	Accepted / Drafted	Acceptance
Short, 256 tokens	32.2 t/s	352.6 t/s	245 / 320	76.6%
Technical, 512 tokens	33.3 t/s	163.1 t/s	462 / 1541	30.0%

Chat-template results:

Case	Official HF AR	Official HF Orthrus	Accepted / Drafted	Acceptance
Technical, 512 tokens	27.4 t/s	48.3 t/s	370 / 4242	8.7%
Long context, 6.5k prompt + 256 output	3.8 t/s wall / 9.0 t/s decode	5.9 t/s wall / 16.0 t/s decode	175 / 2349	7.5%

These numbers are not quantization-to-quantization comparisons against the GGUF below; they use the official BF16 Transformers runtime. They are included because they validate that the source model's intended Orthrus path can produce large throughput gains on favorable raw-completion prompts and smaller but still measurable gains on chat/long-context prompts.

Benchmark Snapshot

Environment:

• RTX 5090
• CUDA 13.3
• -np 1
• -c 40960
• -ngl 999
• -fa on
• raw /completion temp-0 benchmark prompts

Wall TPS:

Config	Short	Technical	Long Context	Average
8B F16 AR	92	84	72	82
8B F16 draft-orthrus auto	138	74	67	93
8B Q4_K_M AR	220	215	134	190
8B Q4_K_M draft-orthrus auto 31	174	319	89	194
8B Q4_K_M draft-orthrus nmax 7	173	222	98	164
8B Q4_K_M draft-orthrus nmax 15	200	340	110	217

Acceptance for the tuned 8B Q4_K_M nmax 15 run:

Prompt	Accepted / Drafted	Acceptance
Short	202 / 778	26.0%
Technical	450 / 896	50.2%
Long context	182 / 1066	17.1%

Interpretation:

• The tuned 8B Q4_K_M Orthrus config was the best average local benchmark.
• Plain Q4_K_M AR was faster on the long-context raw-completion case.
• Acceptance is content-dependent, so benchmark your intended workload.

Source

• Orthrus project: https://github.com/chiennv2000/orthrus
• Source model: https://huggingface.co/chiennv/Orthrus-Qwen3-8B
• llama.cpp reference fork: https://github.com/remesis/orthrus_llamacpp

Citation

@misc{vannguyen2026orthrusmemoryefficientparalleltoken,
      title={Orthrus: Memory-Efficient Parallel Token Generation via Dual-View Diffusion},
      author={Chien Van Nguyen and Chaitra Hegde and Van Cuong Pham and Ryan A. Rossi and Franck Dernoncourt and Thien Huu Nguyen},
      year={2026},
      eprint={2605.12825},
      archivePrefix={arXiv},
      primaryClass={cs.LG},
      url={https://arxiv.org/abs/2605.12825},
}