--- license: mit language: - en base_model: deepreinforce-ai/Ornith-1.0-35B tags: - mlx - mlx-node - quantized - qwen3.5 - moe - hybrid-attention - gated-delta-net - vision-language - coding-agent - apple-silicon - 5-bit - unsloth-dynamic library_name: mlx-node quantized_by: mlx-node pipeline_tag: text-generation model_type: qwen3_5_moe --- # Ornith-1.0-35B — UD-Q5_K_XL (mlx-node) 5-bit base mixed-precision quantization of [deepreinforce-ai/Ornith-1.0-35B](https://huggingface.co/deepreinforce-ai/Ornith-1.0-35B) for Apple Silicon, using the **Unsloth Dynamic** per-tensor bit allocation (without imatrix AWQ) via [mlx-node](https://github.com/mlx-node/mlx-node). [Ornith-1.0](https://deep-reinforce.com/ornith.html) is a self-improving family of open-source **agentic coding** models. The 35B member is a Qwen3.5-VL-MoE (hybrid Gated-DeltaNet + full attention, 256 experts, vision-language) post-train. | | Original (BF16) | This Model | |---|---|---| | **Size** | ~68 GB | **26 GB** | | **Format** | SafeTensors (sharded) | SafeTensors (sharded) | | **Precision** | BF16 uniform | Mixed 5/6/8/8-bit affine + BF16 | ## All Variants | Repo | Format | Size | Decode (tok/s) | |---|---|---|---| | [Brooooooklyn/Ornith-1.0-35B-UD-Q3_K_XL-mlx](https://huggingface.co/Brooooooklyn/Ornith-1.0-35B-UD-Q3_K_XL-mlx) | UD-Q3_K_XL | 17 GB | 111.6 | | [Brooooooklyn/Ornith-1.0-35B-mxfp4-mlx](https://huggingface.co/Brooooooklyn/Ornith-1.0-35B-mxfp4-mlx) | MXFP4 | 20 GB | 107.8 | | [Brooooooklyn/Ornith-1.0-35B-UD-Q4_K_XL-mlx](https://huggingface.co/Brooooooklyn/Ornith-1.0-35B-UD-Q4_K_XL-mlx) | UD-Q4_K_XL | 22 GB | 102.3 | | [Brooooooklyn/Ornith-1.0-35B-nvfp4-mlx](https://huggingface.co/Brooooooklyn/Ornith-1.0-35B-nvfp4-mlx) | NVFP4 | 23 GB | 94.6 | | **[Brooooooklyn/Ornith-1.0-35B-UD-Q5_K_XL-mlx](https://huggingface.co/Brooooooklyn/Ornith-1.0-35B-UD-Q5_K_XL-mlx) (this model)** | **UD-Q5_K_XL** | **26 GB** | **95.4** | | [Brooooooklyn/Ornith-1.0-35B-UD-Q6_K_XL-mlx](https://huggingface.co/Brooooooklyn/Ornith-1.0-35B-UD-Q6_K_XL-mlx) | UD-Q6_K_XL | 31 GB | 93.1 | | [Brooooooklyn/Ornith-1.0-35B-UD-Q8_K_XL-mlx](https://huggingface.co/Brooooooklyn/Ornith-1.0-35B-UD-Q8_K_XL-mlx) | UD-Q8_K_XL | 36 GB | 91.5 | | [Brooooooklyn/Ornith-1.0-35B-mxfp8-mlx](https://huggingface.co/Brooooooklyn/Ornith-1.0-35B-mxfp8-mlx) | MXFP8 | 36 GB | 84.8 | Benchmarked on a **cool Apple M5 Max**: median decode throughput over three 512-token generations, with a 60-second idle GPU cooldown after every generation. (Sustained decode on Apple Silicon is thermally sensitive — back-to-back benchmarking on a hot chip can understate throughput by 20–30%, so every model here was measured from a comparable cool start.) ## Performance Steady-state decode: **95.4 tok/s** (**1.5x** vs BF16) on Apple M5 Max. Decode is memory-bandwidth bound on Apple Silicon — fewer bytes per token directly translates to higher throughput. The MoE architecture activates only 8 of 256 experts per token (~3B active out of 35.9B total), so the active-weight footprint streamed per token is what matters. ## Output Quality Decoded-text quality was verified against the BF16 reference with a multi-judge review of the **actual generated output** (not a heuristic): a 4-turn factual chat plus a Python `is_balanced()` bracket-matching task. This `UD-Q5_K_XL` build produced coherent prose, correct facts, and a correct implementation — no runaway generation, repetition loops, or stray tokens — on par with full precision. (The 2-bit tier is intentionally excluded from this collection: it was the only width that showed coherence breakdown.) ## Per-Tensor Quantization | Weight | Bits | Rationale | |---|---|---| | `embed_tokens` | 8-bit affine | KLD ~0.15 — very low sensitivity | | `lm_head` | 8-bit affine | KLD ~0.05 — safest tensor | | `self_attn.q/k/v_proj` | 8-bit affine | KLD ~1.5–2.9 — attention-sensitive | | `linear_attn.in_proj_qkv/z` | 8-bit affine | KLD ~2.9 — SSM input gates | | `self_attn.o_proj` | 8-bit affine | KLD ~1.5; row-independent qmv for T=0 exactness | | `linear_attn.out_proj` | 8-bit affine | KLD ~6.0 — worst tensor; kept high | | `linear_attn.in_proj_a/b` | 8-bit affine | tiny low-rank GDN projections | | `switch_mlp.down_proj` | 6-bit affine | "slightly more sensitive" than other FFN | | `switch_mlp.gate_proj/up_proj` | 5-bit affine | bulk of the expert budget | | Router gates (`mlp.gate`, `shared_expert_gate`) | 8-bit affine | MoE routing accuracy | | GDN params (`A_log`, `dt_bias`) | **bf16** | state-space dynamics | | `vision_tower.*` | **bf16** | vision encoder kept full precision | ## Quantization Strategy Built on [Unsloth Dynamic 2.0](https://unsloth.ai/docs/models/qwen3.5/gguf-benchmarks) per-tensor KLD analysis: sensitive layers (attention/SSM inputs, down_proj, embeddings/head) get higher bits, while the bulk of FFN expert weights are quantized to the base width. `self_attn.o_proj`, `linear_attn.out_proj`, the split low-rank GDN projections (`in_proj_a/b`) and the MoE router gates are pinned to 8-bit affine (group_size 64). GatedDeltaNet state-space parameters and the vision encoder stay bf16. > **Note:** These ornith quants apply the Unsloth bit *allocation* **without imatrix AWQ pre-scaling** — ornith has no published imatrix, so the attention/SSM channels are quantized directly. Expect a small quality gap versus an imatrix-calibrated build at the lowest bit widths. ## Architecture | Parameter | Value | |---|---| | Total parameters | 35.9B (~3B active per token) | | Hidden size | 2,048 | | Layers | 40 (30 linear GatedDeltaNet + 10 full attention) | | Attention heads | 16 (2 KV heads, GQA 8:1) | | Head dimension | 256 | | Experts | 256 per MoE layer, top-8 routing | | Vocab size | 248,320 | | Vision | yes (Qwen3.5-VL vision tower, kept bf16) | | Max context | 262,144 tokens | ## Usage ```typescript import { loadSession } from '@mlx-node/lm'; const session = await loadSession('./Ornith-1.0-35B-UD-Q5_K_XL-mlx'); for await (const event of session.sendStream('Write a Python function to merge two sorted lists.', { config: { maxNewTokens: 2048, temperature: 0.6, reasoningEffort: 'low' }, })) { if (!event.done) process.stdout.write(event.text); } ``` ## How It Was Made ```bash mlx convert \ -i Ornith-1.0-35B \ -o Ornith-1.0-35B-UD-Q5_K_XL-mlx \ -q --q-recipe unsloth --q-bits 5 ``` The Unsloth recipe's per-tensor bit tiers were applied without imatrix AWQ (no native ornith imatrix). 7-bit tiers are snapped up to 8-bit (MLX affine supports 2/3/4/5/6/8-bit). ## Acknowledgments - **[Unsloth](https://unsloth.ai)** — Per-layer KLD bit-allocation strategy ([Dynamic 2.0](https://unsloth.ai/docs/models/qwen3.5/gguf-benchmarks)) - **[DeepReinforce](https://deep-reinforce.com/ornith.html)** — For the Ornith-1.0 model family - **[Qwen Team](https://huggingface.co/Qwen)** — For the Qwen3.5 base architecture - **[Apple MLX](https://github.com/ml-explore/mlx)** — For the Metal-accelerated ML framework ## License [MIT](https://huggingface.co/deepreinforce-ai/Ornith-1.0-35B/blob/main/LICENSE) (inherited from base model).