---
license: apache-2.0
base_model: thinkingmachines/Inkling-Small
base_model_relation: quantized
pipeline_tag: image-text-to-text
library_name: exllamav3
tags:
  - exl3
  - exllamav3
  - trellis
  - inkling
  - quantized
  - moe
  - multimodal
---

# Inkling-Small-EXL3-3.5bpw

Calibrated EXL3 trellis quantization of the routed MoE experts in
[thinkingmachines/Inkling-Small](https://huggingface.co/thinkingmachines/Inkling-Small), targeting
**3.5 bits per routed-expert weight**.

> **Status:** The assembled weight archive has been uploaded. Consult `EXL3_MANIFEST.json` for the current structural and runtime validation state.

> **Runtime validation:** Text generation, multimodal generation, and MTP
> validation are pending. The uploaded archive and its packed tensors have
> structural validation only.

## What is quantized

| Component | Storage |
|---|---|
| Routed MoE experts, layers 2–41 | EXL3/MCG trellis, 3.5 bpw target |
| Dense MLP layers 0–1 | Source BF16 |
| Shared experts and routers | Source BF16/FP32 |
| Attention, relative-position, and short-convolution tensors | Source precision |
| Embeddings, norms, and LM head | Source precision |
| Vision/audio components | Source precision |
| Eight MTP layers | Source precision |

This fractional target uses whole routed layers encoded at integer EXL3 K=3 or K=4. Twenty of the forty routed layers use each K value. The higher-error layers measured by the K=3 calibration proxy receive K=4. No tensor is labeled with a fractional bit width.

- Calibration: 1,048,576 naturally routed tokens selected with seeded,
  no-repeat axis water-filling across general, legal, code/agentic, and
  reasoning/termination data
- Maximum calibration sequence/sample span: 4,096 tokens
- Routing: Inkling's natural top-6 routed-expert assignments
- Source revision: `b2d4f225a02032c5d154bff748ab5a00c5ca26e4`
- Achieved routed-trellis rate: 3.500000 bpw
- Assembled repository payload: 120.75 GiB
- Per-layer allocation, tensor inventory, sizes, and validation state:
  `EXL3_MANIFEST.json`

## Compatibility and how to use it

Download the repository with:

```bash
hf download 0xSero/Inkling-Small-EXL3-3.5bpw \
  --local-dir Inkling-Small-EXL3-3.5bpw
```

This repository is **not a drop-in Transformers checkpoint**. The routed
experts use EXL3 trellis tensors while the rest of Inkling remains in source
precision. It requires an Inkling-aware EXL3 loader/runtime that understands
the tensor layout described by `quantization_config.json` and
`EXL3_MANIFEST.json`.

Stock ExLlamaV3 v1.2.1 does not yet include an
`InklingForConditionalGeneration` architecture adapter. The upstream BF16
Inkling model has vLLM and SGLang recipes, but those recipes do not by
themselves add support for this experts-only EXL3 layout. Do not infer text,
image, audio, or MTP runtime support from a successful download or structural
assembly alone.

## All EXL3 variants

| Target | Repository |
|---:|---|
| 2.0 bpw | [0xSero/Inkling-Small-EXL3-2.0bpw](https://huggingface.co/0xSero/Inkling-Small-EXL3-2.0bpw) |
| 2.5 bpw | [0xSero/Inkling-Small-EXL3-2.5bpw](https://huggingface.co/0xSero/Inkling-Small-EXL3-2.5bpw) |
| 3.0 bpw | [0xSero/Inkling-Small-EXL3-3.0bpw](https://huggingface.co/0xSero/Inkling-Small-EXL3-3.0bpw) |
| **3.5 bpw** | [0xSero/Inkling-Small-EXL3-3.5bpw](https://huggingface.co/0xSero/Inkling-Small-EXL3-3.5bpw) |
| 4.0 bpw | [0xSero/Inkling-Small-EXL3-4.0bpw](https://huggingface.co/0xSero/Inkling-Small-EXL3-4.0bpw) |
| 4.5 bpw | [0xSero/Inkling-Small-EXL3-4.5bpw](https://huggingface.co/0xSero/Inkling-Small-EXL3-4.5bpw) |
| 5.0 bpw | [0xSero/Inkling-Small-EXL3-5.0bpw](https://huggingface.co/0xSero/Inkling-Small-EXL3-5.0bpw) |

## Method

The source model is loaded once for calibration. Hidden states and natural
expert assignments are captured for all forty routed layers. Each expert's
gate, up, and down projections are calibrated, Hadamard-transformed, and
encoded as EXL3/MCG trellis weights. The full sweep checks finite Hessians and
scales, exact trellis byte counts, safetensor key counts, and per-file
checksums. Before the sweep, a bounded H200 proof on a real Inkling expert
also passed trellis pack/unpack/repack equality and finite reconstruction.
That bounded kernel proof is not a full-model generation test. Integer-K
caches are reused to assemble the seven public variants without repeating
the full model calibration.

The target bpw applies to routed-expert trellis weights. The complete
repository is larger than a whole-model quantization at the same nominal bpw
because attention, shared experts, multimodal components, the LM head, and MTP
remain in source precision.

## Credits

- **Thinking Machines Lab / Inkling team** for releasing
  [Inkling-Small](https://huggingface.co/thinkingmachines/Inkling-Small).
- **TurboDerp** for
  [ExLlamaV3 and the EXL3 format](https://github.com/turboderp-org/exllamav3).
- **JarvisLabs** for providing the 8× NVIDIA H200 compute used for this
  quantization sweep.

This is an independent community quantization and is not an official release
from Thinking Machines Lab, TurboDerp, or JarvisLabs.

## License and use

This derivative follows the upstream
[Apache 2.0 license](https://huggingface.co/thinkingmachines/Inkling-Small) and the upstream
[acceptable-use policy](https://thinkingmachines.ai/model-acceptable-use-policy).
Review the base model card for intended uses, limitations, and safety
information.