---
license: apache-2.0
base_model: thinkingmachines/Inkling-Small
library_name: mlx
tags:
- mlx
- vision
- moe
pipeline_tag: image-text-to-text
---

# ToPo-ToPo/Inkling-Small-mlx-4bit

MLX **4bit** conversion of [`thinkingmachines/Inkling-Small`](https://huggingface.co/thinkingmachines/Inkling-Small)
for Apple Silicon (mlx-vlm). 276B total / 12B active sparse MoE (42 layers, 256 routed experts top-6 + 2 shared),
text + image + audio in, text out.

## Provenance (self-converted from official weights)
- Source: [`thinkingmachines/Inkling-Small`](https://huggingface.co/thinkingmachines/Inkling-Small) (license: apache-2.0, bf16, 531.9 GB)
- Tool: `mlx-vlm 0.6.7` — `mlx_vlm.convert --hf-path <staged> --mlx-path . -q --q-bits 4 --q-group-size 64`
- Effective: **4.506 bits/weight** (138 GiB on disk, ~148.7 GB peak RSS at inference)
- Validation: text and image input both verified end-to-end through an OpenAI-compatible gateway.
  In an agentic CAD pipeline the produced solids are geometrically exact
  (e.g. bushing 22972.896 mm³ and 6-hole disc 22195.352 mm³, both matching the analytic values).

## ⚠ Requires a config translation (already applied here) and a small loader shim

Stock `mlx-vlm` (0.6.7 and 0.6.8) **cannot load an Inkling checkpoint as published**. This repo has the
config-side problems already fixed, but two runtime patches are still needed on the mlx-vlm side.

### Applied to this repo (no action needed)
| Key | Official | Here | Why |
| --- | --- | --- | --- |
| `model_type` | `inkling_mm_model` | `inkling` | no alias in `utils.MODEL_REMAPPING`, so the model type never resolves |
| `text_config.intermediate_size` | 2048 (expert) | 16384 (dense) | mlx-vlm's `TextConfig` uses `intermediate_size` for the **dense** MLP |
| `text_config.moe_intermediate_size` | — (was `intermediate_size`) | 2048 | …and `moe_intermediate_size` for the **experts**. Unswapped, `layers.0.mlp.gate_proj` fails to load |
| `vision_config.text_hidden_size` | `decoder_dmodel` | 4096 | `ModelConfig.__post_init__` wires this, but `utils.update_module_configs` rebuilds the tower configs and discards it (falls back to 6144 = Inkling-975B) |
| `audio_config.text_hidden_size` | `decoder_dmodel` | 4096 | same |
| `vision_config.num_channels` | `n_channels` | 3 | name only |
| `tokenizer_config.pad_token` / `eos_token` | unset | `<\|endoftext\|>` / `<\|content_model_end_sampling\|>` | the official `TokenizersBackend` config sets neither, so transformers raises on any padded call |

`image_token_id` / `audio_token_id` are left at mlx-vlm's defaults (200054 / 200053) — these are **correct**.
They are named `<|unused_*|>` in the tokenizer, but `InklingProcessor` reuses them as the image/audio patch
placeholders (a 640×480 image expands to 204 of them, matching `pixel_values`). Pointing them at
`<|content_image|>` (200005) silently disables vision.

### Still needed at runtime (patch mlx-vlm, or apply the shim below)
1. `mlx_vlm/models/inkling/__init__.py` does not re-export `TextConfig` / `VisionConfig` / `AudioConfig`,
   which `utils.update_module_configs` fetches with `getattr` → `AttributeError` on every load.
2. `prompt_utils.MODEL_CONFIG` has no `inkling` entry, so `apply_chat_template` treats the model as
   text-only and **silently drops image/audio parts** (no error; the model just hallucinates from the text).

```python
from mlx_vlm.models import inkling
from mlx_vlm.models.inkling import config as inkling_config
from mlx_vlm import prompt_utils

for name in ("TextConfig", "VisionConfig", "AudioConfig"):
    setattr(inkling, name, getattr(inkling_config, name))
prompt_utils.MODEL_CONFIG.setdefault("inkling", prompt_utils.MessageFormat.LIST_WITH_IMAGE_FIRST)
```

When serving over the OpenAI-compatible `mlx_vlm.server`, also add Inkling's structural tokens to
`mlx_vlm.server.responses_state._CONTENT_MARKERS` (`<|message_model|>`, `<|content_text|>`,
`<|end_message|>`) and set `MLX_VLM_THINKING_START_TOKEN=<|content_thinking|>` /
`MLX_VLM_THINKING_END_TOKEN=<|end_message|>`, otherwise the reasoning channel and those markers leak
into `content`.

## Reasoning effort

The chat template always injects a `Thinking effort level:` system message (default **0.9**). Control it
with the OpenAI-compatible `reasoning_effort` — `"none"` / `"minimal"` / `"low"` / `"medium"` / `"high"` /
`"max"`, or a float in `[0.0, 0.99]`. `"none"` disables thinking entirely.

## MTP (speculative decoding)

Like every other quantized Inkling repo, the conversion drops the built-in `model.mtp.*` weights
(160 keys in the official bf16). To build a drafter, split it from the **official bf16** — and split from a
checkpoint whose config has already been translated as above, or the drafter hits the same dense/MoE
width swap:

```bash
python -m mlx_vlm.speculative.drafters.inkling_mtp.split \
    --model <translated-bf16> --output Inkling-Small-MTP-bf16
```

As of mlx-vlm 0.6.8 the resulting drafter still cannot be used: the first draft block snapshots an empty
cache and `models/cache.py` dereferences `self.keys` while it is `None`.

## Usage

```python
from mlx_vlm import load, generate
model, processor = load("ToPo-ToPo/Inkling-Small-mlx-4bit")  # apply the shim above first
```

Roughly 44 tok/s on an M3 Ultra (256 GB) once warm; the first load reads 138 GiB, so allow several minutes
and raise any client-side startup timeouts accordingly.