ToPo-ToPo/Inkling-Small-mlx-4bit

MLX 4bit conversion of thinkingmachines/Inkling-Small for Apple Silicon (mlx-vlm). 276B total / 12B active sparse MoE (42 layers, 256 routed experts top-6 + 2 shared), text + image + audio in, text out.

Provenance (self-converted from official weights)

• Source: thinkingmachines/Inkling-Small (license: apache-2.0, bf16, 531.9 GB)
• Tool: mlx-vlm 0.6.7 — mlx_vlm.convert --hf-path <staged> --mlx-path . -q --q-bits 4 --q-group-size 64
• Effective: 4.506 bits/weight (138 GiB on disk, ~148.7 GB peak RSS at inference)
• Validation: text and image input both verified end-to-end through an OpenAI-compatible gateway. In an agentic CAD pipeline the produced solids are geometrically exact (e.g. bushing 22972.896 mm³ and 6-hole disc 22195.352 mm³, both matching the analytic values).

⚠ Requires a config translation (already applied here) and a small loader shim

Stock mlx-vlm (0.6.7 and 0.6.8) cannot load an Inkling checkpoint as published. This repo has the config-side problems already fixed, but two runtime patches are still needed on the mlx-vlm side.

Applied to this repo (no action needed)

Key	Official	Here	Why
model_type	inkling_mm_model	inkling	no alias in utils.MODEL_REMAPPING, so the model type never resolves
text_config.intermediate_size	2048 (expert)	16384 (dense)	mlx-vlm's TextConfig uses intermediate_size for the dense MLP
text_config.moe_intermediate_size	— (was intermediate_size)	2048	…and moe_intermediate_size for the experts. Unswapped, layers.0.mlp.gate_proj fails to load
vision_config.text_hidden_size	decoder_dmodel	4096	ModelConfig.__post_init__ wires this, but utils.update_module_configs rebuilds the tower configs and discards it (falls back to 6144 = Inkling-975B)
audio_config.text_hidden_size	decoder_dmodel	4096	same
vision_config.num_channels	n_channels	3	name only
tokenizer_config.pad_token / eos_token	unset	<|endoftext|> / <|content_model_end_sampling|>	the official TokenizersBackend config sets neither, so transformers raises on any padded call

image_token_id / audio_token_id are left at mlx-vlm's defaults (200054 / 200053) — these are correct. They are named <|unused_*|> in the tokenizer, but InklingProcessor reuses them as the image/audio patch placeholders (a 640×480 image expands to 204 of them, matching pixel_values). Pointing them at <|content_image|> (200005) silently disables vision.

Still needed at runtime (patch mlx-vlm, or apply the shim below)

1. mlx_vlm/models/inkling/__init__.py does not re-export TextConfig / VisionConfig / AudioConfig, which utils.update_module_configs fetches with getattr → AttributeError on every load.
2. prompt_utils.MODEL_CONFIG has no inkling entry, so apply_chat_template treats the model as text-only and silently drops image/audio parts (no error; the model just hallucinates from the text).

from mlx_vlm.models import inkling
from mlx_vlm.models.inkling import config as inkling_config
from mlx_vlm import prompt_utils

for name in ("TextConfig", "VisionConfig", "AudioConfig"):
    setattr(inkling, name, getattr(inkling_config, name))
prompt_utils.MODEL_CONFIG.setdefault("inkling", prompt_utils.MessageFormat.LIST_WITH_IMAGE_FIRST)

When serving over the OpenAI-compatible mlx_vlm.server, also add Inkling's structural tokens to mlx_vlm.server.responses_state._CONTENT_MARKERS (<|message_model|>, <|content_text|>, <|end_message|>) and set MLX_VLM_THINKING_START_TOKEN=<|content_thinking|> / MLX_VLM_THINKING_END_TOKEN=<|end_message|>, otherwise the reasoning channel and those markers leak into content.

Reasoning effort

The chat template always injects a Thinking effort level: system message (default 0.9). Control it with the OpenAI-compatible reasoning_effort — "none" / "minimal" / "low" / "medium" / "high" / "max", or a float in [0.0, 0.99]. "none" disables thinking entirely.

MTP (speculative decoding)

Like every other quantized Inkling repo, the conversion drops the built-in model.mtp.* weights (160 keys in the official bf16). To build a drafter, split it from the official bf16 — and split from a checkpoint whose config has already been translated as above, or the drafter hits the same dense/MoE width swap:

python -m mlx_vlm.speculative.drafters.inkling_mtp.split \
    --model <translated-bf16> --output Inkling-Small-MTP-bf16

As of mlx-vlm 0.6.8 the resulting drafter still cannot be used: the first draft block snapshots an empty cache and models/cache.py dereferences self.keys while it is None.

Usage

from mlx_vlm import load, generate
model, processor = load("ToPo-ToPo/Inkling-Small-mlx-4bit")  # apply the shim above first

Roughly 44 tok/s on an M3 Ultra (256 GB) once warm; the first load reads 138 GiB, so allow several minutes and raise any client-side startup timeouts accordingly.