ndaly/Mistral-Small-4-119B-2603-tt-p300x2

mistral-small-4-119b-p300x2

Mistral-Small-4-119B-2603 (119B-parameter, 6.5B-active MoE with MLA attention, 1,048,576-token context) served on four Blackhole chips (2 x p300c, 1x4 ring, TP=4) through a TTNN autoport and the Tenstorrent vLLM plugin: an OpenAI-compatible chat/completions endpoint for general assistant and coding use.

Runs on p300x2 (mesh P300x2) — 1,048,576-token context, up to 32 concurrent sequences.

Packaged and published with tt-model-manager 0.1.0 (manifest schema 5.1).

Quickstart

tt-model pull  ndaly/Mistral-Small-4-119B-2603-tt-p300x2 --with-weights
tt-model serve ndaly/Mistral-Small-4-119B-2603-tt-p300x2

pull --with-weights downloads the Docker image and the mistralai/Mistral-Small-4-119B-2603 weights at a11f36bebf709121056b1dbcc943d1c6afbe494d (into your HF cache; they are not in the image). serve starts an OpenAI-compatible server on port 20000 (or the next free port, if that one is busy); the first start compiles kernels for your device, which takes several minutes, and the server is ready when it logs Application startup complete.

What to expect

• Hardware: a TT-QuietBox 2 or any host with 2 x p300c (4 Blackhole chips, 32 GB GDDR6 each), Docker, hugepages.
• Weights: 121 GB of HF-format shards are downloaded into your Hugging Face cache on first serve.
• First boot converts and tile-caches the weights for the mesh (tens of minutes, on the order of 100 GB under the mounted kernel cache); later boots load from that cache.
• Measured (stage 11 release run, 2026-09-16): single user TTFT 1.5 s through chat completions (the template adds a ~564-token default system prompt), decode 21-22 ms/token (46 tok/s per user); 32 users ~66 tok/s aggregate at 100-token prompts; prompts up to 1,048,576 tokens accepted (prefill ~2 ms/token).
• Evals through the same server: IFEval 72.83 % prompt-strict (1024-token-capped), HumanEval-instruct 92.68 %. GPQA was not measured. Full evidence: doc/tti_release/RUN_NOTES.md in the tt-metal autoport tree.

Use with your client

Point an OpenAI-compatible client at http://127.0.0.1:20000 (or the port tt-model serve printed) with model id mistralai/Mistral-Small-4-119B-2603.

Provenance

The exact sources the image was built from — code/ in this repo is byte-identical to the model code inside the image:

component	built from
tt-metal	a local checkout — commit not published (dirty tree — the image includes uncommitted changes)
vLLM	vllm-0.1.dev14195+g8c28fcecb.d20260916.empty-py3-none-any.whl — a wheel the author built
vllm-tt-plugin	a local checkout — commit not published (dirty tree — the image includes uncommitted changes)
code/ digest	a4dc23aeb6810dad (sha256, first 16 hex digits)
built	2026-09-16T16:55:50+00:00 by tt-model 0.1.0