gpt-oss-120b-p150x4

Runs on p150x4 (mesh P150x4) — 131,072-token context, up to 32 concurrent sequences.

Packaged and published with tt-model-manager 0.1.0 (manifest schema 5.1).

Quickstart

tt-model pull  tt-hous/gpt-oss-120b-p150x4 --with-weights
tt-model serve tt-hous/gpt-oss-120b-p150x4

pull --with-weights downloads the Docker image and the openai/gpt-oss-120b weights at b5c939de8f754692c1647ca79fbf85e8c1e70f8a (into your HF cache; they are not in the image). serve starts an OpenAI-compatible server on port 20000 (or the next free port, if that one is busy); the first start compiles kernels for your device, which takes several minutes, and the server is ready when it logs Application startup complete.

This package serves openai/gpt-oss-120b with official vLLM and the Tenstorrent vLLM plugin on four P150 boards. Follow startup until the API is ready:

tt-model serve tt-hous/gpt-oss-120b-p150x4 --follow

The first start compiles kernels and model tensors into the persistent package cache and takes about 18 minutes; a restart with a warm cache takes about 8 minutes.

Source

Every line of code in this image is on a public branch; the Provenance table at the end of this card links the exact commits the image was built from.

• tt-metal: hous/gpt-oss-120b-p150x4 on the housTT fork: the GPT-OSS 120B autoport (batched decode, batched and indexed MoE prefill, chunked 131k prefill, sharded decode norms), the ttnn.sparse_matmul extensions it uses (fused per-group bias, optional second in0 sender, two-block synchronisation), the allocator tracker fix, and this card.
• vLLM: tt/gpt-oss-serving on the housTT fork, commit 53b3c0128: upstream vLLM plus a Harmony parser that recovers from an off-grammar token mid-stream (the parsed reasoning is kept, the remaining text is returned as content, a warning names the token) instead of failing the request. The image installs the wheel vllm-0.26.0.dev2+g53b3c0128 built from that commit with VLLM_TARGET_DEVICE=empty; the Tenstorrent plugin supplies the platform.
• vllm-tt-plugin: hous/gpt-oss-120b-serving on the housTT fork, commit 176a8c5: the plugin with capability-gated tensor sampling parameters and host-state versions for decode, which this autoport uses.

Measured P150x4 latency (2026-09-17)

Measured on this package by the bench-sweeps harness (tt-model serve from a cold start): random-token prompts of exactly ISL tokens, ignore_eos, output pinned to OSL tokens, Users concurrent requests held open as a closed loop, N requests per row in total (4 per user up to 4,096 input tokens, 2 per user at 16,384, then 1 per user). TTFT is the time to the first streamed token; TPOT = (E2EL - TTFT) / (OSL - 1) per request; prefill tok/s/u = ISL / TTFT; decode tok/s/u = 1 / TPOT; E2EL is the end-to-end latency of one request; Out tok/s total is the total output tokens per second across all users over the row's wall time. Every value is the mean over the row's N requests. Decode runs one grouped expert pass per step with trace buckets at 1, 4, 8, 16 and 32 users; new requests that arrive in the same scheduler step and share a padded prompt length (up to 2,048 tokens) are prefilled together; the expert gate/up bias is fused into the indexed sparse matmul.

ISL	OSL	Users	N	TTFT (ms)	Prefill (tok/s/u)	TPOT (ms)	Decode (tok/s/u)	E2EL (s)	Out (tok/s total)
128	128	1	4	261	490	15.2	65.7	2.19	58
128	128	2	8	496	258	21.8	45.8	3.27	78
128	128	4	16	584	219	21.8	45.8	3.35	153
128	128	8	32	559	229	32.6	30.6	4.70	218
128	128	16	64	826	155	42.4	23.6	6.21	330
128	128	32	128	1,025	125	50.0	20.0	7.37	556
1,024	256	1	4	347	2,950	15.5	64.6	4.29	60
1,024	256	2	8	620	1,651	22.1	45.3	6.25	82
1,024	256	4	16	1,049	976	22.1	45.2	6.69	153
1,024	256	8	32	1,583	647	32.9	30.4	9.98	205
1,024	256	16	64	2,716	377	42.9	23.3	13.65	300
1,024	256	32	128	5,079	202	50.3	19.9	17.90	458
4,096	256	1	4	747	5,482	15.7	63.8	4.75	54
4,096	256	2	8	1,481	2,766	22.3	44.8	7.17	71
4,096	256	4	16	2,873	1,426	22.3	44.8	8.57	119
4,096	256	8	32	5,920	692	33.4	30.0	14.43	142
4,096	256	16	64	11,763	348	43.5	23.0	22.87	179
4,096	256	32	128	23,125	177	50.4	19.8	35.99	228
16,384	256	1	2	2,435	6,730	16.1	62.1	6.54	39
16,384	256	2	4	4,908	3,338	23.2	43.1	10.83	47
16,384	256	4	8	9,735	1,683	23.2	43.1	15.66	65
16,384	256	8	16	19,439	843	35.0	28.5	28.38	72
16,384	256	16	32	21,949	746	113.8	8.8	50.98	80
16,384	256	32	64	57,790	284	137.2	7.3	92.77	80
32,768	256	1	1	5,206	6,294	16.7	59.9	9.46	27
32,768	256	2	2	10,380	3,157	24.4	40.9	16.61	31
32,768	256	4	4	20,824	1,574	24.4	41.0	27.04	38
32,768	256	8	8	27,247	1,203	93.6	10.7	51.11	40
32,768	256	16	16	51,835	632	118.9	8.4	82.16	40
32,768	256	32	32	102,279	320	132.7	7.5	136.13	40
65,536	256	1	1	11,926	5,495	17.8	56.2	16.47	16
65,536	256	2	2	23,836	2,749	26.7	37.4	30.65	17
65,536	256	4	4	34,481	1,901	59.6	16.8	49.69	17
65,536	256	8	8	62,304	1,052	86.4	11.6	84.34	17
65,536	256	16	16	117,556	557	102.0	9.8	143.58	18
65,536	256	32	32	230,250	285	111.3	9.0	258.64	18
130,816	256	1	1	30,315	4,315	20.0	49.9	35.42	7
130,816	256	2	2	48,150	2,717	20.0	49.9	53.26	7
130,816	256	4	4	83,329	1,570	20.0	50.0	88.43	7
130,816	256	8	8	155,008	844	20.0	50.0	160.11	7
130,816	256	16	16	291,563	449	20.0	50.0	296.66	7
130,816	256	32	32	572,871	228	20.0	50.0	577.97	7

How to read the long-context rows. Prompts above 2,048 tokens are prefilled one at a time, so TTFT at N users is about N times the single-user prefill (the last user of a wave waits for every prefill before it). vLLM admits 148,480 tokens of context across the running requests (the plugin's 4,640 KV blocks are split equally between the full-attention and the sliding-window layer groups, and the full-attention group's 2,320 blocks of 64 tokens set the limit; the server logs this as "maximum concurrency 1.13x at 131,072 tokens"): 8 sequences at 16k, 4 at 32k, 2 at 64k and one at 128k. The rest of a wave waits for admission (that wait lands in TTFT), and each new prompt's prefill stalls the decode steps of the users already generating, which is why per-user decode drops sharply past those counts (16k at 16 users, 32k at 8, 64k at 4). The 130,816-token rows are the context boundary (max_model_len - 256): only one such sequence fits, so the server handles those requests strictly one at a time (30.3 s prefill, then 5.1 s of decode at 20 ms per token). The decode column stays at 50 tok/s per user at every concurrency there because it is the undisturbed single-sequence rate; the concurrency cost is entirely in TTFT (about 35 s per request ahead in the queue) and in the 7 tok/s total output rate.

Known issues

At 130,816-token prompts with 16 or more concurrent requests, one request per batch receives its own prompt streamed back as generated text (one 722,000-character chunk, no echo requested, usage reports 256 completion tokens). Not observed at 8 users or at shorter inputs. Open in the vLLM fork / plugin output path.

Accuracy

Evals run through the OpenAI-compatible chat API against the served model: reasoning_effort medium, greedy (temperature 0, do_sample false), 4,096 max output tokens, seed 42, chat template applied. IFEval is the full 541-prompt set at 32 concurrent requests (14 answers are empty because the reasoning channel used the whole 4,096-token budget); GPQA Diamond, AIME 2025 and MMLU are subsets run at one request at a time.

eval	published	measured on P150x4
IFEval, prompt-level strict (frozebench run of gpt-oss-120b)	78.2%	85.4% (462/541, stderr 1.5)
IFEval, instruction-level strict	not published	89.1%
IFEval, prompt-level loose / instruction-level loose	not published	88.5% / 91.1%
GPQA Diamond, CoT zero-shot (OpenAI model card)	80.1%	100% on a 7-question subset
AIME 2025 (OpenAI model card)	92.5%	86.7% on 15 problems
MMLU, generative (OpenAI model card)	85.9%	84.7% on a 15% subset

Prompts of 512+ tokens use the indexed expert prefill: each expert's routed tokens are gathered into slabs and run through compact indexed sparse matmuls, with the routing and slab layout computed on the host from the device router logits. Prefill program shapes (expert groups and the common batched (users, length) pairs) are compiled at server start; the first prompt of a new length still pays a few seconds of program creation. Prompts longer than 2,048 tokens are prefilled one at a time, so TTFT for many long prompts arriving together is the serialized prefill queue. Prompts longer than 65,536 tokens run the expert prefill in 64k-row chunks, which is how the full 131,072-token context is served (the 130,816 rows above).

P150 and P150x2 are not offered as serve profiles: the fixed resident model state does not fit those configurations. Production serving requires P150x4.

Provenance

The exact sources the image was built from — code/ in this repo is byte-identical to the model code inside the image:

component	built from
tt-metal	47df3bd6d2620a7fd6a29fe44ef75fd7c45d165f (dirty tree — the image includes uncommitted changes)
vLLM	vllm-0.26.0.dev2+g53b3c0128-cp310-cp310-linux_x86_64.whl — a wheel the author built
vllm-tt-plugin	176a8c5865d9abe82d2d8a04a25b5920e78f3fcb
code/ digest	621d3af037d18bd4 (sha256, first 16 hex digits)
built	2026-09-17T20:34:31+00:00 by tt-model 0.1.0