--- tags: - blackhole - p300x2 - tt-model-cache - tt-model-catalog - tt-model-container - vllm-plugin --- # qwen3.8-flash-next-p300x2 Runs on **p300x2** (mesh `(4, 1)`) — 262,144-token context, up to 1 concurrent sequences. Packaged and published with [tt-model-manager](https://github.com/tenstorrent/tt-model-manager) 0.1.0 (manifest schema 5.1). ## Quickstart ```bash tt-model pull tt-hous/qwen3.8-flash-next-p300x2 --with-weights tt-model serve tt-hous/qwen3.8-flash-next-p300x2 ``` `pull --with-weights` downloads the Docker image and the [`Qwen/Qwen3.8-Flash-Next`](https://huggingface.co/Qwen/Qwen3.8-Flash-Next) weights at `f5d08274bafd880402bd16f5e3e6c514136ec06c` (into your HF cache; they are not in the image). `serve` starts an OpenAI-compatible server on port 20000 (or the next free port, if that one is busy); the first start compiles kernels for your device, which takes several minutes, and the server is ready when it logs `Application startup complete`. **Text-only.** This package serves the language model of Qwen3.8-Flash-Next as an OpenAI-compatible chat and completion endpoint on two P300 boards (four Blackhole devices). The checkpoint's vision encoder is not ported: requests with image or video content are rejected. The runtime uses TP4+EP4 with all 512 routed experts resident on device; only the PLE n-gram table and its sparse row assembly remain host-backed. ### Serve on two P300 boards (four chips) ```bash tt-model serve tt-hous/qwen3.8-flash-next-p300x2 ``` `tt-model serve` waits for the model to become ready and prints the selected local endpoint. With the default port, verify the OpenAI-compatible API with: ```bash curl -fsS http://127.0.0.1:20000/v1/chat/completions \ -H 'Content-Type: application/json' \ --data-binary '{"model":"Qwen/Qwen3.8-Flash-Next","messages":[{"role":"user","content":"What is 7 multiplied by 8? Answer in exactly one sentence."}],"temperature":0.0,"max_tokens":64}' ``` ### Tool calling The endpoint serves OpenAI-style function calling. The server starts with `--enable-auto-tool-choice --tool-call-parser qwen3_xml` and the `qwen3` reasoning parser, so a request that passes `tools` gets structured `tool_calls` back and thinking content arrives in `reasoning_content`: ```bash curl -fsS http://127.0.0.1:20000/v1/chat/completions \ -H 'Content-Type: application/json' \ --data-binary '{"model":"Qwen/Qwen3.8-Flash-Next","messages":[{"role":"user","content":"What is the weather in Toronto right now?"}],"tools":[{"type":"function","function":{"name":"get_weather","description":"Get the current weather for a city.","parameters":{"type":"object","properties":{"city":{"type":"string"}},"required":["city"]}}}],"tool_choice":"auto","temperature":0.0,"max_tokens":256}' ``` The package pins the exact Hugging Face checkpoint revision, vLLM 0.24.0, Transformers 5.16.0, and the Qwen3.8-capable TT plugin revision. It exposes one physical batch-1 decode lane and uses resumable 1,024-token prefill chunks split into 512-token model microchunks with 128-token tails. The single serve profile, `long-context`, exposes the full 262,144-token context, so requests may use large `max_tokens` values without being rejected. ### What is implemented Text generation only, on four Blackhole chips as a (4, 1) mesh: dense weights are tensor-parallel over the four chips (TP4) and the 512 routed experts of every layer are resident on device, 128 per chip (EP4). Batch one, 262,144-token context, traced decode, eager prefill in 512-token microchunks (128-token tails) inside 1,024-token vLLM chunks. | component | in this port | weights on device | | --- | --- | --- | | Gated DeltaNet linear attention (36 layers) | one fused decode kernel: causal conv, q/k norm, delta-rule recurrence, gated RMSNorm; recurrent state kept in FP32 | projections BFP8 (HiFi2) | | Query-sparse attention (12 layers) | exact top-512 block selection from the compressed index above 2,048 tokens; dense paged attention below it (same result) | projections BFP8 (HiFi2); K/V page cache BFP8 with BF16 updates; compressed index keys BF16 | | MoE, 512 routed experts, top-10 plus a shared expert | all experts resident, indexed sparse matmuls per chip, fused routing / top-k / softmax / weighted-sum kernels | routed experts BFP4 (LoFi, ~17 GB per chip); shared expert BFP8; router BF16 | | Hyper-connections (4-stream gated residual) | fused mixer kernels; the residual is width-sharded over the four chips | per-layer mixers BF16, final mixer BFP8 | | N-gram (PLE) embedding table, 51.2 B parameters | memory-mapped on the host; the rows a token needs are looked up and uploaded per step | BF16 | | Embedding and LM head | replicated embedding; vocabulary-sharded LM head with an exact global argmax on device | BF16 (LM head HiFi2) | | Norms, activations, residual stream, collectives | RMSNorm at HiFi4; two-link linear collectives | BF16 | Quantization in one sentence: the 123.6 B routed-expert parameters run as BFP4 (block floating point: 4-bit mantissas, one shared exponent per 16 values), derived once from the BF16 checkpoint into the resident cache; the dense attention, DeltaNet and shared-expert projections and the final mixer are BFP8; embeddings, the LM head, norms, routers, per-layer mixers, the n-gram table and all activations stay BF16; the DeltaNet recurrence accumulates in FP32. The policy is declared in `doc/datatype_sweep/selected_precision_config.json` and audited against the live tensors on every accuracy-gate run. Not in this port: the vision encoder and multimodal RoPE (text-only endpoint), YaRN scaling beyond 262,144 tokens, speculative decoding with the checkpoint's MTP head (the weights load and a lossless k=1 verifier exists in the test harness, but it is performance-neutral on this stack until more of the layer is fused, so the server does not use it), prefix caching (the recurrent and n-gram state cannot be rebuilt from KV blocks), on-device stochastic sampling (see Recommended sampling), and batching above one. ### Recommended sampling Requests that omit sampling fields get the checkpoint's own defaults (`generation_config.json`: `temperature 1.0`, `top_p 0.95`, `top_k 20`), which is Qwen's thinking-mode preset. For long reasoning add `presence_penalty` 1.0-1.5. The `temperature: 0.0` in the examples above is for reproducible smoke tests, not a recommendation. Sampling paths: greedy requests (`temperature 0`) sample on device (exact global argmax, ~0.7 ms per token). Every request with `temperature > 0` is sampled by vLLM on the host from the full logits (as are requests with penalties, `logprobs`, an explicit `seed`, `min_p`, `logit_bias`, `bad_words`, `allowed_token_ids` or structured outputs); this adds about 18 ms per generated token over the greedy figures in the table below (41 -> 59 ms/token under 2,048 tokens of context, 72 -> 91 ms/token above; TTFT unchanged). The bounded on-device stochastic sampler is disabled in this revision while a user report of repetition loops under it is investigated. Measured against the vLLM reference the device sampler draws no token outside the requested `top_p` set and stays within 0.02 total-variation distance, so the switch is a precaution, not a fix for a known numerical defect. Greedy decoding was never affected. The first start populates a persistent BFP4 resident-expert tensor cache in tt-model's per-model weight-cache mount. Later starts reuse complete cached layers; model weights remain pinned in the host Hugging Face cache. ### Long context: what to expect The `long-context` profile accepts prompts up to 262,144 tokens and they complete, but treat anything above ~32k tokens as a batch job rather than an interactive request. Latency follows one rule: `end-to-end = TTFT(prompt) + max_tokens x TPOT` - TTFT is linear in prompt length at roughly 2.6 ms per prompt token (prefill ~400 tokens/s): 16k ~ 38 s, 64k ~ 2.7 min, 131k ~ 5.6 min, 262k ~ 11.5 min. Set client timeouts accordingly (at least 10 min for 131k prompts) and use `stream: true` so the first token is visible. - TPOT does not depend on prompt length: about 41 ms/token (24 tok/s) while the context is under the 2,048-token sparse-attention budget and about 72 ms/token (14 tok/s) above it, independent of the 262,144-token capacity. Reducing `max_tokens` saves only 72 ms per removed token (256 -> 64 saves ~14 s on a 3-minute 64k request). - There is no prefix caching: every request, including each turn of a multi-turn chat over a long document, re-prefills the whole prompt. - The first long request after a server start pays a few seconds of one-time kernel compilation. Measured 2026-09-17 on two P300 boards (four devices) against this exact package on the `long-context` profile, with the tt-model community latency-sweep harness: one closed-loop user, random token prompts of exactly ISL tokens, output pinned to OSL tokens with `ignore_eos`, one unrecorded warm-up request per input length, greedy sampling. TTFT and TPOT are means; prefill tok/s = ISL / TTFT, decode tok/s/user = 1 / TPOT. | ISL | OSL | Users | Requests | Mean TTFT | Prefill tok/s | Mean TPOT | Decode tok/s/user | E2EL | | ---: | ---: | ---: | ---: | ---: | ---: | ---: | ---: | ---: | | 128 | 128 | 1 | 4 | 0.34 s | 379 | 41.2 ms | 24.3 | 5.6 s | | 1,024 | 256 | 1 | 4 | 1.59 s | 642 | 41.3 ms | 24.2 | 12.1 s | | 4,096 | 256 | 1 | 4 | 7.99 s | 513 | 72.1 ms | 13.9 | 26.4 s | | 16,384 | 256 | 1 | 2 | 38.27 s | 428 | 72.4 ms | 13.8 | 56.7 s | | 32,768 | 256 | 1 | 1 | 79.25 s | 413 | 72.4 ms | 13.8 | 97.7 s | | 65,536 | 256 | 1 | 1 | 163 s | 402 | 72.5 ms | 13.8 | 182 s | | 131,072 | 256 | 1 | 1 | 338 s | 388 | 72.4 ms | 13.8 | 357 s | Decode is about 41 ms/token (24 tok/s per user) for prompts up to the 2,048-token sparse-attention budget and about 72 ms/token (14 tok/s) beyond it, independent of the configured 262,144-token capacity. The serving profile is intentionally batch one; later requests queue in vLLM instead of time-slicing a virtual state bank. Tool calling was verified end to end on this package (single, parallel, streaming, and tool-result round-trip requests). Queued work: re-enable the bounded on-device sampler for `temperature > 0` requests (removes the ~18 ms/token host-sampling cost below; the sampler now has a distribution gate and an endpoint A/B, and the switch flips back once the reported repetition loops are reproduced or ruled out), a capacity-independent sparse-attention selector (decode above 2k tokens from ~72 toward ~45 ms/token and 10-20 % off TTFT above 2k), then traced prefill (~1.5x TTFT). Prefix caching is not planned: the hybrid DeltaNet / n-gram state cannot be rebuilt from KV blocks. ### Evaluation Measured on this exact package configuration on two P300 boards, against a BF16 Hugging Face reference (Transformers 5.16) of the same checkpoint revision run on the CPU. Agreement numbers are teacher-forced: the reference's tokens are fed in and the port's next-token prediction is compared position by position, so they measure numerical fidelity, not task skill. Teacher set (`doc/correctness/teacher_set/`, 14 prompts, 1,386 decode positions, greedy): | domain | prompts | decode positions | top-1 | ref. token in TT top-5 | | --- | ---: | ---: | ---: | ---: | | mathematics (competition, word problem) | 2 | 198 | 97.0 % | 100.0 % | | code (function, debugging) | 2 | 198 | 97.5 % | 100.0 % | | JSON extraction | 1 | 99 | 97.0 % | 100.0 % | | Chinese reasoning, Japanese summary | 2 | 198 | 97.5 % | 100.0 % | | chat (system persona, instruct) | 2 | 198 | 97.0 % | 100.0 % | | summarization | 1 | 99 | 98.0 % | 100.0 % | | tool-call formatting | 1 | 99 | 93.9 % | 100.0 % | | long documents (3k / 6k / 12k tokens) | 3 | 297 | 93.9 % | 100.0 % | | **all** | 14 | 1,386 | **96.3 %** | **100.0 %** | Prompts up to 2,048 tokens agree at 97.0 % top-1; the long documents run through the sparse-attention selector and agree at 93.9 % (see Known limitations). The reference token is in the port's top-100 at 100.0 % of all positions. - AIME24 competition prompt (201 tokens), 99 teacher-forced steps: top-1 97.98 %, top-5 100 %, top-100 100 %. The free-run greedy continuation matches the reference for its first 5 tokens and stays coherent for 100 tokens (`doc/correctness/evidence_fix_geometry_20260917/full_c512_a1_s1/`). - Prefill, every position (12 prompts, 4,594 positions): top-1 88.8 %, reference top-1 in the port's top-5 98.8 %, mean top-100 logit PCC 0.938. - GSM8K through the served endpoint (100 vendored items, `doc/correctness/gsm8k_100.jsonl`, greedy, thinking off, `max_tokens 512`, exact match on the final number): **92 / 100** (8 answers cut at the token cap, no degenerate or empty replies; `doc/correctness/gsm8k_endpoint_20260917_p1_100.json`). - Sampling: with `temperature 1.0 / top_p 0.95 / top_k 20`, 8 of 8 AIME24 requests through the endpoint finish on their own with the correct answer. Reproduce in the tt-metal checkout (`RUN_QWEN38_ACCURACY=1` and `RUN_QWEN38_TEACHER_SET=1` select the gates in `tests/test_full_model.py`; `demo/run_gsm8k_endpoint.py --limit 100` scores a running server). ### Known limitations - Competition mathematics and very long reasoning: on AIME24-style problems this port scores well below the model card. Chains of thought beyond roughly 25-30k reasoning tokens can drift and lose the final answer even when the mathematics is sound; cap `max_tokens` near 32k for such prompts and prefer several shorter attempts over one very long one. - Long-context accuracy: after prompts above 2,048 tokens the sparse attention path (top-512 block selection) agrees with the BF16 reference at about 94 % top-1 / 100 % top-5 on our 3k-12k-token teacher documents, versus 97 % top-1 for short prompts. Long-document answers can still differ from the reference model more often than short-prompt answers. - Host sampling (precautionary in this revision): every request with `temperature > 0`, penalties, `logprobs` or a seed is sampled by vLLM on the host from the full logits, which costs about 18 ms per generated token more than greedy (17 instead of 24 tok/s under 2k tokens of context, 11 instead of 14 above). Because the checkpoint's own defaults set `temperature 1.0`, this is the path a request that omits sampling fields takes; send `temperature: 0` for the greedy figures in the table. The on-device stochastic sampler can be restored per deployment with `QWEN38_DEVICE_SAMPLING_MAX_TOP_K=32` in the serve environment. - Batch one: concurrent requests queue in vLLM; throughput does not scale with users. ### Checkpoint size The pinned snapshot is 360 GB (335 GiB): 180.0 B parameters in BF16 with nothing duplicated - 123.6 B routed experts, a 51.2 B n-gram (PLE) lookup table that stays memory-mapped on the host, 3.5 B dense / attention / DeltaNet weights, 1.3 B embeddings and LM head, and a 0.45 B vision tower this text-only endpoint does not load. About 6.7 B parameters are active per token; the transformer proper is ~128 B. The first serve converts the experts once into a ~64 GiB BFP4 resident-expert cache (17 GB per device) in tt-model's per-model weight cache; later starts reuse it. Runtime telemetry reported all 48 expert layers resident with 16,986,931,200 expert bytes per device, zero expert host-store bytes, zero expert H2D, and zero route D2H/stall time. Only PLE n-gram row lookup and selected-row DMA remain host-backed. ## Provenance The exact sources the image was built from — `code/` in this repo is byte-identical to the model code inside the image: | component | built from | | --- | --- | | tt-metal | a local checkout — commit not published *(dirty tree — the image includes uncommitted changes)* | | vLLM | [`v0.24.0`](https://github.com/vllm-project/vllm/releases/tag/v0.24.0) | | vllm-tt-plugin | a local checkout — commit not published *(dirty tree — the image includes uncommitted changes)* | | `code/` digest | `1b5c8d17b1caee5f` (sha256, first 16 hex digits) | | built | 2026-09-17T16:37:48+00:00 by tt-model 0.1.0 |