qwen3.8-27b-p150x4
A tt-model container package that serves Qwen/Qwen3.8-27B on four Tenstorrent P150 boards (mesh P150x4). The server is OpenAI-compatible with tool calling and separated reasoning output, a 131,584-token context, fused DeltaNet kernels, decode tracing, and on-device sampling. You need Docker and the boards on the host — no tt-metal, no vLLM, no venv.
Capabilities: tool calling — yes (qwen3_coder); reasoning parsing — yes (qwen3).
Quickstart
tt-model pull tt-hous/qwen3.8-27b-p150x4
tt-model serve tt-hous/qwen3.8-27b-p150x4
pull downloads the Docker image and the model weights. The weights go to your Hugging Face cache; they are not inside the image. serve starts the server on port 8000. The first start compiles kernels for your device, which takes several minutes; the server is ready when it logs Application startup complete. Add --follow to stream the server log.
Current performance
Measured with vLLM at concurrency 1: fixed input/output lengths, greedy sampling, EOS ignored, median of five requests after warmup. E2EL is derived as TTFT + (OSL - 1) * TPOT.
| input tokens | output tokens | TTFT | TPOT | end-to-end | decode tok/s/user |
|---|---|---|---|---|---|
| 128 | 128 | 0.157 s | 30.20 ms | 3.99 s | 33.11 |
| 8,192 | 512 | 1.40 s | 30.74 ms | 17.11 s | 32.53 |
| 32,768 | 512 | 5.96 s | 31.25 ms | 21.93 s | 32.00 |
| 65,536 | 512 | 13.05 s | 32.01 ms | 29.41 s | 31.24 |
| 131,072 | 512 | 30.97 s | 33.46 ms | 48.07 s | 29.89 |
What is in this repo
code/— the model's TT implementation and runtime, byte-identical to the code inside the imageimage/— the Docker image as an exploded OCI layout
The serving stack is vLLM 0.24.0 with the vllm-tt-plugin.
Next steps for optimization
This deployment is functional but not tuned:
- Batching. The server accepts one concurrent sequence. Raising
max_num_seqsand validating batched decode is the largest aggregate-throughput win on a four-board mesh. - Prefill latency. TTFT reaches ~31 s at the full 131k context while decode stays near-flat; prefill is the long-context bottleneck (chunked or traced prefill are the candidates).
- Prefix caching. Not enabled. Agent workloads that reuse long prompts would see a direct TTFT win.
- Decode headroom. Decode holds ~30–33 tok/s/user across all input lengths, so the decode path is stable; per-layer profiling of the DeltaNet blocks is the next place to look for rate improvements.