---
tags:
- blackhole
- p150x4
- tt-model-cache
- tt-model-catalog
- tt-model-container
- vllm-plugin
---

# qwen3.8-27b-p150x4

A tt-model container package that serves [Qwen/Qwen3.8-27B](https://huggingface.co/Qwen/Qwen3.8-27B) on four Tenstorrent P150 boards (mesh `P150x4`). The server is OpenAI-compatible with tool calling and separated reasoning output, a 131,584-token context, fused DeltaNet kernels, decode tracing, and on-device sampling. You need Docker and the boards on the host — no tt-metal, no vLLM, no venv.

**Capabilities:** tool calling — yes (`qwen3_coder`); reasoning parsing — yes (`qwen3`).

## Quickstart

```bash
tt-model pull  tt-hous/qwen3.8-27b-p150x4
tt-model serve tt-hous/qwen3.8-27b-p150x4
```

`pull` downloads the Docker image and the model weights. The weights go to your Hugging Face cache; they are not inside the image. `serve` starts the server on port 8000. The first start compiles kernels for your device, which takes several minutes; the server is ready when it logs `Application startup complete`. Add `--follow` to stream the server log.

## Current performance

Measured with vLLM at concurrency 1: fixed input/output lengths, greedy sampling, EOS ignored, median of five requests after warmup. E2EL is derived as `TTFT + (OSL - 1) * TPOT`.

| input tokens | output tokens | TTFT | TPOT | end-to-end | decode tok/s/user |
| ---: | ---: | ---: | ---: | ---: | ---: |
| 128 | 128 | 0.157 s | 30.20 ms | 3.99 s | 33.11 |
| 8,192 | 512 | 1.40 s | 30.74 ms | 17.11 s | 32.53 |
| 32,768 | 512 | 5.96 s | 31.25 ms | 21.93 s | 32.00 |
| 65,536 | 512 | 13.05 s | 32.01 ms | 29.41 s | 31.24 |
| 131,072 | 512 | 30.97 s | 33.46 ms | 48.07 s | 29.89 |

## What is in this repo

- `code/` — the model's TT implementation and runtime, byte-identical to the code inside the image
- `image/` — the Docker image as an exploded OCI layout

The serving stack is vLLM 0.24.0 with the vllm-tt-plugin.

## Next steps for optimization

This deployment is functional but not tuned:

- **Batching.** The server accepts one concurrent sequence. Raising `max_num_seqs` and validating batched decode is the largest aggregate-throughput win on a four-board mesh.
- **Prefill latency.** TTFT reaches ~31 s at the full 131k context while decode stays near-flat; prefill is the long-context bottleneck (chunked or traced prefill are the candidates).
- **Prefix caching.** Not enabled. Agent workloads that reuse long prompts would see a direct TTFT win.
- **Decode headroom.** Decode holds ~30–33 tok/s/user across all input lengths, so the decode path is stable; per-layer profiling of the DeltaNet blocks is the next place to look for rate improvements.