# Serving LocateAnything-3B on Blackhole with tt-model-manager

This repo is a **tt-model container package source**: `tt-model.yaml` + `code/` build an OCI
image that `tt-model serve` / `tt serve` run on a Blackhole p150a host. The server is
`code/locate_anything/server/app.py` (FastAPI, `kind: tt-dit-server`); the model recipe it
drives is `code/locate_anything/tt/pipeline.py`, lifted from the port's validated demo and
benchmark. Read this file top to bottom before touching the box.

Facts that shape everything below:

- **tt-metal tree**: `/home/deepgadget/experiments/gbp-tt/tt-metal` (main `8b98410e730`,
  v0.78.0-dev20260820-25; torch 2.11.0+cpu, transformers 5.12.1, numpy 1.26.4). The venv at
  `<tree>/python_env` is Python **3.10** (the image uses 3.12); it has ttnn, torch, transformers,
  but **not** fastapi/uvicorn/sniffio.
- **Weights**: `nvidia/LocateAnything-3B @ c32291ca5e996f5a7a485845b4f57a233936bba0` (public,
  ungated, NVIDIA 'other' license). Already in the host HF cache
  (`~/.cache/huggingface/hub/models--nvidia--LocateAnything-3B/snapshots/c32291ca...`).
- The LLM checkpoint tt_transformers loads is **extracted** from that snapshot on first boot
  (~6.2 GB `model.safetensors` + a synthesized Qwen2 `config.json` + tokenizer files) into
  `$TT_DIT_CACHE_DIR/la-qwen2_5-3b`. The app then sets `HF_MODEL` to that dir **before** building
  `ModelArgs` - the launcher's `HF_MODEL=nvidia/LocateAnything-3B` is only the weights pointer.
- One chip, mesh `1x1`, batch 1, one image per request, greedy AR decode with trace replay.

## 1. Run on the HOST (hardware-validation phase)

Everything from the repo dir. `$ROOT` = `/home/deepgadget/experiments/tt-models`.

```bash
ROOT=/home/deepgadget/experiments/tt-models
REPO=$ROOT/models/locate-anything-3b-p150
TREE=/home/deepgadget/experiments/gbp-tt/tt-metal
VENV=$TREE/python_env

# The tree venv lacks the HTTP stack (fastapi, uvicorn, sniffio, typing_inspection). Do NOT
# modify the tree venv: stage them beside it and put that dir first on PYTHONPATH. Note
# `uv pip install --target` also drops in pydantic 2.13.5 etc., which then shadow the venv's
# pydantic 2.9.2 -- harmless (the image resolves pydantic 2.13.5 too); the offline import
# check here was done with only the four missing packages staged, venv pydantic 2.9.2.
mkdir -p /tmp/la-hostdeps
~/.local/bin/uv pip install --python $VENV/bin/python --target /tmp/la-hostdeps fastapi uvicorn sniffio typing_inspection

export TT_METAL_HOME=$TREE
export ARCH_NAME=blackhole
export PYTHONPATH=/tmp/la-hostdeps:$REPO/code:$TREE:$TREE/ttnn:$TREE/tools

# What the tt-dit-server launcher exports inside the container (mirror it):
export HF_MODEL=nvidia/LocateAnything-3B
export MESH_DEVICE=P150
export TT_MESH_SHAPE=1x1
export TT_WEIGHTS_REVISION=c32291ca5e996f5a7a485845b4f57a233936bba0
export TT_METAL_VISIBLE_DEVICES=0
export LA_IN_TOKEN_LIMIT=1024 LA_PREC=accuracy LA_TRACE=1 LA_WARMUP_RUNS=2

# The writable caches tt-model mounts in the container; use the same host dirs so the
# container run later reuses the extracted LLM dir and the BFP8 tensor cache.
CACHE=~/.cache/tt-model/locate-anything-3b-p150
mkdir -p $CACHE/weights $CACHE/tensors $CACHE/cache
export TT_DIT_CACHE_DIR=$CACHE/weights    # -> extracted LLM at $CACHE/weights/la-qwen2_5-3b
export TT_CACHE_PATH=$CACHE/tensors       # -> $CACHE/tensors/P150/tensor_cache_bfp8
export TT_METAL_CACHE=$CACHE/cache        # JIT kernels

cd $REPO
$VENV/bin/python -m uvicorn --host 127.0.0.1 --port 20000 --lifespan on locate_anything.server.app:app
```

Boot log landmarks (stdout): `Loading weights: resolving ...` -> `Loading weights: extracting
...` (first boot only, ~6.2 GB) -> `Opening mesh device (1, 1)` -> `Loading weights: Qwen2.5-3B
LLM ...` (BFP8 conversion on the first boot) -> `Loading weights: MoonViT ...` -> `Warming up` ->
`Warmup run 1/2`, `Warmup run 2/2` -> `Warmup complete` -> uvicorn's `Application startup
complete`. `GET /health` answers `starting` until then. A startup failure raises and uvicorn
exits non-zero (`Application startup failed`) - there is no CPU fallback by design.

Smoke test (stdlib only, from anywhere):

```bash
python $REPO/code/locate_anything/server/smoke_test.py --url http://127.0.0.1:20000 --save /tmp/la-overlay.png
# -> PASS N boxes labels=['car'] tokens=.. eos=True grid=[24, 44] canon=[616, 336] mode=ar_greedy_trace vision=..ms prefill=..ms decode=..ms (.. tok/s) ...
```

Compare `/tmp/la-overlay.png` with `media/demo_ar.png`. Stop with Ctrl-C / SIGTERM: the
lifespan shutdown drops the device tensors and closes the mesh (within the 120 s `tt-model stop`
allows).

Extra knobs while validating (all env, all read at startup):

| var | default | meaning |
|---|---|---|
| `LA_WEIGHTS_DIR` (alias `LA_MODEL_PATH`) | unset | directory holding the snapshot files (skips the HF-cache resolve) |
| `LA_LLM_DIR` | `$TT_DIT_CACHE_DIR/la-qwen2_5-3b` | where the extracted Qwen2.5-3B dir lives; keep 'it'/'instruct' out of the path (ModelArgs heuristic) |
| `LA_FORCE_EXTRACT=1` | off | re-extract even if `model.safetensors` exists |
| `LA_CANON_GRID` | `24x44` (derived) | fixed vision grid `HxW` in 14-px patches, even sides; `LA_CANON_ASPECT=WxH` (default `1920x1080`) picks the aspect it is derived from at `LA_IN_TOKEN_LIMIT` |
| `LA_PREC` | `accuracy` | `bfp8attn` = all-BFP8 attention/KV (stock-only fallback if BFP8-MLP decode does not fit L1, see caveats) |
| `LA_TRACE` | `1` | `0` disables trace-replay decode (7.5 tok/s instead of ~38) |
| `LA_TRACE_REGION_SIZE` | `50000000` | bytes, validated |
| `LA_MAX_SEQ_LEN` | `4096` | LLM context (paged KV 32 x 1024 blocks) |
| `LA_MAX_NEW_TOKENS` / `LA_MAX_NEW_TOKENS_CAP` | `128` / `1024` | request default / hard cap |
| `TT_DEVICE_ID` | unset | pass a physical device id to `open_mesh_device` |

## 2. Package, serve, push (from the repo dir)

```bash
ROOT=/home/deepgadget/experiments/tt-models
cd $ROOT/models/locate-anything-3b-p150
source $ROOT/bin/docker-env.sh                      # rootless Docker 28 + buildx; plain `docker` is podman here

# offline check (must print VALID; run from THIS dir - extra_code.root is CWD-relative)
$ROOT/.venv/bin/python -c "from tt_kernel.container_manifest import load_container_manifest; m = load_container_manifest('tt-model.yaml', check_sources=True); p = m.resolve_profile(); print('VALID', m.name, m.kind, p.hardware, p.mesh_device, m.weights_ref)"

$ROOT/.venv/bin/tt-model package --container tt-model.yaml --out $ROOT/build     # hours cold; --out outside the git checkout
$ROOT/.venv/bin/tt-model serve   $ROOT/build/locate-anything-3b-p150/tt_kernel_manifest.json
python code/locate_anything/server/smoke_test.py --url http://127.0.0.1:<port serve printed>
$ROOT/.venv/bin/tt-model stop    changh95/locate-anything-3b-p150
$ROOT/.venv/bin/tt-model push    $ROOT/build/locate-anything-3b-p150 --publish
```

`tt-model serve` pre-downloads the pinned snapshot (honouring `ignore_patterns`) into the host
HF cache and mounts it at `/hf`; the app resolves it offline with `revision=$TT_WEIGHTS_REVISION`
(a sha-pinned snapshot has no `refs/main`). The container's `/weight-cache`, `/tensor-cache`,
`/cache` are `~/.cache/tt-model/locate-anything-3b-p150/{weights,tensors,cache}` on the
host - the same dirs section 1 used, so a host validation run pre-seeds the container's first
boot. Readiness deadline is 30 min (`Application startup complete`); measured on this host a
cold first boot (extract 5 s + BFP8 conversion 25 s + JIT + warm-up) is a few minutes and a
warm boot ~15 s.

What `push` does to this repo: `code/` and `image/` become exactly the staged trees
(`extra_code.paths` = `locate_anything`, `models` (the `mlp.py` overlay), `scripts`,
`conftest.py` plus tt-metal's `models/{common,tt_transformers,demos/qwen25_vl}`), `README.md`
is replaced by the generated
card (all durable README content lives in `card.description` / `card.quickstart` of
`tt-model.yaml`; the orchestrator restores `license`/`pipeline_tag` tags afterwards), and
`media/`, `SERVING.md`, `.gitattributes`, `tt-model.yaml` survive at the root.

## 3. Request / response contract

`GET /health` -> `{"status": "ok"|"starting", "model": "LocateAnything-3B", "device": "Blackhole p150a mesh 1x1"}` (200 always).

`GET /info` -> model/task/io summary, hardware, `decode_mode`, `precision`, `weights` (repo,
revision, snapshot dir, extracted LLM dir), `source` (repo + commit), `limits` (canonical grid /
size, `in_token_limit`, `max_seq_len`, token caps, query length), `license`.

`GET /v1/models` -> OpenAI-shaped stub (`{"object": "list", "data": [{"id": "nvidia/LocateAnything-3B", ...}]}`)
so the tt-model ready card does not 404. This is not an OpenAI API.

`POST /predict` (JSON):

| field | type | notes |
|---|---|---|
| `image` | str | base64 PNG/JPEG, exactly one image |
| `query` | str | required, 1-1000 chars; several categories joined with `</c>` (`person</c>car`) |
| `max_new_tokens` | int? | default `LA_MAX_NEW_TOKENS` (128), cap 1024; prompt + budget must fit 4096 |
| `return_overlay` | bool | default false; adds `overlay_png_b64` |

Response: `query`, `width`, `height` (original), `canonical_size [W,H]` + `grid_hw [h,w]` (what
the model saw: 616x336 / 24x44 by default), `num_image_tokens` (264), `prompt_tokens`,
`prefill_tokens` (padded prefill length, 512 for short queries), `raw_text` (the generated
`<ref>label</ref><box><x1><y1><x2><y2></box>...<|im_end|>`), `detections: [{label, box:[x1,y1,x2,y2]
in ORIGINAL pixels, box_norm:[0..1000 ints]}]`, `points: [{label, point, point_norm}]` (2-coord
outputs), `num_generated_tokens`, `stopped_on_eos`, `decode_mode` (`ar_greedy_trace` |
`ar_greedy`), `timing_ms {vision, prefill, decode, total, decode_tok_s}`, `overlay_png_b64`
(null unless requested).

Errors: 400 bad image / empty query / prompt too long (`ValueError` from the pipeline), 503
while starting, 500 with `ExceptionType: text` otherwise. Requests are serialized (one
device-worker thread + a lock); a second client waits.

Preprocessing: the HF processor's rescale (bicubic, multiples of 28 px, `in_token_limit=1024`)
runs first, then the image is squash-resized onto the canonical grid; for the demo image the
first step already lands on 616x336 so it is byte-identical to the validated demo run.
Non-16:9 inputs are distorted (boxes still map back exactly); set `LA_CANON_GRID` to match your
input aspect if that matters. The grid is fixed for the life of the server (MoonViT is built
once per grid).

## 4. Caveats and open items for the hardware phase

1. **`mlp.py` overlay (RESOLVED on hardware, 2026-09-12).** Stock gbp-tt
   `models/tt_transformers/tt/mlp.py` fails on one p150a at the FIRST decode step (warm-up), in
   the `w3` `ttnn.linear`: `TT_THROW program.cpp:1925: Statically allocated circular buffers in
   program 161 clash with L1 buffers on core range [0-0 - 7-9]. L1 buffer allocated at 1324800
   and static circular buffer region ends at 1365376` (40 KB short). Forcing the `w1`/`w3`
   outputs to BFP8 alone (the `bench_locate_anything.py` comment) was still 10 KB short (the
   matmul then needs a separate interm CB). What fits is the port README's "spilled to DRAM"
   variant, shipped here as `code/models/tt_transformers/tt/mlp.py` (listed under
   `extra_code.paths` as `models`; extra_code is staged after `source.code` and wins on
   collision): in decode on a single chip (`not TG`, no prefetcher) `w1_out` is moved to DRAM
   *before* the `w3` matmul (freeing its 88 KB L1 shard while the weight-stream CBs are
   allocated) and `w3_out` right after; the gate/up `ttnn.mul` runs DRAM->DRAM and the stock
   `to_memory_config` brings `w2_in` back to L1 for `w2`. Output dtypes stay stock bf16 (numerics
   at least as good as the validated BFP8-intermediate run). Prefill / galaxy / prefetcher paths
   are untouched; the file is otherwise byte-identical to stock, and the image `verify:` asserts
   the overlay is in place (`'spill_ff1_3' in mlp.py`). Measured with it (in the image, same
   host): `media/demo_input.png` + `car` -> `<ref>car</ref><box><282><414><606><794></box>`, the
   same box as `media/demo_ar.png`; vision 49 ms, prefill 58 ms, decode 41 tok/s, 346 ms
   end-to-end. For a host run keep `$REPO/code` ahead of the tree on `PYTHONPATH` so the overlay
   shadows the stock module (section 1 already does). `LA_PREC=bfp8attn` remains a knob but is
   not needed.
2. **transformers 5.12.1 vs the port's 4.53.0.** The tree pins 5.12.1; the extracted
   `config.json` still says `transformers_version 4.53.0` (informational). Offline checks here:
   `AutoTokenizer` loads the snapshot tokenizer as `Qwen2Tokenizer` (0.4 s, no `tokenizer.json`
   upstream), the `<IMG_CONTEXT>` count matches (264), and `ModelArgs.create_tokenizer` passes
   `trust_remote_code=False`. The extractor strips the upstream `auto_map` from
   `tokenizer_config.json` so the extracted dir loads as a plain Qwen2 checkpoint. transformers
   5.12 prints a "incorrect regex pattern ... fix_mistral_regex" warning for this tokenizer; it
   is the same tokenizer the tree's Qwen2.5 demos use. Verify the raw box string on the box.
3. **First-boot budget.** Extraction (6.2 GB write) + bf16 HF load + BFP8 conversion + JIT +
   warm-up must finish inside tt-model's 30-minute readiness deadline. Section 1's host run
   pre-seeds the extracted dir and the tensor cache in the same host directories the container
   mounts, which takes the first two off the container's clock.
4. **Canonical grid.** 24x44 (16:9). Squash distortion for other aspects; `LA_CANON_GRID`.
   Per-request grids (a MoonViT per grid) are deliberately not implemented.
5. **Prefill program shapes.** `preprocess_inputs_prefill` pads the prompt to the next power of
   two (512 for the default ~303-token prompt). A long query that crosses 512 tokens compiles a
   1024-token prefill program once (slow first time, then cached).
6. **Not served**: MTP / Parallel Box Decoding (approximate, not gated), sampling
   (temperature/top-p), multi-image, batch > 1, `in_token_limit` per request.
7. `models/common/utility_functions.py` and `models/tt_transformers/tt/*.py` import `pytest`,
   `loguru`, `tqdm`, `yaml` at module level - hence those `runtime.packages`. `torchvision` is
   deliberately not listed (it would drag its own torch); `la_inputs.py` reimplements
   `to_tensor`/`normalize` in torch (bit-identical, checked offline).
8. The tree venv is Python 3.10; the image is 3.12 (uv dry-run of the exact package set on 3.12
   resolves to torch 2.11.0+cpu / numpy 1.26.4 / transformers 5.12.1, no torchvision).
9. **`TT_FUSED` fused device paths (default ON since the 2026-09-13 device validation).**
   Branch `opt/locate-anything-3b-p150-megakernel` adds fused device paths behind one env knob
   read once at startup (`code/locate_anything/tt/fused.py`): the MoonViT graph as one metal
   trace with the patch merger on device and a row-major pixel upload, the vision->LLM embedding
   merge on device, a traced 36-layer prefill with an eager norm/LM-head tail, a one-row
   first-token readback and on-device greedy sampling in the decode loop. The default fused
   configuration keeps the legacy vision numerics bit for bit (device-verified) and measured
   360.9 -> 304.9 ms per demo request in the host pipeline (see `DEVICE_VALIDATION.md`
   "Results" for the served numbers). `TT_FUSED=0` = the 2026-09-12 shipped device graph (same
   ops, same order, same arguments, same numerics); the only host-visible additions are the
   `"fused"` key in the pipeline `run()` result dict and the `fused` entry of `GET /info` -- the
   `/predict` body is unchanged. With the path on, `LA_TRACE_REGION_SIZE` defaults to 160000000
   and the warm-up must capture the vision and prefill traces before READY (startup error
   otherwise); `/info` reports `fused.config` / `fused.status` and `timing_ms.vision` becomes
   the non-blocking trace submit (compare `total`). Measured opt-ins that change the bf16
   rounding of the vision tower (not default; numbers in `DEVICE_VALIDATION.md`):
   `LA_FUSED_EXACT_SEQ=1 LA_FUSED_SDPA_CHUNKS=96,352 LA_FUSED_SDPA_EXP_APPROX=0` (another
   -13 ms; the demo box becomes the HF reference's `<282><414><607><797>` instead of the shipped
   `<606><794>`) and `LA_FUSED_MATMUL=minimal` (below the 0.99 PCC gate on the port golden).
   Host tests: `code/locate_anything/tests/test_fused_host.py`.