--- tags: - blackhole - p150 - tt-dit-server - tt-model-cache - tt-model-container --- # locate-anything-3b-p150 NVIDIA LocateAnything-3B (MoonViT-SO-400M vision tower + Qwen2.5-3B LLM with a detection vocabulary) doing visual grounding / open-vocabulary detection on a single Tenstorrent Blackhole p150a via tt-nn: image + free-text query -> labelled boxes. Vision tower and LLM both run on the chip; greedy AR decode with trace replay reaches ~38 tok/s, ~53 ms vision, ~64 ms prefill, ~2.3 frames/s end-to-end at PCC 0.9928 vs the torch reference. Weights are fetched from nvidia/LocateAnything-3B (NVIDIA license, 'other' - not OSI; read it before use); the port code (Apache-2.0 SPDX headers) is redistributed under the same upstream terms. Port source: github.com/changh95/tt-locate-anything @ 7a3b7407c4f1268cfb14ce1dd0ed4cdcd877b9a1. Runs on **p150** (mesh `P150`). Packaged and published with [tt-model-manager](https://github.com/tenstorrent/tt-model-manager) 0.1.0 (manifest schema 5.1). ## Quickstart ```bash tt-model pull changh95/locate-anything-3b-p150 --with-weights tt-model serve changh95/locate-anything-3b-p150 ``` `pull --with-weights` downloads the Docker image and the [`nvidia/LocateAnything-3B`](https://huggingface.co/nvidia/LocateAnything-3B) weights at `c32291ca5e996f5a7a485845b4f57a233936bba0` (into your HF cache; they are not in the image). `serve` starts the model's own HTTP server on port 20000 (or the next free port, if that one is busy); the first start compiles kernels for your device, which takes several minutes, and the server is ready when it logs `Application startup complete`. ### tt-cli users ```bash tt serve changh95/locate-anything-3b-p150 # pulls image + weights, boots, prints the port tt model stop changh95/locate-anything-3b-p150 ``` `tt-model stop changh95/locate-anything-3b-p150` does the same without tt-cli. ### What this server is (and is not) The port is the one `serve` printed (20000, or the next free one). This is the model's own HTTP API, **not** an OpenAI-compatible one: `curl /v1/models` only returns a stub so the ready card does not 404, and `tt-model curl` does not apply. The routes are: | route | what | |---|---| | `GET /health` | `{"status": "ok" \| "starting", "model", "device"}` - `ok` only after warm-up | | `GET /info` | model, weights repo + revision loaded, source commit, input limits, license | | `POST /predict` | one image + one query -> labelled boxes (JSON) | ```bash PORT=20000 # the port serve printed curl -s localhost:$PORT/health curl -s localhost:$PORT/info | python -m json.tool # media/demo_input.png (1920x1080) with the README's query "car"; several categories are # joined with , e.g. "personcar". The PNG is 3.6 MB (4.8 MB as base64), far above # the shell's 128 KiB per-argument limit, so an inline "$(base64 -w0 ...)" fails with # "Argument list too long": write the body to a file and post it with -d @file. python3 - <<'EOF' import base64, json img = base64.b64encode(open("media/demo_input.png", "rb").read()).decode() json.dump({"image": img, "query": "car", "max_new_tokens": 128}, open("/tmp/la_req.json", "w")) EOF curl -s localhost:$PORT/predict -H 'Content-Type: application/json' -d @/tmp/la_req.json \ | python3 -m json.tool ``` Request fields: `image` (base64 PNG/JPEG, exactly one image), `query` (required free text, 1-1000 chars; join categories with ``), `max_new_tokens` (optional, default 128, cap 1024; each box costs ~6-8 tokens plus its label), `return_overlay` (optional bool; adds `overlay_png_b64`, the boxes drawn on your image). Response: `detections: [{label, box: [x1, y1, x2, y2], box_norm: [..0..1000..]}]` with `box` in ORIGINAL image pixels, `points` (the model's 2-coordinate outputs, same shape), `raw_text` (the generated `car<282><414><606><794><|im_end|>` string for the demo image), `width`/`height`, `canonical_size` and `grid_hw` (what the model actually saw, see below), `prompt_tokens`, `num_generated_tokens`, `stopped_on_eos`, `decode_mode` (`ar_greedy_trace`), and `timing_ms` (`vision`, `prefill`, `decode`, `total`, `decode_tok_s`). Errors: 400 for a bad image / empty query / prompt too long, 503 while starting, 500 with the exception text otherwise. One request at a time is served (batch 1, stateful KV cache); concurrent calls queue. A one-line check that the hardware phase also runs: `python code/locate_anything/server/smoke_test.py --url http://127.0.0.1:$PORT` prints `PASS ...` with the box count and timings. ### Input handling Every image is preprocessed like the HF processor at `LA_IN_TOKEN_LIMIT=1024` (all README numbers use this cap; upstream's default is 25600) and then squash-resized onto ONE canonical vision grid so MoonViT is built once: by default the grid the demo image (16:9) gets at that cap, 24x44 patches = 336x616 px = 264 image tokens. Boxes are normalized over that view, so they map back to your original pixels exactly; non-16:9 images are distorted before the model sees them. The grid is fixed for the life of the server and comes from the `LA_CANON_GRID=HxW` environment variable (even numbers of 14-px patches, e.g. `32x32` for square inputs). `tt-model serve` passes the container only the manifest's `serve.env` (there is no `--env` flag), so this package always serves the 24x44 grid; to serve another one, run the app on a host with tt-metal (see `SERVING.md`) or repackage with the variable added to `serve.env` in `tt-model.yaml`. ### First boot - `tt-model serve` downloads ~7.7 GB of weights (2 safetensors shards + tokenizer/config) into your HF cache before the container starts. - Inside the container the app writes a vanilla Qwen2.5-3B checkpoint (6.8 GB) to `~/.cache/tt-model/locate-anything-3b-p150/weights/la-qwen2_5-3b` (once), converts the LLM weights to BFP8 into `.../tensors/P150/tensor_cache_bfp8` (once), JIT-compiles kernels into `.../cache` (once), uploads MoonViT and runs two warm-up passes (prefill + 4 decode steps; the decode trace is captured on the first). Measured on the validation host: a cold first boot is a few minutes (extraction ~5 s, BFP8 conversion ~25 s, JIT + warm-up ~1-2 min), later boots ~15 s; `tt-model logs -f changh95/locate-anything-3b-p150` shows `Loading weights`, `Warming up`, `Warmup complete`. Host RAM peak ~10 GB. - Disk: 7.7 GB (HF cache) + 6.8 GB (extracted LLM) + 4.3 GB (BFP8 tensor cache) + ~0.4 GB JIT cache. - The weights repo is public and ungated; no `hf auth login` is needed for it. ### Results (from the port README; single p150a, everything on device, warm, trace decode) | Input (`media/demo_input.png`) | Greedy AR decode (`media/demo_ar.png`) | |:---:|:---:| | ![](media/demo_input.png) | ![](media/demo_ar.png) | Accuracy (PCC vs the torch-CPU golden, gate >= 0.99): | Stage | PCC | |---|---:| | Vision `patch_embed` | 0.99999 | | Vision `encoder_out` (27 blocks) | 0.9809 | | Vision `vit_proj` (after `mlp1`) | 0.9911 | | LLM prefill last-token logits | 0.9922 | | **Full on-device logits (vision -> LLM)** | **0.9928** | Performance: | Metric | Value | |---|---:| | Decode throughput | ~38 tok/s | | Vision (MoonViT + projector) | ~53 ms | | Prefill | ~64 ms | | End-to-end | ~2.32 frames/s | The LLM runs BF16 attention + BFP8 MLP weights (the `accuracy` preset; BFP8 MLP is what the >= 0.99 gate needs, BFP4 reaches ~0.935). Decode is weight-bandwidth bound (~3 GB/token), so trace replay is the big win (7.5 -> 38 tok/s) and a second command queue is not. The experimental on-device Parallel Box Decoding (MTP, `media/demo_mtp.png`) is approximate, not accuracy-gated, and not served. The validated run needed one change to tt-metal's `models/tt_transformers/tt/mlp.py`, and this package ships it as an overlay (`code/models/tt_transformers/tt/mlp.py`, otherwise byte-identical to the stock file): in single-chip decode the `w1`/`w3` outputs are moved to DRAM before the next matmul, because with stock bf16 L1-sharded intermediates the `w3` weight-stream circular buffers do not fit L1 on one p150a (`Statically allocated circular buffers ... clash with L1 buffers` at the first decode step). Measured with the overlay on the validation host: `media/demo_input.png` + `car` -> the same box as `media/demo_ar.png`, vision 49 ms, prefill 58 ms, decode 41 tok/s, 346 ms end-to-end. `LA_PREC=bfp8attn` (all-BFP8 attention, PCC 0.9912) remains available. ### Layout notes `code/locate_anything/` is the port (`tt/vision.py` MoonViT, `tt/model_la.py` the embeds-fed Qwen2.5-3B `Transformer` subclass, `tt/pipeline.py` the serving recipe, `tt/mtp.py` the experimental MTP decoder, `reference/` the torch-CPU reference and the LLM-checkpoint extractor, `tests/` the PCC/benchmark/demo pytest suites, `server/` this app + smoke test). `code/models/{common,tt_transformers,demos/qwen25_vl}` are the tt-metal packages the LLM path reuses (stock Qwen2.5 `Transformer`/`Generator`/paged `Attention`), with the one overlay file `models/tt_transformers/tt/mlp.py` described above. The tests need a device and the torch-CPU goldens (`reference/run_reference.py`), which are not shipped. ### Licensing The **upstream model and weights** are licensed **`other`** (nvidia-license): https://huggingface.co/nvidia/LocateAnything-3B/blob/main/LICENSE - NVIDIA's own licence, not an OSI licence; read it before any use. The weights are **not redistributed** here; they are fetched from the upstream repo under whatever terms that repo sets. The **port code** (Apache-2.0 SPDX headers; written by Hyunggi Chang, github.com/changh95/tt-locate-anything) is published under the same upstream terms, since a port cannot grant more than its upstream does. Backbone: Qwen2.5-3B-Instruct (Apache-2.0); vision tower: MoonViT (Kimi-VL, Moonshot AI); runtime: Tenstorrent tt-metal / tt-nn (Apache-2.0). ## Provenance The exact sources the image was built from — `code/` in this repo is byte-identical to the model code inside the image: | component | built from | | --- | --- | | tt-metal | [`8b98410e730bb504fea43a88609756e34821d91d`](https://github.com/tenstorrent/tt-metal/commit/8b98410e730bb504fea43a88609756e34821d91d) | | `code/` digest | `eb2ce7d7357c84e3` (sha256, first 16 hex digits) | | built | 2026-09-12T12:20:44+00:00 by tt-model 0.1.0 |