README.md

---
tags:
- blackhole
- p150
- tt-dit-server
- tt-model-cache
- tt-model-container
- tenstorrent
- ttnn
- tt-metal
- tt-nn
- visual-grounding
- open-vocabulary
- vlm
- qwen2.5
- tt-model-catalog
license: other
base_model:
- nvidia/LocateAnything-3B
license_link: https://huggingface.co/nvidia/LocateAnything-3B/blob/main/LICENSE
pipeline_tag: object-detection
license_name: nvidia-license
---

# locate-anything-3b-p150

NVIDIA LocateAnything-3B (Eagle-family visual grounding / open-vocabulary detection VLM: MoonViT-SO-400M vision tower + Qwen2.5-3B-Instruct with a detection vocabulary) running entirely on one Tenstorrent Blackhole p150a via tt-nn: image + free-text query in, labelled boxes out.
Weights: [nvidia/LocateAnything-3B](https://huggingface.co/nvidia/LocateAnything-3B) · Paper: [arXiv:2605.27365](https://arxiv.org/abs/2605.27365) · Upstream code: [NVlabs/Eagle (Embodied)](https://github.com/NVlabs/Eagle/tree/main/Embodied) · Port: [changh95/tt-locate-anything](https://github.com/changh95/tt-locate-anything)

Runs on **p150** (mesh `P150`).

Packaged and published with [tt-model-manager](https://github.com/tenstorrent/tt-model-manager) 0.1.0 (manifest schema 5.1).

## Quickstart

```bash
tt-model pull  changh95/locate-anything-3b-p150 --with-weights
tt-model serve changh95/locate-anything-3b-p150
```

- Weights [`nvidia/LocateAnything-3B`](https://huggingface.co/nvidia/LocateAnything-3B) at `c32291ca5e99` go to your HF cache; the image does not contain them.
- Serves on port 20000 (or the next free port); ready when the log says `Application startup complete`.

### Run with tt-cli

```bash
tt serve changh95/locate-anything-3b-p150
{ printf '{"query":"car","image":"'; base64 -w0 media/demo_input.png; printf '"}'; } > req.json
curl -s localhost:20000/predict -H 'Content-Type: application/json' -d @req.json
tt model stop changh95/locate-anything-3b-p150
```

- `POST /predict`: `image` (base64 PNG/JPEG, one image), `query` (free text, 1-1000 chars; join categories with `</c>`, e.g. `person</c>car`); optional `max_new_tokens` (128, cap 1024), `return_overlay` (false).
- `GET /health`, `GET /info`.

### Response

```json
{"query": "car", "width": 1920, "height": 1080, "canonical_size": [616, 336], "grid_hw": [24, 44],
 "raw_text": "<ref>car</ref><box><282><414><606><794></box><|im_end|>",
 "detections": [{"label": "car", "box": [541.44, 447.12, 1163.52, 857.52], "box_norm": [282, 414, 606, 794]}],
 "points": [], "num_generated_tokens": 10, "stopped_on_eos": true, "decode_mode": "ar_greedy_trace_device_sampling",
 "timing_ms": {"vision": 1.4, "prefill": 86.3, "decode": 202.1, "total": 303.9, "decode_tok_s": 44.53}}
```

- `box` is `[x1, y1, x2, y2]` in original image pixels; `box_norm` is the model's own 0..1000 output over the squashed `canonical_size` view. `points` holds 2-coordinate outputs the same way.
- `return_overlay: true` adds `overlay_png_b64`, the boxes drawn on your image as a base64 PNG.

### Demo

| Input (`media/demo_input.png`), query `car` | Greedy decode on p150a (`media/demo_ar.png`) |
|:---:|:---:|
| ![](media/demo_input.png) | ![](media/demo_ar.png) |

### Accuracy and speed

| Metric | Value |
|---|---:|
| Full on-device logits PCC vs torch-CPU reference (vision → LLM) | 0.9919 (gate ≥ 0.99; vision projector 0.9924, LLM prefill 0.9915) — measured 2026-09-13 on the p150a against goldens regenerated with the port's torch 2.7.1 / transformers 4.53.0; identical with `TT_FUSED=0` (the fused default keeps the vision numerics bit for bit) |
| Decoded box string on the demo image | `<ref>car</ref><box><282><414><606><794></box>`, unchanged from the 2026-09-12 package (the HF torch-CPU reference generated with the port's torch 2.7.1 / transformers 4.53.0 gives `<282><414><607><797>`, IoU 0.989) |
| `/predict` served over HTTP (warm, batch 1, `car`, 10 tokens; median of 100 requests, 2026-09-13) | **~305 ms** server-side (304–306; vision-trace submit 2 · prefill incl. the vision wait 86 · decode 202 ms, ~44.5 tok/s) · ~0.34 s wall. `TT_FUSED=0` (the 2026-09-12 graph): ~357 ms (351–365; vision 48 · prefill 62 · decode 232 ms, ~39 tok/s) · ~0.40 s wall |
| Same forward on an RTX 5090 (same host, port's torch reference, eager PyTorch, batch 1, incl. H2D/D2H; bf16 / fp16 autocast) | bf16 native weights 158 ms (83 tok/s decode) → GPU 1.8× faster than the p150a's 290 ms (vision + prefill + decode, 44.5 tok/s); bf16/fp16 autocast over fp32 weights 218 / 215 ms (1.3×); fp32-strict 216 ms (1.3×; vision+prefill 92 ms vs p150a 88 ms — p150a faster); best `torch.compile` 137 ms (2.1×) |

### Caveats

- One image + one query per request, batch 1, requests are serialized. Every image is squash-resized onto a fixed 24×44-patch grid (616×336, 16:9; `LA_IN_TOKEN_LIMIT=1024`, upstream default 25600): non-16:9 images are distorted before the model sees them, boxes still map back to original pixels.
- bf16 vision, BF16 attention + BFP8 MLP LLM; the package ships an `mlp.py` overlay (`code/models/tt_transformers/tt/mlp.py`, decode `w1`/`w3` spilled to DRAM) so the LLM fits L1 on one p150a. Fused device paths (MoonViT as one metal trace with the patch merger on device, device vision→LLM merge, traced prefill, one-row first token, on-device greedy argmax) are on by default since 2026-09-13; `TT_FUSED=0` restores the 2026-09-12 graph, and the vision numerics are unchanged bit for bit either way. Only greedy AR decode is served; the experimental Parallel Box Decoding (MTP) is not.
- Weights are public and ungated but under NVIDIA's own license (not OSI); ~7.7 GB download, and the first boot extracts a 6.8 GB Qwen2.5-3B checkpoint, converts it to BFP8 and JIT-compiles (~100 s on the very first boot, ~50 s with cached weights but an empty kernel cache, ~17 s warm; measured 2026-09-13).
- Not an OpenAI-compatible API; `GET /v1/models` is a stub so the tt-model ready card does not 404.
- Validated on tt-metal `v0.78.0-dev20260820` (main `8b98410e730`), single p150a only.
- GPU comparison: GPU (Qwen2.5-VL-3B in HF transformers) bf16 is 1.8× faster overall; on vision + prefill the p150a is slightly faster than fp32-strict GPU (88 vs 92 ms) and the gap is in decode (44.5 vs 83 tok/s). RTX 5090 rows (2026-09-14): same host, the port's own torch reference (same weights) run eagerly in PyTorch 2.11 cu128 with fp32 weights + autocast unless stated, no TensorRT; medians of 50 iterations after warm-up, H2D/D2H included; the p150a rows are the served bf16 fused path incl. upload/readback. p150a power was not measured, so no efficiency comparison is made. Full table: [`GPU_COMPARISON.md`](GPU_COMPARISON.md).

### Licensing

- Weights: [nvidia/LocateAnything-3B](https://huggingface.co/nvidia/LocateAnything-3B), `other` ([NVIDIA license](https://huggingface.co/nvidia/LocateAnything-3B/blob/main/LICENSE), not OSI); fetched from upstream, not redistributed here.
- Port and serving code (`code/locate_anything`, `code/scripts`): Apache-2.0 SPDX headers, from [changh95/tt-locate-anything](https://github.com/changh95/tt-locate-anything), published under the same upstream terms; the vendored [tt-metal](https://github.com/tenstorrent/tt-metal) overlay `code/models/tt_transformers/tt/mlp.py` is Apache-2.0.

## Provenance

The exact sources the image was built from — `code/` in this repo is byte-identical to the model code inside the image:

| component | built from |
| --- | --- |
| tt-metal | [`8b98410e730bb504fea43a88609756e34821d91d`](https://github.com/tenstorrent/tt-metal/commit/8b98410e730bb504fea43a88609756e34821d91d) |
| `code/` digest | `27dc2b154aa86da9` (sha256, first 16 hex digits) |
| built | 2026-09-13T15:59:45+00:00 by tt-model 0.1.0 |