Card: RTX 5090 comparison row + caveat

5a97ae5 verified 4 days ago

7.69 kB

tags:
  - blackhole
  - p150
  - tt-dit-server
  - tt-model-cache
  - tt-model-container
  - tenstorrent
  - ttnn
  - tt-metal
  - tt-nn
  - visual-grounding
  - open-vocabulary
  - vlm
  - qwen2.5
  - tt-model-catalog
license: other
base_model:
  - nvidia/LocateAnything-3B
license_link: https://huggingface.co/nvidia/LocateAnything-3B/blob/main/LICENSE
pipeline_tag: object-detection
license_name: nvidia-license

locate-anything-3b-p150

NVIDIA LocateAnything-3B (Eagle-family visual grounding / open-vocabulary detection VLM: MoonViT-SO-400M vision tower + Qwen2.5-3B-Instruct with a detection vocabulary) running entirely on one Tenstorrent Blackhole p150a via tt-nn: image + free-text query in, labelled boxes out. Weights: nvidia/LocateAnything-3B · Paper: arXiv:2605.27365 · Upstream code: NVlabs/Eagle (Embodied) · Port: changh95/tt-locate-anything

Runs on p150 (mesh P150).

Packaged and published with tt-model-manager 0.1.0 (manifest schema 5.1).

Quickstart

tt-model pull  changh95/locate-anything-3b-p150 --with-weights
tt-model serve changh95/locate-anything-3b-p150

Weights nvidia/LocateAnything-3B at c32291ca5e99 go to your HF cache; the image does not contain them.
Serves on port 20000 (or the next free port); ready when the log says Application startup complete.

Run with tt-cli

tt serve changh95/locate-anything-3b-p150
{ printf '{"query":"car","image":"'; base64 -w0 media/demo_input.png; printf '"}'; } > req.json
curl -s localhost:20000/predict -H 'Content-Type: application/json' -d @req.json
tt model stop changh95/locate-anything-3b-p150

POST /predict: image (base64 PNG/JPEG, one image), query (free text, 1-1000 chars; join categories with </c>, e.g. person</c>car); optional max_new_tokens (128, cap 1024), return_overlay (false).
GET /health, GET /info.

Response

{"query": "car", "width": 1920, "height": 1080, "canonical_size": [616, 336], "grid_hw": [24, 44],
 "raw_text": "<ref>car</ref><box><282><414><606><794></box><|im_end|>",
 "detections": [{"label": "car", "box": [541.44, 447.12, 1163.52, 857.52], "box_norm": [282, 414, 606, 794]}],
 "points": [], "num_generated_tokens": 10, "stopped_on_eos": true, "decode_mode": "ar_greedy_trace_device_sampling",
 "timing_ms": {"vision": 1.4, "prefill": 86.3, "decode": 202.1, "total": 303.9, "decode_tok_s": 44.53}}

box is [x1, y1, x2, y2] in original image pixels; box_norm is the model's own 0..1000 output over the squashed canonical_size view. points holds 2-coordinate outputs the same way.
return_overlay: true adds overlay_png_b64, the boxes drawn on your image as a base64 PNG.

Demo

Input (`media/demo_input.png`), query `car`	Greedy decode on p150a (`media/demo_ar.png`)

Accuracy and speed

Metric	Value
Full on-device logits PCC vs torch-CPU reference (vision → LLM)	0.9919 (gate ≥ 0.99; vision projector 0.9924, LLM prefill 0.9915) — measured 2026-09-13 on the p150a against goldens regenerated with the port's torch 2.7.1 / transformers 4.53.0; identical with `TT_FUSED=0` (the fused default keeps the vision numerics bit for bit)
Decoded box string on the demo image	`<ref>car</ref><box><282><414><606><794></box>`, unchanged from the 2026-09-12 package (the HF torch-CPU reference generated with the port's torch 2.7.1 / transformers 4.53.0 gives `<282><414><607><797>`, IoU 0.989)
`/predict` served over HTTP (warm, batch 1, `car`, 10 tokens; median of 100 requests, 2026-09-13)	~305 ms server-side (304–306; vision-trace submit 2 · prefill incl. the vision wait 86 · decode 202 ms, ~44.5 tok/s) · ~0.34 s wall. `TT_FUSED=0` (the 2026-09-12 graph): ~357 ms (351–365; vision 48 · prefill 62 · decode 232 ms, ~39 tok/s) · ~0.40 s wall
Same forward on an RTX 5090 (same host, port's torch reference, eager PyTorch, batch 1, incl. H2D/D2H; bf16 / fp16 autocast)	bf16 native weights 158 ms (83 tok/s decode) → GPU 1.8× faster than the p150a's 290 ms (vision + prefill + decode, 44.5 tok/s); bf16/fp16 autocast over fp32 weights 218 / 215 ms (1.3×); fp32-strict 216 ms (1.3×; vision+prefill 92 ms vs p150a 88 ms — p150a faster); best `torch.compile` 137 ms (2.1×)

Caveats

One image + one query per request, batch 1, requests are serialized. Every image is squash-resized onto a fixed 24×44-patch grid (616×336, 16:9; LA_IN_TOKEN_LIMIT=1024, upstream default 25600): non-16:9 images are distorted before the model sees them, boxes still map back to original pixels.
bf16 vision, BF16 attention + BFP8 MLP LLM; the package ships an mlp.py overlay (code/models/tt_transformers/tt/mlp.py, decode w1/w3 spilled to DRAM) so the LLM fits L1 on one p150a. Fused device paths (MoonViT as one metal trace with the patch merger on device, device vision→LLM merge, traced prefill, one-row first token, on-device greedy argmax) are on by default since 2026-09-13; TT_FUSED=0 restores the 2026-09-12 graph, and the vision numerics are unchanged bit for bit either way. Only greedy AR decode is served; the experimental Parallel Box Decoding (MTP) is not.
Weights are public and ungated but under NVIDIA's own license (not OSI); ~~7.7 GB download, and the first boot extracts a 6.8 GB Qwen2.5-3B checkpoint, converts it to BFP8 and JIT-compiles (~~100 s on the very first boot, ~50 s with cached weights but an empty kernel cache, ~17 s warm; measured 2026-09-13).
Not an OpenAI-compatible API; GET /v1/models is a stub so the tt-model ready card does not 404.
Validated on tt-metal v0.78.0-dev20260820 (main 8b98410e730), single p150a only.
GPU comparison: GPU (Qwen2.5-VL-3B in HF transformers) bf16 is 1.8× faster overall; on vision + prefill the p150a is slightly faster than fp32-strict GPU (88 vs 92 ms) and the gap is in decode (44.5 vs 83 tok/s). RTX 5090 rows (2026-09-14): same host, the port's own torch reference (same weights) run eagerly in PyTorch 2.11 cu128 with fp32 weights + autocast unless stated, no TensorRT; medians of 50 iterations after warm-up, H2D/D2H included; the p150a rows are the served bf16 fused path incl. upload/readback. p150a power was not measured, so no efficiency comparison is made. Full table: GPU_COMPARISON.md.

Licensing

Weights: nvidia/LocateAnything-3B, other (NVIDIA license, not OSI); fetched from upstream, not redistributed here.
Port and serving code (code/locate_anything, code/scripts): Apache-2.0 SPDX headers, from changh95/tt-locate-anything, published under the same upstream terms; the vendored tt-metal overlay code/models/tt_transformers/tt/mlp.py is Apache-2.0.

Provenance

The exact sources the image was built from — code/ in this repo is byte-identical to the model code inside the image:

component	built from
tt-metal	`8b98410e730bb504fea43a88609756e34821d91d`
`code/` digest	`27dc2b154aa86da9` (sha256, first 16 hex digits)
built	2026-09-13T15:59:45+00:00 by tt-model 0.1.0