Add files using upload-large-folder tool

README.md

@@ -1,30 +1,16 @@
 ---
 tags:
 - blackhole
-- open-vocabulary
 - p150
-- qwen2.5
-- tenstorrent
 - tt-dit-server
-- tt-metal
 - tt-model-cache
-- tt-model-catalog
 - tt-model-container
-- tt-nn
-- ttnn
-- visual-grounding
-- vlm
-base_model:
-- nvidia/LocateAnything-3B
-license: other
-license_name: nvidia-license
-license_link: https://huggingface.co/nvidia/LocateAnything-3B/blob/main/LICENSE
-pipeline_tag: object-detection
 ---
 
 # locate-anything-3b-p150
 
-NVIDIA LocateAnything-3B (MoonViT-SO-400M vision tower + Qwen2.5-3B LLM with a detection vocabulary) doing visual grounding / open-vocabulary detection on a single Tenstorrent Blackhole p150a via tt-nn: image + free-text query -> labelled boxes. Vision tower and LLM both run on the chip; greedy AR decode with trace replay reaches ~38 tok/s, ~53 ms vision, ~64 ms prefill, ~2.3 frames/s end-to-end at PCC 0.9928 vs the torch reference. Weights are fetched from nvidia/LocateAnything-3B (NVIDIA license, 'other' - not OSI; read it before use); the port code (Apache-2.0 SPDX headers) is redistributed under the same upstream terms. Port source: github.com/changh95/tt-locate-anything @ 7a3b7407c4f1268cfb14ce1dd0ed4cdcd877b9a1.
+NVIDIA LocateAnything-3B (Eagle-family visual grounding / open-vocabulary detection VLM: MoonViT-SO-400M vision tower + Qwen2.5-3B-Instruct with a detection vocabulary) running entirely on one Tenstorrent Blackhole p150a via tt-nn: image + free-text query in, labelled boxes out.
+Weights: [nvidia/LocateAnything-3B](https://huggingface.co/nvidia/LocateAnything-3B) · Paper: [arXiv:2605.27365](https://arxiv.org/abs/2605.27365) · Upstream code: [NVlabs/Eagle (Embodied)](https://github.com/NVlabs/Eagle/tree/main/Embodied) · Port: [changh95/tt-locate-anything](https://github.com/changh95/tt-locate-anything)
 
 Runs on **p150** (mesh `P150`).
 
@@ -39,153 +25,57 @@ tt-model serve changh95/locate-anything-3b-p150
 
 `pull --with-weights` downloads the Docker image and the [`nvidia/LocateAnything-3B`](https://huggingface.co/nvidia/LocateAnything-3B) weights at `c32291ca5e996f5a7a485845b4f57a233936bba0` (into your HF cache; they are not in the image). `serve` starts the model's own HTTP server on port 20000 (or the next free port, if that one is busy); the first start compiles kernels for your device, which takes several minutes, and the server is ready when it logs `Application startup complete`.
 
-### tt-cli users
+### Run with tt-cli
 
 ```bash
-tt serve changh95/locate-anything-3b-p150        # pulls image + weights, boots, prints the port
+tt serve changh95/locate-anything-3b-p150
+{ printf '{"query":"car","image":"'; base64 -w0 media/demo_input.png; printf '"}'; } > req.json
+curl -s localhost:20000/predict -H 'Content-Type: application/json' -d @req.json
 tt model stop changh95/locate-anything-3b-p150
 ```
 
-`tt-model stop changh95/locate-anything-3b-p150` does the same without tt-cli.
+- `POST /predict`: `image` (base64 PNG/JPEG, one image), `query` (free text, 1-1000 chars; join categories with `</c>`, e.g. `person</c>car`); optional `max_new_tokens` (128, cap 1024), `return_overlay` (false).
+- `GET /health`, `GET /info`.
 
-### What this server is (and is not)
+### Response
 
-The port is the one `serve` printed (20000, or the next free one). This is the model's
-own HTTP API, **not** an OpenAI-compatible one: `curl <endpoint>/v1/models` only returns a
-stub so the ready card does not 404, and `tt-model curl` does not apply. The routes are:
+```json
+{"query": "car", "width": 1920, "height": 1080, "canonical_size": [616, 336], "grid_hw": [24, 44],
+ "raw_text": "<ref>car</ref><box><282><414><606><794></box><|im_end|>",
+ "detections": [{"label": "car", "box": [541.44, 447.12, 1163.52, 857.52], "box_norm": [282, 414, 606, 794]}],
+ "points": [], "num_generated_tokens": 10, "stopped_on_eos": true, "decode_mode": "ar_greedy_trace",
+ "timing_ms": {"vision": 47.4, "prefill": 55.8, "decode": 224.0, "total": 343.4, "decode_tok_s": 40.17}}
+```
 
-| route | what |
-|---|---|
-| `GET /health` | `{"status": "ok" \| "starting", "model", "device"}` - `ok` only after warm-up |
-| `GET /info` | model, weights repo + revision loaded, source commit, input limits, license |
-| `POST /predict` | one image + one query -> labelled boxes (JSON) |
+- `box` is `[x1, y1, x2, y2]` in original image pixels; `box_norm` is the model's own 0..1000 output over the squashed `canonical_size` view. `points` holds 2-coordinate outputs the same way.
+- `return_overlay: true` adds `overlay_png_b64`, the boxes drawn on your image as a base64 PNG.
 
-```bash
-PORT=20000   # the port serve printed
-curl -s localhost:$PORT/health
-curl -s localhost:$PORT/info | python -m json.tool
-
-# media/demo_input.png (1920x1080) with the README's query "car"; several categories are
-# joined with </c>, e.g. "person</c>car". The PNG is 3.6 MB (4.8 MB as base64), far above
-# the shell's 128 KiB per-argument limit, so an inline "$(base64 -w0 ...)" fails with
-# "Argument list too long": write the body to a file and post it with -d @file.
-python3 - <<'EOF'
-import base64, json
-img = base64.b64encode(open("media/demo_input.png", "rb").read()).decode()
-json.dump({"image": img, "query": "car", "max_new_tokens": 128}, open("/tmp/la_req.json", "w"))
-EOF
-curl -s localhost:$PORT/predict -H 'Content-Type: application/json' -d @/tmp/la_req.json \
-  | python3 -m json.tool
-```
+### Demo
 
-Request fields: `image` (base64 PNG/JPEG, exactly one image), `query` (required free text,
-1-1000 chars; join categories with `</c>`), `max_new_tokens` (optional, default 128, cap 1024;
-each box costs ~6-8 tokens plus its label), `return_overlay` (optional bool; adds
-`overlay_png_b64`, the boxes drawn on your image).
-
-Response: `detections: [{label, box: [x1, y1, x2, y2], box_norm: [..0..1000..]}]` with `box`
-in ORIGINAL image pixels, `points` (the model's 2-coordinate outputs, same shape), `raw_text`
-(the generated `<ref>car</ref><box><282><414><606><794></box><|im_end|>` string for the demo image),
-`width`/`height`, `canonical_size` and `grid_hw` (what the model actually saw, see below),
-`prompt_tokens`, `num_generated_tokens`, `stopped_on_eos`, `decode_mode`
-(`ar_greedy_trace`), and `timing_ms` (`vision`, `prefill`, `decode`, `total`, `decode_tok_s`).
-Errors: 400 for a bad image / empty query / prompt too long, 503 while starting, 500 with
-the exception text otherwise. One request at a time is served (batch 1, stateful KV cache);
-concurrent calls queue.
-
-A one-line check that the hardware phase also runs:
-`python code/locate_anything/server/smoke_test.py --url http://127.0.0.1:$PORT` prints
-`PASS ...` with the box count and timings.
-
-### Input handling
-
-Every image is preprocessed like the HF processor at `LA_IN_TOKEN_LIMIT=1024` (all README
-numbers use this cap; upstream's default is 25600) and then squash-resized onto ONE canonical
-vision grid so MoonViT is built once: by default the grid the demo image (16:9) gets at that
-cap, 24x44 patches = 336x616 px = 264 image tokens. Boxes are normalized over that view, so
-they map back to your original pixels exactly; non-16:9 images are distorted before the
-model sees them. The grid is fixed for the life of the server and comes from the
-`LA_CANON_GRID=HxW` environment variable (even numbers of 14-px patches, e.g. `32x32` for
-square inputs). `tt-model serve` passes the container only the manifest's `serve.env`
-(there is no `--env` flag), so this package always serves the 24x44 grid; to serve another
-one, run the app on a host with tt-metal (see `SERVING.md`) or repackage with the variable
-added to `serve.env` in `tt-model.yaml`.
-
-### First boot
-
-- `tt-model serve` downloads ~7.7 GB of weights (2 safetensors shards + tokenizer/config)
-  into your HF cache before the container starts.
-- Inside the container the app writes a vanilla Qwen2.5-3B checkpoint (6.8 GB) to
-  `~/.cache/tt-model/locate-anything-3b-p150/weights/la-qwen2_5-3b` (once), converts the
-  LLM weights to BFP8 into `.../tensors/P150/tensor_cache_bfp8` (once), JIT-compiles kernels
-  into `.../cache` (once), uploads MoonViT and runs two warm-up passes (prefill + 4 decode
-  steps; the decode trace is captured on the first). Measured on the validation host: a
-  cold first boot is a few minutes (extraction ~5 s, BFP8 conversion ~25 s, JIT + warm-up
-  ~1-2 min), later boots ~15 s; `tt-model logs -f changh95/locate-anything-3b-p150`
-  shows `Loading weights`, `Warming up`, `Warmup complete`. Host RAM peak ~10 GB.
-- Disk: 7.7 GB (HF cache) + 6.8 GB (extracted LLM) + 4.3 GB (BFP8 tensor cache) + ~0.4 GB JIT cache.
-- The weights repo is public and ungated; no `hf auth login` is needed for it.
-
-### Results (from the port README; single p150a, everything on device, warm, trace decode)
-
-| Input (`media/demo_input.png`) | Greedy AR decode (`media/demo_ar.png`) |
+| Input (`media/demo_input.png`), query `car` | Greedy decode on p150a (`media/demo_ar.png`) |
 |:---:|:---:|
 | ![](media/demo_input.png) | ![](media/demo_ar.png) |
 
-Accuracy (PCC vs the torch-CPU golden, gate >= 0.99):
+### Accuracy and speed
 
-| Stage | PCC |
+| Metric | Value |
 |---|---:|
-| Vision `patch_embed` | 0.99999 |
-| Vision `encoder_out` (27 blocks) | 0.9809 |
-| Vision `vit_proj` (after `mlp1`) | 0.9911 |
-| LLM prefill last-token logits | 0.9922 |
-| **Full on-device logits (vision -> LLM)** | **0.9928** |
+| Full on-device logits PCC vs torch-CPU reference (vision → LLM) | 0.9928 (gate ≥ 0.99; vision projector 0.9911, LLM prefill 0.9922) |
+| Decoded box string on the demo image | matches the HF reference |
+| `/predict` served over HTTP (warm, batch 1, `car`, 10 tokens) | ~343 ms server-side (vision 47 · prefill 56 · decode 224 ms, ~40 tok/s) · ~0.38 s wall |
 
-Performance:
+### Caveats
 
-| Metric | Value |
-|---|---:|
-| Decode throughput | ~38 tok/s |
-| Vision (MoonViT + projector) | ~53 ms |
-| Prefill | ~64 ms |
-| End-to-end | ~2.32 frames/s |
-
-The LLM runs BF16 attention + BFP8 MLP weights (the `accuracy` preset; BFP8 MLP is what the
->= 0.99 gate needs, BFP4 reaches ~0.935). Decode is weight-bandwidth bound (~3 GB/token), so
-trace replay is the big win (7.5 -> 38 tok/s) and a second command queue is not. The
-experimental on-device Parallel Box Decoding (MTP, `media/demo_mtp.png`) is approximate,
-not accuracy-gated, and not served. The validated run needed one change to tt-metal's
-`models/tt_transformers/tt/mlp.py`, and this package ships it as an overlay
-(`code/models/tt_transformers/tt/mlp.py`, otherwise byte-identical to the stock file): in
-single-chip decode the `w1`/`w3` outputs are moved to DRAM before the next matmul, because
-with stock bf16 L1-sharded intermediates the `w3` weight-stream circular buffers do not fit
-L1 on one p150a (`Statically allocated circular buffers ... clash with L1 buffers` at the
-first decode step). Measured with the overlay on the validation host: `media/demo_input.png`
-+ `car` -> the same box as `media/demo_ar.png`, vision 49 ms, prefill 58 ms, decode 41 tok/s,
-346 ms end-to-end. `LA_PREC=bfp8attn` (all-BFP8 attention, PCC 0.9912) remains available.
-
-### Layout notes
-
-`code/locate_anything/` is the port (`tt/vision.py` MoonViT, `tt/model_la.py` the embeds-fed
-Qwen2.5-3B `Transformer` subclass, `tt/pipeline.py` the serving recipe, `tt/mtp.py` the
-experimental MTP decoder, `reference/` the torch-CPU reference and the LLM-checkpoint
-extractor, `tests/` the PCC/benchmark/demo pytest suites, `server/` this app + smoke test).
-`code/models/{common,tt_transformers,demos/qwen25_vl}` are the tt-metal packages the LLM
-path reuses (stock Qwen2.5 `Transformer`/`Generator`/paged `Attention`), with the one
-overlay file `models/tt_transformers/tt/mlp.py` described above. The tests need a device
-and the torch-CPU goldens (`reference/run_reference.py`), which are not shipped.
+- One image + one query per request, batch 1, requests are serialized. Every image is squash-resized onto a fixed 24×44-patch grid (616×336, 16:9; `LA_IN_TOKEN_LIMIT=1024`, upstream default 25600): non-16:9 images are distorted before the model sees them, boxes still map back to original pixels.
+- bf16 vision, BF16 attention + BFP8 MLP LLM; the package ships an `mlp.py` overlay (`code/models/tt_transformers/tt/mlp.py`, decode `w1`/`w3` spilled to DRAM) so the LLM fits L1 on one p150a. Only greedy AR decode is served; the experimental Parallel Box Decoding (MTP) is not.
+- Weights are public and ungated but under NVIDIA's own license (not OSI); ~7.7 GB download, and the first boot extracts a 6.8 GB Qwen2.5-3B checkpoint, converts it to BFP8 and JIT-compiles (~100 s cold, ~25 s afterwards).
+- Not an OpenAI-compatible API; `GET /v1/models` is a stub so the tt-model ready card does not 404.
+- Validated on tt-metal `v0.78.0-dev20260820` (main `8b98410e730`), single p150a only.
 
 ### Licensing
 
-The **upstream model and weights** are licensed **`other`** (nvidia-license):
-https://huggingface.co/nvidia/LocateAnything-3B/blob/main/LICENSE - NVIDIA's own licence,
-not an OSI licence; read it before any use. The weights are **not redistributed** here; they
-are fetched from the upstream repo under whatever terms that repo sets. The **port code**
-(Apache-2.0 SPDX headers; written by Hyunggi Chang, github.com/changh95/tt-locate-anything)
-is published under the same upstream terms, since a port cannot grant more than its upstream
-does. Backbone: Qwen2.5-3B-Instruct (Apache-2.0); vision tower: MoonViT (Kimi-VL, Moonshot AI);
-runtime: Tenstorrent tt-metal / tt-nn (Apache-2.0).
+- Weights: [nvidia/LocateAnything-3B](https://huggingface.co/nvidia/LocateAnything-3B), `other` ([NVIDIA license](https://huggingface.co/nvidia/LocateAnything-3B/blob/main/LICENSE), not OSI); fetched from upstream, not redistributed here.
+- Port and serving code (`code/locate_anything`, `code/scripts`): Apache-2.0 SPDX headers, from [changh95/tt-locate-anything](https://github.com/changh95/tt-locate-anything), published under the same upstream terms; the vendored [tt-metal](https://github.com/tenstorrent/tt-metal) overlay `code/models/tt_transformers/tt/mlp.py` is Apache-2.0.
 
 ## Provenance
 
@@ -195,5 +85,5 @@ The exact sources the image was built from — `code/` in this repo is byte-iden
 | --- | --- |
 | tt-metal | [`8b98410e730bb504fea43a88609756e34821d91d`](https://github.com/tenstorrent/tt-metal/commit/8b98410e730bb504fea43a88609756e34821d91d) |
 | `code/` digest | `eb2ce7d7357c84e3` (sha256, first 16 hex digits) |
-| built | 2026-09-12T12:20:44+00:00 by tt-model 0.1.0 |
+| built | 2026-09-12T13:42:57+00:00 by tt-model 0.1.0 |
 

tt_kernel_manifest.json

@@ -6,7 +6,7 @@
   "device_count": 1,
   "producer": {
     "tt_kernel_version": "0.1.0",
-    "created_at": "2026-09-12T12:20:55.419379+00:00",
+    "created_at": "2026-09-12T13:43:23.393943+00:00",
     "hostname": "deepgadget"
   },
   "weights": {
@@ -102,7 +102,7 @@
       "image": "tt-model/locate-anything-3b-p150:04871c7cfa03",
       "repo": "changh95/locate-anything-3b-p150",
       "tt_model_version": "0.1.0",
-      "created_at": "2026-09-12T12:20:44+00:00",
+      "created_at": "2026-09-12T13:42:57+00:00",
       "tt_metal": {
         "sha": "8b98410e730bb504fea43a88609756e34821d91d",
         "describe": "v0.78.0-dev20260820-25-g8b98410e73",