# SPDX-License-Identifier: Apache-2.0 # tt-model-manager container manifest (schema 5.1) for NVIDIA LocateAnything-3B on one Blackhole p150a. # # Run every tt-model command FROM THIS DIRECTORY (`extra_code.root: code` is CWD-relative): # source $ROOT/bin/docker-env.sh # tt-model package --container tt-model.yaml --out $ROOT/build # amd64 + Docker>=25, hours cold # tt-model serve $ROOT/build/locate-anything-3b-blackhole/tt_kernel_manifest.json # tt-model push $ROOT/build/locate-anything-3b-blackhole --publish # # The port validated against tt-metal main 8b98410e730 (v0.78.0-dev20260820-25): torch 2.11.0+cpu, # transformers 5.12.1. source.tt_metal is that local checkout (absolute; supplies the torch pin). schema: "5.1" repo: changh95/locate-anything-3b-blackhole name: locate-anything-3b-blackhole # A POINTER, pinned. `tt-model serve` downloads it into the consumer's HF cache before boot; # the app resolves the same repo/revision offline (TT_WEIGHTS_REVISION below) and extracts the # vanilla Qwen2.5-3B LLM checkpoint into ~/.cache/tt-model/locate-anything-3b-blackhole/weights. weights: repo: nvidia/LocateAnything-3B revision: c32291ca5e996f5a7a485845b4f57a233936bba0 ignore_patterns: ["assets/*", "training_args.bin"] kind: tt-dit-server arch: blackhole source: tt_metal: /home/deepgadget/experiments/gbp-tt/tt-metal # tt-metal packages the port really imports (module-level closure incl. models/common). code: - models/common - models/tt_transformers - models/demos/qwen25_vl # This repo's code (does not live in the tt-metal tree). Everything listed survives `push`. # `models` holds ONE overlay file, code/models/tt_transformers/tt/mlp.py: extra_code is staged # after source.code and wins on collision, so it replaces the stock file in the image. Stock # mlp.py fails on a single p150a at the first decode step ("Statically allocated circular # buffers ... clash with L1 buffers", w3 linear); the overlay spills the decode w1/w3 outputs # to DRAM before the next matmul, the change the port validated with (PCC 0.9928). extra_code: - root: code paths: - locate_anything - models - scripts - conftest.py ubuntu: "22.04" python: "3.12" runtime: app: locate_anything.server.app:app mesh_shape_env: TT_MESH_SHAPE # On top of the kind's defaults (fastapi, uvicorn, pydantic>=2, pillow) and the auto torch pin. # transformers must equal tt-metal's own pin (ModelArgs loads the Qwen2 checkpoint through it); # loguru/tqdm/pytest/pyyaml are module-level imports of models.tt_transformers / models.common. packages: - "numpy>=1.24.4,<2" - "transformers==5.12.1" - safetensors - huggingface_hub - loguru - tqdm - pytest - pyyaml serve: port: 20000 hardware: p150 mesh_device: P150 env: TT_WEIGHTS_REVISION: "c32291ca5e996f5a7a485845b4f57a233936bba0" TT_METAL_VISIBLE_DEVICES: "0" LA_IN_TOKEN_LIMIT: "1024" LA_PREC: "accuracy" LA_TRACE: "1" LA_WARMUP_RUNS: "2" # Build-time assertions, run INSIDE the finished image as uid 1000 with no device and no weights. verify: - "import locate_anything.server.app as a; assert a.app" - "import locate_anything.tt.pipeline, locate_anything.tt.vision, locate_anything.tt.model_la, locate_anything.reference.la_inputs, locate_anything.reference.extract_llm_checkpoint" - "import models.tt_transformers.tt.model_config, models.tt_transformers.tt.generator, models.demos.qwen25_vl.tt.common, models.common.utility_functions" - "import transformers; assert transformers.__version__ == '5.12.1', transformers.__version__" - "import safetensors, huggingface_hub, loguru, tqdm, pytest, yaml, numpy; assert numpy.__version__.startswith('1.'), numpy.__version__" - "from pathlib import Path; assert Path('/opt/tt-metal/models/tt_transformers/tt/prefetcher/prefetcher_config.yaml').is_file()" - "from pathlib import Path; assert Path('/opt/tt-metal/locate_anything/server/smoke_test.py').is_file()" - "from pathlib import Path; assert 'spill_ff1_3' in Path('/opt/tt-metal/models/tt_transformers/tt/mlp.py').read_text(), 'mlp.py overlay (decode w1/w3 DRAM spill) missing'" card: description: > NVIDIA LocateAnything-3B (MoonViT-SO-400M vision tower + Qwen2.5-3B LLM with a detection vocabulary) doing visual grounding / open-vocabulary detection on a single Tenstorrent Blackhole p150a via tt-nn: image + free-text query -> labelled boxes. Vision tower and LLM both run on the chip; greedy AR decode with trace replay reaches ~38 tok/s, ~53 ms vision, ~64 ms prefill, ~2.3 frames/s end-to-end at PCC 0.9928 vs the torch reference. Weights are fetched from nvidia/LocateAnything-3B (NVIDIA license, 'other' - not OSI; read it before use); the port code (Apache-2.0 SPDX headers) is redistributed under the same upstream terms. Port source: github.com/changh95/tt-locate-anything @ 7a3b7407c4f1268cfb14ce1dd0ed4cdcd877b9a1. quickstart: | ### tt-cli users ```bash tt serve changh95/locate-anything-3b-blackhole # pulls image + weights, boots, prints the port tt model stop changh95/locate-anything-3b-blackhole ``` `tt-model stop changh95/locate-anything-3b-blackhole` does the same without tt-cli. ### What this server is (and is not) The port is the one `serve` printed (20000, or the next free one). This is the model's own HTTP API, **not** an OpenAI-compatible one: `curl /v1/models` only returns a stub so the ready card does not 404, and `tt-model curl` does not apply. The routes are: | route | what | |---|---| | `GET /health` | `{"status": "ok" \| "starting", "model", "device"}` - `ok` only after warm-up | | `GET /info` | model, weights repo + revision loaded, source commit, input limits, license | | `POST /predict` | one image + one query -> labelled boxes (JSON) | ```bash PORT=20000 # the port serve printed curl -s localhost:$PORT/health curl -s localhost:$PORT/info | python -m json.tool # media/demo_input.png (1920x1080) with the README's query "car"; several categories are # joined with , e.g. "personcar". The PNG is 3.6 MB (4.8 MB as base64), far above # the shell's 128 KiB per-argument limit, so an inline "$(base64 -w0 ...)" fails with # "Argument list too long": write the body to a file and post it with -d @file. python3 - <<'EOF' import base64, json img = base64.b64encode(open("media/demo_input.png", "rb").read()).decode() json.dump({"image": img, "query": "car", "max_new_tokens": 128}, open("/tmp/la_req.json", "w")) EOF curl -s localhost:$PORT/predict -H 'Content-Type: application/json' -d @/tmp/la_req.json \ | python3 -m json.tool ``` Request fields: `image` (base64 PNG/JPEG, exactly one image), `query` (required free text, 1-1000 chars; join categories with ``), `max_new_tokens` (optional, default 128, cap 1024; each box costs ~6-8 tokens plus its label), `return_overlay` (optional bool; adds `overlay_png_b64`, the boxes drawn on your image). Response: `detections: [{label, box: [x1, y1, x2, y2], box_norm: [..0..1000..]}]` with `box` in ORIGINAL image pixels, `points` (the model's 2-coordinate outputs, same shape), `raw_text` (the generated `car<282><414><606><794><|im_end|>` string for the demo image), `width`/`height`, `canonical_size` and `grid_hw` (what the model actually saw, see below), `prompt_tokens`, `num_generated_tokens`, `stopped_on_eos`, `decode_mode` (`ar_greedy_trace`), and `timing_ms` (`vision`, `prefill`, `decode`, `total`, `decode_tok_s`). Errors: 400 for a bad image / empty query / prompt too long, 503 while starting, 500 with the exception text otherwise. One request at a time is served (batch 1, stateful KV cache); concurrent calls queue. A one-line check that the hardware phase also runs: `python code/locate_anything/server/smoke_test.py --url http://127.0.0.1:$PORT` prints `PASS ...` with the box count and timings. ### Input handling Every image is preprocessed like the HF processor at `LA_IN_TOKEN_LIMIT=1024` (all README numbers use this cap; upstream's default is 25600) and then squash-resized onto ONE canonical vision grid so MoonViT is built once: by default the grid the demo image (16:9) gets at that cap, 24x44 patches = 336x616 px = 264 image tokens. Boxes are normalized over that view, so they map back to your original pixels exactly; non-16:9 images are distorted before the model sees them. The grid is fixed for the life of the server and comes from the `LA_CANON_GRID=HxW` environment variable (even numbers of 14-px patches, e.g. `32x32` for square inputs). `tt-model serve` passes the container only the manifest's `serve.env` (there is no `--env` flag), so this package always serves the 24x44 grid; to serve another one, run the app on a host with tt-metal (see `SERVING.md`) or repackage with the variable added to `serve.env` in `tt-model.yaml`. ### First boot - `tt-model serve` downloads ~7.6 GB of weights (2 safetensors shards + tokenizer/config) into your HF cache before the container starts. - Inside the container the app writes a vanilla Qwen2.5-3B checkpoint (6.8 GB) to `~/.cache/tt-model/locate-anything-3b-blackhole/weights/la-qwen2_5-3b` (once), converts the LLM weights to BFP8 into `.../tensors/P150/tensor_cache_bfp8` (once), JIT-compiles kernels into `.../cache` (once), uploads MoonViT and runs two warm-up passes (prefill + 4 decode steps; the decode trace is captured on the first). Measured on the validation host: a cold first boot is a few minutes (extraction ~5 s, BFP8 conversion ~25 s, JIT + warm-up ~1-2 min), later boots ~15 s; `tt-model logs -f changh95/locate-anything-3b-blackhole` shows `Loading weights`, `Warming up`, `Warmup complete`. Host RAM peak ~10 GB. - Disk: 7.2 GB (HF cache) + 6.8 GB (extracted LLM) + 4 GB (BFP8 tensor cache) + ~0.4 GB JIT cache. - The weights repo is public and ungated; no `hf auth login` is needed for it. ### Results (from the port README; single p150a, everything on device, warm, trace decode) | Input (`media/demo_input.png`) | Greedy AR decode (`media/demo_ar.png`) | |:---:|:---:| | ![](media/demo_input.png) | ![](media/demo_ar.png) | Accuracy (PCC vs the torch-CPU golden, gate >= 0.99): | Stage | PCC | |---|---:| | Vision `patch_embed` | 0.99999 | | Vision `encoder_out` (27 blocks) | 0.9809 | | Vision `vit_proj` (after `mlp1`) | 0.9911 | | LLM prefill last-token logits | 0.9922 | | **Full on-device logits (vision -> LLM)** | **0.9928** | Performance: | Metric | Value | |---|---:| | Decode throughput | ~38 tok/s | | Vision (MoonViT + projector) | ~53 ms | | Prefill | ~64 ms | | End-to-end | ~2.32 frames/s | The LLM runs BF16 attention + BFP8 MLP weights (the `accuracy` preset; BFP8 MLP is what the >= 0.99 gate needs, BFP4 reaches ~0.935). Decode is weight-bandwidth bound (~3 GB/token), so trace replay is the big win (7.5 -> 38 tok/s) and a second command queue is not. The experimental on-device Parallel Box Decoding (MTP, `media/demo_mtp.png`) is approximate, not accuracy-gated, and not served. The validated run needed one change to tt-metal's `models/tt_transformers/tt/mlp.py`, and this package ships it as an overlay (`code/models/tt_transformers/tt/mlp.py`, otherwise byte-identical to the stock file): in single-chip decode the `w1`/`w3` outputs are moved to DRAM before the next matmul, because with stock bf16 L1-sharded intermediates the `w3` weight-stream circular buffers do not fit L1 on one p150a (`Statically allocated circular buffers ... clash with L1 buffers` at the first decode step). Measured with the overlay on the validation host: `media/demo_input.png` + `car` -> the same box as `media/demo_ar.png`, vision 49 ms, prefill 58 ms, decode 41 tok/s, 346 ms end-to-end. `LA_PREC=bfp8attn` (all-BFP8 attention, PCC 0.9912) remains available. ### Layout notes `code/locate_anything/` is the port (`tt/vision.py` MoonViT, `tt/model_la.py` the embeds-fed Qwen2.5-3B `Transformer` subclass, `tt/pipeline.py` the serving recipe, `tt/mtp.py` the experimental MTP decoder, `reference/` the torch-CPU reference and the LLM-checkpoint extractor, `tests/` the PCC/benchmark/demo pytest suites, `server/` this app + smoke test). `code/models/{common,tt_transformers,demos/qwen25_vl}` are the tt-metal packages the LLM path reuses (stock Qwen2.5 `Transformer`/`Generator`/paged `Attention`), with the one overlay file `models/tt_transformers/tt/mlp.py` described above. The tests need a device and the torch-CPU goldens (`reference/run_reference.py`), which are not shipped. ### Licensing The **upstream model and weights** are licensed **`other`** (nvidia-license): https://huggingface.co/nvidia/LocateAnything-3B/blob/main/LICENSE - NVIDIA's own licence, not an OSI licence; read it before any use. The weights are **not redistributed** here; they are fetched from the upstream repo under whatever terms that repo sets. The **port code** (Apache-2.0 SPDX headers; written by Hyunggi Chang, github.com/changh95/tt-locate-anything) is published under the same upstream terms, since a port cannot grant more than its upstream does. Backbone: Qwen2.5-3B-Instruct (Apache-2.0); vision tower: MoonViT (Kimi-VL, Moonshot AI); runtime: Tenstorrent tt-metal / tt-nn (Apache-2.0).