Add files using upload-large-folder tool

README.md

@@ -1,348 +1,184 @@
 ---
-pipeline_tag: object-detection
 tags:
-- tenstorrent
 - blackhole
 - p150
-- ttnn
-- tt-metal
-- tt-nn
-- visual-grounding
-- open-vocabulary
-- vlm
-- qwen2.5
-base_model:
-- nvidia/LocateAnything-3B
-license: other
-license_name: nvidia-license
-license_link: https://huggingface.co/nvidia/LocateAnything-3B/blob/main/LICENSE
+- tt-dit-server
+- tt-model-cache
+- tt-model-container
 ---
 
-# NVIDIA LocateAnything-3B on Tenstorrent Blackhole
+# locate-anything-3b-blackhole
 
-NVIDIA LocateAnything-3B, an Eagle-family visual-grounding / open-vocabulary detection VLM (MoonViT-SO-400M vision tower plus a Qwen2.5-3B-Instruct backbone with an extended detection vocabulary), on a single Blackhole p150a.
+NVIDIA LocateAnything-3B (MoonViT-SO-400M vision tower + Qwen2.5-3B LLM with a detection vocabulary) doing visual grounding / open-vocabulary detection on a single Tenstorrent Blackhole p150a via tt-nn: image + free-text query -> labelled boxes. Vision tower and LLM both run on the chip; greedy AR decode with trace replay reaches ~38 tok/s, ~53 ms vision, ~64 ms prefill, ~2.3 frames/s end-to-end at PCC 0.9928 vs the torch reference. Weights are fetched from nvidia/LocateAnything-3B (NVIDIA license, 'other' - not OSI; read it before use); the port code (Apache-2.0 SPDX headers) is redistributed under the same upstream terms. Port source: github.com/changh95/tt-locate-anything @ 7a3b7407c4f1268cfb14ce1dd0ed4cdcd877b9a1.
 
-**TT-NN MoonViT tower plus an experimental on-device Parallel Box Decoding (MTP) decoder; overlays a tt-metal checkout.**
+Runs on **p150** (mesh `P150`).
 
-| | |
-|---|---|
-| Hardware | Tenstorrent Blackhole **p150a** (single chip) |
-| Runtime | [tt-metal](https://github.com/tenstorrent/tt-metal) / tt-nn |
-| Upstream model | [huggingface.co/nvidia/LocateAnything-3B](https://huggingface.co/nvidia/LocateAnything-3B) |
-| Port source | [github.com/changh95/tt-locate-anything](https://github.com/changh95/tt-locate-anything) |
-
-> [!IMPORTANT]
-> This repo holds **model code, not weights.** It is a tt-nn port that runs against a
-> built `tt-metal` checkout on a machine with a Blackhole card; weights are fetched
-> from the upstream repo above. See *Licensing* at the end for terms.
-
----
-
-## Project README
-
-End-to-end port of NVIDIA
-[LocateAnything-3B](https://huggingface.co/nvidia/LocateAnything-3B) — an Eagle-family
-visual-grounding / open-vocabulary detection VLM — to Tenstorrent **tt-metal**
-(tt-nn + tt-metallium), running on a single Blackhole **p150a** chip.
-
-The model is a **MoonViT-SO-400M** vision tower + a 2×2 patch merger + an `mlp1`
-projector feeding a **Qwen2.5-3B-Instruct** language model with an extended
-detection vocabulary. Given an image and a free-text query ("locate all the
-instances that match …"), it emits `<ref>label</ref><box><x1><y1><x2><y2></box>`
-token sequences that decode to pixel boxes.
-
-This repository contains:
-
-- a TT-NN MoonViT vision tower + projector (`locate_anything/tt/vision.py`),
-- a thin `Transformer` subclass that drives the Qwen2.5-3B backbone from
-  pre-merged image+text embeddings (`locate_anything/tt/model_la.py`),
-- an experimental on-device **Parallel Box Decoding** (MTP) decoder
-  (`locate_anything/tt/mtp.py`),
-- a self-contained torch-CPU reference + golden/oracle builders
-  (`locate_anything/reference/`),
-- pytest suites for per-stage vision PCC, an end-to-end baseline benchmark with a
-  PCC accuracy gate, an MTP fidelity test, and two image-in / boxes-out demos.
-
-Unlike a from-scratch ttnn model, **LocateAnything's LLM backbone reuses
-tt-metal's own model libraries** — `models.tt_transformers` (the stock Qwen2.5
-`Transformer` / `Generator` / paged-KV `Attention` / `MLP`) and
-`models.demos.qwen25_vl` (vision-token merge + prefill prep). So this repo is an
-*overlay* on a tt-metal checkout, not a standalone reimplementation: point Python
-at a built tt-metal and run from here. See **Environment setup**.
-
----
-
-## Demo
+Packaged and published with [tt-model-manager](https://github.com/tenstorrent/tt-model-manager) 0.1.0 (manifest schema 5.1).
 
-Greedy autoregressive decode and the experimental hybrid-MTP path, both running
-the full pipeline (MoonViT vision + Qwen2.5-3B LLM) on one Blackhole p150a.
-Reproduce with `pytest locate_anything/tests/test_demo_visualize.py` (AR) or
-`test_demo_mtp_visualize.py` (MTP); query and image are set via `LA_QUERY` /
-`LA_IMAGE`.
+## Quickstart
 
-| Input (`media/demo_input.png`) | AR decode (`media/demo_ar.png`) | Hybrid-MTP (`media/demo_mtp.png`) |
-|:---:|:---:|:---:|
-| ![](media/demo_input.png) | ![](media/demo_ar.png) | ![](media/demo_mtp.png) |
-
----
-
-## Contents
-
-```
-locate_anything/
-├── reference/
-│   ├── la_inputs.py             # image preprocess + chat-template build (no cv2/lmdb/decord)
-│   ├── extract_llm_checkpoint.py# LocateAnything-3B → vanilla Qwen2.5-3B HF dir for tt_transformers
-│   ├── run_reference.py         # HF torch-CPU golden dump (vision + prefill logits) → golden.pt
-│   ├── mtp_cpu_loop.py          # correct bsz=1 hybrid/fast MTP loop (torch CPU); the device blueprint
-│   └── mtp_oracle.py            # picks a box-yielding (image,query); dumps MTP oracle → mtp_oracle.pt
-├── tt/
-│   ├── vision.py                # TT-NN MoonViT-SO-400M + mlp1 projector (bf16 / HiFi4)
-│   ├── model_la.py              # LATransformer: Qwen2.5-3B with embeds-driven prefill
-│   └── mtp.py                   # MTPDecoder: on-device Parallel Box Decoding (experimental)
-└── tests/
-    ├── test_vision.py           # incremental vision PCC vs golden (gate ≥ 0.99)
-    ├── bench_locate_anything.py # baseline benchmark: prefill PCC + greedy AR decode + metrics
-    ├── test_mtp.py              # device-MTP vs torch-CPU-MTP logit PCC (fidelity)
-    ├── test_demo_visualize.py   # image → boxes (greedy AR) visualization
-    └── test_demo_mtp_visualize.py # image → boxes (hybrid MTP) visualization
-scripts/
-└── download_weights.sh         # pull LocateAnything-3B + extract the Qwen2.5-3B LLM dir
-conftest.py                     # loads tt-metal's device fixtures (mesh_device, device_params, …)
+```bash
+tt-model pull  changh95/locate-anything-3b-blackhole --with-weights
+tt-model serve changh95/locate-anything-3b-blackhole
 ```
 
-This repo does **not** vendor the tt-metal monorepo, the model weights, or the
-torch goldens. The first come from your tt-metal build; the second from the Hugging
-Face Hub; the third are generated locally.
+`pull --with-weights` downloads the Docker image and the [`nvidia/LocateAnything-3B`](https://huggingface.co/nvidia/LocateAnything-3B) weights at `c32291ca5e996f5a7a485845b4f57a233936bba0` (into your HF cache; they are not in the image). `serve` starts the model's own HTTP server on port 20000 (or the next free port, if that one is busy); the first start compiles kernels for your device, which takes several minutes, and the server is ready when it logs `Application startup complete`.
 
----
+### tt-cli users
 
-## Environment setup
-
-1. **Build tt-metal** with its Python bindings (and Tracy if you plan to profile).
-   Instructions: https://github.com/tenstorrent/tt-metal. This is the source of
-   `ttnn`, `models.tt_transformers`, and `models.demos.qwen25_vl`, all of which
-   this repo imports directly.
-
-2. **Install the Python deps** (the tt-metal `python_env` already has most of
-   them). A working set on top of `ttnn`:
-
-   ```bash
-   pip install torch torchvision transformers safetensors pillow numpy loguru \
-               huggingface_hub matplotlib
-   ```
-
-   `torch` can be CPU-only — it is used for the reference, host embedding lookup,
-   the vision-token merge, and host argmax sampling.
-
-3. **Point Python at tt-metal** and set the runtime env. The single most important
-   variable is `TT_VISIBLE_DEVICES`, which constrains UMD to a single chip:
-
-   ```bash
-   export TT_METAL_HOME=/path/to/tt-metal
-   export ARCH_NAME=blackhole
-   export MESH_DEVICE=N150                  # single-chip 1x1 mesh
-   export PYTHONPATH=$PWD:$TT_METAL_HOME:$TT_METAL_HOME/ttnn:$TT_METAL_HOME/tools
-   # Run on exactly ONE Blackhole chip. Set BOTH so UMD opens only this chip:
-   export TT_VISIBLE_DEVICES=0
-   export TT_METAL_VISIBLE_DEVICES=0
-   ```
-
-   `conftest.py` re-uses tt-metal's own pytest fixtures and hooks (so
-   `mesh_device` / `device_params` / `reset_seeds` behave identically to running
-   inside the tt-metal tree); it requires `TT_METAL_HOME` to be set.
-
-4. **Download the weights** and extract the LLM directory:
+```bash
+tt serve changh95/locate-anything-3b-blackhole        # pulls image + weights, boots, prints the port
+tt model stop changh95/locate-anything-3b-blackhole
+```
 
-   ```bash
-   bash scripts/download_weights.sh
-   # → LocateAnything-3B snapshot (LA_MODEL_PATH) + extracted Qwen2.5-3B (HF_MODEL)
-   export HF_MODEL=~/.cache/locate_anything/LA-Qwen2.5-3B
-   ```
+`tt-model stop changh95/locate-anything-3b-blackhole` does the same without tt-cli.
 
-5. **Generate the torch-CPU goldens** the PCC tests compare against:
+### What this server is (and is not)
 
-   ```bash
-   python locate_anything/reference/run_reference.py --in-token-limit 1024  # → golden.pt
-   python locate_anything/reference/mtp_oracle.py    --in-token-limit 1024  # → mtp_oracle.pt (MTP test only)
-   ```
+The port is the one `serve` printed (20000, or the next free one). This is the model's
+own HTTP API, **not** an OpenAI-compatible one: `curl <endpoint>/v1/models` only returns a
+stub so the ready card does not 404, and `tt-model curl` does not apply. The routes are:
 
----
-
-## Running the tests
+| route | what |
+|---|---|
+| `GET /health` | `{"status": "ok" \| "starting", "model", "device"}` - `ok` only after warm-up |
+| `GET /info` | model, weights repo + revision loaded, source commit, input limits, license |
+| `POST /predict` | one image + one query -> labelled boxes (JSON) |
 
 ```bash
-# 1) Vision tower per-stage PCC vs the torch golden (gate ≥ 0.99 on vit_proj)
-pytest -svq locate_anything/tests/test_vision.py
-
-# 2) Baseline benchmark: prefill PCC vs golden + greedy AR decode + metrics
-#    Prints greppable: inference_speed=, accuracy=, peak_dram=, decode_tok_s=, vision_ms=, prefill_ms=
-pytest -svq locate_anything/tests/bench_locate_anything.py
-
-# 3) Experimental MTP (Parallel Box Decoding): device-MTP vs torch-CPU-MTP logit PCC
-pytest -svq locate_anything/tests/test_mtp.py
-
-# 4) Image → boxes demos (set LA_QUERY / LA_IMAGE / LA_OUT)
-LA_QUERY="car" pytest -svq locate_anything/tests/test_demo_visualize.py        # greedy AR
-LA_QUERY="car" pytest -svq locate_anything/tests/test_demo_mtp_visualize.py    # hybrid MTP
-
-# 5) Tracy-profiled run (requires a Tracy-enabled tt-metal build)
-python -m tracy --no-runtime-analysis --collect-noc-traces \
-    --profiler-capture-perf-counters=all -v -r -o ./tracy_out \
-    -m pytest locate_anything/tests/bench_locate_anything.py
+PORT=20000   # the port serve printed
+curl -s localhost:$PORT/health
+curl -s localhost:$PORT/info | python -m json.tool
+
+# media/demo_input.png (1920x1080) with the README's query "car"; several categories are
+# joined with </c>, e.g. "person</c>car". The PNG is 3.6 MB (4.8 MB as base64), far above
+# the shell's 128 KiB per-argument limit, so an inline "$(base64 -w0 ...)" fails with
+# "Argument list too long": write the body to a file and post it with -d @file.
+python3 - <<'EOF'
+import base64, json
+img = base64.b64encode(open("media/demo_input.png", "rb").read()).decode()
+json.dump({"image": img, "query": "car", "max_new_tokens": 128}, open("/tmp/la_req.json", "w"))
+EOF
+curl -s localhost:$PORT/predict -H 'Content-Type: application/json' -d @/tmp/la_req.json \
+  | python3 -m json.tool
 ```
 
-`bench_locate_anything.py` env knobs: `LA_PREC` (`accuracy` default — BF16
-attention + BFP8 MLP — or `bfp8attn`), `LA_TRACE` (`1` default, trace-replay
-decode), `LA_VISION` (`device` default, or `golden` to feed the CPU vision golden
-and isolate LLM PCC).
-
----
-
-## Results
-
-All numbers are for the fixed golden workload (one image + one query) on a single
-p150a, **everything on device** (MoonViT vision + Qwen2.5-3B LLM), warm-measured
-with trace-replay decode. The accuracy gate is **PCC ≥ 0.99** against the torch-CPU
-reference; the decoded box string matches the HF reference.
-
-### Accuracy (PCC vs torch-CPU golden)
+Request fields: `image` (base64 PNG/JPEG, exactly one image), `query` (required free text,
+1-1000 chars; join categories with `</c>`), `max_new_tokens` (optional, default 128, cap 1024;
+each box costs ~6-8 tokens plus its label), `return_overlay` (optional bool; adds
+`overlay_png_b64`, the boxes drawn on your image).
+
+Response: `detections: [{label, box: [x1, y1, x2, y2], box_norm: [..0..1000..]}]` with `box`
+in ORIGINAL image pixels, `points` (the model's 2-coordinate outputs, same shape), `raw_text`
+(the generated `<ref>car</ref><box><282><414><606><794></box><|im_end|>` string for the demo image),
+`width`/`height`, `canonical_size` and `grid_hw` (what the model actually saw, see below),
+`prompt_tokens`, `num_generated_tokens`, `stopped_on_eos`, `decode_mode`
+(`ar_greedy_trace`), and `timing_ms` (`vision`, `prefill`, `decode`, `total`, `decode_tok_s`).
+Errors: 400 for a bad image / empty query / prompt too long, 503 while starting, 500 with
+the exception text otherwise. One request at a time is served (batch 1, stateful KV cache);
+concurrent calls queue.
+
+A one-line check that the hardware phase also runs:
+`python code/locate_anything/server/smoke_test.py --url http://127.0.0.1:$PORT` prints
+`PASS ...` with the box count and timings.
+
+### Input handling
+
+Every image is preprocessed like the HF processor at `LA_IN_TOKEN_LIMIT=1024` (all README
+numbers use this cap; upstream's default is 25600) and then squash-resized onto ONE canonical
+vision grid so MoonViT is built once: by default the grid the demo image (16:9) gets at that
+cap, 24x44 patches = 336x616 px = 264 image tokens. Boxes are normalized over that view, so
+they map back to your original pixels exactly; non-16:9 images are distorted before the
+model sees them. The grid is fixed for the life of the server and comes from the
+`LA_CANON_GRID=HxW` environment variable (even numbers of 14-px patches, e.g. `32x32` for
+square inputs). `tt-model serve` passes the container only the manifest's `serve.env`
+(there is no `--env` flag), so this package always serves the 24x44 grid; to serve another
+one, run the app on a host with tt-metal (see `SERVING.md`) or repackage with the variable
+added to `serve.env` in `tt-model.yaml`.
+
+### First boot
+
+- `tt-model serve` downloads ~7.6 GB of weights (2 safetensors shards + tokenizer/config)
+  into your HF cache before the container starts.
+- Inside the container the app writes a vanilla Qwen2.5-3B checkpoint (6.8 GB) to
+  `~/.cache/tt-model/locate-anything-3b-blackhole/weights/la-qwen2_5-3b` (once), converts the
+  LLM weights to BFP8 into `.../tensors/P150/tensor_cache_bfp8` (once), JIT-compiles kernels
+  into `.../cache` (once), uploads MoonViT and runs two warm-up passes (prefill + 4 decode
+  steps; the decode trace is captured on the first). Measured on the validation host: a
+  cold first boot is a few minutes (extraction ~5 s, BFP8 conversion ~25 s, JIT + warm-up
+  ~1-2 min), later boots ~15 s; `tt-model logs -f changh95/locate-anything-3b-blackhole`
+  shows `Loading weights`, `Warming up`, `Warmup complete`. Host RAM peak ~10 GB.
+- Disk: 7.2 GB (HF cache) + 6.8 GB (extracted LLM) + 4 GB (BFP8 tensor cache) + ~0.4 GB JIT cache.
+- The weights repo is public and ungated; no `hf auth login` is needed for it.
+
+### Results (from the port README; single p150a, everything on device, warm, trace decode)
+
+| Input (`media/demo_input.png`) | Greedy AR decode (`media/demo_ar.png`) |
+|:---:|:---:|
+| ![](media/demo_input.png) | ![](media/demo_ar.png) |
+
+Accuracy (PCC vs the torch-CPU golden, gate >= 0.99):
 
 | Stage | PCC |
 |---|---:|
-| Vision `patch_embed`                  | 0.99999 |
-| Vision `encoder_out` (27 blocks)      | 0.9809 |
-| Vision `vit_proj` (after `mlp1`)      | 0.9911 |
-| LLM prefill last-token logits         | 0.9922 |
-| **Full on-device logits (vision→LLM)**| **0.9928** |
-
-The vision encoder is computed in bf16 with HiFi4 (fp32 dest accumulate); the
-golden is taken in fp32 so the bf16 port has a fair high-precision target (a 27-layer
-bf16 tower drifts on its own). The `mlp1` LayerNorm/GELU lifts the projector PCC
-above the encoder's. The LLM uses **BF16 attention + BFP8 MLP weights**; BFP8 MLP is
-required for the ≥0.99 gate (BFP4 MLP only reaches ~0.935), and the decode `w1`/`w3`
-outputs are spilled to DRAM so the BFP8 weight-stream circular buffers fit L1 on one
-chip.
-
-### Performance (warm, trace-replay decode, everything on device)
+| Vision `patch_embed` | 0.99999 |
+| Vision `encoder_out` (27 blocks) | 0.9809 |
+| Vision `vit_proj` (after `mlp1`) | 0.9911 |
+| LLM prefill last-token logits | 0.9922 |
+| **Full on-device logits (vision -> LLM)** | **0.9928** |
+
+Performance:
 
 | Metric | Value |
 |---|---:|
-| Decode throughput        | **~38 tok/s** |
-| Vision (MoonViT + proj)  | ~53 ms |
-| Prefill                  | ~64 ms |
-| End-to-end               | **~2.32 frames/s** |
-
-Decode is **weight-bandwidth bound** (~3 GB/token streamed), not host-bound — which
-is why the trace win is large and the second command queue is not.
-
-### Optimization trajectory
-
-Each row was verified to improve throughput while holding PCC ≥ 0.99 (the baseline
-row is below the gate and is the starting point, not a kept result). Discarded
-experiments are listed below the table.
-
-| # | Commit | Change | FPS | decode tok/s | PCC |
-|---:|---|---|---:|---:|---:|
-| 0 | `e99e3bb` | Initial e2e port: Qwen2.5-3B LLM on 1×p150a (perf / BFP4 MLP), CPU vision golden, greedy AR decode | 0.19 | — | 0.935¹ |
-| 1 | `e9d4899` | BFP8 MLP (accuracy preset); DRAM-spill decode `w1`/`w3` to fit L1 → meets the gate | 0.57 | 7.5 | 0.992 |
-| 2 | `eb03f7b` | Move MoonViT vision + `mlp1` projector on device — full pipeline on one chip | 1.29 | 7.5 | 0.993 |
-| 3 | `08af3aa` | Trace-replay decode (5× faster decode) | **2.32** | **38** | 0.993 |
-
-¹ Below the 0.99 gate — kept only as the bring-up baseline; MLP weight precision
-(BFP4→BFP8) dominates LLM accuracy, not attention precision.
-
-Rejected after measurement:
-
-- **Second command queue (2CQ).** No gain — the stock `tt_transformers` generator
-  only issues on `cq_id=0`, and decode is device-compute / weight-bandwidth bound.
-- **All-BFP8 attention + KV.** Slower *and* lower PCC (0.9928 → 0.9912); BF16
-  attention is already efficient on Blackhole.
-- **MTP / Parallel Box Decoding as the default.** ~60 tok/s over 16 forwards
-  (~1.7× decode), but greedy MTP is intrinsically approximate — it degenerates
-  under temperature 0 and does **not** reproduce greedy-AR boxes (NVIDIA's hybrid
-  mode itself uses sampling). It fails the strict ≥0.99 / match-AR gate, so AR +
-  trace stays the default. MTP is kept as an optional fast mode whose **device port
-  is faithful to the torch-CPU MTP** (end-to-end device-vs-torch logit PCC ~0.986).
-
-**Within the strict ≥0.99-PCC / deterministic-match gate, trace-AR decode
-(38 tok/s, 0.9928 PCC, everything on device) is the speed ceiling.**
-
----
-
-## Architecture notes
-
-- **MoonViT-SO-400M** (`tt/vision.py`): hidden 1152, 27 encoder layers, 16 heads
-  (head_dim 72, padded to 96 for tile alignment), intermediate 4304, patch 14,
-  GELU-tanh. `Conv2d` patch embed expressed as a single `(588, 1152)` matmul +
-  host-precomputed bicubic-interpolated 2D position embedding; **interleaved-complex
-  2D RoPE** mapped onto `ttnn.experimental.rotary_embedding_llama`; full bidirectional
-  packed attention via masked SDPA; final LayerNorm; **2×2 patch merge** on host
-  (a pure reshape) → `mlp1` projector `LayerNorm(4608) → Linear → GELU → Linear` to
-  2048-d. PCC validated at 0.99999 (patch embed) / 0.9911 (projector).
-- **Qwen2.5-3B LLM** (`tt/model_la.py`): the stock `tt_transformers` `Transformer`,
-  with one added method that feeds **pre-merged image+text embeddings** into prefill
-  instead of token ids (standard 1D RoPE, `rope_theta=1e6`; *no* mrope). hidden 2048,
-  36 layers, 16 heads / 2 KV heads (GQA), head_dim 128, intermediate 11008, tied
-  embeddings, extended vocab 152681. Image embeds are scattered into the text
-  embeds at `image_token_index` (151665) on host, then prefill runs on device with
-  a paged KV cache; greedy decode runs through the stock `Generator` with trace.
-- **Parallel Box Decoding / MTP** (`tt/mtp.py`, experimental): predicts a whole
-  6-token box per forward using the block-bidirectional "generation window"
-  attention from NVIDIA's reference, with lazy KV commit. Closely reproduces the
-  torch-CPU MTP loop in `reference/mtp_cpu_loop.py`.
-
-**Host vs device.** The forward keeps the heavy compute on chip; host work is the
-image layout reshape and 2×2 patch merge, the text-token embedding lookup and the
-vision↔text embedding merge, and argmax sampling (the 152681-wide vocab exceeds the
-on-device-sampling 64K/split limit, but host argmax is negligible since decode is
-device-bound).
-
----
-
-## Known caveats
-
-- **Overlay on tt-metal, not standalone.** The LLM path imports `tt_transformers`
-  and `qwen25_vl` directly; `conftest.py` reuses tt-metal's device fixtures. You
-  need a built tt-metal checkout on `PYTHONPATH` (`TT_METAL_HOME` set). This is by
-  design — the port deliberately reuses the tuned stock Qwen2.5 implementation
-  rather than re-deriving it.
-- **One image per forward, single chip.** Batch is 1 and the model runs on exactly
-  one p150a; nothing is sharded across chips. Set `TT_VISIBLE_DEVICES` to isolate
-  the chip you want.
-- **MTP is approximate and not accuracy-gated.** It is a faithful device port of the
-  torch-CPU MTP (logit PCC ~0.986) and runs the model's intended fast path, but
-  greedy MTP ≠ greedy AR by construction; for deterministic, gate-passing detection
-  use the default AR path. The MTP demo accepts `LA_TEMP` / `LA_TOP_P` /
-  `LA_REP_PEN` for the model's intended sampling.
-- **Goldens are generated, not shipped.** `golden.pt` / `mtp_oracle.pt` depend on
-  your torch/transformers build; regenerate them with the `reference/` scripts if
-  you change the image, query, or token-limit.
-- **BFP8 MLP is mandatory** for the ≥0.99 gate; the decode `w1`/`w3` DRAM spill is
-  what makes it fit L1 on a single chip (the alternative BFP4 MLP drops PCC to
-  ~0.935).
-
----
-
-## License
-
-Apache 2.0 (matches upstream LocateAnything, Qwen2.5, MoonViT/Kimi-VL, and tt-metal).
-
----
-
-## Acknowledgements
-
-- Original model: NVIDIA **LocateAnything-3B** — https://huggingface.co/nvidia/LocateAnything-3B
-- Language backbone: **Qwen2.5-3B-Instruct** — https://huggingface.co/Qwen/Qwen2.5-3B-Instruct
-- Vision tower: **MoonViT** (Kimi-VL / Moonshot AI), SigLIP-SO400M shape
-- Runtime: Tenstorrent **tt-metal / tt-nn** — https://github.com/tenstorrent/tt-metal
-
----
-
-## Licensing
-
-The **upstream model** is licensed **`other`** (nvidia-license) - see [the licence](https://huggingface.co/nvidia/LocateAnything-3B/blob/main/LICENSE).
-
-Governed by NVIDIA's own licence, not an OSI licence - read it before any use.
+| Decode throughput | ~38 tok/s |
+| Vision (MoonViT + projector) | ~53 ms |
+| Prefill | ~64 ms |
+| End-to-end | ~2.32 frames/s |
+
+The LLM runs BF16 attention + BFP8 MLP weights (the `accuracy` preset; BFP8 MLP is what the
+>= 0.99 gate needs, BFP4 reaches ~0.935). Decode is weight-bandwidth bound (~3 GB/token), so
+trace replay is the big win (7.5 -> 38 tok/s) and a second command queue is not. The
+experimental on-device Parallel Box Decoding (MTP, `media/demo_mtp.png`) is approximate,
+not accuracy-gated, and not served. The validated run needed one change to tt-metal's
+`models/tt_transformers/tt/mlp.py`, and this package ships it as an overlay
+(`code/models/tt_transformers/tt/mlp.py`, otherwise byte-identical to the stock file): in
+single-chip decode the `w1`/`w3` outputs are moved to DRAM before the next matmul, because
+with stock bf16 L1-sharded intermediates the `w3` weight-stream circular buffers do not fit
+L1 on one p150a (`Statically allocated circular buffers ... clash with L1 buffers` at the
+first decode step). Measured with the overlay on the validation host: `media/demo_input.png`
++ `car` -> the same box as `media/demo_ar.png`, vision 49 ms, prefill 58 ms, decode 41 tok/s,
+346 ms end-to-end. `LA_PREC=bfp8attn` (all-BFP8 attention, PCC 0.9912) remains available.
+
+### Layout notes
+
+`code/locate_anything/` is the port (`tt/vision.py` MoonViT, `tt/model_la.py` the embeds-fed
+Qwen2.5-3B `Transformer` subclass, `tt/pipeline.py` the serving recipe, `tt/mtp.py` the
+experimental MTP decoder, `reference/` the torch-CPU reference and the LLM-checkpoint
+extractor, `tests/` the PCC/benchmark/demo pytest suites, `server/` this app + smoke test).
+`code/models/{common,tt_transformers,demos/qwen25_vl}` are the tt-metal packages the LLM
+path reuses (stock Qwen2.5 `Transformer`/`Generator`/paged `Attention`), with the one
+overlay file `models/tt_transformers/tt/mlp.py` described above. The tests need a device
+and the torch-CPU goldens (`reference/run_reference.py`), which are not shipped.
+
+### Licensing
+
+The **upstream model and weights** are licensed **`other`** (nvidia-license):
+https://huggingface.co/nvidia/LocateAnything-3B/blob/main/LICENSE - NVIDIA's own licence,
+not an OSI licence; read it before any use. The weights are **not redistributed** here; they
+are fetched from the upstream repo under whatever terms that repo sets. The **port code**
+(Apache-2.0 SPDX headers; written by Hyunggi Chang, github.com/changh95/tt-locate-anything)
+is published under the same upstream terms, since a port cannot grant more than its upstream
+does. Backbone: Qwen2.5-3B-Instruct (Apache-2.0); vision tower: MoonViT (Kimi-VL, Moonshot AI);
+runtime: Tenstorrent tt-metal / tt-nn (Apache-2.0).
+
+## Provenance
+
+The exact sources the image was built from — `code/` in this repo is byte-identical to the model code inside the image:
+
+| component | built from |
+| --- | --- |
+| tt-metal | [`8b98410e730bb504fea43a88609756e34821d91d`](https://github.com/tenstorrent/tt-metal/commit/8b98410e730bb504fea43a88609756e34821d91d) |
+| `code/` digest | `eb3aa8433050cc46` (sha256, first 16 hex digits) |
+| built | 2026-09-12T06:57:00+00:00 by tt-model 0.1.0 |
 
-The **port code** here was written by [Hyunggi Chang](https://github.com/changh95) and is published under the same terms, since a port cannot grant more than its upstream does. The **weights are not redistributed** in this repository - they are fetched from the upstream repo, under whatever terms that repo sets.

code/models/common/auto_compose.py

@@ -0,0 +1,161 @@
+# SPDX-FileCopyrightText: © 2025 Tenstorrent USA, Inc.
+# SPDX-License-Identifier: Apache-2.0
+
+"""
+Automatic composition of multi-device sharded tensors using TensorTopology.
+
+This module provides utilities to infer the correct MeshToTensor composer from a
+sharded ttnn.Tensor's topology metadata and use it to compose shards on host.
+"""
+
+from typing import Optional
+
+import torch
+from loguru import logger
+
+import ttnn
+
+# ======================================================================================
+# Public API
+# ======================================================================================
+
+
+def to_torch_auto_compose(tensor: ttnn.Tensor, device: Optional[ttnn.MeshDevice] = None) -> torch.Tensor:
+    """
+    Convert a (possibly multi-device) TTNN tensor to torch, automatically
+    composing shards based on the tensor's topology.
+
+    Args:
+        tensor: The distributed tensor to convert
+        device: Optional MeshDevice to use when the tensor lives on host
+
+    Returns:
+        PyTorch tensor with shards composed
+    """
+    composer = _infer_mesh_composer_from_topology(tensor, device=device)
+    try:
+        return ttnn.to_torch(tensor, mesh_composer=composer)
+    except Exception as e:
+        logger.error(f"Failed to convert tensor to torch with mesh_composer: {e}")
+        raise
+
+
+def extract_tensor_topology_info(
+    tensor: ttnn.Tensor,
+) -> tuple[list[object], list[int]]:
+    """
+    Extract placements and distribution shape from a tensor's topology.
+
+    Returns:
+        (placements, dist_shape)
+    """
+    topology = tensor.tensor_topology()
+    placements = topology.placements()
+    dist_shape = list(topology.distribution_shape())
+    return placements, dist_shape
+
+
+def get_device_from_tensor(tensor: ttnn.Tensor) -> Optional[ttnn.MeshDevice]:
+    """Get device from tensor or fallback to provided mesh_device."""
+    device = tensor.device()
+    # tensor.device() returns None if the tensor is on the host (ttnn/core/tensor/tensor.cpp --> Tensor::device())
+    if device is None:
+        logger.debug("tensor.device() returns None, tensor is on the host")
+    else:
+        logger.debug(f"tensor.device() returns {device}")
+
+    return device
+
+
+# ======================================================================================
+# Private Implementation
+# ======================================================================================
+
+
+def _infer_mesh_composer_from_topology(
+    tensor: ttnn.Tensor, *, device: Optional[ttnn.MeshDevice] = None
+) -> Optional[ttnn.CppMeshToTensor]:
+    """
+    Return a MeshToTensor composer inferred from the tensor's TensorTopology,
+    or None if no composition is needed (fully replicated, single-device).
+
+    Note: For ND meshes with replicated dimensions, the composer will concatenate
+    all replicas, resulting in duplicated data. Callers may want to slice the
+    result if only one copy is desired.
+
+    Args:
+        tensor: The distributed tensor to infer composer for
+
+    Returns:
+        MeshToTensor composer or None if no composition needed
+    """
+    placements, dist_shape = extract_tensor_topology_info(tensor)
+
+    # No distribution or trivial 1-device case
+    if len(dist_shape) == 0 or (len(dist_shape) == 1 and dist_shape[0] == 1):
+        return None
+
+    tensor_device = get_device_from_tensor(tensor)
+    mesh_device = tensor_device or device
+    if mesh_device is None:
+        # As a last resort, try default device for backward-compatibility
+        mesh_device = ttnn.GetDefaultDevice()
+        if mesh_device is None:
+            raise RuntimeError(
+                "Tensor is on host and no mesh_device provided. "
+                "Pass device=... to to_torch_auto_compose or set a default via ttnn.SetDefaultDevice(...)."
+            )
+
+    # Must match length (should be guaranteed by C++ TT_FATAL in ttnn/core/distributed/distributed_tensor.cpp)
+    assert len(dist_shape) == len(placements)
+
+    if len(dist_shape) == 1 and mesh_device.shape.dims() == 1:
+        return _compose_1d_sharded(mesh_device, placements, dist_shape)
+    else:
+        # N >= 2 dimensions
+        return _compose_nd_sharded(mesh_device, placements, dist_shape)
+
+
+def _compose_1d_sharded(
+    device: ttnn.MeshDevice,
+    placements: list[object],
+    dist_shape: list[int],
+) -> Optional[ttnn.CppMeshToTensor]:
+    """Handle 1D case - returns None if fully replicated."""
+    p = placements[0]
+    if isinstance(p, ttnn.PlacementShard):
+        # Use ND composer with shape override to match the tensor's distribution
+        composer_cfg = ttnn.MeshComposerConfig(dims=[p.dim], mesh_shape_override=ttnn.MeshShape(dist_shape))
+        return ttnn.create_mesh_composer(device, composer_cfg)
+    # Fully replicated - no composition needed
+    return None
+
+
+def _compose_nd_sharded(
+    device: ttnn.MeshDevice,
+    placements: list[object],
+    dist_shape: list[int],
+) -> ttnn.CppMeshToTensor:
+    """
+    Handle ND (N>=2) case.
+
+    For replicated mesh dims, we use dim 0 as convention (the composed result
+    will include all replicas concatenated, which is typically not desired but
+    is how the C++ API works).
+    """
+    dims = []
+    shape_override = []
+    for i, p in enumerate(placements):
+        if isinstance(p, ttnn.PlacementShard):
+            dims.append(p.dim)
+            shape_override.append(dist_shape[i])
+        else:
+            assert isinstance(p, ttnn.PlacementReplicate)
+            # [INFO] steal from TensorDistribution2x4Test test case in test_distributed_tensor.cpp
+            # Replicated: use dim 0 as convention
+            dims.append(0)
+            # Replicated: use shape 1 to skip concatenation
+            shape_override.append(1)
+
+    composer_cfg = ttnn.MeshComposerConfig(dims=dims, mesh_shape_override=ttnn.MeshShape(shape_override))
+    return ttnn.create_mesh_composer(device, composer_cfg)

code/models/common/rmsnorm.py

@@ -0,0 +1,226 @@
+# SPDX-FileCopyrightText: © 2023 Tenstorrent USA, Inc.
+
+# SPDX-License-Identifier: Apache-2.0
+import ttnn
+from models.common.lightweightmodule import LightweightModule
+from models.tt_transformers.tt.common import Mode
+
+TILE = 32
+SHARD_HEIGHT = TILE  # Current ttnn.rms_norm implementation requires shard height to be a single tile
+
+
+class RMSNorm(LightweightModule):
+    """
+    RMSNorm supporting replication over a MeshDevice and sharding within devices.
+
+    This class implements a Root Mean Square Normalization (RMSNorm) that can be
+    distributed across multiple devices and cores. If the `device` parameter is a
+    MeshDevice, the weights and computations are replicated across all devices in
+    the mesh. Expects an interleaved input tensor, can optionally output a sharded tensor.
+
+    Args:
+        device: The device or MeshDevice on which to perform the computations.
+        state_dict: The state dictionary containing the model parameters.
+        dim: Input dimension (e.g. model hidden dimension size).
+        layer_num: The layer number to determine the weight key in the state dictionary.
+        weight_key: The key for retrieving the weight from the state dictionary.
+        weight_cache_path: Optional path for caching the tilized weights.
+        weight_memory_config: Configuration for the weight memory, default is DRAM_MEMORY_CONFIG.
+        weight_dtype: The data type for the tensors, bfp8_b hits >0.999 PCC in the models we tested.
+        model_config: Optional configuration dictionary for the model.
+        eps (float): Small value to avoid division by zero in normalization, default is 1e-05.
+
+    If model_config is provided, it must specify SHARDED_NORM_INPUT_MEMCFG, SHARDED_NORM_PRGM_CFG
+    and SHARDED_NORM_OUTPUT_MEMCFG. If not provided, default configurations will be generated.
+    """
+
+    def __init__(
+        self,
+        device,
+        dim,
+        state_dict,
+        weight_key,
+        layer_num=None,
+        state_dict_prefix=None,
+        weight_cache_path=None,
+        weight_memory_config=ttnn.DRAM_MEMORY_CONFIG,
+        weight_dtype=ttnn.bfloat16,
+        is_distributed=None,
+        eps: float = 1e-05,
+        add_unit_offset=False,
+        sharded_program_config=None,
+        sharded_output_config=None,
+        output_mem_config=None,
+        ccl_topology=ttnn.Topology.Ring,
+        tt_ccl=None,
+        fp32_dest_acc_en=True,
+    ):
+        super().__init__()
+        self.device = device
+        self.eps = eps
+        self.is_distributed = is_distributed
+        self.ccl_topology = ccl_topology
+        self.tt_ccl = tt_ccl
+
+        if state_dict_prefix:
+            weight_name = f"{state_dict_prefix}{weight_key}.weight"
+        else:
+            if layer_num is None:
+                weight_name = f"{weight_key}.weight"
+            else:
+                weight_name = f"layers.{layer_num}.{weight_key}.weight"
+
+        torch_weight = (
+            state_dict[weight_name].unsqueeze(0).view(1, 1, dim).reshape([1, 1, dim // SHARD_HEIGHT, SHARD_HEIGHT])
+        )
+
+        # Add offset before caching
+        if add_unit_offset:
+            torch_weight = torch_weight + 1.0
+
+        # Compatibility with models that don't use mesh devices (e.g. single-chip Mistral-7b)
+        is_mesh_device = device.__class__.__name__ == "MeshDevice"
+
+        self.weight = ttnn.as_tensor(
+            torch_weight,
+            device=device,
+            dtype=weight_dtype,
+            layout=ttnn.ROW_MAJOR_LAYOUT,
+            memory_config=weight_memory_config,
+            cache_file_name=None if weight_cache_path is None else weight_cache_path / weight_name,
+            mesh_mapper=ttnn.ReplicateTensorToMesh(device) if is_mesh_device else None,
+        )
+
+        if self.is_distributed:
+            self.weight_distributed = ttnn.as_tensor(
+                torch_weight,
+                device=device,
+                dtype=weight_dtype,
+                layout=ttnn.ROW_MAJOR_LAYOUT,
+                memory_config=weight_memory_config,
+                cache_file_name=(
+                    None if weight_cache_path is None else weight_cache_path / (weight_name + "_distributed")
+                ),
+                mesh_mapper=(
+                    ttnn.ShardTensor2dMesh(device, dims=(None, 2), mesh_shape=list(device.shape))
+                    if is_mesh_device
+                    else None
+                ),
+            )
+
+        self.sharded_output_config = sharded_output_config
+        self.sharded_program_config = sharded_program_config
+        self.output_mem_config = output_mem_config
+
+        self.compute_kernel_config_hifi2 = ttnn.WormholeComputeKernelConfig(
+            math_fidelity=ttnn.MathFidelity.HiFi2,
+            math_approx_mode=False,
+            fp32_dest_acc_en=fp32_dest_acc_en,
+            packer_l1_acc=True,
+        )
+
+    def forward(
+        self,
+        x: ttnn.Tensor,
+        mode: Mode | str,
+        in_sharded=False,
+        out_sharded=False,
+        norm_config=None,
+    ) -> ttnn.Tensor:
+        if isinstance(mode, str):
+            try:
+                mode = Mode(mode)
+            except ValueError:
+                raise ValueError(f"Invalid mode: {mode}")
+        elif not isinstance(mode, Mode):
+            raise ValueError(f"Invalid mode: {mode}")
+
+        sharded_program_config = norm_config.get("sharded_program_config") if norm_config else None
+        sharded_output_config = norm_config.get("sharded_output_config") if norm_config else None
+        output_mem_config = norm_config.get("output_mem_config") if norm_config else None
+        # Optional L1 placement for the distributed 3-op outputs (pre/gather/post); None -> DRAM default.
+        distributed_out_mc = norm_config.get("distributed_output_mem_config") if norm_config else None
+
+        # If input is sharded do sharded RMSNorm and optionally return sharded output
+        program_config = sharded_program_config if in_sharded else None
+        memory_config = sharded_output_config if out_sharded else None
+        distributed = self.is_distributed and self.is_distributed(mode)
+        weight = self.weight_distributed if distributed else self.weight
+
+        if in_sharded:
+            assert not distributed, "Distributed RMSNorm does not support sharded inputs"
+        else:
+            assert not out_sharded, "Non-sharded version of RMSNorm cannot output a sharded tensor"
+
+        if distributed:
+            x = self._distributed_rmsnorm(
+                x,
+                epsilon=self.eps,
+                weight=weight,
+                compute_kernel_config=self.compute_kernel_config_hifi2,
+                output_memory_config=distributed_out_mc,
+            )
+        else:
+            x = ttnn.rms_norm(
+                x,
+                epsilon=self.eps,
+                weight=weight,
+                program_config=program_config,
+                memory_config=memory_config,
+                compute_kernel_config=self.compute_kernel_config_hifi2,
+            )
+
+        if in_sharded and not out_sharded:
+            return ttnn.sharded_to_interleaved(x)
+        else:
+            if output_mem_config is not None:
+                x = ttnn.to_memory_config(x, output_mem_config)
+            return x
+
+    def _distributed_rmsnorm(
+        self,
+        inp,
+        epsilon=None,
+        weight=None,
+        program_config=None,
+        memory_config=None,
+        compute_kernel_config=None,
+        output_memory_config=None,
+    ):
+        assert program_config is None, "Distributed RMSNorm does not support sharded inputs"
+        assert memory_config is None, "Distributed RMSNorm does not support sharded outputs"
+        assert self.tt_ccl is not None, "Distributed RMSNorm requires tt_ccl"
+
+        # Interleaved output placement for the 3 ops; default DRAM (matches the prior hardcoded behavior).
+        mc = output_memory_config if output_memory_config is not None else ttnn.DRAM_MEMORY_CONFIG
+
+        # Run distributed rmsnorm part 1
+        tt_stats = ttnn.rms_norm_pre_all_gather(
+            inp, compute_kernel_config=compute_kernel_config, dtype=ttnn.bfloat16, memory_config=mc
+        )
+        # AllGather stats
+        tt_stats = ttnn.experimental.all_gather_async(
+            tt_stats,
+            persistent_output_buffer=None,
+            dim=3,
+            multi_device_global_semaphore=self.tt_ccl.get_and_cycle_ag_semaphore_handles(),
+            num_links=1,
+            topology=self.ccl_topology,
+            memory_config=mc,
+            barrier_semaphore=self.tt_ccl.get_and_cycle_barrier_semaphore_handle(),
+            chunks_per_sync=10,
+            num_workers_per_link=2,
+            num_buffers_per_channel=2,
+        )
+        # Run distributed rmsnorm part 2
+        tt_out = ttnn.rms_norm_post_all_gather(
+            inp,
+            tt_stats,
+            epsilon=epsilon,
+            weight=weight,
+            compute_kernel_config=compute_kernel_config,
+            memory_config=mc,
+        )
+        tt_stats.deallocate(True)
+
+        return tt_out

code/models/common/tensor_utils.py

@@ -0,0 +1,229 @@
+# SPDX-FileCopyrightText: © 2025 Tenstorrent USA, Inc.
+# SPDX-License-Identifier: Apache-2.0
+
+"""
+Tensor utility functions for TTTv2 modules.
+"""
+
+import json
+import math
+import re
+
+import torch
+
+import ttnn
+
+# Standard tile size - hardware constant
+TILE_SIZE = ttnn.TILE_SIZE  # 32
+
+
+def nearest_multiple(value: int, multiple: int) -> int:
+    return math.ceil(value / multiple) * multiple
+
+
+def nearest_32(value: int) -> int:
+    return nearest_multiple(value, TILE_SIZE)
+
+
+def num_to_core_range_set(num_cores: int):
+    assert num_cores < 8 or num_cores % 8 == 0
+    num_x = min(num_cores, 8)
+    num_y = num_cores // num_x
+    assert num_x * num_y == num_cores
+    return ttnn.CoreRangeSet(
+        {
+            ttnn.CoreRange(
+                ttnn.CoreCoord(0, 0),
+                ttnn.CoreCoord(num_x - 1, num_y - 1),
+            )
+        }
+    )
+
+
+def get_out_subblock_w(per_core_n: int, out_subblock_h: int = 1) -> int:
+    out_subblock_w = 4
+    while out_subblock_w > 1:
+        if out_subblock_w * out_subblock_h <= 4 and per_core_n % out_subblock_w == 0:
+            break
+        out_subblock_w -= 1
+    return out_subblock_w
+
+
+def align_shape_to_tile(shape, tile_size: int = TILE_SIZE):
+    """Round up the last two dimensions of *shape* to multiples of *tile_size*.
+
+    This is the recommended replacement for the deprecated ``ttnn.pad_to_tile_shape``.
+
+    Args:
+        shape: An iterable of dimension sizes (list, tuple, or ttnn.Shape).
+        tile_size: Tile dimension to align to (default 32).
+
+    Returns:
+        List[int]: A new shape with the last two dims tile-aligned.
+    """
+    import math
+
+    result = list(shape)
+    if len(result) >= 1:
+        result[-1] = math.ceil(result[-1] / tile_size) * tile_size
+    if len(result) >= 2:
+        result[-2] = math.ceil(result[-2] / tile_size) * tile_size
+    return result
+
+
+def get_rot_transformation_mat(dhead: int = TILE_SIZE) -> torch.Tensor:
+    """
+    Create rotation transformation matrix for RoPE.
+
+    Constructs a permutation matrix that pairs adjacent dimensions with
+    signs (+1, -1) for the RoPE rotation:
+        [0, 1] → +1 at (0,1), -1 at (1,0)
+        [2, 3] → +1 at (2,3), -1 at (3,2)
+        ...
+
+    Used by ttnn.experimental.rotary_embedding_llama.
+
+    Args:
+        dhead: Matrix dimension. Must equal TILE_SIZE. Use TILE_SIZE for decode.
+
+    Returns:
+        torch.Tensor of shape [1, 1, dhead, dhead].
+    """
+    rot_emb_matrix = torch.zeros(1, 1, dhead, dhead)
+    rot_emb_matrix[..., torch.arange(0, dhead, 2), torch.arange(1, dhead, 2)] = 1
+    rot_emb_matrix[..., torch.arange(1, dhead, 2), torch.arange(0, dhead, 2)] = -1
+    return rot_emb_matrix
+
+
+def zeros_like_kv_cache(batch_size: int, n_kv_heads: int, max_seq_len: int, head_dim: int) -> torch.Tensor:
+    """Create zeros tensor for standard KV cache."""
+    return torch.zeros((batch_size, n_kv_heads, max_seq_len, head_dim))
+
+
+def zeros_like_paged_cache(paged_config, n_kv_heads: int, head_dim: int) -> torch.Tensor:
+    """Create zeros tensor for paged KV cache."""
+    return torch.zeros((paged_config.max_num_blocks, n_kv_heads, paged_config.block_size, head_dim))
+
+
+# todo)) add a on-device pad_dim_to_size function?
+def pad_dim_to_size(x: "torch.Tensor", dim: int, size: int) -> "torch.Tensor":
+    """Pads the specified dimension of the input tensor with zeros."""
+    if dim < 0:
+        dim = x.dim() + dim
+    current_size = x.size(dim)
+    pad_size = size - current_size
+
+    if pad_size < 0:
+        raise ValueError(f"Target size {size} is smaller than current size {current_size} on dim {dim}")
+
+    if pad_size == 0:
+        return x
+
+    pad = [0] * (2 * x.dim())
+    pad_index = 2 * (x.dim() - dim - 1)
+    pad[pad_index + 1] = pad_size
+
+    return torch.nn.functional.pad(x, pad, mode="constant", value=0)
+
+
+def pad_to_shape(x: "torch.Tensor", target_shape: tuple[int, ...], pad_value: float = 0.0) -> "torch.Tensor":
+    """Pad tensor to target_shape in a single F.pad call (more efficient than per-dim padding)."""
+    if x.shape == target_shape:
+        return x
+
+    # F.pad expects: (left_last, right_last, left_second_last, right_second_last, ...)
+    pad = []
+    for orig, target in zip(reversed(x.shape), reversed(target_shape)):
+        if target < orig:
+            raise ValueError(f"Target size {target} is smaller than current size {orig}")
+        pad.extend([0, target - orig])
+
+    return torch.nn.functional.pad(x, pad, mode="constant", value=pad_value)
+
+
+def get_padded_hidden_dim(hidden_dim: int, num_devices: int, tile_size: int = 32) -> int:
+    """
+    Compute padded hidden_dim to satisfy ttnn.from_torch's tile alignment constraint.
+
+    ttnn.from_torch requires physical shard shapes to be tile-aligned. When sharding
+    a tensor across devices, each shard_dim = hidden_dim / num_devices must be
+    divisible by tile_size.
+
+    We pad the global tensor first, then shard evenly so only the last shard has padding.
+    """
+    shard_dim = hidden_dim // num_devices
+    padded_shard = ((shard_dim + tile_size - 1) // tile_size) * tile_size
+    return padded_shard * num_devices
+
+
+def parse_shard_dims_from_mesh_mapper_config(mesh_mapper_config: ttnn.MeshMapperConfig) -> list[int]:
+    """
+    Parse shard dimensions from MeshMapperConfig's repr.
+
+    MeshMapperConfig doesn't expose .placements directly, but repr shows them:
+        'MeshMapperConfig(placements: [PlacementShard(-1)], mesh_shape_override=MeshShape([8]))'
+
+    This parses out the shard dimensions (e.g., [-1]) from PlacementShard entries.
+    Returns empty list if no PlacementShard found (e.g., replicated).
+
+    Note: This is a workaround until TTNN exposes .placements directly.
+    """
+    config_repr = repr(mesh_mapper_config)
+    matches = re.findall(r"PlacementShard\((-?\d+)\)", config_repr)
+    return [int(d) for d in matches]
+
+
+def memory_config_to_dict(memory_config: ttnn.MemoryConfig):
+    # Convert to plain types for deterministic serialization.
+    return {
+        "memory_layout": str(memory_config.memory_layout),
+        "buffer_type": str(memory_config.buffer_type),
+        "shard_spec": str(memory_config.shard_spec),
+        "is_sharded": bool(memory_config.is_sharded()),
+        "interleaved": bool(memory_config.interleaved),
+        "hash": int(memory_config.__hash__()),
+    }
+
+
+def compute_kernel_config_to_str(compute_kernel_config: ttnn.WormholeComputeKernelConfig):
+    # Backward compat shim; prefer compute_kernel_config_to_dict + serialize_config.
+    cfg = compute_kernel_config_to_dict(compute_kernel_config)
+    return serialize_config(cfg)
+
+
+def compute_kernel_config_to_dict(compute_kernel_config: ttnn.WormholeComputeKernelConfig):
+    return {
+        "math_fidelity": str(compute_kernel_config.math_fidelity),
+        "math_approx_mode": str(compute_kernel_config.math_approx_mode),
+        "fp32_dest_acc_en": bool(compute_kernel_config.fp32_dest_acc_en),
+        "packer_l1_acc": bool(compute_kernel_config.packer_l1_acc),
+        "dst_full_sync_en": bool(compute_kernel_config.dst_full_sync_en),
+        "throttle_level": str(compute_kernel_config.throttle_level),
+    }
+
+
+def program_config_to_str(program_config: ttnn.MatmulMultiCoreReuseMultiCastDRAMShardedProgramConfig):
+    # Backward compat shim; prefer program_config_to_dict + serialize_config.
+    cfg = program_config_to_dict(program_config)
+    return serialize_config(cfg)
+
+
+def program_config_to_dict(program_config):
+    if hasattr(program_config, "to_json"):
+        d = json.loads(program_config.to_json())
+        d["type"] = type(program_config).__name__
+        return d
+    else:
+        return {"type": type(program_config).__name__, "repr": repr(program_config)}
+
+
+def serialize_config(cfg_dict: dict, fmt: str = "json") -> str:
+    if fmt == "json":
+        return json.dumps(cfg_dict, sort_keys=True)
+    if fmt == "yaml":
+        try:
+            import yaml
+        except ImportError as exc:  # pragma: no cover - optional dependency
+            raise RuntimeError("PyYAML is required for yaml serialization") from exc
+        return yaml.safe_dump(cfg_dict, sort_keys=True)
+    raise ValueError(f"Unsupported format: {fmt}")

code/models/common/validation_tools.py

@@ -0,0 +1,719 @@
+# SPDX-FileCopyrightText: © 2025 Tenstorrent USA, Inc.
+# SPDX-License-Identifier: Apache-2.0
+
+"""
+TTNN Validation Framework
+
+A decorator-based validation system for comparing TTNN implementations against
+reference implementations (in PyTorch). Supports automatic input/output
+mapping, metric computation, and result collection.
+
+Key Features:
+- Automatic comparison of TTNN vs reference implementations
+- TTNN-native metric computation (stays on device until final scalar)
+- Flexible input/output mapping
+- Built-in metrics: max_abs_error, mean_abs_error, cosine_similarity
+- Performance tracking
+- Result registry for batch reporting
+"""
+
+import time
+from dataclasses import dataclass, field
+from enum import Enum
+from functools import wraps
+from typing import Any, Callable, Dict, List, Optional
+
+import torch
+
+import ttnn
+
+from .auto_compose import to_torch_auto_compose
+from .distribute_as import from_torch_dist_as
+from .metrics import DEFAULT_METRICS
+
+# ============================================================================
+# Public API
+# ============================================================================
+
+# Module exports are defined at the package level in __init__.py
+
+
+def get_validation_registry() -> "ValidationRegistry":
+    """Get the global validation registry"""
+    return _validation_registry
+
+
+def enable_validation(enabled: bool = True):
+    """Enable or disable validation globally"""
+    _validation_registry.enabled = enabled
+
+
+def clear_validation_results():
+    """Clear all validation results"""
+    _validation_registry.results.clear()
+
+
+def compare_to_ttnn(
+    reference_fn: Callable,
+    *,
+    input_to_ttnn: Optional[Callable] = None,
+    output_to_ttnn: Optional[Callable] = None,
+    metric_tolerances: Optional[Dict[Any, Any]] = None,
+    enabled: bool = True,
+    raise_exceptions: bool = False,
+    return_reference_output: bool = False,
+):
+    """
+    Convenience wrapper for TTNN-on-device comparison. Provides useful visual cue to users that the reference function is a TTNN-native function.
+
+    Args:
+        reference_fn: Reference function to compare against
+        input_to_ttnn: Maps decorated function inputs to reference function inputs
+        output_to_ttnn: Maps decorated function outputs to reference function outputs
+        metric_tolerances: Dictionary specifying tolerances and optionally custom metrics.
+        enabled: Whether validation is enabled (can disable globally via registry)
+        raise_exceptions: When True, re-raise any exceptions encountered during
+            reference execution, output mapping, or metric computation instead
+            of logging them into validation results.
+
+    Examples:
+        @compare_to_ttnn(
+            reference_fn=lambda self, x: ttnn.matmul(x, self.weight),
+            input_to_ttnn=lambda self, x: (self, x),
+        )
+        def __call__(self, x):
+            return torch.matmul(x, self.torch_weight)
+            # alternatively, the decorated function can return a TTNN tensor: return ttnn.from_torch(x) @ self.weight
+
+    NOTES:
+        - The reference function is expected to accepts TTNN tensors and returns a TTNN tensor
+        - The decorated function inputs/outputs TTNN tensors, Torch tensors, or mixed TTNN and Torch tensors
+        - When decorated function returns torch tensors:
+          - the reference function's inputs will be constructed through either input_to_ttnn or from_torch(decorated function inputs, device=ttnn.GetDefaultDevice())
+          - the metric on output tensor will be computed on the host
+        - Experimental support for on-device metric computation is provided and used when both the decorated function and the reference function return TTNN tensors
+    """
+
+    # Default converters: recursively convert any TTNN tensors to torch, auto-compose shards.
+    # Non-tensor objects are passed through unchanged.
+
+    def _to_ttnn_auto(x: Any) -> Any:
+        if torch.is_tensor(x):
+            # Use auto-compose; relies on tensor.device() or a globally-set default device
+            assert (
+                ttnn.GetDefaultDevice() is not None
+            ), "Default device is not set. It is required by compare_to_ttnn. Please set it via ttnn.SetDefaultDevice(...)."
+            return ttnn.from_torch(x, device=ttnn.GetDefaultDevice())
+        return x
+
+    def _default_input_map(*args, **kwargs):
+        ref_args = _map_structure(args, _to_ttnn_auto)
+        ref_kwargs = _map_structure(kwargs, _to_ttnn_auto)
+        return ref_args, ref_kwargs
+
+    map_fn_to_match_sig = lambda tt_tensor, filler: to_torch_auto_compose(tt_tensor)
+
+    return __validate_against(
+        reference_fn=reference_fn,
+        input_map=input_to_ttnn or _default_input_map,
+        output_map=output_to_ttnn,
+        metric_tolerances=metric_tolerances,
+        enabled=enabled,
+        raise_exceptions=raise_exceptions,
+        reference_output_map_fn=map_fn_to_match_sig if return_reference_output else None,
+    )
+
+
+def compare_to_torch(
+    reference_fn: Callable,
+    *,
+    input_to_torch: Optional[Callable] = None,
+    output_to_torch: Optional[Callable] = None,
+    metric_tolerances: Optional[Dict[Any, Any]] = None,
+    enabled: bool = True,
+    raise_exceptions: bool = False,
+    return_reference_output: Optional[Callable[..., bool] | bool] = False,
+):
+    """
+    Convenience wrapper for host/CPU comparison using torch.
+
+    # Args:
+    #     reference_fn: Reference function to compare against
+    #     input_to_torch: Maps decorated function inputs to reference function inputs
+    #     output_to_torch: Maps decorated function outputs to reference function outputs
+    #     metric_tolerances: Dictionary specifying tolerances and optionally custom metrics.
+    #     enabled: Whether validation is enabled (can disable globally via registry)
+    #     raise_exceptions: When True, re-raise any exceptions encountered during
+    #         reference execution, output mapping, or metric computation instead
+    #         of logging them into validation results.
+    #
+    # Notes:
+    # - compare_to_torch is used when the reference function is a PyTorch function
+    # - the reference function takes as inputs to_torch_auto_compose(decorated function inputs) and compares the outputs with to_torch_auto_compose(decorated function outputs)
+    # - the decorated function inputs/outputs TTNN tensors, Torch tensors, or mixed TTNN and Torch tensors
+    """
+
+    # Default converters: recursively convert any TTNN tensors to torch, auto-compose shards.
+    # Non-tensor objects are passed through unchanged.
+
+    def _to_torch_auto(x: Any) -> Any:
+        if isinstance(x, ttnn.Tensor):
+            # Use auto-compose; relies on tensor.device() or a globally-set default device
+            return to_torch_auto_compose(x)
+        return x
+
+    def _default_input_map(*args, **kwargs):
+        ref_args = _map_structure(args, _to_torch_auto)
+        ref_kwargs = _map_structure(kwargs, _to_torch_auto)
+        return ref_args, ref_kwargs
+
+    def _default_output_map(output):
+        return _map_structure(output, _to_torch_auto)
+
+    return __validate_against(
+        reference_fn=reference_fn,
+        input_map=input_to_torch or _default_input_map,
+        output_map=output_to_torch or _default_output_map,
+        metric_tolerances=metric_tolerances,
+        enabled=enabled,
+        raise_exceptions=raise_exceptions,
+        reference_output_map_fn=from_torch_dist_as if return_reference_output else None,
+    )
+
+
+# ============================================================================
+# Data Structures
+# ============================================================================
+
+
+@dataclass
+class MetricResult:
+    """Per-metric validation outcome"""
+
+    value: float = float("inf")
+    passed: bool = False
+    error: str = ""
+
+
+@dataclass
+class ValidationResult:
+    """Results from a single validation run"""
+
+    function_name: str
+    passed: bool
+    # Map of metric name to its result (value/pass/fail/error)
+    metrics: Dict[Any, MetricResult] = field(default_factory=dict)
+    execution_time_impl: float = 0.0
+    execution_time_ref: float = 0.0
+    timestamp: float = field(default_factory=time.time)
+    logs: List[str] = field(default_factory=list)
+
+
+class ValidationRegistry:
+    """Global registry for validation results"""
+
+    def __init__(self):
+        self.results: List[ValidationResult] = []
+        self.enabled = True
+
+    def add_result(self, result: ValidationResult):
+        self.results.append(result)
+
+    def get_summary(self) -> Dict[str, Any]:
+        """Get summary statistics of all validations"""
+        if not self.results:
+            return {"total": 0, "passed": 0, "failed": 0}
+
+        passed = sum(1 for r in self.results if r.passed)
+        failed = len(self.results) - passed
+
+        return {
+            "total": len(self.results),
+            "passed": passed,
+            "failed": failed,
+            "pass_rate": passed / len(self.results) if self.results else 0.0,
+            "avg_speedup": (
+                sum(r.execution_time_ref / r.execution_time_impl for r in self.results if r.execution_time_impl > 0)
+                / len(self.results)
+                if self.results
+                else 0.0
+            ),
+        }
+
+    def print_report(self, verbose: bool = False):
+        """Print detailed validation report"""
+        summary = self.get_summary()
+        print("\n" + "=" * 80)
+        print("VALIDATION REPORT")
+        print("=" * 80)
+        print()
+
+        for result in self.results:
+            status = "✓ PASS" if result.passed else "✗ FAIL"
+            print(f"{status} - {result.function_name}")
+            print(
+                f"  Execution time: impl={result.execution_time_impl*1000:.2f}ms, ref={result.execution_time_ref*1000:.2f}ms"
+            )
+
+            if result.metrics:
+                print(f"  Metrics:")
+                for metric_name, mres in result.metrics.items():
+                    # Use enum value for readability if metric is an Enum
+                    name_str = metric_name.value if hasattr(metric_name, "value") else str(metric_name)
+                    if mres.value is not None:
+                        try:
+                            val_str = f"{mres.value:.6f}"
+                        except Exception:
+                            val_str = str(mres.value)
+                    else:
+                        val_str = "-"
+                    status = "PASS" if mres.passed else "FAIL"
+                    print(f"    {name_str}: {val_str} — {status}")
+                    if mres.error:
+                        print(f"      error: {mres.error}")
+
+            # Print any collected logs for this validation
+            if result.logs and verbose:
+                print("  Logs:")
+                for entry in result.logs:
+                    try:
+                        msg = str(entry)
+                    except Exception:
+                        msg = "<unprintable log entry>"
+                    print(f"    {msg}")
+
+            # All errors are reported via per-metric entries
+            print()
+
+        print("-" * 36 + "Summary:" + "-" * 36)
+        print(f"Total validations: {summary['total']}")
+        print(f"Passed: {summary['passed']} ({summary['pass_rate']*100:.1f}%)")
+        print(f"Failed: {summary['failed']}")
+        print(f"Average speedup: {summary['avg_speedup']:.2f}x")
+        print()
+        print("=" * 80 + "\n")
+
+
+# Global validation registry
+_validation_registry = ValidationRegistry()
+
+
+# ============================================================================
+# Validation Decorator
+# ============================================================================
+
+
+class Metric(str, Enum):
+    """Enumeration of supported metric names, values match current string keys."""
+
+    MAX_ABS_ERROR = "max_abs_error"
+    MEAN_ABS_ERROR = "mean_abs_error"
+    PCC = "pcc"
+
+
+@dataclass
+class MetricSpec:
+    """Metric specification: name, tolerance, direction, and compute function."""
+
+    tolerance: float
+    higher_is_better: bool
+    compute_fn: Callable[[Any, Any], float]
+    name: str = field(default="")
+
+
+# Registry of built-in metrics with defaults. Tolerances here are sensible
+# defaults; callers can override per-validation via `tolerances`.
+METRIC_SPECS: Dict[Metric, MetricSpec] = {
+    Metric.MAX_ABS_ERROR: MetricSpec(
+        name=Metric.MAX_ABS_ERROR.value,
+        tolerance=0.0,
+        higher_is_better=False,
+        compute_fn=DEFAULT_METRICS[Metric.MAX_ABS_ERROR.value],
+    ),
+    Metric.MEAN_ABS_ERROR: MetricSpec(
+        name=Metric.MEAN_ABS_ERROR.value,
+        tolerance=0.0,
+        higher_is_better=False,
+        compute_fn=DEFAULT_METRICS[Metric.MEAN_ABS_ERROR.value],
+    ),
+    Metric.PCC: MetricSpec(
+        name=Metric.PCC.value,
+        tolerance=0.0,
+        higher_is_better=True,
+        compute_fn=DEFAULT_METRICS[Metric.PCC.value],
+    ),
+}
+
+# Convenience groupings for quick checks
+HIGHER_IS_BETTER_METRICS = {m.value for m, spec in METRIC_SPECS.items() if spec.higher_is_better}
+LOWER_IS_BETTER_METRICS = {m.value for m, spec in METRIC_SPECS.items() if not spec.higher_is_better}
+
+
+# Helper: prefer Metric enum as dict key when possible
+def _metric_key(key: Any) -> Any:
+    try:
+        return Metric(key)
+    except Exception:
+        return key
+
+
+# Helper: Build active metrics map (name -> compute fn). Accept Metric enum keys for tolerances.
+def _normalize_key(k: Any) -> str:
+    try:
+        # Enum or similar objects with .value as canonical string
+        return k.value if hasattr(k, "value") else str(k)
+    except Exception:
+        return str(k)
+
+
+# Helper: Prepare metrics, tolerances, and directionality
+def _prepare_metric_config(metric_tolerances_input):
+    metrics_map = {name: fn for name, fn in DEFAULT_METRICS.items()}
+    hib = set(HIGHER_IS_BETTER_METRICS)
+    logs_local: List[str] = []
+    tol_map: Dict[str, float] = {}
+
+    if not isinstance(metric_tolerances_input, dict):
+        logs_local.append(f"metric_tolerances_input must be a dict, got {type(metric_tolerances_input)}")
+        metric_tolerances_input = dict()
+
+    if not metric_tolerances_input:
+        logs_local.append("no metric tolerances provided")
+        metric_tolerances_input = dict()
+
+    for raw_key, spec in metric_tolerances_input.items():
+        name = _normalize_key(raw_key)
+        if isinstance(spec, MetricSpec):
+            tol_map[name] = float(spec.tolerance)
+            metrics_map[name] = spec.compute_fn
+            spec.name = name if spec.name == "" else spec.name
+            if spec.higher_is_better:
+                hib.add(name)
+            else:
+                hib.discard(name)
+            continue
+        try:
+            tol_map[name] = float(spec)
+        except Exception:
+            logs_local.append(f"unrecognized tolerance: {raw_key}: {spec}")
+
+    return metrics_map, hib, tol_map, logs_local
+
+
+# todo)) also allow raise an exception from the a failed metric!
+
+# todo)) add support for multiple outputs from the reference function and the decorated function!
+# e.g., return logits, past_key_values, etc.
+
+# todo)) make sure the dtypes are taken care of in the validate_against decorator!
+# e.g., if the decorated function is of dtype bfp4, what is the dtype of the to_torch_auto_compose output?
+
+# todo)) add file line number to the validation results!
+
+# todo)) add function to export the validation results to a csv file!
+
+# todo)) enhance report to use file line number as index to summarize the validation results
+# e.g., ✗ FAIL - __main__.Attention.__call__ (line 100) -> 100 failed validations
+
+# todo)) remove compile time from speed up calculation -- e.g., 9118.15ms should be removed in the example below:
+# ================================================================================
+# VALIDATION REPORT
+# ================================================================================
+# Total validations: 1400
+# Passed: 1400 (100.0%)
+# Failed: 0
+# Average speedup: 0.97x
+
+# ✓ PASS - __main__.TransformerBlock.__call__
+#   Execution time: impl=9118.15ms, ref=14.84ms
+#   Metrics:
+#     pcc: 0.999743 — PASS
+
+# ✓ PASS - __main__.TransformerBlock.__call__
+#   Execution time: impl=3.05ms, ref=12.73ms
+#   Metrics:
+#     pcc: 0.999913 — PASS
+
+# ✓ PASS - __main__.TransformerBlock.__call__
+#   Execution time: impl=3.31ms, ref=12.48ms
+#   Metrics:
+#     pcc: 0.999962 — PASS
+
+# ✓ PASS - __main__.TransformerBlock.__call__
+#   Execution time: impl=3.11ms, ref=12.89ms
+#   Metrics:
+#     pcc: 1.000000 — PASS
+
+# ✓ PASS - __main__.TransformerBlock.__call__
+#   Execution time: impl=3.16ms, ref=12.97ms
+#   Metrics:
+#     pcc: 0.999998 — PASS
+
+
+# todo)) stretch goals:
+# - generate unit test automatically from the failed validations
+def __validate_against(
+    reference_fn: Callable,
+    *,
+    input_map: Optional[Callable] = None,
+    output_map: Optional[Callable] = None,
+    metric_tolerances: Optional[Dict[Any, Any]] = None,
+    enabled: bool = True,
+    raise_exceptions: bool = False,
+    reference_output_map_fn: Optional[Callable] = None,
+):
+    """
+    Decorator to validate a function against a reference implementation.
+
+    Args:
+        reference_fn: Reference function to compare against
+        input_map: Maps decorated function inputs to reference function inputs
+                   Signature: (args, kwargs) -> (ref_args, ref_kwargs)
+                   If None, inputs are passed as-is
+        output_map: Converts impl output to match ref output's type
+                    Signature: (output) -> comparable_output
+                    Applied ONLY to impl_output to convert it to ref_output's type
+                    Common use: lambda x: ttnn.to_torch(x).squeeze() to convert ttnn → torch
+                    If None, outputs are used as-is (both must already be same type)
+        metric_tolerances: Dictionary specifying tolerances and optionally custom metrics.
+            Accepts the following per metric key (str or Metric):
+              - float: tolerance only (uses built-in compute + direction)
+              - MetricSpec instance
+            Validation fails if any metric exceeds its tolerance
+        enabled: Whether validation is enabled (can disable globally via registry)
+        raise_exceptions: When True, re-raise any exceptions encountered during
+            reference execution, output mapping, or metric computation instead
+            of logging them into validation results.
+
+    Examples:
+        # Pattern 1: TTNN-native metrics (recommended, 100-1000× faster!)
+        # Both impl and ref return ttnn.Tensor, no output_map needed
+        def _reference_impl(self, x):
+            x_torch = ttnn.to_torch(x).squeeze(0)
+            result_torch = torch.matmul(x_torch, self.weight_torch)
+            # Convert back to TTNN for on-device metrics!
+            return ttnn.from_torch(result_torch.unsqueeze(0), device=self.device, ...)
+
+        @validate_against(
+            reference_fn=lambda self, x: self._reference_impl(x),
+            tolerances={'max_abs_error': 1e-3}
+        )
+        def __call__(self, x):
+            return ttnn.matmul(x, self.weight)
+
+        # Pattern 2: PyTorch metrics (when reference returns torch.Tensor)
+        # Use output_map to convert impl output (ttnn.Tensor) to match ref (torch.Tensor)
+        @validate_against(
+            reference_fn=torch.nn.functional.rms_norm,
+            input_map=lambda args, kwargs: (
+                (ttnn.to_torch(args[1]).squeeze(),),
+                {'eps': args[0].eps}
+            ),
+            output_map=lambda x: ttnn.to_torch(x).squeeze(),  # Convert impl: ttnn → torch
+            tolerances={'max_abs_error': 1e-3}
+        )
+        def __call__(self, x):
+            return ttnn.rms_norm(x, self.weight, self.eps)  # Returns ttnn.Tensor
+    """
+
+    if metric_tolerances is None:
+        metric_tolerances = {
+            Metric.MAX_ABS_ERROR: 1e-2,
+            Metric.PCC: 0.99,
+        }
+
+    metrics_to_use, higher_is_better_effective, tolerances_map, pre_logs = _prepare_metric_config(metric_tolerances)
+
+    def decorator(func):
+        @wraps(func)
+        def wrapper(*args, **kwargs):
+            # Check if validation is enabled
+            if not enabled or not _validation_registry.enabled:
+                return func(*args, **kwargs)
+
+            # Execute implementation
+            start_time = time.perf_counter()
+            impl_output = func(*args, **kwargs)
+            impl_time = time.perf_counter() - start_time
+            logs: List[str] = pre_logs.copy()
+
+            # Map inputs for reference function: prefer input_map, else pass-through
+            if input_map:
+                _nm = getattr(input_map, "__name__", None) or type(input_map).__name__
+                logs.append(f"input_map={_nm}")
+                try:
+                    mapped = input_map(*args, **kwargs)
+                except Exception as e:
+                    # If input mapping fails, log error, record result, and return impl output
+                    logs.append(f"input_mapping_error={str(e)}")
+                    result = ValidationResult(
+                        function_name=f"{func.__module__}.{func.__qualname__}",
+                        passed=False,
+                        metrics={
+                            "input_mapping": MetricResult(
+                                value=None, passed=False, error=f"Input mapping failed: {str(e)}"
+                            )
+                        },
+                        execution_time_impl=impl_time,
+                        execution_time_ref=0.0,
+                        logs=logs,
+                    )
+                    _validation_registry.add_result(result)
+                    # Re-raise exception if raise_exceptions is True
+                    if raise_exceptions:
+                        raise
+                    return impl_output
+                # Normalize mapper output:
+                # - If (ref_args, ref_kwargs) with kwargs as dict, use directly
+                # - Otherwise, treat return as positional args and use empty kwargs
+                if isinstance(mapped, tuple) and len(mapped) == 2 and isinstance(mapped[1], dict):
+                    ref_args, ref_kwargs = mapped
+                else:
+                    ref_args = mapped if isinstance(mapped, (list, tuple)) else (mapped,)
+                    ref_kwargs = {}
+            else:
+                logs.append("input_map=pass-through")
+                ref_args, ref_kwargs = args, kwargs
+
+            # Execute reference
+            try:
+                start_time = time.perf_counter()
+                ref_output = reference_fn(*ref_args, **ref_kwargs)
+                ref_time = time.perf_counter() - start_time
+            except Exception as e:
+                # If reference fails, just return impl output and log error via metrics
+                logs.append(f"reference_execution_error={str(e)}")
+                # Record elapsed time until failure
+                ref_time = time.perf_counter() - start_time
+                result = ValidationResult(
+                    function_name=f"{func.__module__}.{func.__qualname__}",
+                    passed=False,
+                    metrics={
+                        "reference_execution": MetricResult(
+                            value=None, passed=False, error=f"Reference execution failed: {str(e)}"
+                        )
+                    },
+                    execution_time_impl=impl_time,
+                    execution_time_ref=ref_time,
+                    logs=logs,
+                )
+                _validation_registry.add_result(result)
+                # Re-raise exception if raise_exceptions is True
+                if raise_exceptions:
+                    raise
+                return impl_output
+
+            # Map outputs for comparison
+            # Note: output_map only applies to impl_output to convert it to match ref_output's type
+            try:
+                _nm = getattr(output_map, "__name__", None) or type(output_map).__name__
+                logs.append(f"output_map={_nm}")
+                impl_comparable = output_map(impl_output) if output_map else impl_output
+                ref_comparable = ref_output  # Reference output is always used as-is
+            except Exception as e:
+                logs.append(f"output_mapping_error={str(e)}")
+                result = ValidationResult(
+                    function_name=f"{func.__module__}.{func.__qualname__}",
+                    passed=False,
+                    metrics={
+                        "output_mapping": MetricResult(
+                            value=None, passed=False, error=f"Output mapping failed: {str(e)}"
+                        )
+                    },
+                    execution_time_impl=impl_time,
+                    execution_time_ref=ref_time,
+                    logs=logs,
+                )
+                _validation_registry.add_result(result)
+                # Re-raise exception if raise_exceptions is True
+                if raise_exceptions:
+                    raise
+                return impl_output
+
+            # Compute metrics
+            computed_metrics: Dict[Any, MetricResult] = {}
+            passed = True
+
+            for metric_name, threshold in tolerances_map.items():
+                try:
+                    metric_fn = metrics_to_use.get(metric_name)
+
+                    # Store results keyed by enum when available
+                    metric_key = _metric_key(metric_name)
+                    # If metric function isn't known, record an error
+                    if metric_fn is None:
+                        computed_metrics[metric_key] = MetricResult(
+                            value=None, passed=False, error=f"Unknown metric: {metric_name}"
+                        )
+                        passed = False
+                        continue
+
+                    value = metric_fn(impl_comparable, ref_comparable)
+
+                    # Determine direction using registry when available
+                    if metric_name in higher_is_better_effective:
+                        ok = value >= threshold
+                        err = None
+                        if not ok:
+                            passed = False
+                            err = f"{metric_name}={value:.6e} below threshold {threshold:.6e}"
+                        computed_metrics[metric_key] = MetricResult(value=value, passed=ok, error=err)
+                    else:
+                        ok = value <= threshold
+                        err = None
+                        if not ok:
+                            passed = False
+                            err = f"{metric_name}={value:.6e} exceeds tolerance {threshold:.6e}"
+                        computed_metrics[metric_key] = MetricResult(value=value, passed=ok, error=err)
+                except Exception as e:
+                    msg = f"Metric {metric_name} failed: {str(e)}"
+                    computed_metrics[metric_key] = MetricResult(value=None, passed=False, error=msg)
+                    passed = False
+                    if raise_exceptions:
+                        raise
+
+            # Optionally return the (aligned) reference output instead of impl output
+            backup_impl_output = impl_output
+            try:
+                if reference_output_map_fn:
+                    impl_output = reference_output_map_fn(ref_output, impl_output)
+                    logs.append(f"reference_output_mapping_fn={reference_output_map_fn.__name__}")
+            except Exception as e:
+                # If alignment fails, fall back to impl output
+                impl_output = backup_impl_output
+                # Re-raise exception if raise_exceptions is True after logging the error
+                logs.append(f"reference_output_mapping_error={str(e)}")
+                if raise_exceptions:
+                    raise
+
+            # Record results
+            pass_count = sum(1 for v in computed_metrics.values() if v.passed)
+            fail_count = sum(1 for v in computed_metrics.values() if not v.passed)
+            logs.append(f"metrics={pass_count}_pass,{fail_count}_fail")
+            result = ValidationResult(
+                function_name=f"{func.__module__}.{func.__qualname__}",
+                passed=passed,
+                metrics=computed_metrics,
+                execution_time_impl=impl_time,
+                execution_time_ref=ref_time,
+                logs=logs,
+            )
+            _validation_registry.add_result(result)
+
+            return impl_output
+
+        return wrapper
+
+    return decorator
+
+
+def _map_structure(obj: Any, fn: Callable[[Any], Any]) -> Any:
+    """
+    Map a structure of objects to a new structure using a function.
+    """
+    if isinstance(obj, (list, tuple)):
+        mapped = [_map_structure(x, fn) for x in obj]
+        return type(obj)(mapped)
+    if isinstance(obj, dict):
+        return {k: _map_structure(v, fn) for k, v in obj.items()}
+    return fn(obj)

code/models/tt_transformers/Host_Mem_Profile.md

@@ -0,0 +1,88 @@
+# Model memory on host for performance and accuracy
+
+[Host memory usage results](../sample_data/host_mem_profiling/) at peak memory (in MB) are collected from host_mem_profiler.py by running the target demo under memory_profiler at
+100 ms sampling intervals. You can generate these results by running the script with --mesh-device, --hf-model, and
+--test arguments in [demo/simple_text_demo.py](demo/simple_text_demo.py); it produces a timestamped PNG plot under profiling_results/ and prints peak and baseline RSS to the
+console.
+
+Note that all measurements include child processes spawned by the TT device runtime, giving a realistic view of total
+host-side footprint. Peak memory is defined as the maximum RSS observed across the full test lifetime — including
+model weight loading, KV-cache allocation, and decode iterations. Baseline is the RSS at process start before any
+model work begins.
+
+Host memory (RSS) was profiled across three configurations — [Performance](#performance), [Accuracy](#accuracy), and [Multimodal](#multimodal-models) — using up to 200
+decode iterations on T3K (8-chip) and N300 (2-chip) devices.
+
+All [numbers](../sample_data/host_mem_profiling/) below were captured with a maximum generation of 200 tokens (200 decode iterations) to match the
+conditions used for centralized targets in [models/model_targets.yaml](../model_targets.yaml).
+
+## Performance
+
+This configuration uses bfp4 MLP and bfp8 attention weights for all models except:
+* Qwen-2.5-7B, which uses bfp8 MLP and bfp16 attention weights in all decoder layers
+* Llama-3.1-8B which uses bfp8 MLP in only the 32nd decoder layer and bfp4 MLP elsewhere
+
+| Model             | Device      | batch-1 max mem (MB)  | batch-32 max mem (MB) | Speed (t/s/u) | TTFT (ms) |
+|-------------------|-------------|-----------------------|-----------------------|---------------|-----------|
+| Llama-3.2-1B      | T3K         |         3532          |         2712          | 119.8         | 32        |
+| Llama-3.2-3B      | T3K         |         7846          |         5388          | 68.5          | 52        |
+| Llama-3.1-8B      | T3K         |         13683         |         6922          | 64.3          | 53        |
+| Llama-3.2-11B     | T3K         |         13885         |         7254          | 62.7          | 47        |
+| Llama-3.1-70B     | T3K         |         35820         |         35187         | 16.6          | 164       |
+| Llama-3.3-70B     | T3K         |         101640        |         36176         | 16.6          | 164       |
+| Llama-3.2-90B     | T3K         |         34427         |         34842         | 6             | 5535      |
+| Qwen2.5-7B        | N300        |         16597         |         17754         | 24.6          | 92        |
+| Qwen2.5-72B       | T3K         |         147106        |         145432        | 15.2          | 225       |
+| Qwen2.5-32B       | T3K         |         53434         |         15594         | 22.4          | 190       |
+| Qwen3-32B         | T3K         |         56033         |         20422         | 22.9          | 123       |
+| QwQ-32B           | T3K         |         69900         |         67186         | 20.7          | 105       |
+
+## Accuracy
+
+This configuration uses bfp8 MLP and BF16 attention weights (70B+ models use bfp8 attention and bfp4 MLP).
+Llama 3 models test as insensitive to attention precision and so we use bfp8 attention and kv-cache for them even in accuracy mode.
+
+| Model             | Device      | batch-1 max mem (MB)  | batch-32 max mem (MB) | Speed (t/s/u) | TTFT (ms) |
+|-------------------|-------------|----------------------|------------------------|---------------|-----------|
+| Llama-3.2-1B      | T3K         |        4654          |          2707          | 120.5         | 28        |
+| Llama-3.2-3B      | T3K         |        10374         |          5270          | 67.9          | 69        |
+| Llama-3.1-8B      | T3K         |        20406         |          7704          | 60.8          | 81        |
+| Llama-3.2-11B     | T3K         |        20265         |          7051          | 61.4          | 53        |
+| Llama-3.1-70B     | T3K         |        154122        |          37370         | 16.5          | 168       |
+| Llama-3.3-70B     | T3K         |        100112        |          35026         | 16.5          | 168       |
+| Llama-3.2-90B     | T3K         |        155139        |          34997         | 6             | 5600      |
+| Qwen2.5-7B        | N300        |        20189         |          17956         | 24.6          | 92        |
+| Qwen2.5-72B       | T3K         |        146204        |          147032        | 15.1          | 216       |
+| Qwen2.5-32B       | T3K         |        74162         |          14811         | 19.7          | 183       |
+| Qwen3-32B         | T3K         |        76288         |          20785         | 19.6          | 119       |
+| QwQ-32B           | T3K         |        72627         |          67278         | 18.3          | 120       |
+
+
+## Multimodal Models
+The results are collected using this script [demo/simple_vision_demo.py](demo/simple_vision_demo.py)); with host memory profiler and evaluated on both language and vision branches.
+The memory is reported in MB.
+
+| Model             | Device      | batch-1 max mem (MB)  | batch-32 max mem (MB) | Speed (t/s/u) | TTFT (ms) |
+|-------------------|-------------|-----------------------|-----------------------|---------------|-----------|
+| Llama-3.2-11B     | T3K         |         14309         |         8610          | 61.4          | 53        |
+| Llama-3.2-90B     | T3K         |         75451         |         -             | 6             | 5600      |
+| Qwen2.5-VL-3B     | N300        |         9319          |         9068          | 24.6          | 92        |
+| Qwen2.5-VL-7B     | N300        |         18745         |         10946         | 24.6          | 92        |
+| Qwen2.5-VL-72B    | T3K         |         160437        |         147925        | 15.1          | 216       |
+| Qwen2.5-VL-32B    | T3K         |         71960         |         68464         | 19.7          | 183       |
+| Qwen3-VL-32B      | T3K         |         76297         |         21514         | 19.6          | 119       |
+
+
+[Performance](#performance) vs. [Accuracy](#accuracy) modes trade memory for precision: accuracy mode uses BF16 attention weights versus bfp8/bfp4
+in performance mode, and typically raises peak memory. The increase is clearest at batch-1, where most models grow noticeably Llama-3.1-8B rises from 13.7 GB to 20.4 GB and Qwen2.5-32B from 53 GB to 74 GB — while 70B+ models show a smaller relative increase since attention weight precision is already reduced in both modes. At batch-32 the picture is mixed: KV-cache and activations dominate, and several models (Llama-3.2-1B/3B/11B, Llama-3.3-70B, Qwen2.5-32B) report slightly lower peaks in accuracy mode rather than higher.
+
+Memory scales roughly with model size, ranging from ~3–4 GB for 1B models up to ~147–155 GB for 72–90B models. The
+largest models (Qwen2.5-72B, Llama-3.2-90B) show little difference between batch-1 and batch-32, suggesting KV-cache
+and weight loading dominate over activation memory. Mid-size models (32B class) exhibit a larger batch-1 to batch-32
+gap, likely due to KV-cache growth with sequence length at batch-1.
+
+Throughput and TTFT follow expected trends: smaller models deliver higher tokens/s/user (up to 120 t/s/u for 1B) and
+low TTFT (~30 ms), while 90B models drop to 6 t/s/u with TTFT exceeding 5.5 seconds.
+
+[Multimodal](#multimodal-models) models carry a modest overhead versus their text-only counterparts — Llama-3.2-11B adds ~400 MB at batch-1
+though Qwen2.5-VL-72B reaches the highest overall footprint at ~160 GB.

code/models/tt_transformers/PERF.md

@@ -0,0 +1,247 @@
+Note: DEPRECATED. PLEASE CHECK THE YAML FILE [models/model_targets.yaml](models/model_targets.yaml)
+
+# Model performance and accuracy
+
+Performance and token accuracy using teacher forcing is collected from [demo/simple_text_demo.py](demo/simple_text_demo.py) with the `ci-token-matching` test case. You can generate this table by running these tests with the `lt` tool (tell it to run `table` or `pareto`) and pressing `m` whilst in the results section to export to markdown.
+
+Note that token accuracy parses the below to determine expected values +- 0.5. In May 2025 we switched the default to measuring the accuracy by prefilling 512 tokens and generating another 511, rather than generating 128 tokens in earlier versions. This caused overall accuracy values to drop slightly.
+
+Also note that all the performance metrics below were taken for a maximum generation of 200 tokens, i.e., 200 decode iterations.
+
+## Performance
+
+This configuration uses bfp4 MLP and bfp8 attention weights for all models except:
+* Qwen-2.5-7B, which uses bfp8 MLP and bfp16 attention weights in all decoder layers
+* Llama-3.1-8B which uses bfp8 MLP in only the 32nd decoder layer and bfp4 MLP elsewhere
+
+| Model             | Device      | Top-1 (%) | Top-5 (%) | Speed (t/s/u) | TTFT (ms) |
+|-------------------|-------------|-----------|-----------|---------------|-----------|
+| Llama-3.2-1B      | N150        | 79        | 97        | 87.8          | 26        |
+| Llama-3.2-1B      | N300        | 79        | 97        | 105.9         | 22        |
+| Llama-3.2-1B      | T3K         | 80        | 97        | 119.8         | 32        |
+| Llama-3.2-1B      | TG          | 77        | 96        | 51.0          |           |
+| Llama-3.2-3B      | N150        | 89        | 98        | 54.0          | 55        |
+| Llama-3.2-3B      | N300        | 89        | 98        | 68.0          | 39        |
+| Llama-3.2-3B      | T3K         | 91        | 99        | 68.5          | 52        |
+| Llama-3.2-3B      | TG          | 87        | 97        | 33.5          |           |
+| Llama-3.1-8B      | N150        | 90        | 97        | 28.3          | 104       |
+| Llama-3.1-8B      | N300        | 90        | 97        | 44.2          | 67        |
+| Llama-3.1-8B      | P100        | 90        | 98        | 29.5          | 84        |
+| Llama-3.1-8B      | P150        | 90        | 98        | 33.6          | 76        |
+| Llama-3.1-8B      | T3K         | 90        | 98        | 64.3          | 53        |
+| Llama-3.1-8B      | T3K  (DP=4) |           |           | 39.6          | 58        |
+| Llama-3.1-8B      | T3K  (DP=8) |           |           | 24.9          | 86        |
+| Llama-3.1-8B      | TG          | 88        | 97        | 29.5          |           |
+| Llama-3.2-11B     | N150        | 90        | 98        | 55.5          | 58        |
+| Llama-3.2-11B     | N300        | 90        | 98        | 44.1          | 67        |
+| Llama-3.2-11B     | T3K         | 90        | 98        | 62.7          | 47        |
+| Llama-3.2-11B     | TG          | 87        | 97        | 29.5          |           |
+| Llama-3.1-70B     | T3K         | 96        | 100       | 16.6          | 164       |
+| Llama-3.3-70B     | T3K         | 96        | 100       | 16.6          | 164       |
+| Llama-3.1-70B     | TG          | 95        | 100       | 12.7          |           |
+| Llama-3.3-70B     | TG          | 95        | 100       | 12.7          |           |
+| Llama-3.1-70B     | TG   (DP=4) |           |           | 14.8          | 189       |
+| Llama-3.2-90B     | T3K         | 96        | 100       | 6             | 5535      |
+| Qwen2.5-7B        | N300        | 84        | 96        | 24.6          | 92        |
+| Qwen2.5-72B       | T3K         | 99        | 100       | 15.2          | 225       |
+| Qwen2.5-32B       | T3K         | 98        | 99        | 22.4          | 190       |
+| Qwen2.5-72B       | P150x8      | 99        | 100       |               |           |
+| Qwen2.5-32B       | P150x8      | 97        | 99        |               |           |
+| Qwen2.5-Coder-32B | T3K         | 96        | 99        | 22.4          | 190       |
+| Qwen3-32B         | T3K         | 89        | 97        | 22.9          | 123       |
+| QwQ-32B           | T3K         | 96        | 100       | 20.7          | 105       |
+| Phi3.5-mini       | N150        |           |           | 43.2          | 98        |
+| Phi3.5-mini       | N300        |           |           | 57.8          | 62        |
+| Phi3.5-mini       | T3K         |           |           | 48.8          | 51        |
+| Mistral-7B        | N150        | 95        | 99        | 29.75         | 100.24    |
+| Mistral-7B        | N300        | 95        | 100       | 47.01         | 65.95     |
+| Mistral-7B        | T3K         | 95        | 100       | 67.82         | 53.93     |
+| Mistral-Small-3.1-24B | T3K      | 95        | 99        |               |           |
+| Phi-3-mini-128k-instruct | N150        | 89        | 99        | 45.0          | 73.32     |
+| Phi-3-mini-128k-instruct | N300        | 89        | 99        | 60.87         | 114.94    |
+| Phi-4 | N300 | 97 | 100 | 37.34 | 123.33 |
+| Mixtral-8x7B-v0.1 | T3K         | 98        | 100       | 67.82         | 53.93     |
+| Ministral-8B      | N300        | 93        | 98        | 22.15         | 79.3      |
+
+## Accuracy
+
+This configuration uses bfp8 MLP and BF16 attention weights (70B+ models use bfp8 attention and bfp4 MLP).
+Llama 3 models test as insensitive to attention precision and so we use bfp8 attention and kv-cache for them even in accuracy mode.
+
+| Model             | Device      | Top-1 (%) | Top-5 (%) | Speed (t/s/u) | TTFT (ms) |
+|-------------------|-------------|-----------|-----------|---------------|-----------|
+| Llama-3.2-1B      | N150        | 87        | 99        | 84.7          | 29        |
+| Llama-3.2-1B      | N300        | 87        | 98        | 102.8         | 21        |
+| Llama-3.2-1B      | T3K         | 88        | 99        | 120.5         | 28        |
+| Llama-3.2-1B      | TG          | 85        | 98        | 48.4          |           |
+| Llama-3.2-3B      | N150        | 96        | 100       | 47.6          | 63        |
+| Llama-3.2-3B      | N300        | 96        | 100       | 63.5          | 41        |
+| Llama-3.2-3B      | T3K         | 96        | 100       | 67.9          | 69        |
+| Llama-3.2-3B      | TG          | 92        | 99        | 33.6          |           |
+| Llama-3.1-8B      | N150        | 96        | 100       | 25.2          | 138       |
+| Llama-3.1-8B      | N300        | 96        | 100       | 38.8          | 79        |
+| Llama-3.1-8B      | T3K         | 97        | 100       | 60.8          | 81        |
+| Llama-3.1-8B      | TG          | 95        | 100       | 29.5          |           |
+| Llama-3.2-11B     | N150        | 95        | 100       | 56.7          | 62        |
+| Llama-3.2-11B     | N300        | 95        | 100       | 38.3          | 78        |
+| Llama-3.2-11B     | T3K         | 96        | 100       | 61.4          | 53        |
+| Llama-3.2-11B     | TG          | 94        | 100       | 29.5          |           |
+| Llama-3.1-70B     | T3K         | 96        | 100       | 16.5          | 168       |
+| Llama-3.1-70B     | TG          | 95        | 100       | 12.7          |           |
+| Llama-3.2-90B     | T3K         | 96        | 100       | 6             | 5600      |
+| Qwen2.5-7B        | N300        | 84        | 96        | 24.6          | 92        |
+| Qwen2.5-72B       | T3K         | 99        | 100       | 15.1          | 216       |
+| Qwen2.5-32B       | T3K         | 95        | 99        | 19.7          | 183       |
+| Qwen2.5-72B       | P150x8      | 99        | 100       |               |           |
+| Qwen2.5-32B       | P150x8      | 97        | 99        |               |           |
+| Qwen2.5-Coder-32B | T3K         | 95        | 99        | 19.7          | 183       |
+| Qwen3-32B         | T3K         | 95        | 100       | 19.6          | 119       |
+| QwQ-32B           | T3K         | 99        | 100       | 18.3          | 120       |
+| Phi3.5-mini       | N150        |           |           | 38.8          | 92        |
+| Phi3.5-mini       | N300        |           |           | 53.9          | 63        |
+| Phi3.5-mini       | T3K         |           |           | 48.6          | 53        |
+| Mistral-7B        | N150        | 96        | 100       | 29.75         | 100.24    |
+| Mistral-7B        | N300        | 97        | 100       | 47.01         | 65.95     |
+| Mistral-7B        | T3K         | 98        | 100       | 67.82         | 53.93     |
+| Mistral-Small-3.1-24B | T3K      | 95        | 99        |               |           |
+| Phi-3-mini-128k-instruct | N150        | 94        | 99        | 40.41         | 82.58     |
+| Phi-3-mini-128k-instruct | N300        | 94        | 99        | 57.0          | 115.36    |
+| Phi-4 | N300 | 99 | 100 | 20.48 | 146.32 |
+| Mixtral-8x7B-v0.1 | T3K         | 95        | 100       | 67.82         | 53.93     |
+| Ministral-8B      | N300        | 97        | 100       | 19.95         | 93.22     |
+
+##  Long-context (64K Tokens)
+
+This configuration uses bfp4 MLP FF1+FF3 for all models. **Batch_size=1 and prefill_length is 64k tokens.**
+
+| Model          | Device | Speed (t/s/u) | TTFT (ms) |
+|----------------|--------|---------------|-----------|
+| Llama-3.2-1B   | N150   | 53.0          | 20066     |
+| Llama-3.2-1B   | N300   | 65.2          | 10949     |
+| Llama-3.2-1B   | T3K    | 73.7          | 5271      |
+| Llama-3.2-1B   | TG     |               |           |
+| Llama-3.2-3B   | N150   | 25.3          | 46743     |
+| Llama-3.2-3B   | N300   | 34.8          | 22921     |
+| Llama-3.2-3B   | T3K    | 41.0          | 10677     |
+| Llama-3.2-3B   | TG     |               |           |
+| Llama-3.1-8B   | N150   | 16.9          | 64385     |
+| Llama-3.1-8B   | N300   | 26.1          | 36229     |
+| Llama-3.1-8B   | T3K    | 38.1          | 16165     |
+| Llama-3.1-8B   | TG     |               |           |
+| Llama-3.2-11B  | N300   | 26.1          | 36247     |
+| Llama-3.2-11B  | T3K    | 38.4          | 16167     |
+| Llama-3.2-11B  | TG     |               |           |
+| Llama-3.1-70B  | T3K    | 11.9          | 74363     |
+| Llama-3.1-70B  | TG     |               |           |
+| Qwen2.5-7B     | N300   |               |           |
+| Qwen2.5-72B    | T3K    |               |           |
+
+##  Long-context (32K Tokens)
+
+This configuration uses bfp4 MLP FF1+FF3 for all models. **Batch_size=1 and prefill_length is 32k tokens.**
+
+| Model                     | Device | Speed (t/s/u) | TTFT (ms) |
+|---------------------------|--------|---------------|-----------|
+| Phi-3-mini-128k-instruct  | N300   | 26.1          | 10072     |
+
+## Short-Context, Batch-32
+
+This configuration uses bfp4 MLP FF1+FF3 for all models. **Batch_size=32 and prefill_length is 128 tokens.**
+
+| Model          | Device | Speed (t/s/u) | avg TTFT (ms) |
+|----------------|--------|---------------|---------------|
+| Llama-3.2-1B   | N150   | 54.7          | 38            |
+| Llama-3.2-1B   | N300   | 64.2          | 34            |
+| Llama-3.2-1B   | T3K    | 69.9          | 42            |
+| Llama-3.2-1B   | TG     |               |               |
+| Llama-3.2-3B   | N150   | 36.5          | 69            |
+| Llama-3.2-3B   | N300   | 45.8          | 51            |
+| Llama-3.2-3B   | T3K    | 47.8          | 63            |
+| Llama-3.2-3B   | TG     |               |               |
+| Llama-3.1-8B   | N150   | 22.3          | 119           |
+| Llama-3.1-8B   | N300   | 33.5          | 80            |
+| Llama-3.1-8B   | T3K    | 45.6          | 64            |
+| Llama-3.1-8B   | TG     |               |               |
+| Llama-3.2-11B  | N300   | 33.4          | 79            |
+| Llama-3.2-11B  | T3K    | 45.1          | 64            |
+| Llama-3.2-11B  | TG     |               |               |
+| Llama-3.1-70B  | T3K    | 14.8          | 192           |
+| Llama-3.1-70B  | TG     |               |               |
+| Qwen2.5-7B     | N300   |               |               |
+| Qwen2.5-72B    | T3K    |               |               |
+| Phi-3-mini-128k-instruct  | 150    | 25.66         | 68.58         |
+| Phi-3-mini-128k-instruct  | N300   | 39.4          | 85.99         |
+
+# Llama 3 model precision and math fidelity
+
+## precision_cfg = {ff1_3: bfp4, ff2: bfp4, wqkv: bfp8, wo: bfp8, kv_cache: bfp8, activation: mixed}, fidelity_cfg = {li_ff1_3: lofi, li_ff2: lofi, li_qkv_decode: hifi2, li_o_decode: hifi2, sdpa_decode: hifi2na, li_qkv_prefill: hifi2, li_o_prefill: hifi2fp16, sdpa_prefill: hifi4}
+
+| Model          | Device | Top-1 (%) | Top-5 (%) | Speed (t/s/u) | TTFT (ms) |
+|----------------|--------|-----------|-----------|---------------|-----------|
+| Llama-3.2-1B   | N300   | 85        | 98        | 100.3         | 69        |
+
+## precision_cfg = {ff1_3: bfp4, ff2: bfp8, wqkv: bfp8, wo: bfp8, kv_cache: bfp8, activation: mixed}, fidelity_cfg = {li_ff1_3: lofi, li_ff2: hifi2, li_qkv_decode: hifi2, li_o_decode: hifi2, sdpa_decode: hifi2na, li_qkv_prefill: hifi2, li_o_prefill: hifi2fp16, sdpa_prefill: hifi4}
+
+| Model          | Device | Top-1 (%) | Top-5 (%) | Speed (t/s/u) | TTFT (ms) |
+|----------------|--------|-----------|-----------|---------------|-----------|
+| Llama-3.2-1B   | N300   | 88        | 98        | 100.3         | 55        |
+
+## precision_cfg = {ff1_3: bfp4, ff2: bf16, wqkv: bfp8, wo: bfp8, kv_cache: bfp8, activation: mixed}, fidelity_cfg = {li_ff1_3: lofi, li_ff2: hifi4, li_qkv_decode: hifi2, li_o_decode: hifi2, sdpa_decode: hifi2na, li_qkv_prefill: hifi2, li_o_prefill: hifi2fp16, sdpa_prefill: hifi4}
+
+| Model          | Device | Top-1 (%) | Top-5 (%) | Speed (t/s/u) | TTFT (ms) |
+|----------------|--------|-----------|-----------|---------------|-----------|
+| Llama-3.2-1B   | N300   | 87        | 98        | 96.8          | 51        |
+
+## precision_cfg = {ff1_3: bfp8, ff2: bfp4, wqkv: bfp8, wo: bfp8, kv_cache: bfp8, activation: mixed}, fidelity_cfg = {li_ff1_3: hifi2, li_ff2: lofi, li_qkv_decode: hifi2, li_o_decode: hifi2, sdpa_decode: hifi2na, li_qkv_prefill: hifi2, li_o_prefill: hifi2fp16, sdpa_prefill: hifi4}
+
+| Model          | Device | Top-1 (%) | Top-5 (%) | Speed (t/s/u) | TTFT (ms) |
+|----------------|--------|-----------|-----------|---------------|-----------|
+| Llama-3.2-1B   | N300   | 87        | 98        | 98.5          | 50        |
+
+## precision_cfg = {ff1_3: bfp8, ff2: bfp8, wqkv: bfp8, wo: bfp8, kv_cache: bfp8, activation: mixed}, fidelity_cfg = {li_ff1_3: hifi2, li_ff2: hifi2, li_qkv_decode: hifi2, li_o_decode: hifi2, sdpa_decode: hifi2na, li_qkv_prefill: hifi2, li_o_prefill: hifi2fp16, sdpa_prefill: hifi4}
+
+| Model          | Device | Top-1 (%) | Top-5 (%) | Speed (t/s/u) | TTFT (ms) |
+|----------------|--------|-----------|-----------|---------------|-----------|
+| Llama-3.2-1B   | N300   | 91        | 98        | 99.0          | 60        |
+
+## precision_cfg = {ff1_3: bfp8, ff2: bf16, wqkv: bfp8, wo: bfp8, kv_cache: bfp8, activation: mixed}, fidelity_cfg = {li_ff1_3: hifi2, li_ff2: hifi4, li_qkv_decode: hifi2, li_o_decode: hifi2, sdpa_decode: hifi2na, li_qkv_prefill: hifi2, li_o_prefill: hifi2fp16, sdpa_prefill: hifi4}
+
+| Model          | Device | Top-1 (%) | Top-5 (%) | Speed (t/s/u) | TTFT (ms) |
+|----------------|--------|-----------|-----------|---------------|-----------|
+| Llama-3.2-1B   | N300   | 89        | 99        | 95.2          | 49        |
+
+## precision_cfg = {ff1_3: bf16, ff2: bfp4, wqkv: bfp8, wo: bfp8, kv_cache: bfp8, activation: mixed}, fidelity_cfg = {li_ff1_3: hifi4, li_ff2: lofi, li_qkv_decode: hifi2, li_o_decode: hifi2, sdpa_decode: hifi2na, li_qkv_prefill: hifi2, li_o_prefill: hifi2fp16, sdpa_prefill: hifi4}
+
+| Model          | Device | Top-1 (%) | Top-5 (%) | Speed (t/s/u) | TTFT (ms) |
+|----------------|--------|-----------|-----------|---------------|-----------|
+| Llama-3.2-1B   | N300   | 89        | 98        | 95.2          | 53        |
+
+## precision_cfg = {ff1_3: bf16, ff2: bfp8, wqkv: bfp8, wo: bfp8, kv_cache: bfp8, activation: mixed}, fidelity_cfg = {li_ff1_3: hifi4, li_ff2: hifi2, li_qkv_decode: hifi2, li_o_decode: hifi2, sdpa_decode: hifi2na, li_qkv_prefill: hifi2, li_o_prefill: hifi2fp16, sdpa_prefill: hifi4}
+
+| Model          | Device | Top-1 (%) | Top-5 (%) | Speed (t/s/u) | TTFT (ms) |
+|----------------|--------|-----------|-----------|---------------|-----------|
+| Llama-3.2-1B   | N300   | 91        | 98        | 94.4          | 57        |
+
+## precision_cfg = {ff1_3: bf16, ff2: bf16, wqkv: bfp8, wo: bfp8, kv_cache: bfp8, activation: mixed}, fidelity_cfg = {li_ff1_3: hifi4, li_ff2: hifi4, li_qkv_decode: hifi2, li_o_decode: hifi2, sdpa_decode: hifi2na, li_qkv_prefill: hifi2, li_o_prefill: hifi2fp16, sdpa_prefill: hifi4}
+
+| Model          | Device | Top-1 (%) | Top-5 (%) | Speed (t/s/u) | TTFT (ms) |
+|----------------|--------|-----------|-----------|---------------|-----------|
+| Llama-3.2-1B   | N300   | 90        | 98        | 91.2          | 60        |
+
+## precision_cfg = {ff1_3: bfp8, ff2: bfp8, wqkv: bfp8, wo: bfp4, kv_cache: bfp8, activation: bf16}, fidelity_cfg = {li_ff1_3: hifi2, li_ff2: hifi2, li_qkv_decode: hifi2, li_o_decode: lofi, sdpa_decode: hifi2na, li_qkv_prefill: hifi2, li_o_prefill: lofi, sdpa_prefill: hifi4}
+
+| Model          | Device | Top-1 (%) | Top-5 (%) | Speed (t/s/u) | TTFT (ms) |
+|----------------|--------|-----------|-----------|---------------|-----------|
+| Llama-3.2-1B   | N300   | 88        | 98        | 98.2          | 45        |
+
+## precision_cfg = {ff1_3: bfp8, ff2: bfp8, wqkv: bfp8, wo: bfp4, kv_cache: bfp8, activation: bfp8}, fidelity_cfg = {li_ff1_3: hifi2, li_ff2: hifi2, li_qkv_decode: hifi2, li_o_decode: lofi, sdpa_decode: hifi2na, li_qkv_prefill: hifi2, li_o_prefill: lofi, sdpa_prefill: hifi4}
+
+| Model          | Device | Top-1 (%) | Top-5 (%) | Speed (t/s/u) | TTFT (ms) |
+|----------------|--------|-----------|-----------|---------------|-----------|
+| Llama-3.2-1B   | N300   | 90        | 98        | 101.0         | 57        |
+
+## precision_cfg = {ff1_3: bfp8, ff2: bfp8, wqkv: bfp8, wo: bfp4, kv_cache: bfp8, activation: mixed}, fidelity_cfg = {li_ff1_3: hifi2, li_ff2: hifi2, li_qkv_decode: hifi2, li_o_decode: lofi, sdpa_decode: hifi2na, li_qkv_prefill: hifi2, li_o_prefill: lofi, sdpa_prefill: hifi4}
+
+| Model          | Device | Top-1 (%) | Top-5 (%) | Speed (t/s/u) | TTFT (ms) |
+|----------------|--------|-----------|-----------|---------------|-----------|
+| Llama-3.2-1B   | N300   | 90        | 98        | 99.4          | 73        |

code/models/tt_transformers/README.md

@@ -0,0 +1,388 @@
+# TT-Transformers
+
+This code can run large language models such as the Llama3 family, Qwen2.5, Mistral, DeepSeek-R1-Distill variants and similar. Tensor-parallelism automatically distributes workloads across all available chips.
+
+The current version is verified to work with the following models:
+| Model                                                                                            | Hardware                    | <org/model>                                      |
+|--------------------------------------------------------------------------------------------------|-----------------------------|-------------------------------------------------|
+| [DeepSeek R1 Distill Llama 70B](https://huggingface.co/deepseek-ai/DeepSeek-R1-Distill-Llama-70B)| LoudBox / QuietBox / Galaxy | ```deepseek-ai/DeepSeek-R1-Distill-Llama-70B``` |
+| [Llama 3.1 8B](https://huggingface.co/meta-llama/Llama-3.1-8B)                                   | n150 / p100 / p150          | ```meta-llama/Llama-3.1-8B```                   |
+| [Llama 3.1 70B](https://huggingface.co/meta-llama/Llama-3.1-70B)                                 | LoudBox / QuietBox / Galaxy | ```meta-llama/Llama-3.1-70B```                  |
+| [Llama 3.2 1B](https://huggingface.co/meta-llama/Llama-3.2-1B)                                   | n150                        | ```meta-llama/Llama-3.2-1B```                   |
+| [Llama 3.2 3B](https://huggingface.co/meta-llama/Llama-3.2-3B)                                   | n150                        | ```meta-llama/Llama-3.2-3B```                   |
+| [Llama 3.2 11B Vision](https://huggingface.co/meta-llama/Llama-3.2-11B-Vision)                   | n300                        | ```meta-llama/Llama-3.2-11B-Vision```           |
+| [Llama 3.2 90B Vision](https://huggingface.co/meta-llama/Llama-3.2-90B-Vision)                   | LoudBox / QuietBox          | ```meta-llama/Llama-3.2-90B-Vision```           |
+| [Mistral 7B Instruct v0.3](https://huggingface.co/mistralai/Mistral-7B-Instruct-v0.3)            | n150                        | ```mistralai/Mistral-7B-Instruct-v0.3```        |
+| [Mistral Small 3.1 24B Instruct](https://huggingface.co/mistralai/Mistral-Small-3.1-24B-Instruct-2503) | T3K                   | ```mistralai/Mistral-Small-3.1-24B-Instruct-2503``` |
+| [Mixtral 8x7B Instruct v0.1](https://huggingface.co/mistralai/Mixtral-8x7B-Instruct-v0.1)        | LoudBox / QuietBox          | ```mistralai/Mixtral-8x7B-Instruct-v0.1```        |
+| [Qwen 2.5 7B](https://huggingface.co/Qwen/Qwen2.5-7B)                                            | n300                        | ```Qwen/Qwen2.5-7B```                           |
+| [Qwen 2.5 Coder 32B](https://huggingface.co/Qwen/Qwen2.5-Coder-32B)                              | LoudBox / QuietBox          | ```Qwen/Qwen2.5-Coder-32B```                    |
+| [Qwen 2.5 72B](https://huggingface.co/Qwen/Qwen2.5-72B)                                          | LoudBox / QuietBox          | ```Qwen/Qwen2.5-72B```                          |
+| [Qwen 3 32B](https://huggingface.co/Qwen/Qwen3-32B)                                              | LoudBox / QuietBox          | ```Qwen/Qwen3-32B```                            |
+| [Phi-3-mini-128k-instruct](https://huggingface.co/microsoft/Phi-3-mini-128k-instruct)            | n150 / n300                 | ```microsoft/Phi-3-mini-128k-instruct```        |
+
+<details>
+<summary>  Explore an extended list of compatible models (experimental) </summary>
+
+| Model                                                                                                                     | Hardware | <org/model>.                                                 |
+|---------------------------------------------------------------------------------------------------------------------------|----------|-------------------------------------------------------------|
+| [Bellatrix-Tiny-0.5B](https://huggingface.co/prithivMLmods/Bellatrix-Tiny-0.5B)                                           | N150     | `prithivMLmods/Bellatrix-Tiny-0.5B`                         |
+| [Bellatrix-Tiny-1B](https://huggingface.co/prithivMLmods/Bellatrix-Tiny-1B)                                               | N150     | `prithivMLmods/Bellatrix-Tiny-1B`                           |
+| [Bellatrix-Tiny-1B-R1](https://huggingface.co/prithivMLmods/Bellatrix-Tiny-1B-R1)                                         | N150     | `prithivMLmods/Bellatrix-Tiny-1B-R1`                        |
+| [deepseek-coder-6.7b-instruct](https://huggingface.co/deepseek-ai/deepseek-coder-6.7b-instruct)                           | N300     | `deepseek-ai/deepseek-coder-6.7b-instruct`                  |
+| [deepseek-llm-7b-base](https://huggingface.co/deepseek-ai/deepseek-llm-7b-base)                                           | N300     | `deepseek-ai/deepseek-llm-7b-base`                          |
+| [deepseek-llm-7b-chat](https://huggingface.co/deepseek-ai/deepseek-llm-7b-chat)                                           | N300     | `deepseek-ai/deepseek-llm-7b-chat`                          |
+| [DeepSeek-R1-Distill-Llama-8B](https://huggingface.co/deepseek-ai/DeepSeek-R1-Distill-Llama-8B)                           | N300     | `deepseek-ai/DeepSeek-R1-Distill-Llama-8B`                  |
+| [DeepSeek-R1-Distill-Llama-8B](https://huggingface.co/unsloth/DeepSeek-R1-Distill-Llama-8B)                               | N300     | `unsloth/DeepSeek-R1-Distill-Llama-8B`                      |
+| [DeepSeek-R1-Distill-Llama-8B-abliterated](https://huggingface.co/huihui-ai/DeepSeek-R1-Distill-Llama-8B-abliterated)     | N300     | `huihui-ai/DeepSeek-R1-Distill-Llama-8B-abliterated`        |
+| [DeepSeek-R1-Distill-Qwen-1.5B](https://huggingface.co/deepseek-ai/DeepSeek-R1-Distill-Qwen-1.5B)                         | N150     | `deepseek-ai/DeepSeek-R1-Distill-Qwen-1.5B`                 |
+| [DeepSeek-R1-Distill-Qwen-32B](https://huggingface.co/deepseek-ai/DeepSeek-R1-Distill-Qwen-32B)                           | T3K      | `deepseek-ai/DeepSeek-R1-Distill-Qwen-32B`                  |
+| [DeepSeek-R1-Distill-Qwen-32B-Japanese](https://huggingface.co/cyberagent/DeepSeek-R1-Distill-Qwen-32B-Japanese)          | T3K      | `cyberagent/DeepSeek-R1-Distill-Qwen-32B-Japanese`          |
+| [DeepSeek-R1-Distill-Qwen-32B-abliterated](https://huggingface.co/huihui-ai/DeepSeek-R1-Distill-Qwen-32B-abliterated)     | T3K      | `huihui-ai/DeepSeek-R1-Distill-Qwen-32B-abliterated`        |
+| [DeepSeek-R1-Distill-Qwen-7B](https://huggingface.co/deepseek-ai/DeepSeek-R1-Distill-Qwen-7B)                             | N300     | `deepseek-ai/DeepSeek-R1-Distill-Qwen-7B`                   |
+| [DeepSeek-R1-Distill-Qwen-7B-abliterated-v2](https://huggingface.co/huihui-ai/DeepSeek-R1-Distill-Qwen-7B-abliterated-v2) | N300     | `huihui-ai/DeepSeek-R1-Distill-Qwen-7B-abliterated-v2`      |
+| [DeepSeek-R1-Distill-Qwen-7B-Japanese](https://huggingface.co/lightblue/DeepSeek-R1-Distill-Qwen-7B-Japanese)             | N300     | `lightblue/DeepSeek-R1-Distill-Qwen-7B-Japanese`            |
+| [Dobby-Mini-Leashed-Llama-3.1-8B](https://huggingface.co/SentientAGI/Dobby-Mini-Leashed-Llama-3.1-8B)                     | N300     | `SentientAGI/Dobby-Mini-Leashed-Llama-3.1-8B`               |
+| [Dobby-Mini-Unhinged-Llama-3.1-8B](https://huggingface.co/SentientAGI/Dobby-Mini-Unhinged-Llama-3.1-8B)                   | N300     | `SentientAGI/Dobby-Mini-Unhinged-Llama-3.1-8B`              |
+| [Dolphin3.0-Mistral-24B](https://huggingface.co/cognitivecomputations/Dolphin3.0-Mistral-24B)                             | T3K      | `cognitivecomputations/Dolphin3.0-Mistral-24B`              |
+| [Dolphin3.0-R1-Mistral-24B](https://huggingface.co/cognitivecomputations/Dolphin3.0-R1-Mistral-24B)                       | T3K      | `cognitivecomputations/Dolphin3.0-R1-Mistral-24B`           |
+| [EuroLLM-9B-Instruct](https://huggingface.co/utter-project/EuroLLM-9B-Instruct)                                           | N300     | `utter-project/EuroLLM-9B-Instruct`                         |
+| [GPT4chan-24B](https://huggingface.co/v2ray/GPT4chan-24B)                                                                 | T3K      | `v2ray/GPT4chan-24B`                                        |
+| [Llama-2-7b-chat-hf](https://huggingface.co/meta-llama/Llama-2-7b-hf)                                                     | N300     | `meta-llama/Llama-2-7b-hf`                                  |
+| [Llama-2-7b-hf](https://huggingface.co/meta-llama/Llama-2-7b-hf)                                                          | N300     | `meta-llama/Llama-2-7b-hf`                                  |
+| [Llama-3.1-8B-Instruct](https://huggingface.co/meta-llama/Llama-3.1-8B-Instruct)                                          | N300     | `meta-llama/Llama-3.1-8B-Instruct`                          |
+| [Llama-3.2-1B-Instruct](https://huggingface.co/meta-llama/Llama-3.2-1B-Instruct)                                          | N150     | `meta-llama/Llama-3.2-1B-Instruct`                          |
+| [Llama-3.2-3B-Instruct](https://huggingface.co/meta-llama/Llama-3.2-3B-Instruct)                                          | N150     | `meta-llama/Llama-3.2-3B-Instruct`                          |
+| [Llama-3.2-6B-AlgoCode](https://huggingface.co/prithivMLmods/Llama-3.2-6B-AlgoCode)                                       | N150     | `prithivMLmods/Llama-3.2-6B-AlgoCode`                       |
+| [Llama-Krikri-8B-Base](https://huggingface.co/ilsp/Llama-Krikri-8B-Base)                                                  | N150     | `ilsp/Llama-Krikri-8B-Base`                                 |
+| [Llama-Krikri-8B-Instruct](https://huggingface.co/ilsp/Llama-Krikri-8B-Instruct)                                          | N300     | `ilsp/Llama-Krikri-8B-Instruct`                             |
+| [LwQ-10B-Instruct](https://huggingface.co/prithivMLmods/LwQ-10B-Instruct)                                                 | N300     | `prithivMLmods/LwQ-10B-Instruct`                            |
+| [Magnum-v4-Cydonia-vXXX-22B](https://huggingface.co/Kaoeiri/Magnum-v4-Cydonia-vXXX-22B)                                   | T3K      | `Kaoeiri/Magnum-v4-Cydonia-vXXX-22B`                        |
+| [Megatron-Opus-7B-Exp](https://huggingface.co/prithivMLmods/Megatron-Opus-7B-Exp)                                         | N300     | `prithivMLmods/Megatron-Opus-7B-Exp`                        |
+| [Meta-Llama-3-8B](https://huggingface.co/meta-llama/Meta-Llama-3-8B)                                                      | N300     | `meta-llama/Meta-Llama-3-8B`                                |
+| [Meta-Llama-3-8B-Instruct](https://huggingface.co/meta-llama/Meta-Llama-3-8B-Instruct)                                    | N300     | `meta-llama/Meta-Llama-3-8B-Instruct`                       |
+| [Mistral-7B-Instruct-v0.1](https://huggingface.co/mistralai/Mistral-7B-Instruct-v0.1)                                     | N150     | `mistralai/Mistral-7B-Instruct-v0.1`                        |
+| [Mistral-7B-Instruct-v0.2](https://huggingface.co/mistralai/Mistral-7B-Instruct-v0.2)                                     | N150     | `mistralai/Mistral-7B-Instruct-v0.2`                        |
+| [Mistral-7B-v0.1](https://huggingface.co/mistralai/Mistral-7B-v0.1)                                                       | N150     | `mistralai/Mistral-7B-v0.1`                                 |
+| [Mistral-Small-24B-Base-2501](https://huggingface.co/mistralai/Mistral-Small-24B-Base-2501)                               | T3K      | `mistralai/Mistral-Small-24B-Base-2501`                     |
+| [Mistral-Small-24B-Instruct-2501](https://huggingface.co/mistralai/Mistral-Small-24B-Instruct-2501)                       | T3K      | `mistralai/Mistral-Small-24B-Instruct-2501`                 |
+| [Mistral-Small-3.1-24B-Instruct-2503](https://huggingface.co/mistralai/Mistral-Small-3.1-24B-Instruct-2503)             | T3K      | `mistralai/Mistral-Small-3.1-24B-Instruct-2503`             |
+| [NuminaMath-7B-TIR](https://huggingface.co/AI-MO/NuminaMath-7B-TIR)                                                       | N300     | `AI-MO/NuminaMath-7B-TIR`                                   |
+| [OpenThinker-7B](https://huggingface.co/open-thoughts/OpenThinker-7B)                                                     | N300     | `open-thoughts/OpenThinker-7B`                              |
+| [Primal-Mini-3B-Exp](https://huggingface.co/prithivMLmods/Primal-Mini-3B-Exp)                                             | N150     | `prithivMLmods/Primal-Mini-3B-Exp`                          |
+| [Qwen2.5-0.5B](https://huggingface.co/Qwen/Qwen2.5-0.5B)                                                                  | N150     | `Qwen/Qwen2.5-0.5B`                                         |
+| [Qwen2.5-0.5B-Instruct](https://huggingface.co/Qwen/Qwen2.5-0.5B-Instruct)                                                | N150     | `Qwen/Qwen2.5-0.5B-Instruct`                                |
+| [Qwen2.5-1.5B](https://huggingface.co/Qwen/Qwen2.5-1.5B)                                                                  | N150     | `Qwen/Qwen2.5-1.5B`                                         |
+| [Qwen2.5-1.5B-Instruct](https://huggingface.co/Qwen/Qwen2.5-1.5B-Instruct)                                                | N150     | `Qwen/Qwen2.5-1.5B-Instruct`                                |
+| [Qwen2.5-3B-Instruct](https://huggingface.co/Qwen/Qwen2.5-3B-Instruct)                                                    | N150     | `Qwen/Qwen2.5-3B-Instruct`                                  |
+| [Qwen2.5-32B](https://huggingface.co/Qwen/Qwen2.5-32B)                                                                    | T3K      | `Qwen/Qwen2.5-32B`                                          |
+| [Qwen2.5-Coder-7B-Instruct](https://huggingface.co/Qwen/Qwen2.5-Coder-7B-Instruct)                                        | N300     | `Qwen/Qwen2.5-Coder-7B-Instruct`                            |
+| [Qwen2.5-Math-1.5B](https://huggingface.co/Qwen/Qwen2.5-Math-1.5B)                                                        | N150     | `Qwen/Qwen2.5-Math-1.5B`                                    |
+| [Qwen2.5-Math-7B](https://huggingface.co/Qwen/Qwen2.5-Math-7B)                                                            | N300     | `Qwen/Qwen2.5-Math-7B`                                      |
+| [Reasoning-Distilled-ta-7B](https://huggingface.co/prithivMLmods/Reasoning-Distilled-ta-7B)                               | N300     | `prithivMLmods/Reasoning-Distilled-ta-7B`                   |
+| [s1-32B](https://huggingface.co/simplescaling/s1-32B)                                                                     | T3K      | `simplescaling/s1-32B`                                      |
+| [Selene-1-Mini-Llama-3.1-8B](https://huggingface.co/AtlaAI/Selene-1-Mini-Llama-3.1-8B)                                    | N300     | `AtlaAI/Selene-1-Mini-Llama-3.1-8B`                         |
+| [Sky-T1-32B-Preview](https://huggingface.co/NovaSky-AI/Sky-T1-32B-Preview)                                                | T3K      | `NovaSky-AI/Sky-T1-32B-Preview`                             |
+| [SmallThinker-3B-Preview](https://huggingface.co/PowerInfer/SmallThinker-3B-Preview)                                      | N150     | `PowerInfer/SmallThinker-3B-Preview`                        |
+| [SmolLM2-1.7B-Instruct](https://huggingface.co/HuggingFaceTB/SmolLM2-1.7B-Instruct)                                       | N150     | `HuggingFaceTB/SmolLM2-1.7B-Instruct`                       |
+| [Sqweeks-7B-Instruct](https://huggingface.co/prithivMLmods/Sqweeks-7B-Instruct)                                           | N300     | `prithivMLmods/Sqweeks-7B-Instruct`                         |
+| [TinyLlama-1.1B-Chat-v1.0](https://huggingface.co/TinyLlama/TinyLlama-1.1B-Chat-v1.0)                                     | N150     | `TinyLlama/TinyLlama-1.1B-Chat-v1.0`                        |
+| [TinySwallow-1.5B](https://huggingface.co/SakanaAI/TinySwallow-1.5B)                                                      | N150     | `SakanaAI/TinySwallow-1.5B`                                 |
+| [TinySwallow-1.5B-Instruct](https://huggingface.co/SakanaAI/TinySwallow-1.5B-Instruct)                                    | N150     | `SakanaAI/TinySwallow-1.5B-Instruct`                        |
+| [Triangulum-v2-10B](https://huggingface.co/prithivMLmods/Triangulum-v2-10B)                                               | N300     | `prithivMLmods/Triangulum-v2-10B`                           |
+| [WebMind-7B-v0.1](https://huggingface.co/prithivMLmods/WebMind-7B-v0.1)                                                   | N300     | `prithivMLmods/WebMind-7B-v0.1`                             |
+| [WizardLM-7B-Uncensored](https://huggingface.co/cognitivecomputations/WizardLM-7B-Uncensored)                             | N300     | `cognitivecomputations/WizardLM-7B-Uncensored`              |
+| [Xwen-7B-Chat](https://huggingface.co/xwen-team/Xwen-7B-Chat)                                                             | N300     | `xwen-team/Xwen-7B-Chat`                                    |
+| [YuE-s1-7B-anneal-en-cot](https://huggingface.co/m-a-p/YuE-s1-7B-anneal-en-cot)                                           | N300     | `m-a-p/YuE-s1-7B-anneal-en-cot`                             |
+| [YuE-s2-1B-general](https://huggingface.co/m-a-p/YuE-s2-1B-general)                                                       | N150     | `m-a-p/YuE-s2-1B-general`                                   |
+
+</details>
+
+## Prerequisites
+
+Install [TT-Metalium and TTNN](../../INSTALLING.md).
+
+
+## How to Run
+
+To configure the weights and run a demo, choose [Automatic Download](#automatic-download) for a quick setup or [Manual Download](#manual-download) for more control over the files.
+
+---
+
+### Automatic Setup
+
+#### Login to HuggingFace using your token
+
+Use `huggingface-cli login` or set the token with the command `export HF_TOKEN=<token>`
+- To obtain a HuggingFace token visit: https://huggingface.co/docs/hub/security-tokens
+
+#### Choose your Model
+
+Set the `HF_MODEL` environment variable to the HuggingFace org/name of the model to be run.
+
+This will automatically download the weights into your HuggingFace cache directory and run the model directly.
+
+- Check the models chart on the top of the page and substitute the `<org/model>` on the following command:
+```
+export HF_MODEL=<org/name>
+```
+
+#### You are all set!
+
+- Skip to section: [Run the Demo](#run-the-demo)
+
+---
+
+### Manual Setup
+
+If you wish, you can manually download the weights [from HuggingFace](#download-weights-from-huggingface) as described by the following section:
+
+#### Download Weights from HuggingFace
+
+Navigate to [HuggingFace](https://huggingface.co) and download the weights of the model you wish to run (check [models chart](#tt-transformers) for compatibility and direct links to HuggingFace models).
+
+- Ensure your model directory has the following structure:
+```
+/path_to/org/model/
+    config.json
+    generation_config.json
+    model-00001-of-00062.safetensors
+    ...
+```
+
+- Set `HF_MODEL` to the directory of the downloaded weights:
+```
+export HF_MODEL=<path_to_downloaded_directory>
+```
+
+#### You are all set!
+
+- Skip to section: [Run the Demo](#run-the-demo)
+
+---
+
+### Run the Demo
+
+- Make sure `HF_MODEL` has been set:
+
+`export HF_MODEL=<hf_model_name or hf_downloaded_directory>`
+
+- Run the Demo
+
+These are some example commands on how to run the demo. For more information about the parameters, check [parameters description.](#parameters-description)
+
+```
+# Batch-1
+pytest models/tt_transformers/demo/simple_text_demo.py -k "performance and batch-1"
+
+# Batch-32
+pytest models/tt_transformers/demo/simple_text_demo.py -k "performance and batch-32"
+
+# Long context with custom parameters
+pytest models/tt_transformers/demo/simple_text_demo.py -k "long-context" --max_seq_len=16384
+
+# Long-context
+pytest models/tt_transformers/demo/simple_text_demo.py -k "performance and long"
+```
+
+The above examples are run in `ModelOptimizations.performance` mode. You can override this by setting the `optimizations` or the `decoder_config_file` argument in the demo. To use instead the accuracy mode you can call the above tests with `-k "accuracy and ..."` instead of performance.
+
+NOTE: trace region sizes are declared in [`models/model_trace_region_sizes.yaml`](../model_trace_region_sizes.yaml) and resolved at device-open time via [`get_supported_trace_region_size`](demo/trace_region_config.py) (which delegates to [`resolve_trace_region_size`](../demos/utils/trace_region_sizes.py)). A `(model, SKU)` pair without a YAML entry is not an error: resolution logs an info message and falls back to `TRACE_REGION_SIZE_DYNAMIC` (`0`, dynamic allocation). Add an explicit entry when a model needs a fixed reserved trace region.
+
+## Details
+
+### Extra compatibility settings for non-Llama models
+
+If you are bringing up a new model that is similar to these but is not listed above, you may also need to set additional environment variables:
+- `MAX_PREFILL_CHUNK_SIZE` - this determines how many thousands of tokens are prefilled in one go. For optimal performance pick 128. Depending on the model dimensions and hardware you're running on, there may not be enough L1 to prefill 128K tokens at once, in which case you can reduce this in powers of 2 down to 4.
+- `PAD_MLP_CORES` - models with a hidden_dim that is not a nice power of 2 may not have a valid layout or may run with lower performance. You can set this to a multiple of 8 between 8 and 64; `16` and `32` commonly work well if this is required.
+
+You should also watch out for:
+- RoPE encoding style. `llama3`, `yarn` and of course `none` are supported. HuggingFace models encode the complex numbers in RoPE as r1, r2, ..., i1, i2, ... whereas Meta models encode the complex numbers as r1, i1, r2, i2, ... - TTT uses a Meta-style implementation of the RoPE op and when loading a HuggingFace it will reshuffle the weights of the pre-rope attention weights to interleave their outputs in this style (see `reverse_permute` in [load_checkpoints.py](tt/load_checkpoints.py)). When _using_ TTT this happens invisibly and should not affect you, but when bringing up a new model or modifying TTT being aware of this will make some otherwise confusing things clearer.
+- Our [accuracy test](demo/simple_text_demo.py -k ci-token-matching) will require you to [generate some reference logits](tests/generate_reference_hf.py) and perhaps update the test to use them.
+- We parallelise attention over the number of heads. If this number is e.g. 14 then you will not be able to run it on more than 2 chips (because 14/2=7, a prime number). We do not support head-padding or similar mitigations at this time but a PR would be cool.
+
+Huggingface models specify their architecture in the `config.json` file. The following architectures are known to work:
+
+- LlamaForCausalLM
+- Qwen2ForCausalLM
+- Qwen3ForCausalLM
+- MistralForCausalLM
+- Mistral3ForConditionalGeneration
+- Phi3ForCausalLM
+
+At the time of writing this covers the majority of popular HuggingFace text-generation models. If you find another architecture that works or extend TT-Transformers to support one we would love to accept a PR!
+
+---
+
+### Environment Variables Description
+
+- `HF_MODEL` is the HuggingFace org/name of the model you want to run or the path to the downloaded Huggingface weights.
+- `TT_CACHE_PATH` is optional. It sets the path for ttnn's weight cache files. See below for more details.
+- `MESH_DEVICE` is optional. It allows you to use fewer devices than are available. See below for more details.
+
+On the first execution of each model, TTNN will create weight cache files for that model, to speed up future runs. These cache files only need to be created once for each model and device. These files are stored in one of three places:
+
+1. `TT_CACHE_PATH` if you have set it.
+2. `HF_MODEL/device_name` if a path to downloaded weights was specified using `HF_MODEL`.
+3. `model_cache/HF_MODEL/device_name` if a HuggingFace model name was specified using `HF_MODEL`.
+
+The device name used is:
+
+- `N150` for N150
+- `N300` for N300
+- `T3K` for LoudBox / QuietBox
+- `TG` for Galaxy
+
+By default tensor parallelism is used to run the model over all available chips. You can instead run on a smaller mesh either for testing or for performance reasons (for very small models the communication overhead of tensor parallelism may be larger than the performance gained). To use a smaller mesh, set `MESH_DEVICE` to one of the supported devices: `N150`, `N300`, `T3K` or `TG`.
+
+Example: `export MESH_DEVICE=N150`, will enable running one a single chip of a multi-chip system.
+
+---
+
+### Parameters Description
+
+The `simple_text_demo.py` script includes the following main modes of operation and is parametrized to support other configurations.
+
+- `batch-1`: Runs a small prompt (128 tokens) for a single user
+- `batch-32`: Runs a small prompt (128 tokens) for a a batch of 32 users
+- `long-context`: Runs a large prompt (64k tokens) for a single user
+- `reasoning-1`: Runs a reasoning prompt for a single user (generates up to 15k tokens)
+
+If you want to provide your own demo configuration, please take a look at the pytest parametrize calls in `models/tt_transformers/demo/simple_text_demo.py`. For convenience we list all the supported params below:
+
+- `input_prompts (string)`: input json file with prompts to process. See `models/tt_transformers/demo/*.json` for a list of input files
+- `instruct (bool)`: Whether to use Llama instruct weights or general weights
+- `repeat_batches (int)`: Number of consecutive batches of users to run (default: 1)
+- `max_seq_len (int)`: Maximum context length supported by the model (refer to the table above)
+- `batch_size (int)`: Number of users in a batch (Supports 1/2/4/8/16/32 batches)
+- `max_generated_tokens (int)`: Maximum number of tokens to generate for each user (Note that the users will stop generation before this limit if they reach a eos token)
+- `paged_attention (bool)`: Whether to use paged attention or default attention (vLLM support (WIP) requires paged attention)
+- `page_params (dict)`: Page parameters for paged attention - [`block_size`, `max_num_blocks`]. For smaller context lengths use `block_size=32` and `max_num_blocks=1024`, for larger context use block_size=64 and max_num_blocks=2048
+- `sampling_params (dict)`: Sampling parameters for decoding -[`temperature`, `top_p`]. If temperature is set to 0, argmax (greedy decode) is used.
+- `stop_at_eos (bool)`: Flag to stop decoding when the model generates an EoS token
+- `optimizations (ModelOptimizations)`: Optimization level to use for the model [`accuracy`, `performance`]. Applied uniformly across all decoders unless an override config exists in `models/tt_transformers/model_params/<model-name>`
+- `decoder_config_file (DecodersPrecision)`: Fine-grained optimization control that allows specifying a configuration file to set different settings for each decoder.
+
+Please note that using `argmax` with `batch_size > 1` or using `top-p` sampling with any batch size, these ops will be run on host. This is because those ops are not yet fully supported on device. A decrease in performance is expected when these configurations are enabled.
+
+---
+
+### Optimization overrides
+Some models require a unique set of optimizations defined in `models/tt_transformers/model_params/<model-name>`. To override the default optimizations, you can define files named `models/tt_transformers/tt/model_config/PERFORMANCE_DECODER_CONFIG_FILENAME` and `models/tt_transformers/tt/model_config/ACCURACY_DECODER_CONFIG_FILENAME` in the appropriate `models/tt_transformers/model_params/<model-name>` directory to override the `ModelOptimizations.performance` and `ModelOptimizations.accuracy` optimizations respectively. For example, to override the default "performance" optimizations for Llama3.1-8B-Instruct, a file named `performance_decoder_config.json` has been created in the `models/tt_transformers/model_params/Llama3.1-8B-Instruct` directory. The content to write in override files is described in [the custom optimizations section](#custom-optimizations). Optimizations are applied with the following prioritization:
+1. from override config (if it exists)
+2. from the `optimizations` argument
+
+---
+
+### Custom input arguments
+To facilitate testing different configurations, `simple_text_demo.py` supports argument overrides. The full list of overrides is included in `models/tt_transformers/demo/conftest.py`.
+
+An example usage where the `batch-1` test is modified to run with 16 users and keep generating tokens until 1024 are generated:
+
+```
+pytest models/tt_transformers/demo/simple_text_demo.py -k "performance and batch-1" --batch_size 16 --max_generated_tokens 1024 --stop_at_eos 0
+```
+
+---
+
+### Custom optimizations
+To apply the same settings across all decoders, the `optimizations` argument can be used. `optimizations` offers a wide range of configurations for precision and math fidelity. The user can override the configurations of the data types of the weight tensors and activation tensors and the math fidelity of the kernels that works on those tensors, using the `--optimizations` argument on the command line. For example:
+
+```
+pytest models/tt_transformers/demo/simple_text_demo.py -k "accuracy and batch-1" --optimizations 'precision_cfg = {ff1_3: bfp4, ff2: bfp4, wqkv: bfp8, wo: bfp8, kv_cache: bfp8, activation: mixed}, fidelity_cfg = {li_ff1_3: hifi2, li_ff2: lofi, li_qkv_decode: hifi2, li_o_decode: hifi2, sdpa_decode: hifi2na, li_qkv_prefill: hifi2, li_o_prefill: hifi2fp16, sdpa_prefill: hifi4}'
+```
+
+Please refer to [model_config.py](models/tt_transformers/tt/model_config.py) for the full list of supported key-value pairs in the `--optimizations` argument. Centralized performance and accuracy targets are defined in [models/model_targets.yaml](../model_targets.yaml). The `lt` utility can still export markdown snapshots for local Pareto analysis.
+
+To apply non-uniform settings across the decoders, the user can provide a JSON file using the `decoder_config_file` argument to specify the configuration for each decoder. For example
+
+```
+pytest models/tt_transformers/demo/simple_text_demo.py -k "performance and batch-1" --decoder_config_file 'models/tt_transformers/demo/config_16_decoders.json'
+```
+
+When a component is not specified (e.g., FF2 is missing for decoder 2 in `models/tt_transformers/demo/config_16_decoders.json`), the baseline configuration is used for that component.
+
+---
+
+### Expected performance and accuracy
+
+See [models/model_targets.yaml](../model_targets.yaml) for expected performance and accuracy targets across supported configurations.
+Accuracy of the network architectures is measured by exact token matching using teacher forcing method. During inference the previous token is replaced by the ground truth token while the network generates the next token. This allows to avoid accumulating errors when comparisons on a finer level (tokens) assessed in comparison to other known metrics that compare quality and context of the answer. Token accuracy can be reported by passing the argument shown below:
+
+```
+pytest models/tt_transformers/demo/simple_text_demo.py -k "performance and batch-1" --token_accuracy True
+```
+
+---
+
+### Implementation notes
+
+**Chunked prefill (text-only)**: All of the compatible model/device combinations support a max prefill context-length of 128k, with the exception of Llama3.1-8B and Llama3.2-11B on N150 which have a max of 32k (due to a lack of memory). To support these large max context-lengths, chunked prefill is performed with different max chunk sizes as shown in the table below.
+
+Max Prefill Chunk Sizes (text-only):
+|              |      N150     |      N300     |      T3K       |      TG     |
+|--------------|---------------|---------------|----------------|-------------|
+| Llama3.2-1B  | 128k tokens   | 128k tokens   | 128k tokens    | 128k tokens |
+| Llama3.2-3B  | 8k tokens     | 128k tokens   | 128k tokens    | 128k tokens |
+| Qwen2.5-7B | 4k tokens | 32k tokens | 128k tokens | 128k tokens |
+| Llama3.1-8B  | 4k tokens     | 64k tokens    | 128k tokens    | 128k tokens |
+| Llama3.2-11B | 4k tokens     | 64k tokens    | 128k tokens    | 128k tokens |
+| Mistral-Small-3.1-24B | 8k tokens | 128k tokens | 128k tokens | 128k tokens |
+| Llama3.1-70B | Not supported | Not supported | 32k tokens     | 128k tokens |
+| Llama3.2-90B | Not supported | Not supported | 32k tokens     | Not supported |
+| DeepSeek-R1-Distill-Llama3.3-70B | Not supported | Not supported | 32k tokens | 128k tokens |
+
+
+- These max chunk sizes are specific to max context length 128k and are configured via `MAX_PREFILL_CHUNK_SIZES_DIV1024` in [model_config.py](https://github.com/tenstorrent/tt-metal/blob/main/models/demos/llama3/tt/model_config.py). If the max context length is set to a smaller value using the `max_seq_len` flag (see [Run the demo](#run-the-demo)), these chunk sizes can possibly be increased due to using a smaller KV cache.
+
+**Chunked prefill (Llama3.2-11B multimodal)**: Llama3.2-11B multimodal is currently only supported on N300 and T3000. On N300, a max prefill context length of 8k is supported, while T3000 supports a max context length of 128k.
+
+**Chunked prefill (Mistral-Small-3.1-24B multimodal)**: Mistral-Small-3.1-24B-Instruct-2503 (Pixtral vision) is currently supported on T3000. On T3000, a max prefill context length of 128k is supported.
+
+---
+
+### Memory Optimization
+
+#### HuggingFace Model Caching Control
+
+To help manage memory usage, you can control whether the HuggingFace model is cached in memory using the `cache_hf` parameter via command line or code:
+
+```python
+# Default: disables caching to conserve memory usage
+model_args = ModelArgs(
+    mesh_device,
+    cache_hf=False,  # Default: Reduces memory usage by not keeping HF model in memory
+    max_batch_size=1,
+    max_seq_len=2048
+)
+
+# Optional: enables caching for faster repeated access
+model_args = ModelArgs(
+    mesh_device,
+    cache_hf=True,  # Cache HF model for better performance running reference tests
+    max_batch_size=4,
+    max_seq_len=4096
+)
+```
+
+**When to disable caching (`cache_hf=False`):**
+- Running on systems with limited memory (< 256GB)
+- Loading large models (70B+ parameters)
+- Using the model for single inference runs
+- When you don't need reference model comparisons
+
+**When to keep caching enabled (`cache_hf=True`, default):**
+- Sufficient memory available
+- Comparisons with torch model is needed
+- Minimizing test duration is prioritized over memory usage
+- Running reference model tests
+
+The `cache_hf` parameter affects:
+- `load_state_dict()` method: Controls whether HF model is cached after loading
+- `reference_transformer()` method: Controls whether to reuse cached model or load fresh
+
+**Memory Impact:**
+- Disabling caching saves approximately the full model size in memory
+- For a 70B model, this can save ~140GB+ of memory usage
+- Increased test duration as model needs to be reloaded for reference operations

code/models/tt_transformers/conftest.py

@@ -0,0 +1,31 @@
+# SPDX-FileCopyrightText: © 2025 Tenstorrent USA, Inc.
+
+# SPDX-License-Identifier: Apache-2.0
+import os
+
+import pytest
+
+import ttnn
+
+
+@pytest.fixture
+def device_params(request, galaxy_type):
+    # Get param dict passed in from test parametrize (or default to empty dict).
+    # Any TRACE_MODEL_KEY_PARAM is left in place; the mesh_device fixture resolves it
+    # to trace_region_size using the logical submesh SKU.
+    params = getattr(request, "param", {}).copy()
+
+    mesh_device = {"N150": (1, 1), "N300": (1, 2), "N150x4": (1, 4), "T3K": (1, 8), "TG": (8, 4), "P150x8": (1, 8)}.get(
+        os.environ.get("MESH_DEVICE"), len(ttnn.get_device_ids())
+    )
+    is_single_device = (mesh_device == (1, 1)) if isinstance(mesh_device, tuple) else (mesh_device == 1)
+
+    if "fabric_config" in params:
+        if is_single_device:
+            params["fabric_config"] = None
+        elif params["fabric_config"] == True:
+            params["fabric_config"] = (
+                ttnn.FabricConfig.FABRIC_1D_RING if galaxy_type == "6U" else ttnn.FabricConfig.FABRIC_1D
+            )
+
+    return params

code/models/tt_transformers/host_mem_profiler.py

@@ -0,0 +1,188 @@
+# SPDX-FileCopyrightText: © 2025 Tenstorrent AI ULC
+
+# SPDX-License-Identifier: Apache-2.0
+
+"""
+host_mem_profiler.py — Host-side memory profiler for model tests
+===============================================================
+
+What this script does
+---------------------
+Wraps a pytest run with `memory_profiler` to record the host process
+memory (RSS) at 100 ms intervals while the model test executes.
+After the run it produces a PNG plot showing memory over time with a
+peak-memory reference line.  The script exits with pytest's own return
+code, so CI pipelines see failures correctly and no plot is saved for a
+failed run.
+
+Measured memory includes all child processes spawned by the test
+(e.g. the TT device runtime), giving a realistic view of total host
+footprint during inference.
+
+Dependencies
+------------
+`memory-profiler` and `matplotlib` are auto-installed at startup if
+absent.  Everything else must already be present in the active
+environment (loguru, pytest, and the tt-metal Python stack).
+
+    pip install memory-profiler matplotlib   # manual one-time install
+
+Arguments
+---------
+  --mesh-device   Value forwarded as the MESH_DEVICE environment variable
+                  to the test process (e.g. N150, N300, T3K, TG).
+                  Default: N150
+  --hf-model      Hugging Face model ID forwarded as HF_MODEL.
+                  Default: meta-llama/Llama-3.2-1B-Instruct
+  --test          Pytest target — a file path, directory, or node id.
+                  Default: models/tt_transformers/demo/simple_text_demo.py
+  -k              Optional pytest -k filter expression to select a subset
+                  of tests within the target.
+
+Output
+------
+A timestamped directory is created under profiling_results/:
+
+    profiling_results/<name>_<YYYYMMDD_HHMMSS>/
+        memory_profile.png   ← time-series plot (peak annotated in red)
+
+The script exits with pytest's return code; no plot is saved on failure.
+
+Usage examples
+--------------
+# Minimal — profile the default Llama-3.2-1B demo on a single N150 card:
+    python host_mem_profiler.py
+
+# Llama-3.1-8B on a T3K (8-chip) host, full test file:
+     python host_mem_profiler.py \\
+        --mesh-device T3K \\
+         --hf-model    meta-llama/Llama-3.1-8B-Instruct \\
+         --test        models/tt_transformers/demo/simple_text_demo.py
+
+# Same model but only the decode token accuracy test case:
+     python host_mem_profiler.py \\
+         --mesh-device T3K \\
+        --hf-model    meta-llama/Llama-3.1-8B-Instruct \\
+        --test        models/tt_transformers/demo/simple_text_demo.py \\
+        -k            performance-ci-token-mathcing
+"""
+
+import argparse
+import importlib
+import os
+import subprocess
+import sys
+from datetime import datetime
+from pathlib import Path
+
+from loguru import logger
+
+_ALLOWED_PACKAGES = {"memory-profiler", "matplotlib"}
+
+
+def _require(package, import_name=None):
+    if package not in _ALLOWED_PACKAGES:
+        raise ValueError(f"Package '{package}' is not in the allowed list: {_ALLOWED_PACKAGES}")
+    import_name = import_name or package
+    try:
+        importlib.import_module(import_name)
+    except ImportError:
+        print(f"Installing missing dependency: {package}")
+        subprocess.check_call([sys.executable, "-m", "pip", "install", package])
+        importlib.invalidate_caches()  # flush finder cache so the new package is visible
+
+
+_require("memory-profiler", "memory_profiler")
+_require("matplotlib")
+
+import matplotlib.pyplot as plt
+from memory_profiler import memory_usage
+
+PYTEST_EXTRA_ARGS = ["-v", "-s"]
+OUTPUT_ROOT = Path("profiling_results")
+
+
+# ── Runner ────────────────────────────────────────────────────────────────────
+def make_runner(model_cfg, k_filter=None):
+    """Return a zero-arg callable that runs a model's test suite."""
+
+    def run():
+        env = os.environ.copy()
+        env.update(model_cfg["env"])
+        cmd = [sys.executable, "-m", "pytest", model_cfg["test"], *PYTEST_EXTRA_ARGS]
+        if k_filter:
+            cmd += ["-k", k_filter]
+        result = subprocess.run(cmd, capture_output=False, env=env)
+        return result.returncode
+
+    return run
+
+
+# ── Plot ──────────────────────────────────────────────────────────────────────
+def plot_results(name, elapsed, mib, out_dir):
+    fig, ax = plt.subplots(figsize=(14, 6))
+    ax.plot(elapsed, mib, "+-k", linewidth=1.0, markersize=4, label=name)
+    ax.axhline(max(mib), color="red", linestyle="--", linewidth=0.8, label=f"Peak: {max(mib):.1f} MiB")
+
+    ax.set_xlabel("Time (s)")
+    ax.set_ylabel("Memory (MiB)")
+    ax.set_title("Memory Usage Over Time")
+    ax.legend()
+    fig.tight_layout()
+    fig.savefig(os.path.join(out_dir, "memory_profile.png"), dpi=150)
+    plt.close(fig)
+    logger.info(f"Saved at {os.path.join(out_dir, 'memory_profile.png')}")
+
+
+# ── Main ──────────────────────────────────────────────────────────────────────
+if __name__ == "__main__":
+    parser = argparse.ArgumentParser(description="Profile host-side memory usage of a model test")
+    parser.add_argument("--mesh-device", default="N150", help="MESH_DEVICE value (e.g. N150, T3K)")
+    parser.add_argument("--hf-model", default="meta-llama/Llama-3.2-1B-Instruct", help="Hugging Face model ID")
+    parser.add_argument("--test", default="models/tt_transformers/demo/simple_text_demo.py", help="Pytest target path")
+    parser.add_argument("-k", default=None, help="pytest -k filter expression")
+    args = parser.parse_args()
+
+    name = args.hf_model.split("/")[-1] if "/" in args.hf_model else args.hf_model
+    if args.k:
+        name = f"{name}_{args.k}"
+
+    model = {
+        "name": name,
+        "env": {
+            "MESH_DEVICE": args.mesh_device,
+            "HF_MODEL": args.hf_model,
+        },
+        "test": args.test,
+    }
+
+    logger.info(f"\n{'='*60}\nProfiling: {model['name']}\n{'='*60}")
+
+    mem_ts, returncode = memory_usage(
+        (make_runner(model, args.k), [], {}),
+        interval=0.1,
+        retval=True,
+        timestamps=True,
+        include_children=True,
+    )
+    if returncode:
+        logger.error(f"pytest exited with code {returncode} — skipping plot")
+        sys.exit(returncode)
+
+    mib = [m for m, _ in mem_ts]
+    ts = [t for _, t in mem_ts]
+    t0 = ts[0]
+    elapsed = [t - t0 for t in ts]
+
+    logger.info(f"  Peak:     {max(mib):.1f} MiB")
+    logger.info(f"  Baseline: {min(mib):.1f} MiB")
+
+    safe_name = name.replace("/", "_").replace(" ", "_")
+    timestamp = datetime.now().strftime("%Y%m%d_%H%M%S")
+    output_root = OUTPUT_ROOT.resolve()
+    out_dir = (output_root / f"{safe_name}_{timestamp}").resolve()
+    if not out_dir.is_relative_to(output_root):
+        raise ValueError(f"Refusing to write outside output root: {out_dir}")
+    out_dir.mkdir(parents=True, exist_ok=True)
+
+    plot_results(model["name"], elapsed, mib, out_dir)

image/blobs/sha256/0690f7dcdac59b821f2eff8a565951a2120a112c6b9613f43be03bb143f8875b

@@ -0,0 +1 @@
+{"id":"d0bbcfefb7467acdbf3aef2851c41c9c57d0eb61a01bdc4ce5e7e58bfa34e90c","parent":"155ab0b6484fdeee7919c43f2a38a56151bd05a8b48891fa53929c0e45577105","created":"1970-01-01T09:00:00+09:00","container_config":{"Hostname":"","Domainname":"","User":"","AttachStdin":false,"AttachStdout":false,"AttachStderr":false,"Tty":false,"OpenStdin":false,"StdinOnce":false,"Env":null,"Cmd":null,"Image":"","Volumes":null,"WorkingDir":"","Entrypoint":null,"OnBuild":null,"Labels":null},"os":"linux"}

image/blobs/sha256/0b3b7bf707349d15afaf4582949a95082c50449ac8e4f7346d1bbee5bc7ba1c5

@@ -0,0 +1 @@
+{"id":"443ab363fe01ac0190ecbaf3fa1afbc9676ce328e3a82321261b28d57844fa11","parent":"94055894755a230626deff4f0207669ea80cf6f5e7990ddf2950bfb2560a8c68","created":"2026-09-12T15:59:37.210009429+09:00","container_config":{"Hostname":"","Domainname":"","User":"","AttachStdin":false,"AttachStdout":false,"AttachStderr":false,"Tty":false,"OpenStdin":false,"StdinOnce":false,"Env":null,"Cmd":null,"Image":"","Volumes":null,"WorkingDir":"","Entrypoint":null,"OnBuild":null,"Labels":null},"config":{"Hostname":"","Domainname":"","User":"tt","AttachStdin":false,"AttachStdout":false,"AttachStderr":false,"Tty":false,"OpenStdin":false,"StdinOnce":false,"Env":["PATH=/opt/tt-venv/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin","VENV=/opt/tt-venv","VIRTUAL_ENV=/opt/tt-venv","TT_METAL_RUNTIME_ROOT=/opt/tt-metal","TT_METAL_HOME=/opt/tt-metal","PYTHONPATH=/opt/tt-metal","LD_LIBRARY_PATH=/opt/tt-metal/build/lib:/opt/openmpi-v5.0.7-ulfm/lib","EXTRA_MODELS_DIR=","TT_VLLM_BUILTIN_MODELS=","TT_MODEL_KIND=tt-dit-server","HF_HOME=/hf","TT_METAL_CACHE=/cache","HOME=/home/tt","USER=tt","LOGNAME=tt"],"Cmd":["/usr/local/bin/serve-default.sh"],"ArgsEscaped":true,"Image":"","Volumes":null,"WorkingDir":"/home/tt/work","Entrypoint":["/usr/local/bin/entrypoint.sh"],"OnBuild":null,"Labels":{"org.opencontainers.image.revision":"8b98410e730bb504fea43a88609756e34821d91d","org.opencontainers.image.version":"22.04","org.tenstorrent.tt-model":"locate-anything-3b-blackhole","org.tenstorrent.tt-model.arch":"blackhole","org.tenstorrent.tt-model.kind":"tt-dit-server","org.tenstorrent.tt-model.plugin":"","org.tenstorrent.tt-model.profiles":"default","org.tenstorrent.tt-model.repo":"changh95/locate-anything-3b-blackhole","org.tenstorrent.tt-model.tt-metal":"v0.78.0-dev20260820-25-g8b98410e73","org.tenstorrent.tt-model.weights":"nvidia/LocateAnything-3B"}},"architecture":"amd64","os":"linux"}

image/blobs/sha256/18b68d787d3c7446ef9eb0eba334950a837818ace17924cb42e6e6a2fac47a17

@@ -0,0 +1 @@
+{"id":"87cb9513aafc6caae0d486fd87639985a32e69f3f8c9b8c6ede95f9cccfaa667","parent":"525c1f4cf166cfba577e017bb1da9678a6667fa9bbd56a2608bf2e3585e0cf8a","created":"1970-01-01T09:00:00+09:00","container_config":{"Hostname":"","Domainname":"","User":"","AttachStdin":false,"AttachStdout":false,"AttachStderr":false,"Tty":false,"OpenStdin":false,"StdinOnce":false,"Env":null,"Cmd":null,"Image":"","Volumes":null,"WorkingDir":"","Entrypoint":null,"OnBuild":null,"Labels":null},"os":"linux"}

image/blobs/sha256/18f46874ca6617cba588e37f39ecd58f44b02065d3c3ea3909c403d2260b686d

@@ -0,0 +1 @@
+{"id":"d1597874ea56b73b5fdf623f9323c8d1620b2625e4b8924ee358c61652b82be5","parent":"b58135a7607f916cd7221128976d64d0f39fc917e1ba780df8efe3e2c4c7d23a","created":"1970-01-01T09:00:00+09:00","container_config":{"Hostname":"","Domainname":"","User":"","AttachStdin":false,"AttachStdout":false,"AttachStderr":false,"Tty":false,"OpenStdin":false,"StdinOnce":false,"Env":null,"Cmd":null,"Image":"","Volumes":null,"WorkingDir":"","Entrypoint":null,"OnBuild":null,"Labels":null},"os":"linux"}

image/blobs/sha256/1d805657f0ef573b66e654d64007281d8aadf4dfa8aa1eb88672c886293dfe13

@@ -0,0 +1 @@
+{"id":"0e81830e949af791d050288892e16a4dd59392bbcc7479a00ab3629ae8e529e6","parent":"d0bbcfefb7467acdbf3aef2851c41c9c57d0eb61a01bdc4ce5e7e58bfa34e90c","created":"1970-01-01T09:00:00+09:00","container_config":{"Hostname":"","Domainname":"","User":"","AttachStdin":false,"AttachStdout":false,"AttachStderr":false,"Tty":false,"OpenStdin":false,"StdinOnce":false,"Env":null,"Cmd":null,"Image":"","Volumes":null,"WorkingDir":"","Entrypoint":null,"OnBuild":null,"Labels":null},"os":"linux"}

image/blobs/sha256/1f179d27e2f48842a37b1dbe6300e0e9d83c174ae4c6d879f38b473d31ff159f

@@ -0,0 +1 @@
+{"id":"d08f3a800926365edee4ef1f3c9d6d42ca6762b65af44fe80261a5543cd58905","parent":"b2319e94f380633a4827308b9cb5ac756723a5f05cefd50d47dafdcba43981b2","created":"1970-01-01T09:00:00+09:00","container_config":{"Hostname":"","Domainname":"","User":"","AttachStdin":false,"AttachStdout":false,"AttachStderr":false,"Tty":false,"OpenStdin":false,"StdinOnce":false,"Env":null,"Cmd":null,"Image":"","Volumes":null,"WorkingDir":"","Entrypoint":null,"OnBuild":null,"Labels":null},"os":"linux"}

image/blobs/sha256/28b873d0d487580ae6a42f1c52c28160516c4f2e4d82abf93e6f6b7c95e49bc7

@@ -0,0 +1 @@
+{"id":"f3d345315bfbedaf1b9e3924e84e2a596382cae91d99fac771a0d023a3f0d356","parent":"5a70f6b5efbd972a1364d6f6d00f598c5fe6c68f620defce025d5d7c0e89e158","created":"1970-01-01T09:00:00+09:00","container_config":{"Hostname":"","Domainname":"","User":"","AttachStdin":false,"AttachStdout":false,"AttachStderr":false,"Tty":false,"OpenStdin":false,"StdinOnce":false,"Env":null,"Cmd":null,"Image":"","Volumes":null,"WorkingDir":"","Entrypoint":null,"OnBuild":null,"Labels":null},"os":"linux"}

image/blobs/sha256/3dcb0a4c97491bdc43addf1e7fe76c9dd7a714314fc4729a32cf94c89455a2a5

@@ -0,0 +1 @@
+{"id":"2dcd7c4c5dcd97f8c692bc32701ed37ed86c9683cc3afff3156da3af639b7f52","parent":"0c37cc42f3bf2829965c4c3a62507e4e540086b724c37f43bf5e53243e230a2b","created":"1970-01-01T09:00:00+09:00","container_config":{"Hostname":"","Domainname":"","User":"","AttachStdin":false,"AttachStdout":false,"AttachStderr":false,"Tty":false,"OpenStdin":false,"StdinOnce":false,"Env":null,"Cmd":null,"Image":"","Volumes":null,"WorkingDir":"","Entrypoint":null,"OnBuild":null,"Labels":null},"os":"linux"}

image/blobs/sha256/4b7d28cf86399d4e9f4f1eeba9a85df52bfc3e360e1c831def2915ad8cff54db

@@ -0,0 +1 @@
+{"id":"b2319e94f380633a4827308b9cb5ac756723a5f05cefd50d47dafdcba43981b2","parent":"e618d643825753f4550a54eacfd500e6e7652078cf4168c9ef3f869701db7fc5","created":"1970-01-01T09:00:00+09:00","container_config":{"Hostname":"","Domainname":"","User":"","AttachStdin":false,"AttachStdout":false,"AttachStderr":false,"Tty":false,"OpenStdin":false,"StdinOnce":false,"Env":null,"Cmd":null,"Image":"","Volumes":null,"WorkingDir":"","Entrypoint":null,"OnBuild":null,"Labels":null},"os":"linux"}

image/blobs/sha256/5628e13ce208b83d793af15fa41c28d1702e7a43e27c8c72944442d01e959a71

@@ -0,0 +1 @@
+{"id":"e618d643825753f4550a54eacfd500e6e7652078cf4168c9ef3f869701db7fc5","created":"1970-01-01T09:00:00+09:00","container_config":{"Hostname":"","Domainname":"","User":"","AttachStdin":false,"AttachStdout":false,"AttachStderr":false,"Tty":false,"OpenStdin":false,"StdinOnce":false,"Env":null,"Cmd":null,"Image":"","Volumes":null,"WorkingDir":"","Entrypoint":null,"OnBuild":null,"Labels":null},"os":"linux"}

image/blobs/sha256/718b6feb51a3973517c6ac89780d58b88478d3ddbdc41353af609f6c944dc085

@@ -0,0 +1 @@
+{"id":"94055894755a230626deff4f0207669ea80cf6f5e7990ddf2950bfb2560a8c68","parent":"56c06abc3c98b8e8b28ca977494b08c3467fc2ef2ba5ef89bc016a6c5aaed462","created":"1970-01-01T09:00:00+09:00","container_config":{"Hostname":"","Domainname":"","User":"","AttachStdin":false,"AttachStdout":false,"AttachStderr":false,"Tty":false,"OpenStdin":false,"StdinOnce":false,"Env":null,"Cmd":null,"Image":"","Volumes":null,"WorkingDir":"","Entrypoint":null,"OnBuild":null,"Labels":null},"os":"linux"}

image/blobs/sha256/77b04a4822168bcd77a83c78c22a390734a35e4232b7c9f2485b2663b5fefb7c

@@ -0,0 +1 @@
+{"id":"0c37cc42f3bf2829965c4c3a62507e4e540086b724c37f43bf5e53243e230a2b","parent":"0e81830e949af791d050288892e16a4dd59392bbcc7479a00ab3629ae8e529e6","created":"1970-01-01T09:00:00+09:00","container_config":{"Hostname":"","Domainname":"","User":"","AttachStdin":false,"AttachStdout":false,"AttachStderr":false,"Tty":false,"OpenStdin":false,"StdinOnce":false,"Env":null,"Cmd":null,"Image":"","Volumes":null,"WorkingDir":"","Entrypoint":null,"OnBuild":null,"Labels":null},"os":"linux"}

image/blobs/sha256/7f2d9782a5ff3e0bf36c4b028cdf5b4e59db0cdf2122cea0669de2841c5d05f7

@@ -0,0 +1 @@
+{"id":"69f6359ca11460b168172b1ad4ba59ef64db4fe7ef455e604d5987c89ec68fea","parent":"87cb9513aafc6caae0d486fd87639985a32e69f3f8c9b8c6ede95f9cccfaa667","created":"1970-01-01T09:00:00+09:00","container_config":{"Hostname":"","Domainname":"","User":"","AttachStdin":false,"AttachStdout":false,"AttachStderr":false,"Tty":false,"OpenStdin":false,"StdinOnce":false,"Env":null,"Cmd":null,"Image":"","Volumes":null,"WorkingDir":"","Entrypoint":null,"OnBuild":null,"Labels":null},"os":"linux"}

image/blobs/sha256/89fcc824b4db414c4ba9f1a86324bd6bfa92505480f7c02fd87f4e3c79ebf1de

@@ -0,0 +1 @@
+{"id":"5a70f6b5efbd972a1364d6f6d00f598c5fe6c68f620defce025d5d7c0e89e158","parent":"2dcd7c4c5dcd97f8c692bc32701ed37ed86c9683cc3afff3156da3af639b7f52","created":"1970-01-01T09:00:00+09:00","container_config":{"Hostname":"","Domainname":"","User":"","AttachStdin":false,"AttachStdout":false,"AttachStderr":false,"Tty":false,"OpenStdin":false,"StdinOnce":false,"Env":null,"Cmd":null,"Image":"","Volumes":null,"WorkingDir":"","Entrypoint":null,"OnBuild":null,"Labels":null},"os":"linux"}

image/blobs/sha256/8a28dfa1a683cbd7674d79f7ac878f4ba04d0a72bccfe56cec0f0f008b325bdd

@@ -0,0 +1 @@
+{"id":"56c06abc3c98b8e8b28ca977494b08c3467fc2ef2ba5ef89bc016a6c5aaed462","parent":"c142e2f41d7cc33a0e167943cc3f43767c01aa6576edb983bd44e3590d9deaec","created":"1970-01-01T09:00:00+09:00","container_config":{"Hostname":"","Domainname":"","User":"","AttachStdin":false,"AttachStdout":false,"AttachStderr":false,"Tty":false,"OpenStdin":false,"StdinOnce":false,"Env":null,"Cmd":null,"Image":"","Volumes":null,"WorkingDir":"","Entrypoint":null,"OnBuild":null,"Labels":null},"os":"linux"}

image/blobs/sha256/bc34115d8ab3da3bb2818457b8bc493ed594968b9ae39c48e30c22b37294974b

@@ -0,0 +1 @@
+{"architecture":"amd64","config":{"User":"tt","Env":["PATH=/opt/tt-venv/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin","VENV=/opt/tt-venv","VIRTUAL_ENV=/opt/tt-venv","TT_METAL_RUNTIME_ROOT=/opt/tt-metal","TT_METAL_HOME=/opt/tt-metal","PYTHONPATH=/opt/tt-metal","LD_LIBRARY_PATH=/opt/tt-metal/build/lib:/opt/openmpi-v5.0.7-ulfm/lib","EXTRA_MODELS_DIR=","TT_VLLM_BUILTIN_MODELS=","TT_MODEL_KIND=tt-dit-server","HF_HOME=/hf","TT_METAL_CACHE=/cache","HOME=/home/tt","USER=tt","LOGNAME=tt"],"Entrypoint":["/usr/local/bin/entrypoint.sh"],"Cmd":["/usr/local/bin/serve-default.sh"],"WorkingDir":"/home/tt/work","Labels":{"org.opencontainers.image.revision":"8b98410e730bb504fea43a88609756e34821d91d","org.opencontainers.image.version":"22.04","org.tenstorrent.tt-model":"locate-anything-3b-blackhole","org.tenstorrent.tt-model.arch":"blackhole","org.tenstorrent.tt-model.kind":"tt-dit-server","org.tenstorrent.tt-model.plugin":"","org.tenstorrent.tt-model.profiles":"default","org.tenstorrent.tt-model.repo":"changh95/locate-anything-3b-blackhole","org.tenstorrent.tt-model.tt-metal":"v0.78.0-dev20260820-25-g8b98410e73","org.tenstorrent.tt-model.weights":"nvidia/LocateAnything-3B"},"ArgsEscaped":true},"created":"2026-09-12T15:59:37.210009429+09:00","history":[{"created":"2026-09-03T12:14:51.139348462Z","created_by":"/bin/sh -c #(nop)  ARG RELEASE","empty_layer":true},{"created":"2026-09-03T12:14:51.170427469Z","created_by":"/bin/sh -c #(nop)  ARG LAUNCHPAD_BUILD_ARCH","empty_layer":true},{"created":"2026-09-03T12:14:51.200307479Z","created_by":"/bin/sh -c #(nop)  LABEL org.opencontainers.image.version=22.04","empty_layer":true},{"created":"2026-09-03T12:14:53.528689977Z","created_by":"/bin/sh -c #(nop) ADD file:81c01921c5f642ac2fcbfae682e489e8e64b347467d9fa1587707e310e64d790 in / "},{"created":"2026-09-03T12:14:54.045714841Z","created_by":"/bin/sh -c #(nop)  CMD [\"/bin/bash\"]","empty_layer":true},{"created":"2026-09-12T15:18:22.627027282+09:00","created_by":"ARG OMPI_DIR=/opt/openmpi-v5.0.7-ulfm","comment":"buildkit.dockerfile.v0","empty_layer":true},{"created":"2026-09-12T15:18:22.627027282+09:00","created_by":"ARG EXTRA_MODELS_DIR=","comment":"buildkit.dockerfile.v0","empty_layer":true},{"created":"2026-09-12T15:18:22.627027282+09:00","created_by":"ARG TT_MODEL_KIND","comment":"buildkit.dockerfile.v0","empty_layer":true},{"created":"2026-09-12T15:18:22.627027282+09:00","created_by":"ARG MODEL_NAME","comment":"buildkit.dockerfile.v0","empty_layer":true},{"created":"2026-09-12T15:18:22.627027282+09:00","created_by":"ARG MODEL_REPO","comment":"buildkit.dockerfile.v0","empty_layer":true},{"created":"2026-09-12T15:18:22.627027282+09:00","created_by":"ARG MODEL_WEIGHTS","comment":"buildkit.dockerfile.v0","empty_layer":true},{"created":"2026-09-12T15:18:22.627027282+09:00","created_by":"ARG MODEL_ARCH","comment":"buildkit.dockerfile.v0","empty_layer":true},{"created":"2026-09-12T15:18:22.627027282+09:00","created_by":"ARG MODEL_PROFILES","comment":"buildkit.dockerfile.v0","empty_layer":true},{"created":"2026-09-12T15:18:22.627027282+09:00","created_by":"ARG MODEL_TT_METAL_SHA","comment":"buildkit.dockerfile.v0","empty_layer":true},{"created":"2026-09-12T15:18:22.627027282+09:00","created_by":"ARG MODEL_TT_METAL_DESCRIBE","comment":"buildkit.dockerfile.v0","empty_layer":true},{"created":"2026-09-12T15:18:22.627027282+09:00","created_by":"ARG MODEL_PLUGIN_SHA","comment":"buildkit.dockerfile.v0","empty_layer":true},{"created":"2026-09-12T15:18:22.627027282+09:00","created_by":"RUN |11 OMPI_DIR=/opt/openmpi-v5.0.7-ulfm EXTRA_MODELS_DIR= TT_MODEL_KIND=tt-dit-server MODEL_NAME=locate-anything-3b-blackhole MODEL_REPO=changh95/locate-anything-3b-blackhole MODEL_WEIGHTS=nvidia/LocateAnything-3B MODEL_ARCH=blackhole MODEL_PROFILES=default MODEL_TT_METAL_SHA=8b98410e730bb504fea43a88609756e34821d91d MODEL_TT_METAL_DESCRIBE=v0.78.0-dev20260820-25-g8b98410e73 MODEL_PLUGIN_SHA= /bin/sh -c apt-get update \u0026\u0026 apt-get install -y --no-install-recommends     libhwloc15 libnuma1 libatomic1 libudev1 libcap2 zlib1g     libmpc3 libmpfr6 libgmp10 libzstd1     libevent-core-2.1-7 libevent-pthreads-2.1-7     libgl1 libsndfile1 ca-certificates     \u0026\u0026 apt-get clean \u0026\u0026 rm -rf /var/lib/apt/lists/* # buildkit","comment":"buildkit.dockerfile.v0"},{"created":"2026-09-12T15:18:22.929268734+09:00","created_by":"RUN |11 OMPI_DIR=/opt/openmpi-v5.0.7-ulfm EXTRA_MODELS_DIR= TT_MODEL_KIND=tt-dit-server MODEL_NAME=locate-anything-3b-blackhole MODEL_REPO=changh95/locate-anything-3b-blackhole MODEL_WEIGHTS=nvidia/LocateAnything-3B MODEL_ARCH=blackhole MODEL_PROFILES=default MODEL_TT_METAL_SHA=8b98410e730bb504fea43a88609756e34821d91d MODEL_TT_METAL_DESCRIBE=v0.78.0-dev20260820-25-g8b98410e73 MODEL_PLUGIN_SHA= /bin/sh -c existing=\"$(getent passwd 1000 | cut -d: -f1)\"     \u0026\u0026 if [ -n \"$existing\" ]; then          userdel -r \"$existing\" 2\u003e/dev/null || userdel \"$existing\";        fi     \u0026\u0026 useradd --uid 1000 --create-home --home-dir /home/tt --shell /bin/bash tt     \u0026\u0026 mkdir -p /home/tt/work/logs /cache /opt/tt-metal     \u0026\u0026 chown -R tt:tt /home/tt /cache /opt/tt-metal     \u0026\u0026 chmod 1777 /home/tt /home/tt/work /home/tt/work/logs /cache # buildkit","comment":"buildkit.dockerfile.v0"},{"created":"2026-09-12T15:20:34.958721745+09:00","created_by":"COPY /opt/openmpi-v5.0.7-ulfm /opt/openmpi-v5.0.7-ulfm # buildkit","comment":"buildkit.dockerfile.v0"},{"created":"2026-09-12T15:20:40.413645687+09:00","created_by":"COPY /opt/tenstorrent /opt/tenstorrent # buildkit","comment":"buildkit.dockerfile.v0"},{"created":"2026-09-12T15:59:09.695650358+09:00","created_by":"COPY /usr/local/share/uv /usr/local/share/uv # buildkit","comment":"buildkit.dockerfile.v0"},{"created":"2026-09-12T15:59:13.968046982+09:00","created_by":"COPY /opt/tt-venv /opt/tt-venv # buildkit","comment":"buildkit.dockerfile.v0"},{"created":"2026-09-12T15:59:14.05489038+09:00","created_by":"COPY /opt/vllm /opt/vllm # buildkit","comment":"buildkit.dockerfile.v0"},{"created":"2026-09-12T15:59:14.837070708+09:00","created_by":"COPY --chown=tt:tt /opt/tt-metal/runtime /opt/tt-metal/runtime # buildkit","comment":"buildkit.dockerfile.v0"},{"created":"2026-09-12T15:59:16.252685022+09:00","created_by":"COPY --chown=tt:tt /opt/tt-metal/build_Release /opt/tt-metal/build_Release # buildkit","comment":"buildkit.dockerfile.v0"},{"created":"2026-09-12T15:59:17.681704755+09:00","created_by":"COPY --chown=tt:tt /opt/tt-metal/build /opt/tt-metal/build # buildkit","comment":"buildkit.dockerfile.v0"},{"created":"2026-09-12T15:59:20.295619738+09:00","created_by":"COPY --chown=tt:tt /opt/tt-metal/tt_metal /opt/tt-metal/tt_metal # buildkit","comment":"buildkit.dockerfile.v0"},{"created":"2026-09-12T15:59:21.696437386+09:00","created_by":"COPY --chown=tt:tt /opt/tt-metal/ttnn /opt/tt-metal/ttnn # buildkit","comment":"buildkit.dockerfile.v0"},{"created":"2026-09-12T15:59:21.905622694+09:00","created_by":"COPY --chown=tt:tt /opt/tt-metal/tools /opt/tt-metal/tools # buildkit","comment":"buildkit.dockerfile.v0"},{"created":"2026-09-12T15:59:22.036864821+09:00","created_by":"COPY --chown=tt:tt /opt/tt-metal/setup.py /opt/tt-metal/pyproject.toml /opt/tt-metal/ # buildkit","comment":"buildkit.dockerfile.v0"},{"created":"2026-09-12T15:59:22.212260176+09:00","created_by":"COPY --chown=tt:tt code/ /opt/tt-metal/ # buildkit","comment":"buildkit.dockerfile.v0"},{"created":"2026-09-12T15:59:22.296404672+09:00","created_by":"COPY entrypoint.sh /usr/local/bin/entrypoint.sh # buildkit","comment":"buildkit.dockerfile.v0"},{"created":"2026-09-12T15:59:22.381901539+09:00","created_by":"COPY --chmod=0755 serve-default.sh /usr/local/bin/serve-default.sh # buildkit","comment":"buildkit.dockerfile.v0"},{"created":"2026-09-12T15:59:22.381901539+09:00","created_by":"ENV VENV=/opt/tt-venv","comment":"buildkit.dockerfile.v0","empty_layer":true},{"created":"2026-09-12T15:59:22.381901539+09:00","created_by":"ENV VIRTUAL_ENV=/opt/tt-venv","comment":"buildkit.dockerfile.v0","empty_layer":true},{"created":"2026-09-12T15:59:22.381901539+09:00","created_by":"ENV PATH=/opt/tt-venv/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin","comment":"buildkit.dockerfile.v0","empty_layer":true},{"created":"2026-09-12T15:59:22.381901539+09:00","created_by":"ENV TT_METAL_RUNTIME_ROOT=/opt/tt-metal","comment":"buildkit.dockerfile.v0","empty_layer":true},{"created":"2026-09-12T15:59:22.381901539+09:00","created_by":"ENV TT_METAL_HOME=/opt/tt-metal","comment":"buildkit.dockerfile.v0","empty_layer":true},{"created":"2026-09-12T15:59:22.381901539+09:00","created_by":"ENV PYTHONPATH=/opt/tt-metal","comment":"buildkit.dockerfile.v0","empty_layer":true},{"created":"2026-09-12T15:59:22.381901539+09:00","created_by":"ENV LD_LIBRARY_PATH=/opt/tt-metal/build/lib:/opt/openmpi-v5.0.7-ulfm/lib","comment":"buildkit.dockerfile.v0","empty_layer":true},{"created":"2026-09-12T15:59:22.381901539+09:00","created_by":"ENV EXTRA_MODELS_DIR=","comment":"buildkit.dockerfile.v0","empty_layer":true},{"created":"2026-09-12T15:59:22.381901539+09:00","created_by":"ARG TT_VLLM_BUILTIN_MODELS=","comment":"buildkit.dockerfile.v0","empty_layer":true},{"created":"2026-09-12T15:59:22.381901539+09:00","created_by":"ENV TT_VLLM_BUILTIN_MODELS=","comment":"buildkit.dockerfile.v0","empty_layer":true},{"created":"2026-09-12T15:59:22.381901539+09:00","created_by":"ENV TT_MODEL_KIND=tt-dit-server","comment":"buildkit.dockerfile.v0","empty_layer":true},{"created":"2026-09-12T15:59:22.381901539+09:00","created_by":"ENV HF_HOME=/hf","comment":"buildkit.dockerfile.v0","empty_layer":true},{"created":"2026-09-12T15:59:22.381901539+09:00","created_by":"ENV TT_METAL_CACHE=/cache","comment":"buildkit.dockerfile.v0","empty_layer":true},{"created":"2026-09-12T15:59:22.381901539+09:00","created_by":"ENV HOME=/home/tt","comment":"buildkit.dockerfile.v0","empty_layer":true},{"created":"2026-09-12T15:59:22.381901539+09:00","created_by":"ENV USER=tt","comment":"buildkit.dockerfile.v0","empty_layer":true},{"created":"2026-09-12T15:59:22.381901539+09:00","created_by":"ENV LOGNAME=tt","comment":"buildkit.dockerfile.v0","empty_layer":true},{"created":"2026-09-12T15:59:22.381901539+09:00","created_by":"USER tt","comment":"buildkit.dockerfile.v0","empty_layer":true},{"created":"2026-09-12T15:59:22.47282321+09:00","created_by":"WORKDIR /home/tt/work","comment":"buildkit.dockerfile.v0"},{"created":"2026-09-12T15:59:22.557362177+09:00","created_by":"COPY verify.sh /ctx/verify.sh # buildkit","comment":"buildkit.dockerfile.v0"},{"created":"2026-09-12T15:59:36.937396823+09:00","created_by":"RUN |12 OMPI_DIR=/opt/openmpi-v5.0.7-ulfm EXTRA_MODELS_DIR= TT_MODEL_KIND=tt-dit-server MODEL_NAME=locate-anything-3b-blackhole MODEL_REPO=changh95/locate-anything-3b-blackhole MODEL_WEIGHTS=nvidia/LocateAnything-3B MODEL_ARCH=blackhole MODEL_PROFILES=default MODEL_TT_METAL_SHA=8b98410e730bb504fea43a88609756e34821d91d MODEL_TT_METAL_DESCRIBE=v0.78.0-dev20260820-25-g8b98410e73 MODEL_PLUGIN_SHA= TT_VLLM_BUILTIN_MODELS= /bin/sh -c bash /ctx/verify.sh # buildkit","comment":"buildkit.dockerfile.v0"},{"created":"2026-09-12T15:59:36.937396823+09:00","created_by":"USER root","comment":"buildkit.dockerfile.v0","empty_layer":true},{"created":"2026-09-12T15:59:37.210009429+09:00","created_by":"RUN |12 OMPI_DIR=/opt/openmpi-v5.0.7-ulfm EXTRA_MODELS_DIR= TT_MODEL_KIND=tt-dit-server MODEL_NAME=locate-anything-3b-blackhole MODEL_REPO=changh95/locate-anything-3b-blackhole MODEL_WEIGHTS=nvidia/LocateAnything-3B MODEL_ARCH=blackhole MODEL_PROFILES=default MODEL_TT_METAL_SHA=8b98410e730bb504fea43a88609756e34821d91d MODEL_TT_METAL_DESCRIBE=v0.78.0-dev20260820-25-g8b98410e73 MODEL_PLUGIN_SHA= TT_VLLM_BUILTIN_MODELS= /bin/sh -c chmod -R a+rwX /home/tt # buildkit","comment":"buildkit.dockerfile.v0"},{"created":"2026-09-12T15:59:37.210009429+09:00","created_by":"USER tt","comment":"buildkit.dockerfile.v0","empty_layer":true},{"created":"2026-09-12T15:59:37.210009429+09:00","created_by":"LABEL org.tenstorrent.tt-model=locate-anything-3b-blackhole org.tenstorrent.tt-model.repo=changh95/locate-anything-3b-blackhole org.tenstorrent.tt-model.weights=nvidia/LocateAnything-3B org.tenstorrent.tt-model.arch=blackhole org.tenstorrent.tt-model.kind=tt-dit-server org.tenstorrent.tt-model.profiles=default org.opencontainers.image.revision=8b98410e730bb504fea43a88609756e34821d91d org.tenstorrent.tt-model.tt-metal=v0.78.0-dev20260820-25-g8b98410e73 org.tenstorrent.tt-model.plugin=","comment":"buildkit.dockerfile.v0","empty_layer":true},{"created":"2026-09-12T15:59:37.210009429+09:00","created_by":"ENTRYPOINT [\"/usr/local/bin/entrypoint.sh\"]","comment":"buildkit.dockerfile.v0","empty_layer":true},{"created":"2026-09-12T15:59:37.210009429+09:00","created_by":"CMD [\"/usr/local/bin/serve-default.sh\"]","comment":"buildkit.dockerfile.v0","empty_layer":true}],"os":"linux","rootfs":{"type":"layers","diff_ids":["sha256:ea16cace89338c84eb6bcb91a7efdfcae6838fff359efe951858227436486c34","sha256:e64f34e6f7fc770276a58ce68b18755913e8f6a38b80274ed30b138ba6db8cef","sha256:3f6ad10d5ba93a17af361223d9d9cd8c53c34d4156cd69af1e9b9af1bebcd4bb","sha256:5b60873863c0500f341a4c237ab1c18b929b3ac1571a2808b87fcacfdd1f2168","sha256:0311ceefbd13017ef17eb9844fbfd15250b1bb099d408687eef4372ad2c65ee5","sha256:661b4446add5adf8c9d4eaab02ab1fe419f17114b8c7a7a381707be7973467e5","sha256:2f1823fe031c57258362a517aa11e3c4a55eccca738913d0cc87587ccfb6fd1a","sha256:e1272a58679febd696c70d77b53ffeb43cba62b4ddb3d97dd4e632c6848d2856","sha256:d1a9879f27dd477b46fe73ced95ea58803d70fc7c2bb42b875c605b8fc744e14","sha256:184743fecd7fc3deb4e6f457a2a602932fe813f75148eec4b6e75ff88a9f2be2","sha256:59b37b1999541425b0a7cda1ce01f95f1be228fa97cd09f6631acfb2fab990fd","sha256:ca297dc8905fb0b70eebf7e9c58d50df4d8d38c5ece2cc866c22798cfc2dfd01","sha256:4f4c468252314917a1847056077663ed71f3abc1dc386c03bca471690a23c83b","sha256:c94435d8cb1f57c0296304800fe6e7376c511a75484d5da38e0eceb4a8cba02d","sha256:639c9dd1747f941795c619f445de593cc6d7957060442b3dbe0148bd1be1eeb3","sha256:082a9ded324a46a5cc7fe15bde1b9d83c03804f0f84eac7ce6bb51229b894f1f","sha256:8529443804912d6639846836d1b7ab793de42b54e3b4058e14a059f0d5fe4ade","sha256:96f3b28657c091e4818e2407f66db33853828fdb8abef3177cdf5c8d525f7dbf","sha256:5f70bf18a086007016e948b04aed3b82103a36bea41755b6cddfaf10ace3c6ef","sha256:c0d475ded7d3e60cdc094385f92036f940b605a0438578608e031030a8819ae9","sha256:53deb2ae079afabdafec52b959453dccca54d8c7eb6315862fcab21d186bcbdd","sha256:5116d36f79d2da6aa08f8ffcff866e39d7c0f5be5bbf3d74c6e8c8caebf2c5fd"]}}

image/blobs/sha256/cf8ccef7cd12a5a0492c4e3cde07fef8807fe0cd228de6257516ba32f752276c

@@ -0,0 +1 @@
+{"id":"ad0118a7016e58d45485fd05788ec2150a90083b48a6e651e81346550adf57bb","parent":"d08f3a800926365edee4ef1f3c9d6d42ca6762b65af44fe80261a5543cd58905","created":"1970-01-01T09:00:00+09:00","container_config":{"Hostname":"","Domainname":"","User":"","AttachStdin":false,"AttachStdout":false,"AttachStderr":false,"Tty":false,"OpenStdin":false,"StdinOnce":false,"Env":null,"Cmd":null,"Image":"","Volumes":null,"WorkingDir":"","Entrypoint":null,"OnBuild":null,"Labels":null},"os":"linux"}

image/blobs/sha256/dfbe9006d67d8efd33cd4834af36a9acd4030ef4f5a9aca44181d9555d590526

@@ -0,0 +1 @@
+{"id":"525c1f4cf166cfba577e017bb1da9678a6667fa9bbd56a2608bf2e3585e0cf8a","parent":"d1597874ea56b73b5fdf623f9323c8d1620b2625e4b8924ee358c61652b82be5","created":"1970-01-01T09:00:00+09:00","container_config":{"Hostname":"","Domainname":"","User":"","AttachStdin":false,"AttachStdout":false,"AttachStderr":false,"Tty":false,"OpenStdin":false,"StdinOnce":false,"Env":null,"Cmd":null,"Image":"","Volumes":null,"WorkingDir":"","Entrypoint":null,"OnBuild":null,"Labels":null},"os":"linux"}

image/blobs/sha256/e71053098ec0882ae447db33f4d23f6455424ec7cf649519ac98455342a3ffbe

@@ -0,0 +1 @@
+{"id":"c142e2f41d7cc33a0e167943cc3f43767c01aa6576edb983bd44e3590d9deaec","parent":"69f6359ca11460b168172b1ad4ba59ef64db4fe7ef455e604d5987c89ec68fea","created":"1970-01-01T09:00:00+09:00","container_config":{"Hostname":"","Domainname":"","User":"","AttachStdin":false,"AttachStdout":false,"AttachStderr":false,"Tty":false,"OpenStdin":false,"StdinOnce":false,"Env":null,"Cmd":null,"Image":"","Volumes":null,"WorkingDir":"","Entrypoint":null,"OnBuild":null,"Labels":null},"os":"linux"}

image/blobs/sha256/e78a28551be8e30ee61eba8b7627859a67e05827f5119754228edfe4ca005e48

@@ -0,0 +1 @@
+{"id":"12e1b52d50de6740b0ae582760feedf780e2affcaa1c5d15bef6df3f7da931d4","parent":"0308415c0d87b5deb13640aea7b13b37d66d085488d40063d6304dcc2de92520","created":"1970-01-01T09:00:00+09:00","container_config":{"Hostname":"","Domainname":"","User":"","AttachStdin":false,"AttachStdout":false,"AttachStderr":false,"Tty":false,"OpenStdin":false,"StdinOnce":false,"Env":null,"Cmd":null,"Image":"","Volumes":null,"WorkingDir":"","Entrypoint":null,"OnBuild":null,"Labels":null},"os":"linux"}

image/blobs/sha256/e94cd641692a2de981d616a078e01282ac9e8cef9fb44bfa26cbab8bc16d9812

@@ -0,0 +1 @@
+{"id":"0308415c0d87b5deb13640aea7b13b37d66d085488d40063d6304dcc2de92520","parent":"ad0118a7016e58d45485fd05788ec2150a90083b48a6e651e81346550adf57bb","created":"1970-01-01T09:00:00+09:00","container_config":{"Hostname":"","Domainname":"","User":"","AttachStdin":false,"AttachStdout":false,"AttachStderr":false,"Tty":false,"OpenStdin":false,"StdinOnce":false,"Env":null,"Cmd":null,"Image":"","Volumes":null,"WorkingDir":"","Entrypoint":null,"OnBuild":null,"Labels":null},"os":"linux"}

image/blobs/sha256/ec0a7658dda53cfa39e9a61f8506fce49e56b206fee8ab604f17daa0a8f42f43

@@ -0,0 +1 @@
+{"schemaVersion":2,"mediaType":"application/vnd.oci.image.manifest.v1+json","config":{"mediaType":"application/vnd.oci.image.config.v1+json","digest":"sha256:bc34115d8ab3da3bb2818457b8bc493ed594968b9ae39c48e30c22b37294974b","size":14594},"layers":[{"mediaType":"application/vnd.oci.image.layer.v1.tar","digest":"sha256:ea16cace89338c84eb6bcb91a7efdfcae6838fff359efe951858227436486c34","size":80641024},{"mediaType":"application/vnd.oci.image.layer.v1.tar","digest":"sha256:e64f34e6f7fc770276a58ce68b18755913e8f6a38b80274ed30b138ba6db8cef","size":204086784},{"mediaType":"application/vnd.oci.image.layer.v1.tar","digest":"sha256:3f6ad10d5ba93a17af361223d9d9cd8c53c34d4156cd69af1e9b9af1bebcd4bb","size":352256},{"mediaType":"application/vnd.oci.image.layer.v1.tar","digest":"sha256:5b60873863c0500f341a4c237ab1c18b929b3ac1571a2808b87fcacfdd1f2168","size":33464832},{"mediaType":"application/vnd.oci.image.layer.v1.tar","digest":"sha256:0311ceefbd13017ef17eb9844fbfd15250b1bb099d408687eef4372ad2c65ee5","size":454126592},{"mediaType":"application/vnd.oci.image.layer.v1.tar","digest":"sha256:661b4446add5adf8c9d4eaab02ab1fe419f17114b8c7a7a381707be7973467e5","size":187652608},{"mediaType":"application/vnd.oci.image.layer.v1.tar","digest":"sha256:2f1823fe031c57258362a517aa11e3c4a55eccca738913d0cc87587ccfb6fd1a","size":1099616768},{"mediaType":"application/vnd.oci.image.layer.v1.tar","digest":"sha256:e1272a58679febd696c70d77b53ffeb43cba62b4ddb3d97dd4e632c6848d2856","size":2048},{"mediaType":"application/vnd.oci.image.layer.v1.tar","digest":"sha256:d1a9879f27dd477b46fe73ced95ea58803d70fc7c2bb42b875c605b8fc744e14","size":454599168},{"mediaType":"application/vnd.oci.image.layer.v1.tar","digest":"sha256:184743fecd7fc3deb4e6f457a2a602932fe813f75148eec4b6e75ff88a9f2be2","size":130853888},{"mediaType":"application/vnd.oci.image.layer.v1.tar","digest":"sha256:59b37b1999541425b0a7cda1ce01f95f1be228fa97cd09f6631acfb2fab990fd","size":130853888},{"mediaType":"application/vnd.oci.image.layer.v1.tar","digest":"sha256:ca297dc8905fb0b70eebf7e9c58d50df4d8d38c5ece2cc866c22798cfc2dfd01","size":191707136},{"mediaType":"application/vnd.oci.image.layer.v1.tar","digest":"sha256:4f4c468252314917a1847056077663ed71f3abc1dc386c03bca471690a23c83b","size":65318400},{"mediaType":"application/vnd.oci.image.layer.v1.tar","digest":"sha256:c94435d8cb1f57c0296304800fe6e7376c511a75484d5da38e0eceb4a8cba02d","size":4678144},{"mediaType":"application/vnd.oci.image.layer.v1.tar","digest":"sha256:639c9dd1747f941795c619f445de593cc6d7957060442b3dbe0148bd1be1eeb3","size":23040},{"mediaType":"application/vnd.oci.image.layer.v1.tar","digest":"sha256:082a9ded324a46a5cc7fe15bde1b9d83c03804f0f84eac7ce6bb51229b894f1f","size":8888832},{"mediaType":"application/vnd.oci.image.layer.v1.tar","digest":"sha256:8529443804912d6639846836d1b7ab793de42b54e3b4058e14a059f0d5fe4ade","size":4608},{"mediaType":"application/vnd.oci.image.layer.v1.tar","digest":"sha256:96f3b28657c091e4818e2407f66db33853828fdb8abef3177cdf5c8d525f7dbf","size":4608},{"mediaType":"application/vnd.oci.image.layer.v1.tar","digest":"sha256:5f70bf18a086007016e948b04aed3b82103a36bea41755b6cddfaf10ace3c6ef","size":1024},{"mediaType":"application/vnd.oci.image.layer.v1.tar","digest":"sha256:c0d475ded7d3e60cdc094385f92036f940b605a0438578608e031030a8819ae9","size":4096},{"mediaType":"application/vnd.oci.image.layer.v1.tar","digest":"sha256:53deb2ae079afabdafec52b959453dccca54d8c7eb6315862fcab21d186bcbdd","size":1162240},{"mediaType":"application/vnd.oci.image.layer.v1.tar","digest":"sha256:5116d36f79d2da6aa08f8ffcff866e39d7c0f5be5bbf3d74c6e8c8caebf2c5fd","size":39936}]}

image/blobs/sha256/fd7308f35b11a131ac41df34684d400e1400055c0e48edb78f7f3292ce84db55

@@ -0,0 +1 @@
+{"id":"b58135a7607f916cd7221128976d64d0f39fc917e1ba780df8efe3e2c4c7d23a","parent":"f3d345315bfbedaf1b9e3924e84e2a596382cae91d99fac771a0d023a3f0d356","created":"1970-01-01T09:00:00+09:00","container_config":{"Hostname":"","Domainname":"","User":"","AttachStdin":false,"AttachStdout":false,"AttachStderr":false,"Tty":false,"OpenStdin":false,"StdinOnce":false,"Env":null,"Cmd":null,"Image":"","Volumes":null,"WorkingDir":"","Entrypoint":null,"OnBuild":null,"Labels":null},"os":"linux"}

image/blobs/sha256/fff7562e27cd6bcd475d31d119da76f39d239cfc59ab03ff6579a8f10ca6e05c

@@ -0,0 +1 @@
+{"id":"155ab0b6484fdeee7919c43f2a38a56151bd05a8b48891fa53929c0e45577105","parent":"12e1b52d50de6740b0ae582760feedf780e2affcaa1c5d15bef6df3f7da931d4","created":"1970-01-01T09:00:00+09:00","container_config":{"Hostname":"","Domainname":"","User":"","AttachStdin":false,"AttachStdout":false,"AttachStderr":false,"Tty":false,"OpenStdin":false,"StdinOnce":false,"Env":null,"Cmd":null,"Image":"","Volumes":null,"WorkingDir":"","Entrypoint":null,"OnBuild":null,"Labels":null},"os":"linux"}

image/index.json

@@ -0,0 +1 @@
+{"schemaVersion":2,"mediaType":"application/vnd.oci.image.index.v1+json","manifests":[{"mediaType":"application/vnd.oci.image.manifest.v1+json","digest":"sha256:ec0a7658dda53cfa39e9a61f8506fce49e56b206fee8ab604f17daa0a8f42f43","size":3615,"annotations":{"io.containerd.image.name":"docker.io/tt-model/locate-anything-3b-blackhole:bc34115d8ab3","org.opencontainers.image.ref.name":"bc34115d8ab3"}}]}

image/manifest.json

@@ -0,0 +1 @@
+[{"Config":"blobs/sha256/bc34115d8ab3da3bb2818457b8bc493ed594968b9ae39c48e30c22b37294974b","RepoTags":["tt-model/locate-anything-3b-blackhole:bc34115d8ab3"],"Layers":["blobs/sha256/ea16cace89338c84eb6bcb91a7efdfcae6838fff359efe951858227436486c34","blobs/sha256/e64f34e6f7fc770276a58ce68b18755913e8f6a38b80274ed30b138ba6db8cef","blobs/sha256/3f6ad10d5ba93a17af361223d9d9cd8c53c34d4156cd69af1e9b9af1bebcd4bb","blobs/sha256/5b60873863c0500f341a4c237ab1c18b929b3ac1571a2808b87fcacfdd1f2168","blobs/sha256/0311ceefbd13017ef17eb9844fbfd15250b1bb099d408687eef4372ad2c65ee5","blobs/sha256/661b4446add5adf8c9d4eaab02ab1fe419f17114b8c7a7a381707be7973467e5","blobs/sha256/2f1823fe031c57258362a517aa11e3c4a55eccca738913d0cc87587ccfb6fd1a","blobs/sha256/e1272a58679febd696c70d77b53ffeb43cba62b4ddb3d97dd4e632c6848d2856","blobs/sha256/d1a9879f27dd477b46fe73ced95ea58803d70fc7c2bb42b875c605b8fc744e14","blobs/sha256/184743fecd7fc3deb4e6f457a2a602932fe813f75148eec4b6e75ff88a9f2be2","blobs/sha256/59b37b1999541425b0a7cda1ce01f95f1be228fa97cd09f6631acfb2fab990fd","blobs/sha256/ca297dc8905fb0b70eebf7e9c58d50df4d8d38c5ece2cc866c22798cfc2dfd01","blobs/sha256/4f4c468252314917a1847056077663ed71f3abc1dc386c03bca471690a23c83b","blobs/sha256/c94435d8cb1f57c0296304800fe6e7376c511a75484d5da38e0eceb4a8cba02d","blobs/sha256/639c9dd1747f941795c619f445de593cc6d7957060442b3dbe0148bd1be1eeb3","blobs/sha256/082a9ded324a46a5cc7fe15bde1b9d83c03804f0f84eac7ce6bb51229b894f1f","blobs/sha256/8529443804912d6639846836d1b7ab793de42b54e3b4058e14a059f0d5fe4ade","blobs/sha256/96f3b28657c091e4818e2407f66db33853828fdb8abef3177cdf5c8d525f7dbf","blobs/sha256/5f70bf18a086007016e948b04aed3b82103a36bea41755b6cddfaf10ace3c6ef","blobs/sha256/c0d475ded7d3e60cdc094385f92036f940b605a0438578608e031030a8819ae9","blobs/sha256/53deb2ae079afabdafec52b959453dccca54d8c7eb6315862fcab21d186bcbdd","blobs/sha256/5116d36f79d2da6aa08f8ffcff866e39d7c0f5be5bbf3d74c6e8c8caebf2c5fd"],"LayerSources":{"sha256:0311ceefbd13017ef17eb9844fbfd15250b1bb099d408687eef4372ad2c65ee5":{"mediaType":"application/vnd.oci.image.layer.v1.tar","size":454126592,"digest":"sha256:0311ceefbd13017ef17eb9844fbfd15250b1bb099d408687eef4372ad2c65ee5"},"sha256:082a9ded324a46a5cc7fe15bde1b9d83c03804f0f84eac7ce6bb51229b894f1f":{"mediaType":"application/vnd.oci.image.layer.v1.tar","size":8888832,"digest":"sha256:082a9ded324a46a5cc7fe15bde1b9d83c03804f0f84eac7ce6bb51229b894f1f"},"sha256:184743fecd7fc3deb4e6f457a2a602932fe813f75148eec4b6e75ff88a9f2be2":{"mediaType":"application/vnd.oci.image.layer.v1.tar","size":130853888,"digest":"sha256:184743fecd7fc3deb4e6f457a2a602932fe813f75148eec4b6e75ff88a9f2be2"},"sha256:2f1823fe031c57258362a517aa11e3c4a55eccca738913d0cc87587ccfb6fd1a":{"mediaType":"application/vnd.oci.image.layer.v1.tar","size":1099616768,"digest":"sha256:2f1823fe031c57258362a517aa11e3c4a55eccca738913d0cc87587ccfb6fd1a"},"sha256:3f6ad10d5ba93a17af361223d9d9cd8c53c34d4156cd69af1e9b9af1bebcd4bb":{"mediaType":"application/vnd.oci.image.layer.v1.tar","size":352256,"digest":"sha256:3f6ad10d5ba93a17af361223d9d9cd8c53c34d4156cd69af1e9b9af1bebcd4bb"},"sha256:4f4c468252314917a1847056077663ed71f3abc1dc386c03bca471690a23c83b":{"mediaType":"application/vnd.oci.image.layer.v1.tar","size":65318400,"digest":"sha256:4f4c468252314917a1847056077663ed71f3abc1dc386c03bca471690a23c83b"},"sha256:5116d36f79d2da6aa08f8ffcff866e39d7c0f5be5bbf3d74c6e8c8caebf2c5fd":{"mediaType":"application/vnd.oci.image.layer.v1.tar","size":39936,"digest":"sha256:5116d36f79d2da6aa08f8ffcff866e39d7c0f5be5bbf3d74c6e8c8caebf2c5fd"},"sha256:53deb2ae079afabdafec52b959453dccca54d8c7eb6315862fcab21d186bcbdd":{"mediaType":"application/vnd.oci.image.layer.v1.tar","size":1162240,"digest":"sha256:53deb2ae079afabdafec52b959453dccca54d8c7eb6315862fcab21d186bcbdd"},"sha256:59b37b1999541425b0a7cda1ce01f95f1be228fa97cd09f6631acfb2fab990fd":{"mediaType":"application/vnd.oci.image.layer.v1.tar","size":130853888,"digest":"sha256:59b37b1999541425b0a7cda1ce01f95f1be228fa97cd09f6631acfb2fab990fd"},"sha256:5b60873863c0500f341a4c237ab1c18b929b3ac1571a2808b87fcacfdd1f2168":{"mediaType":"application/vnd.oci.image.layer.v1.tar","size":33464832,"digest":"sha256:5b60873863c0500f341a4c237ab1c18b929b3ac1571a2808b87fcacfdd1f2168"},"sha256:5f70bf18a086007016e948b04aed3b82103a36bea41755b6cddfaf10ace3c6ef":{"mediaType":"application/vnd.oci.image.layer.v1.tar","size":1024,"digest":"sha256:5f70bf18a086007016e948b04aed3b82103a36bea41755b6cddfaf10ace3c6ef"},"sha256:639c9dd1747f941795c619f445de593cc6d7957060442b3dbe0148bd1be1eeb3":{"mediaType":"application/vnd.oci.image.layer.v1.tar","size":23040,"digest":"sha256:639c9dd1747f941795c619f445de593cc6d7957060442b3dbe0148bd1be1eeb3"},"sha256:661b4446add5adf8c9d4eaab02ab1fe419f17114b8c7a7a381707be7973467e5":{"mediaType":"application/vnd.oci.image.layer.v1.tar","size":187652608,"digest":"sha256:661b4446add5adf8c9d4eaab02ab1fe419f17114b8c7a7a381707be7973467e5"},"sha256:8529443804912d6639846836d1b7ab793de42b54e3b4058e14a059f0d5fe4ade":{"mediaType":"application/vnd.oci.image.layer.v1.tar","size":4608,"digest":"sha256:8529443804912d6639846836d1b7ab793de42b54e3b4058e14a059f0d5fe4ade"},"sha256:96f3b28657c091e4818e2407f66db33853828fdb8abef3177cdf5c8d525f7dbf":{"mediaType":"application/vnd.oci.image.layer.v1.tar","size":4608,"digest":"sha256:96f3b28657c091e4818e2407f66db33853828fdb8abef3177cdf5c8d525f7dbf"},"sha256:c0d475ded7d3e60cdc094385f92036f940b605a0438578608e031030a8819ae9":{"mediaType":"application/vnd.oci.image.layer.v1.tar","size":4096,"digest":"sha256:c0d475ded7d3e60cdc094385f92036f940b605a0438578608e031030a8819ae9"},"sha256:c94435d8cb1f57c0296304800fe6e7376c511a75484d5da38e0eceb4a8cba02d":{"mediaType":"application/vnd.oci.image.layer.v1.tar","size":4678144,"digest":"sha256:c94435d8cb1f57c0296304800fe6e7376c511a75484d5da38e0eceb4a8cba02d"},"sha256:ca297dc8905fb0b70eebf7e9c58d50df4d8d38c5ece2cc866c22798cfc2dfd01":{"mediaType":"application/vnd.oci.image.layer.v1.tar","size":191707136,"digest":"sha256:ca297dc8905fb0b70eebf7e9c58d50df4d8d38c5ece2cc866c22798cfc2dfd01"},"sha256:d1a9879f27dd477b46fe73ced95ea58803d70fc7c2bb42b875c605b8fc744e14":{"mediaType":"application/vnd.oci.image.layer.v1.tar","size":454599168,"digest":"sha256:d1a9879f27dd477b46fe73ced95ea58803d70fc7c2bb42b875c605b8fc744e14"},"sha256:e1272a58679febd696c70d77b53ffeb43cba62b4ddb3d97dd4e632c6848d2856":{"mediaType":"application/vnd.oci.image.layer.v1.tar","size":2048,"digest":"sha256:e1272a58679febd696c70d77b53ffeb43cba62b4ddb3d97dd4e632c6848d2856"},"sha256:e64f34e6f7fc770276a58ce68b18755913e8f6a38b80274ed30b138ba6db8cef":{"mediaType":"application/vnd.oci.image.layer.v1.tar","size":204086784,"digest":"sha256:e64f34e6f7fc770276a58ce68b18755913e8f6a38b80274ed30b138ba6db8cef"},"sha256:ea16cace89338c84eb6bcb91a7efdfcae6838fff359efe951858227436486c34":{"mediaType":"application/vnd.oci.image.layer.v1.tar","size":80641024,"digest":"sha256:ea16cace89338c84eb6bcb91a7efdfcae6838fff359efe951858227436486c34"}}}]

image/oci-layout

@@ -0,0 +1 @@
+{"imageLayoutVersion": "1.0.0"}

image/repositories

@@ -0,0 +1 @@
+{"tt-model/locate-anything-3b-blackhole":{"bc34115d8ab3":"5116d36f79d2da6aa08f8ffcff866e39d7c0f5be5bbf3d74c6e8c8caebf2c5fd"}}

requirements.lock

@@ -0,0 +1,86 @@
+annotated-doc==0.0.5
+annotated-types==0.8.0
+anyio==4.15.1
+certifi==2026.7.22
+cfgv==3.5.0
+charset-normalizer==3.5.1
+click==8.5.0
+contourpy==1.3.3
+cycler==0.12.1
+distlib==0.4.3
+distro==1.9.0
+elastic-transport==9.4.2
+elasticsearch==9.5.1
+fastapi==0.141.1
+filelock==3.32.6
+fonttools==4.65.0
+fsspec==2026.7.0
+graphviz==0.21
+h11==0.16.0
+hf-xet==1.6.0
+httpcore==1.0.9
+httpx==0.28.1
+huggingface_hub==1.31.0
+identify==2.6.19
+idna==3.19
+iniconfig==2.3.0
+Jinja2==3.1.6
+kiwisolver==1.5.1
+linkify-it-py==2.2.0
+loguru==0.7.3
+markdown-it-py==4.2.0
+MarkupSafe==3.0.3
+matplotlib==3.11.2
+mdit-py-plugins==0.6.1
+mdurl==0.1.2
+ml_dtypes==0.5.4
+mpmath==1.3.0
+networkx==3.6.1
+nodeenv==1.10.0
+numpy==1.26.4
+packaging==26.3
+pandas==3.0.5
+pillow==12.3.0
+platformdirs==4.11.8
+pluggy==1.6.0
+pre_commit==4.6.2
+psutil==7.2.2
+pydantic==2.13.5
+pydantic_core==2.46.5
+Pygments==2.21.0
+pyluwen==0.9.0
+pyparsing==3.3.2
+pytest==9.1.1
+python-dateutil==2.9.0.post0
+python-discovery==1.6.0
+PyYAML==6.0.3
+regex==2026.9.10
+requests==2.34.2
+rich==15.0.0
+safetensors==0.8.0
+seaborn==0.13.2
+setuptools==80.10.2
+setuptools-scm==8.1.0
+shellingham==1.5.4
+six==1.17.0
+sniffio==1.3.1
+starlette==1.6.0
+sympy==1.14.0
+textual==8.2.8
+tokenizers==0.22.2
+tomli==2.4.1
+torch==2.11.0+cpu
+tqdm==4.70.1
+transformers==5.12.1
+tt-smi==6.5.0
+tt-tools-common==1.6.0
+tt-umd==0.9.10
+ttnn==0.65.2.dev9100
+ttnn==0.75.0rc10.dev657+g8b98410e730
+typer==0.27.2
+typing-inspection==0.4.4
+typing_extensions==4.16.0
+urllib3==2.7.0
+uvicorn==0.52.4
+virtualenv==21.7.9
+wheel==0.48.0

tt_kernel_manifest.json

@@ -0,0 +1,120 @@
+{
+  "schema_version": "5.1",
+  "name": "locate-anything-3b-blackhole",
+  "tt_metal_version": "0.65.2.dev9100",
+  "arch": "blackhole",
+  "device_count": 1,
+  "producer": {
+    "tt_kernel_version": "0.1.0",
+    "created_at": "2026-09-12T06:59:49.276222+00:00",
+    "hostname": "deepgadget"
+  },
+  "weights": {
+    "repo_id": "nvidia/LocateAnything-3B",
+    "revision": "c32291ca5e996f5a7a485845b4f57a233936bba0",
+    "allow_patterns": null,
+    "ignore_patterns": [
+      "assets/*",
+      "training_args.bin"
+    ],
+    "repo_type": "model"
+  },
+  "mesh": null,
+  "entrypoint": null,
+  "resources": null,
+  "capabilities": null,
+  "env": {},
+  "bundled": null,
+  "deps": null,
+  "container": {
+    "image": {
+      "registry": "hf",
+      "repository": "locate-anything-3b-blackhole",
+      "tag": "tt-model/locate-anything-3b-blackhole:bc34115d8ab3",
+      "digest": "sha256:bc34115d8ab3da3bb2818457b8bc493ed594968b9ae39c48e30c22b37294974b"
+    },
+    "kind": "tt-dit-server",
+    "runtime": {
+      "app": "locate_anything.server.app:app",
+      "mesh_shape_env": "TT_MESH_SHAPE",
+      "packages": [
+        "numpy>=1.24.4,<2",
+        "transformers==5.12.1",
+        "safetensors",
+        "huggingface_hub",
+        "loguru",
+        "tqdm",
+        "pytest",
+        "pyyaml"
+      ],
+      "lock": "requirements.lock"
+    },
+    "serve": {
+      "hardware": "p150",
+      "mesh_device": "P150",
+      "port": 20000,
+      "max_model_len": null,
+      "max_num_seqs": null,
+      "block_size": null,
+      "server_timeout": null,
+      "capabilities": null,
+      "additional_config": {},
+      "args": [],
+      "env": {
+        "TT_WEIGHTS_REVISION": "c32291ca5e996f5a7a485845b4f57a233936bba0",
+        "TT_METAL_VISIBLE_DEVICES": "0",
+        "LA_IN_TOKEN_LIMIT": "1024",
+        "LA_PREC": "accuracy",
+        "LA_TRACE": "1",
+        "LA_WARMUP_RUNS": "2"
+      }
+    },
+    "serve_profiles": [
+      {
+        "hardware": null,
+        "mesh_device": null,
+        "port": null,
+        "max_model_len": null,
+        "max_num_seqs": null,
+        "block_size": null,
+        "server_timeout": null,
+        "capabilities": null,
+        "additional_config": {},
+        "args": [],
+        "env": {},
+        "name": "default",
+        "description": null
+      }
+    ],
+    "default_profile": null,
+    "code_dir": "code",
+    "verify": [
+      "import locate_anything.server.app as a; assert a.app",
+      "import locate_anything.tt.pipeline, locate_anything.tt.vision, locate_anything.tt.model_la, locate_anything.reference.la_inputs, locate_anything.reference.extract_llm_checkpoint",
+      "import models.tt_transformers.tt.model_config, models.tt_transformers.tt.generator, models.demos.qwen25_vl.tt.common, models.common.utility_functions",
+      "import transformers; assert transformers.__version__ == '5.12.1', transformers.__version__",
+      "import safetensors, huggingface_hub, loguru, tqdm, pytest, yaml, numpy; assert numpy.__version__.startswith('1.'), numpy.__version__",
+      "from pathlib import Path; assert Path('/opt/tt-metal/models/tt_transformers/tt/prefetcher/prefetcher_config.yaml').is_file()",
+      "from pathlib import Path; assert Path('/opt/tt-metal/locate_anything/server/smoke_test.py').is_file()",
+      "from pathlib import Path; assert 'spill_ff1_3' in Path('/opt/tt-metal/models/tt_transformers/tt/mlp.py').read_text(), 'mlp.py overlay (decode w1/w3 DRAM spill) missing'"
+    ],
+    "built": {
+      "image": "tt-model/locate-anything-3b-blackhole:bc34115d8ab3",
+      "repo": "changh95/locate-anything-3b-blackhole",
+      "tt_model_version": "0.1.0",
+      "created_at": "2026-09-12T06:57:00+00:00",
+      "tt_metal": {
+        "sha": "8b98410e730bb504fea43a88609756e34821d91d",
+        "describe": "v0.78.0-dev20260820-25-g8b98410e73",
+        "dirty": false,
+        "scm_version": "0.65.2.dev9100",
+        "mode": "local",
+        "remote": "https://github.com/tenstorrent/tt-metal.git",
+        "branch": "main",
+        "pushed": true
+      },
+      "code_sha256": "eb3aa8433050cc460c7a398efeb03b287d79aec9d177cda4a18583f71484065a",
+      "image_digest": "sha256:bc34115d8ab3da3bb2818457b8bc493ed594968b9ae39c48e30c22b37294974b"
+    }
+  }
+}