tt-model authoring files (fused default, megakernel pass)

tt-model.yaml

@@ -75,6 +75,9 @@ serve:
     LA_PREC: "accuracy"
     LA_TRACE: "1"
     LA_WARMUP_RUNS: "2"
+    # Fused device paths (code default since the 2026-09-13 device validation; explicit here so the
+    # served configuration is visible in `tt-model info`). "0" = the 2026-09-12 legacy graph.
+    TT_FUSED: "1"
 
 # Build-time assertions, run INSIDE the finished image as uid 1000 with no device and no weights.
 verify:
@@ -114,8 +117,8 @@ card:
     {"query": "car", "width": 1920, "height": 1080, "canonical_size": [616, 336], "grid_hw": [24, 44],
      "raw_text": "<ref>car</ref><box><282><414><606><794></box><|im_end|>",
      "detections": [{"label": "car", "box": [541.44, 447.12, 1163.52, 857.52], "box_norm": [282, 414, 606, 794]}],
-     "points": [], "num_generated_tokens": 10, "stopped_on_eos": true, "decode_mode": "ar_greedy_trace",
-     "timing_ms": {"vision": 47.4, "prefill": 55.8, "decode": 224.0, "total": 343.4, "decode_tok_s": 40.17}}
+     "points": [], "num_generated_tokens": 10, "stopped_on_eos": true, "decode_mode": "ar_greedy_trace_device_sampling",
+     "timing_ms": {"vision": 1.4, "prefill": 86.3, "decode": 202.1, "total": 303.9, "decode_tok_s": 44.53}}
     ```
 
     - `box` is `[x1, y1, x2, y2]` in original image pixels; `box_norm` is the model's own 0..1000 output over the squashed `canonical_size` view. `points` holds 2-coordinate outputs the same way.
@@ -131,15 +134,15 @@ card:
 
     | Metric | Value |
     |---|---:|
-    | Full on-device logits PCC vs torch-CPU reference (vision → LLM) | 0.9928 (gate ≥ 0.99; vision projector 0.9911, LLM prefill 0.9922) |
-    | Decoded box string on the demo image | matches the HF reference |
-    | `/predict` served over HTTP (warm, batch 1, `car`, 10 tokens) | ~343 ms server-side (vision 47 · prefill 56 · decode 224 ms, ~40 tok/s) · ~0.38 s wall |
+    | Full on-device logits PCC vs torch-CPU reference (vision → LLM) | 0.9919 (gate ≥ 0.99; vision projector 0.9924, LLM prefill 0.9915) — measured 2026-09-13 on the p150a against goldens regenerated with the port's torch 2.7.1 / transformers 4.53.0; identical with `TT_FUSED=0` (the fused default keeps the vision numerics bit for bit) |
+    | Decoded box string on the demo image | `<ref>car</ref><box><282><414><606><794></box>`, unchanged from the 2026-09-12 package (the HF torch-CPU reference generated with the port's torch 2.7.1 / transformers 4.53.0 gives `<282><414><607><797>`, IoU 0.989) |
+    | `/predict` served over HTTP (warm, batch 1, `car`, 10 tokens; median of 100 requests, 2026-09-13) | **~305 ms** server-side (304–306; vision-trace submit 2 · prefill incl. the vision wait 86 · decode 202 ms, ~44.5 tok/s) · ~0.34 s wall. `TT_FUSED=0` (the 2026-09-12 graph): ~357 ms (351–365; vision 48 · prefill 62 · decode 232 ms, ~39 tok/s) · ~0.40 s wall |
 
     ### Caveats
 
     - One image + one query per request, batch 1, requests are serialized. Every image is squash-resized onto a fixed 24×44-patch grid (616×336, 16:9; `LA_IN_TOKEN_LIMIT=1024`, upstream default 25600): non-16:9 images are distorted before the model sees them, boxes still map back to original pixels.
-    - bf16 vision, BF16 attention + BFP8 MLP LLM; the package ships an `mlp.py` overlay (`code/models/tt_transformers/tt/mlp.py`, decode `w1`/`w3` spilled to DRAM) so the LLM fits L1 on one p150a. Only greedy AR decode is served; the experimental Parallel Box Decoding (MTP) is not.
-    - Weights are public and ungated but under NVIDIA's own license (not OSI); ~7.7 GB download, and the first boot extracts a 6.8 GB Qwen2.5-3B checkpoint, converts it to BFP8 and JIT-compiles (~100 s cold, ~25 s afterwards).
+    - bf16 vision, BF16 attention + BFP8 MLP LLM; the package ships an `mlp.py` overlay (`code/models/tt_transformers/tt/mlp.py`, decode `w1`/`w3` spilled to DRAM) so the LLM fits L1 on one p150a. Fused device paths (MoonViT as one metal trace with the patch merger on device, device vision→LLM merge, traced prefill, one-row first token, on-device greedy argmax) are on by default since 2026-09-13; `TT_FUSED=0` restores the 2026-09-12 graph, and the vision numerics are unchanged bit for bit either way. Only greedy AR decode is served; the experimental Parallel Box Decoding (MTP) is not.
+    - Weights are public and ungated but under NVIDIA's own license (not OSI); ~7.7 GB download, and the first boot extracts a 6.8 GB Qwen2.5-3B checkpoint, converts it to BFP8 and JIT-compiles (~100 s on the very first boot, ~50 s with cached weights but an empty kernel cache, ~17 s warm; measured 2026-09-13).
     - Not an OpenAI-compatible API; `GET /v1/models` is a stub so the tt-model ready card does not 404.
     - Validated on tt-metal `v0.78.0-dev20260820` (main `8b98410e730`), single p150a only.