tt-model authoring files (fused default, megakernel pass)
Browse files- SERVING.md +20 -0
SERVING.md
CHANGED
|
@@ -218,3 +218,23 @@ once per grid).
|
|
| 218 |
`to_tensor`/`normalize` in torch (bit-identical, checked offline).
|
| 219 |
8. The tree venv is Python 3.10; the image is 3.12 (uv dry-run of the exact package set on 3.12
|
| 220 |
resolves to torch 2.11.0+cpu / numpy 1.26.4 / transformers 5.12.1, no torchvision).
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 218 |
`to_tensor`/`normalize` in torch (bit-identical, checked offline).
|
| 219 |
8. The tree venv is Python 3.10; the image is 3.12 (uv dry-run of the exact package set on 3.12
|
| 220 |
resolves to torch 2.11.0+cpu / numpy 1.26.4 / transformers 5.12.1, no torchvision).
|
| 221 |
+
9. **`TT_FUSED` fused device paths (default ON since the 2026-09-13 device validation).**
|
| 222 |
+
Branch `opt/locate-anything-3b-p150-megakernel` adds fused device paths behind one env knob
|
| 223 |
+
read once at startup (`code/locate_anything/tt/fused.py`): the MoonViT graph as one metal
|
| 224 |
+
trace with the patch merger on device and a row-major pixel upload, the vision->LLM embedding
|
| 225 |
+
merge on device, a traced 36-layer prefill with an eager norm/LM-head tail, a one-row
|
| 226 |
+
first-token readback and on-device greedy sampling in the decode loop. The default fused
|
| 227 |
+
configuration keeps the legacy vision numerics bit for bit (device-verified) and measured
|
| 228 |
+
360.9 -> 304.9 ms per demo request in the host pipeline (see `DEVICE_VALIDATION.md`
|
| 229 |
+
"Results" for the served numbers). `TT_FUSED=0` = the 2026-09-12 shipped device graph (same
|
| 230 |
+
ops, same order, same arguments, same numerics); the only host-visible additions are the
|
| 231 |
+
`"fused"` key in the pipeline `run()` result dict and the `fused` entry of `GET /info` -- the
|
| 232 |
+
`/predict` body is unchanged. With the path on, `LA_TRACE_REGION_SIZE` defaults to 160000000
|
| 233 |
+
and the warm-up must capture the vision and prefill traces before READY (startup error
|
| 234 |
+
otherwise); `/info` reports `fused.config` / `fused.status` and `timing_ms.vision` becomes
|
| 235 |
+
the non-blocking trace submit (compare `total`). Measured opt-ins that change the bf16
|
| 236 |
+
rounding of the vision tower (not default; numbers in `DEVICE_VALIDATION.md`):
|
| 237 |
+
`LA_FUSED_EXACT_SEQ=1 LA_FUSED_SDPA_CHUNKS=96,352 LA_FUSED_SDPA_EXP_APPROX=0` (another
|
| 238 |
+
-13 ms; the demo box becomes the HF reference's `<282><414><607><797>` instead of the shipped
|
| 239 |
+
`<606><794>`) and `LA_FUSED_MATMUL=minimal` (below the 0.99 PCC gate on the port golden).
|
| 240 |
+
Host tests: `code/locate_anything/tests/test_fused_host.py`.
|