changh95 commited on
Commit
92ecbff
·
verified ·
1 Parent(s): 970bd09

tt-model authoring files (fused default, megakernel pass)

Browse files
Files changed (1) hide show
  1. SERVING.md +20 -0
SERVING.md CHANGED
@@ -218,3 +218,23 @@ once per grid).
218
  `to_tensor`/`normalize` in torch (bit-identical, checked offline).
219
  8. The tree venv is Python 3.10; the image is 3.12 (uv dry-run of the exact package set on 3.12
220
  resolves to torch 2.11.0+cpu / numpy 1.26.4 / transformers 5.12.1, no torchvision).
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
218
  `to_tensor`/`normalize` in torch (bit-identical, checked offline).
219
  8. The tree venv is Python 3.10; the image is 3.12 (uv dry-run of the exact package set on 3.12
220
  resolves to torch 2.11.0+cpu / numpy 1.26.4 / transformers 5.12.1, no torchvision).
221
+ 9. **`TT_FUSED` fused device paths (default ON since the 2026-09-13 device validation).**
222
+ Branch `opt/locate-anything-3b-p150-megakernel` adds fused device paths behind one env knob
223
+ read once at startup (`code/locate_anything/tt/fused.py`): the MoonViT graph as one metal
224
+ trace with the patch merger on device and a row-major pixel upload, the vision->LLM embedding
225
+ merge on device, a traced 36-layer prefill with an eager norm/LM-head tail, a one-row
226
+ first-token readback and on-device greedy sampling in the decode loop. The default fused
227
+ configuration keeps the legacy vision numerics bit for bit (device-verified) and measured
228
+ 360.9 -> 304.9 ms per demo request in the host pipeline (see `DEVICE_VALIDATION.md`
229
+ "Results" for the served numbers). `TT_FUSED=0` = the 2026-09-12 shipped device graph (same
230
+ ops, same order, same arguments, same numerics); the only host-visible additions are the
231
+ `"fused"` key in the pipeline `run()` result dict and the `fused` entry of `GET /info` -- the
232
+ `/predict` body is unchanged. With the path on, `LA_TRACE_REGION_SIZE` defaults to 160000000
233
+ and the warm-up must capture the vision and prefill traces before READY (startup error
234
+ otherwise); `/info` reports `fused.config` / `fused.status` and `timing_ms.vision` becomes
235
+ the non-blocking trace submit (compare `total`). Measured opt-ins that change the bf16
236
+ rounding of the vision tower (not default; numbers in `DEVICE_VALIDATION.md`):
237
+ `LA_FUSED_EXACT_SEQ=1 LA_FUSED_SDPA_CHUNKS=96,352 LA_FUSED_SDPA_EXP_APPROX=0` (another
238
+ -13 ms; the demo box becomes the HF reference's `<282><414><607><797>` instead of the shipped
239
+ `<606><794>`) and `LA_FUSED_MATMUL=minimal` (below the 0.99 PCC gate on the port golden).
240
+ Host tests: `code/locate_anything/tests/test_fused_host.py`.