gaze-lle-p150

Gaze-LLE (the gazelle_dinov2_vitb14_inout checkpoint: DINOv2 ViT-B/14 backbone plus gaze decoder) running entirely on one Tenstorrent Blackhole p150a via tt-nn: one image plus N head boxes in, N 64×64 gaze heatmaps and in-frame scores out. Weights: changh95/gaze-lle-weights · Paper: arXiv:2412.09586 · Upstream code: fkryan/gazelle · Port: changh95/tt-Gaze-LLE

Runs on p150 (mesh P150).

Packaged and published with tt-model-manager 0.1.0 (manifest schema 5.1).

Quickstart

tt-model pull  changh95/gaze-lle-p150 --with-weights
tt-model serve changh95/gaze-lle-p150

• Weights changh95/gaze-lle-weights at f83e53f0f17d go to your HF cache; the image does not contain them.
• Serves on port 20000 (or the next free port); ready when the log says Application startup complete.

Run with tt-cli

tt serve changh95/gaze-lle-p150
printf '{"image":"%s","bboxes":[[116,28,176,104],[269,37,301,81],[87,57,107,76]]}' "$(base64 -w0 media/source_1.png)" > req.json
curl -s localhost:20000/predict -H 'Content-Type: application/json' -d @req.json
tt model stop changh95/gaze-lle-p150

• POST /predict: image (base64 PNG/JPEG), bboxes (1–10 head boxes [x1,y1,x2,y2]); optional bbox_format (pixels | normalized), heatmap_format (png | npz | json | none).
• GET /health, GET /info.

Response

{"model": "gaze-lle-p150", "image_size": [500, 334], "input_size": [448, 448], "num_heads": 3,
 "heatmap_size": [64, 64], "heatmap_format": "png",
 "predictions": [
   {"bbox": [116.0, 28.0, 176.0, 104.0], "bbox_normalized": [0.232, 0.083832, 0.352, 0.311377], "inout": 0.025,
    "gaze_target": {"x": 289.1, "y": 172.2, "x_norm": 0.578, "y_norm": 0.516, "heatmap_row": 33, "heatmap_col": 37, "score": 0.373},
    "heatmap_png": "..."},
   {"bbox": [269.0, 37.0, 301.0, 81.0], "inout": 0.777,
    "gaze_target": {"x": 296.9, "y": 99.2, "heatmap_row": 19, "heatmap_col": 38, "score": 0.355}, "heatmap_png": "..."}],
 "timing_ms": {"inference": 6.38, "total": 14.77}}

• bbox is echoed as sent; inout is P(target inside the frame); gaze_target.x/y is the heatmap argmax mapped back to original pixels (x = col/64 · W, upstream convention). The 64×64 heatmap covers the whole original image.
• heatmap_png is a base64 8-bit grayscale PNG (uint8(round(p·255))); npz returns one base64 .npz (heatmap float32 (N,64,64), inout float32 (N,)) in heatmap_npz.

Demo

GazeFollow test-set scenes; boxes come from RetinaFace, each same-coloured arrow/× is that person's predicted gaze target.

Input (media/source_N.png)	Prediction on p150a (media/target_N.png)

Accuracy and speed

Metric	Value
GazeFollow test AUC (4,782 images)	0.9540 vs 0.9543 torch reference (paper 0.9560)
GazeFollow Avg L2 / Min L2	0.1119 / 0.0502 vs 0.1103 / 0.0491 torch reference
Inference, served over HTTP (warm, batch 1, 1 head, 640×514; median of 30 requests)	5.7 ms device · 16.6 ms server total · 20.2 ms client wall (legacy path TT_FUSED=0: 9.4 ms · 20.3 ms · 24.2 ms)
Inference, served over HTTP (warm, batch 1, 3 heads, 500×334; median of 30 requests)	6.4 ms device · 14.8 ms server total · 16.5 ms client wall (legacy path: 12.6 ms · 20.8 ms · 22.7 ms)
Inference, served over HTTP (warm, batch 1, 5 heads, 500×334; 6-head trace bucket)	7.4 ms device · 16.8 ms server total · 18.6 ms client wall
benchmark.py --impl ttnn (host harness, random weights, 1 head, 50 iters)	181 FPS = 5.5 ms per forward (legacy path 106 FPS = 9.4 ms)
Same forward on an RTX 5090 (same host, port's torch reference, eager PyTorch, batch 1, incl. H2D/D2H; bf16 / fp16 autocast)	5.0 / 5.8 ms vs the p150a's 5.7 ms (N=1) → GPU bf16 1.1× faster, fp16 parity; fp32-strict GPU 9.2 ms (p150a 1.6× faster); best torch.compile 3.8 ms (1.5×)

Caveats

• Every image is squashed to 448×448; one image per request, 1–10 head boxes. All heads share one backbone pass; the gaze decoder runs all heads in one batched, metal-traced pass (head-count buckets 1,2,3,4,6,8,10 are captured at boot, so N=5 runs the 6-head trace).
• bf16/bfp8 on device: backbone stage PCC vs torch is 0.972–0.976 with the pretrained weights (0.999 with random weights); heatmap PCC is 0.94–0.99 per image on face-centred boxes and the argmax can move on very small or off-face boxes; on the GazeFollow test set the device path is within 0.002 AUC / 0.002 L2 of the torch reference.
• Fused, metal-traced pipeline (default; TT_FUSED=0 in the container env restores the original eager per-head path): device-validated on the p150a on 2026-09-13, numbers above are from that run.
• Only the vitb14 checkpoint is ported and the compute grid is hard-coded to the p150a's 10×13 cores; hardware: p150 only.
• Not an OpenAI-compatible API; GET /v1/models is a stub so the tt-model ready card does not 404.
• Validated on tt-metal v0.78.0-dev20260820 (main 8b98410e730), single p150a only.
• GPU comparison: GPU fp32-strict eager is 1.6× slower than the p150a; GPU bf16 is 1.1× faster, compiled 1.5×. RTX 5090 rows (2026-09-14): same host, the port's own torch reference (same weights) run eagerly in PyTorch 2.11 cu128 with fp32 weights + autocast unless stated, no TensorRT; medians of 50 iterations after warm-up, H2D/D2H included; the p150a rows are the served bf16 fused path incl. upload/readback. p150a power was not measured, so no efficiency comparison is made. Full table: GPU_COMPARISON.md.

Licensing

• Weights: changh95/gaze-lle-weights mirrors the two upstream checkpoints unchanged — Gaze-LLE decoder MIT, DINOv2 ViT-B/14 backbone Apache-2.0.
• Port and serving code (code/): from changh95/tt-Gaze-LLE; source files carry SPDX-License-Identifier: Apache-2.0 headers and the repo is published under the upstream MIT terms.

Provenance

The exact sources the image was built from — code/ in this repo is byte-identical to the model code inside the image:

component	built from
tt-metal	8b98410e730bb504fea43a88609756e34821d91d
code/ digest	99ab6c3ad7b4ea9e (sha256, first 16 hex digits)
built	2026-09-13T15:37:45+00:00 by tt-model 0.1.0