vggt-1b-p150

VGGT-1B (Meta's feed-forward multi-view 3D reconstruction transformer) running on one Tenstorrent Blackhole p150a via tt-nn: 1-4 views of a scene in, per-pixel depth, world points, confidences and camera poses out. Weights: facebook/VGGT-1B · Paper: arXiv:2503.11651 · Upstream code: facebookresearch/vggt · Port: changh95/tt-vggt

Runs on p150 (mesh P150).

Packaged and published with tt-model-manager 0.1.0 (manifest schema 5.1).

Quickstart

tt-model pull  changh95/vggt-1b-p150 --with-weights
tt-model serve changh95/vggt-1b-p150

• Weights facebook/VGGT-1B at 860abec7937d go to your HF cache; the image does not contain them.
• Serves on port 20000 (or the next free port); ready when the log says Application startup complete.

Run with tt-cli

tt serve changh95/vggt-1b-p150
printf '{"images":["%s"]}' "$(base64 -w0 media/input.png)" > req.json
curl -s localhost:20000/predict -H 'Content-Type: application/json' -d @req.json
tt model stop changh95/vggt-1b-p150

• POST /predict: images (1-4 base64 PNG/JPEG views of one scene, ordered; image is a single-view alias); optional output_format (npz | json), dtype (float16 | float32), outputs (subset of depth, depth_conf, world_points, world_points_conf), conf_threshold, json_stride (8).
• GET /health, GET /info.

Response

{"model": "VGGT-1B", "num_views": 1, "image_size": [518, 518],
 "preprocess": {"mode": "pad", "per_view": ["..."]},
 "pose_enc": [[0.0, -0.0, -0.0, 0.0, -0.0, 0.0, 0.998, 0.866, 0.866]],
 "extrinsic": ["..."], "intrinsic": ["..."],
 "dense": {"encoding": "npz", "dtype": "float16",
           "arrays": {"depth": {"shape": [1, 518, 518], "dtype": "float16"},
                      "depth_conf": {"shape": [1, 518, 518], "dtype": "float16"},
                      "world_points": {"shape": [1, 518, 518, 3], "dtype": "float32"},
                      "world_points_conf": {"shape": [1, 518, 518], "dtype": "float16"}},
           "data": "..."},
 "timing_ms": {"preprocess": 5.9, "forward": 411.7, "encode": 106.0, "total": 523.7}}

• pose_enc is [tx, ty, tz, qx, qy, qz, qw, fov_h, fov_w] per view; extrinsic (S,3,4) is OpenCV cam-from-world with frame 0 as the world frame; intrinsic (S,3,3) and every dense array are in 518×518 padded pixels (x_518 = x_orig*scale_x + pad_left, from preprocess.per_view).
• dense.data is a base64 np.savez_compressed blob: np.load(io.BytesIO(base64.b64decode(data))) yields depth (S,518,518), depth_conf, world_points (S,518,518,3) and world_points_conf. output_format: "json" returns nested lists subsampled by json_stride instead.

Demo

Kitchen frames 00 / 03 (VGGT example scene) → the union of both views' world_points in the camera-1 frame, coloured by the source pixels, world_points_conf ≥ 1.5 (served npz output, code/make_demo.py).

Accuracy and speed

Metric	Value
min-PCC vs fp32 torch reference, synthetic input (test_vggt.py, S=1 / 2 / 3 / 4)	0.9952 / 0.9981 / 0.9984 / 0.9979 (legacy path 0.9950 / 0.9991 / 0.9978 / 0.9975)
min-PCC vs fp32 torch reference, 7 real scenes S=1..3 (mean / worst)	0.9989 / 0.9977 (legacy path 0.9990 / 0.9979)
AUC@30° camera-pose accuracy (CO3Dv2 apple, S=2, 3 scenes; port author, legacy path)	86.1 vs 87.2 reference
Inference, served over HTTP (warm, S=1, 518×518, npz; 30 requests, median / min / max)	411 / 408 / 435 ms forward · 526 / 521 / 548 ms end-to-end
Inference, served over HTTP (warm, S=2; 10 requests)	881 / 875 / 887 ms forward · 1091 / 1085 / 1097 ms end-to-end
Inference, test_vggt.py best-of-N (S=3 / S=4)	1945 ms / 2759 ms
Same forward on an RTX 5090 (same host, port's torch reference, eager PyTorch, batch 1, incl. H2D/D2H; bf16 / fp16 autocast)	S=1 65.6 / 63.3 ms, S=2 103.4 / 95.7 ms → GPU 6.4–6.6× (S=1) and 8.6–9.3× (S=2) faster than the p150a's 420 / 887 ms; fp32-strict 134 / 245 ms (3.1× / 3.6×); fp16 weights resident 52.3 / 76.8 ms; torch.compile was slower than eager here

Caveats

• Every view is resized and white-padded to 518×518 server-side; 1-4 ordered views of one scene per request (VGGT_S_MAX=4), batch 1, requests serialised on the one chip.
• bf16 on device with fp32 residual/softmax accumulation: outputs differ slightly from the fp32 reference (min-PCC 0.995-0.998 synthetic, 0.998-0.9998 on real images; the port author measured about 1.1 AUC@30° points below the reference on CO3Dv2 with the legacy path).
• The whole forward is device-resident and metal-traced (one trace per view count, TT_FUSED=1, default): warm-up captures the four traces before READY (22 s with a warm kernel cache). TT_FUSED=0 restores the previous host-orchestrated path (1.6 s / 2.9 s forward at S=1 / S=2).
• Weights are CC-BY-NC-4.0 (non-commercial only); the 5.0 GB fp32 model.safetensors is fetched into your HF cache, and the server keeps the torch model (~5 GB host RAM) as the driver.
• Dense outputs are npz-encoded by default (see Response); point tracking is not exposed (track head not ported).
• Not an OpenAI-compatible API; GET /v1/models is a stub so the tt-model ready card does not 404. Validated on tt-metal v0.78.0-dev20260820 (main 8b98410e730), single p150a only.
• GPU comparison: GPU bf16/fp16 is 6–9× faster: the 1B-parameter aggregator is compute-bound on the p150a (one metal-trace per S, no host syncs), so the gap reflects raw throughput, not dispatch overhead. RTX 5090 rows (2026-09-14): same host, the port's own torch reference (same weights) run eagerly in PyTorch 2.11 cu128 with fp32 weights + autocast unless stated, no TensorRT; medians of 50 iterations after warm-up, H2D/D2H included; the p150a rows are the served bf16 fused path incl. upload/readback. p150a power was not measured, so no efficiency comparison is made. Full table: GPU_COMPARISON.md.

Licensing

• Weights: facebook/VGGT-1B, CC-BY-NC-4.0 (non-commercial only; not redistributed here).
• Port and serving code (code/models/): Apache-2.0 per changh95/tt-vggt, bound by the weights' non-commercial terms; vendored upstream code/vggt/ is Meta's under the VGGT License v1 (code/LICENSE.txt, with its Acceptable Use Policy).

Provenance

The exact sources the image was built from — code/ in this repo is byte-identical to the model code inside the image:

component	built from
tt-metal	8b98410e730bb504fea43a88609756e34821d91d
code/ digest	e847a1f38266f7c7 (sha256, first 16 hex digits)
built	2026-09-14T06:48:14+00:00 by tt-model 0.1.0