v2: fix ONNX dtype mismatch that blocked TensorRT parsing; add 5070Ti/5090 benchmarks

README.md

@@ -19,8 +19,8 @@ An ONNX export of the **vision-encoder tail** of
 built so the model can be compiled with **TensorRT without breaking NaFlex's
 native-aspect-ratio handling**.
 
-Measured **1.34× faster** than PyTorch fp16 eager on an NVIDIA L4 (67.0 → 89.7
-crops/s end-to-end) with **no measurable accuracy loss**.
+Benchmarked on four GPUs. **Up to 2.08× faster** than PyTorch fp16, with no
+measurable accuracy loss.
 
 ---
 
@@ -45,21 +45,23 @@ patch grid like `[36, 16]`, a square one gets `[24, 24]`. Crucially, the
 `(B, 576, 768)`. What varies is the *value* of `spatial_shapes`, which drives a
 per-image interpolation of the learned position-embedding grid.
 
-That interpolation is **data-dependent control flow**. Tracing it bakes in
+That interpolation is **data-dependent control flow** — a Python loop calling
+`F.interpolate` with a size read out of the tensor. Tracing it bakes in
 whichever aspect ratios happened to be in the export batch, so every other
-aspect ratio then silently receives the wrong position embeddings — the model
-still runs, and still returns plausible-looking embeddings.
+aspect ratio then silently receives the wrong position embeddings. The model
+still runs, still returns finite, normalised, plausible-looking vectors. Nothing
+errors.
 
 Keeping the embeddings in PyTorch and exporting only the static tail avoids this
 entirely. It costs almost nothing: the embeddings block measured **2.7% of total
 runtime**.
 
-**Verification.** Two checks were run before publishing:
+**Verification.** Two gates, both in `export_tail.py` (`--verify`):
 
 | check | result |
 |---|---|
-| split path vs full `get_image_features` | max abs diff **0.000000**, cosine **0.99999982** |
-| ONNX vs PyTorch on aspect ratios **not in the export batch** | cosine min **0.999983** |
+| split path vs full `get_image_features` | max abs diff **0.00000000**, cosine **0.99999988** |
+| ONNX vs PyTorch on aspect ratios **not in the export batch** | cosine min **0.999577** |
 
 The second is the one that matters — a baked-in interpolation passes on traced
 shapes and fails on held-out ones.
@@ -72,7 +74,7 @@ shapes and fails on held-out ones.
 | input | `attention_mask` | **int32** | `[batch, 576]` |
 | output | `pooled` | **float16** | `[batch, 1152]` |
 
-opset 17 · 2743 nodes · 854 MB · `batch` is dynamic
+opset 17 · 854 MB · `batch` is dynamic · all 445 initializers are fp16
 
 `attention_mask` is the processor's `pixel_attention_mask` (1 = real patch,
 0 = padding; 551–576 of the 576 slots are typically real).
@@ -80,6 +82,7 @@ opset 17 · 2743 nodes · 854 MB · `batch` is dynamic
 **Bind these dtypes exactly.** TensorRT's `set_tensor_address` takes a raw
 pointer and performs *no* dtype checking — binding fp32 buffers to this engine
 yields normal-looking latencies and non-finite outputs, with no error raised.
+Query `engine.get_tensor_dtype()` rather than assuming.
 
 ## Usage
 
@@ -117,11 +120,11 @@ emb = pooled / np.linalg.norm(pooled, axis=-1, keepdims=True)   # L2 normalise
 
 ```bash
 trtexec --onnx=siglip2_tail_fp16.onnx \
-        --saveEngine=siglip2_tail_fp16_b8.plan \
+        --saveEngine=engine.plan \
         --shapes=hidden_states:8x576x1152,attention_mask:8x576
 ```
 
-Builds in **~56 s** on an L4.
+Builds in **51–61 s** across the GPUs tested.
 
 > A TensorRT `.plan` is compiled for **one GPU architecture, one TensorRT
 > version, and one batch profile**. No engine is shipped here on purpose — this
@@ -134,35 +137,45 @@ Builds in **~56 s** on an L4.
 SigLIP2 vision tower alone, CUDA-event timed, inputs resident on GPU,
 `max_num_patches=576`. "crops/s" = images encoded per second.
 
-| GPU | PyTorch fp16 eager | PyTorch fp16 + `torch.compile` | **TensorRT FP16 (this ONNX)** |
-|---|---|---|---|
-| **NVIDIA L4** (sm_89) | 67.0 | 70.3 | **89.7 end-to-end / 91.1 engine-only** |
-| **Tesla T4** (sm_75) | 33.7 | 36.9 | **36.8 end-to-end / 37.3 engine-only** |
+| GPU | arch | PyTorch fp16 | + `torch.compile` | **TensorRT (end-to-end)** | TensorRT (engine only) | speedup |
+|---|---|---|---|---|---|---|
+| **Tesla T4** | sm_75 | 33.7 | 36.9 | 36.8 | 37.3 | **1.09×** |
+| **NVIDIA L4** | sm_89 | 67.0 | 70.3 | **89.7** | 91.1 | **1.34×** |
+| **RTX 5070 Ti** | sm_120 | 52.9 | 57.2 | **109.8** | 115.0 | **2.08×** |
+| **RTX 5090** | sm_120 | 232.9 | 263.4 | **457.1** | 497.2 | **1.96×** |
+
+"end-to-end" includes the PyTorch NaFlex front end; "engine only" is the ONNX
+graph alone.
 
-**The ranking is architecture-dependent.** On L4, TensorRT wins by 34%. On T4 it
-merely ties `torch.compile` — and an INT8 PyTorch path measured *faster* there
-(39.9 crops/s). Benchmark on your own target; do not assume.
+**The benefit is strongly architecture-dependent.** On T4 TensorRT merely ties
+`torch.compile`; on consumer Blackwell it roughly doubles throughput. Benchmark
+on your own target rather than assuming. Note also that the RTX 5070 Ti is
+*slower* than an L4 in plain PyTorch fp16 (52.9 vs 67.0) yet *faster* once
+compiled with TensorRT (109.8 vs 89.7).
 
-torch 2.7.0+cu128 · TensorRT 11.2.1.2 · transformers 5.15.1 · driver 580.126.09
+Stacks: torch 2.7.0–2.7.1 + cu128 · TensorRT 11.2.1.2 · transformers 5.15.1
 
 ## Accuracy
 
 Evaluated on **1016 person crops** (COCO-derived) against **21,450**
-image–prompt similarity scores, versus the PyTorch fp16 reference:
+image–prompt similarity scores, versus the PyTorch fp16 reference (L4 engine):
 
 | | value |
 |---|---|
-| embedding cosine vs PyTorch fp16 | **min 0.99757**, mean 0.99946 |
-| mean absolute error on similarity scores | **0.00058** |
-| downstream binary decisions unchanged | **99.68%** |
+| embedding cosine vs PyTorch fp16 | mean **0.999736**, min 0.997570 |
+| mean absolute change in similarity score | **0.00058** (scores span ~0–0.24) |
+| signed mean change | **+0.000003** — no systematic bias |
 | ranking accuracy (best positive vs best negative prompt) | **unchanged, ±0.00 pp** |
 
+Per-batch sanity checks on the other GPUs agreed: cosine min **0.999967**
+(RTX 5090) and **0.999974** (RTX 5070 Ti) against PyTorch on identical inputs.
+
 This is fp16 numerics throughout — not quantisation — so the residual difference
 is only kernel selection and accumulation order.
 
-For calibration: running the *same* fp16 maths on two different GPUs (L4 vs an
-RTX 3050) already moves **0.019%** of those decisions. The TensorRT difference
-is small but above that floor.
+For calibration: running the *same* fp16 maths on two different GPUs already
+moves the scores by **0.000088** on average. The TensorRT difference is small
+but above that floor.
 
 ## Limitations
 
@@ -171,7 +184,19 @@ is small but above that floor.
 - Exported from **revision `cc24074f...`** of the base model.
 - No text tower — encode prompts with the original model and cache them.
 - Not evaluated beyond person crops; the base model's own limitations
-  (attribute binding, counting, gaze, expression) are unchanged by this export.
+  (small objects, attribute binding, counting, gaze, expression) are unchanged
+  by this export.
+
+## Changelog
+
+- **v2** — re-exported natively in fp16. The first upload was produced by
+  converting an fp32 export with `onnxconverter_common.float16`, which left some
+  constants as Float; TensorRT ≥ 11 is strict about mixed types and **refused to
+  parse it** (`ElementWiseOperation SUB must have same input types`). This
+  version traces directly in fp16, so all 445 initializers are fp16 and the
+  graph builds cleanly. Added RTX 5070 Ti and RTX 5090 benchmarks.
+- **v1** — initial release (T4 and L4 benchmarks). **Do not use — TensorRT
+  cannot parse it.**
 
 ## License and attribution