KEVIN04087/edgevl-ovc-scannet

EdgeVL OVC ScanNet — INT8 QAT Cross-Modal Backbones

PyTorch checkpoints for the Ours (single-stage QAT distillation w/ cross-modal attention) open-vocabulary classification (OVC) experiments on ScanNet, distilled from CLIP ViT-g-14.

File	Backbone	best RGB+Depth acc (test)	Cross-attn
swint_best_rgb_depth.pth	Swin-T (28M)	0.5039	1-layer, 8.4M params
vit_best_rgb_depth.pth	ViT-B/16 (86M)	0.5116	1-layer, 8.4M params
datt_best_rgb_depth.pth	DAT++ Tiny (29M)	0.5178	1-layer, 8.4M params

All three were trained with the same recipe (BASE_LR=2e-4, EPOCHS=30, WARMUP=3, jacob INT8 W8A8) and the same v4 cross-modal attention setup:

fused = rgb_patches.mean() + fusion_gamma * CrossAttn(query=rgb, key=rgb, value=depth)

fusion_gamma is a per-channel learnable scalar [clip_dim] (init 0.5).

Checkpoint contents

Each .pth is a dict with:

• model: backbone state_dict
• cross_attention: MultiLayerCrossModalAttention state_dict
• fusion_gamma: per-channel blend tensor [clip_dim]
• best_metric: "rgb_depth"
• best_value: float
• epoch: int

How to load

import torch
ckpt = torch.load("swint_best_rgb_depth.pth", map_location="cpu", weights_only=False)
model.load_state_dict(ckpt["model"], strict=False)
cross_attention.load_state_dict(ckpt["cross_attention"], strict=False)
fusion_gamma.data.copy_(ckpt["fusion_gamma"])

See the project repo for full pipeline:

• Trainer: trainer.py:Trainer
• ONNX/TRT INT8 export: onnx_trt/onnx_converter.py, onnx_converter_vit.py

Eval (real ScanNet test set, batch=1, A100, INT8 TRT)

Backbone	Engine size	Accuracy	FPS
Swin-T	67.4 MB	0.5021	376
ViT-B	99.0 MB	0.5091	448

(Full INT8 deployment via trtexec --int8 after exporting QDQ ONNX with quant_nn.TensorQuantizer.use_fb_fake_quant = True.)

Distillation losses (per branch: rgb / depth / fused)

total = 1.0 * RKD + 3.0 * InfoNCE + 3.0 * ClassAwareSupCon

Teacher: CLIP ViT-g-14 image features (frozen).

License

Apache 2.0.