README.md

metadata

license: mit
language:
  - en
tags:
  - referring-image-segmentation
  - reranking
  - failure-detection
  - multi-scale
  - vision-language
  - pytorch
pipeline_tag: image-segmentation
base_model:
  - OpenGVLab/BEiT-3
  - microsoft/swin-large-patch4-window12-384-22k
library_name: pytorch
datasets:
  - refcoco
  - refcocog

Venice-H1: Failure-Aware Query Re-Ranking for Referring Image Segmentation

Nicolò Savioli, Ph.D. — OdaxAI Research
nicolo.savioli@odaxai.com

---

Model Description

Venice-H1 is a lightweight, backbone-decoupled re-ranking module for Referring Image Segmentation (RIS). Operating on top of a frozen DeRIS-L backbone, it detects when the default query selection fails and selects a better alternative using:

• Multi-Scale Grid Signatures: 4×4, 8×8, 16×16 spatial pooling → 675-dim descriptors
• Failure Gate: binary classifier predicting whether Query 0 is suboptimal
• Gain Predictor: IoU-regression head estimating improvement per alternative query
• Backbone-decoupled: works on cached features, no retraining of DeRIS required

Architecture: 3-layer pre-norm Transformer encoder, 8 heads, hidden_dim=512
Parameters: 11,296,258 (11.3M) — matches paper exactly

---

External Dependencies

Venice-H1 operates as a post-hoc re-ranker on top of:

Component	Model	Paper
Backbone	DeRIS-L	Dai et al. (2025)
Visual Encoder	Swin-Large	Liu et al. (2021)
Language Encoder	BEiT-3	Wang et al. (2023)
Mask Generator	Mask2Former	Cheng et al. (2022)

Venice-H1 does not include these weights. You need a running DeRIS-L instance to extract features.

---

Checkpoint: venice_h1_deris_l.pt

Trained on RefCOCO/RefCOCO+/RefCOCOg using DeRIS-L features. Evaluated on RefCOCO val split.

Metric	Value
Parameters	11,296,258
Backbone	DeRIS-L
Epoch	15
τ (threshold)	0.90
Δ_fail (mIoU on failures)	+1.824
AUC (failure detection)	0.778
Δ_full (overall mIoU)	+0.039
Q0 mIoU	86.469
Selected mIoU	86.509
Oracle mIoU	89.691

---

Quick Start

import torch
from huggingface_hub import hf_hub_download

# Download checkpoint
ckpt_path = hf_hub_download(repo_id="OdaxAI/venice-h1", filename="venice_h1_deris_l.pt")
ckpt = torch.load(ckpt_path, map_location="cpu", weights_only=False)

print("Config:", ckpt["config"])
print("Metrics:", ckpt["metrics"])
print("Parameters:", sum(v.numel() for v in ckpt["model"].values() if hasattr(v, "numel")))

Reproduce paper results (no dataset needed — verifies architecture + metrics):

git clone https://github.com/odaxai/Venice-H1.git
cd Venice-H1 && pip install -r requirements.txt
python reproduce_results.py --verify_only

Expected output:

── Architecture Verification ──────────────────────────────
  Parameters     : 11,296,258  ✓ MATCH

── Paper Cross-Check (RefCOCO val) ─────────────────────────
  ✓ delta_fail   : 1.8244  (paper: 1.824)
  ✓ auc_fail     : 0.7776  (paper: 0.778)
  ✓ delta_full   : 0.0392  (paper: 0.039)

Full inference pipeline (requires DeRIS-L features — see GitHub README):

from venice_h1.model.reranker import VeniceH1Reranker
import torch
from huggingface_hub import hf_hub_download

ckpt_path = hf_hub_download(repo_id="OdaxAI/venice-h1", filename="venice_h1_deris_l.pt")
ckpt = torch.load(ckpt_path, map_location="cpu", weights_only=False)
cfg  = ckpt["config"]

model = VeniceH1Reranker(
    query_feat_dim=cfg["query_feat_dim"],
    hidden_dim=cfg["hidden_dim"],
    n_layers=cfg["n_layers"],
    n_heads=cfg["n_heads"],
    tau=cfg["tau"],
)
model.load_state_dict(ckpt["model"], strict=False)
model.eval()

# features: (B, N=10, 936) — from scripts/extract_features.py
with torch.no_grad():
    out      = model(features, det_scores, mask_means)
    p_fail   = out["p_fail"]    # (B,) failure probability
    selected = model.rerank(features, det_scores, mask_means)  # (B,) best query

---

Files in Repository

File	Description
venice_h1_deris_l.pt	Trained checkpoint — 11.3M params, DeRIS-L backbone
venice_h1_deris_l_metrics.json	Full evaluation metrics
config.yaml	Training hyperparameters
venice_h1/	Python package (model code)
train.py	Training script
evaluate.py	Evaluation script
reproduce_results.py	One-command paper reproduction
scripts/extract_features.py	Feature extraction from DeRIS-L

---

Training Details

Parameter	Value
Optimizer	AdamW
Learning rate	5e-4
Weight decay	1e-4
Batch size	512
Epochs	20
Scheduler	Cosine + 3 epoch warmup
Loss: L_gate	Focal BCE (γ=2.0)
Loss: L_gain	Smooth-L1 (λ=5.0)
Mixed precision	FP16
Seed	42

---

Citation

@article{savioli2026veniceh1,
  title   = {Venice-H1: Failure-Aware Query Re-Ranking with Multi-Scale Grid Signatures
             for Referring Image Segmentation},
  author  = {Savioli, Nicol\`{o}},
  journal = {arXiv preprint arXiv:2606.22546},
  year    = {2026},
  note    = {OdaxAI Research},
}

---

License

MIT License. © 2026 OdaxAI Research. All research conducted by Nicolò Savioli, Ph.D.