README.md

---
license: mit
language:
- en
tags:
- referring-image-segmentation
- reranking
- failure-detection
- multi-scale
- vision-language
- pytorch
pipeline_tag: image-segmentation
base_model:
- OpenGVLab/BEiT-3
- microsoft/swin-large-patch4-window12-384-22k
library_name: pytorch
datasets:
- refcoco
- refcocog
---

# Venice-H1: Failure-Aware Query Re-Ranking for Referring Image Segmentation

[![Paper](https://img.shields.io/badge/arXiv-2606.22546-b31b1b.svg)](https://arxiv.org/abs/2606.22546)
[![GitHub](https://img.shields.io/badge/GitHub-odaxai%2FVenice--H1-181717?logo=github)](https://github.com/odaxai/Venice-H1)
[![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](https://opensource.org/licenses/MIT)

**Nicolò Savioli, Ph.D.** — OdaxAI Research  
nicolo.savioli@odaxai.com

---

## Model Description

Venice-H1 is a lightweight, backbone-decoupled re-ranking module for Referring Image Segmentation (RIS). Operating on top of a frozen DeRIS-L backbone, it detects when the default query selection fails and selects a better alternative using:

- **Multi-Scale Grid Signatures**: 4×4, 8×8, 16×16 spatial pooling → 675-dim descriptors
- **Failure Gate**: binary classifier predicting whether Query 0 is suboptimal
- **Gain Predictor**: IoU-regression head estimating improvement per alternative query
- **Backbone-decoupled**: works on cached features, no retraining of DeRIS required

**Architecture**: 3-layer pre-norm Transformer encoder, 8 heads, hidden_dim=512  
**Parameters**: 11,296,258 (11.3M) — matches paper exactly

---

## External Dependencies

Venice-H1 operates as a post-hoc re-ranker on top of:

| Component | Model | Paper |
|-----------|-------|-------|
| Backbone | DeRIS-L | Dai et al. (2025) |
| Visual Encoder | Swin-Large | Liu et al. (2021) |
| Language Encoder | BEiT-3 | Wang et al. (2023) |
| Mask Generator | Mask2Former | Cheng et al. (2022) |

Venice-H1 does **not** include these weights. You need a running DeRIS-L instance to extract features.

---

## Checkpoint: `venice_h1_deris_l.pt`

Trained on RefCOCO/RefCOCO+/RefCOCOg using DeRIS-L features. Evaluated on RefCOCO val split.

| Metric | Value |
|--------|-------|
| Parameters | **11,296,258** |
| Backbone | DeRIS-L |
| Epoch | 15 |
| τ (threshold) | 0.90 |
| **Δ_fail (mIoU on failures)** | **+1.824** |
| **AUC (failure detection)** | **0.778** |
| **Δ_full (overall mIoU)** | **+0.039** |
| Q0 mIoU | 86.469 |
| Selected mIoU | 86.509 |
| Oracle mIoU | 89.691 |

---

## Quick Start

```python
import torch
from huggingface_hub import hf_hub_download

# Download checkpoint
ckpt_path = hf_hub_download(repo_id="OdaxAI/venice-h1", filename="venice_h1_deris_l.pt")
ckpt = torch.load(ckpt_path, map_location="cpu", weights_only=False)

print("Config:", ckpt["config"])
print("Metrics:", ckpt["metrics"])
print("Parameters:", sum(v.numel() for v in ckpt["model"].values() if hasattr(v, "numel")))
```

**Reproduce paper results** (no dataset needed — verifies architecture + metrics):

```bash
git clone https://github.com/odaxai/Venice-H1.git
cd Venice-H1 && pip install -r requirements.txt
python reproduce_results.py --verify_only
```

Expected output:
```
── Architecture Verification ──────────────────────────────
  Parameters     : 11,296,258  ✓ MATCH

── Paper Cross-Check (RefCOCO val) ─────────────────────────
  ✓ delta_fail   : 1.8244  (paper: 1.824)
  ✓ auc_fail     : 0.7776  (paper: 0.778)
  ✓ delta_full   : 0.0392  (paper: 0.039)
```

**Full inference pipeline** (requires DeRIS-L features — see [GitHub README](https://github.com/odaxai/Venice-H1)):

```python
from venice_h1.model.reranker import VeniceH1Reranker
import torch
from huggingface_hub import hf_hub_download

ckpt_path = hf_hub_download(repo_id="OdaxAI/venice-h1", filename="venice_h1_deris_l.pt")
ckpt = torch.load(ckpt_path, map_location="cpu", weights_only=False)
cfg  = ckpt["config"]

model = VeniceH1Reranker(
    query_feat_dim=cfg["query_feat_dim"],
    hidden_dim=cfg["hidden_dim"],
    n_layers=cfg["n_layers"],
    n_heads=cfg["n_heads"],
    tau=cfg["tau"],
)
model.load_state_dict(ckpt["model"], strict=False)
model.eval()

# features: (B, N=10, 936) — from scripts/extract_features.py
with torch.no_grad():
    out      = model(features, det_scores, mask_means)
    p_fail   = out["p_fail"]    # (B,) failure probability
    selected = model.rerank(features, det_scores, mask_means)  # (B,) best query
```

---

## Files in Repository

| File | Description |
|------|-------------|
| `venice_h1_deris_l.pt` | **Trained checkpoint** — 11.3M params, DeRIS-L backbone |
| `venice_h1_deris_l_metrics.json` | Full evaluation metrics |
| `config.yaml` | Training hyperparameters |
| `venice_h1/` | Python package (model code) |
| `train.py` | Training script |
| `evaluate.py` | Evaluation script |
| `reproduce_results.py` | One-command paper reproduction |
| `scripts/extract_features.py` | Feature extraction from DeRIS-L |

---

## Training Details

| Parameter | Value |
|-----------|-------|
| Optimizer | AdamW |
| Learning rate | 5e-4 |
| Weight decay | 1e-4 |
| Batch size | 512 |
| Epochs | 20 |
| Scheduler | Cosine + 3 epoch warmup |
| Loss: L_gate | Focal BCE (γ=2.0) |
| Loss: L_gain | Smooth-L1 (λ=5.0) |
| Mixed precision | FP16 |
| Seed | 42 |

---

## Citation

```bibtex
@article{savioli2026veniceh1,
  title   = {Venice-H1: Failure-Aware Query Re-Ranking with Multi-Scale Grid Signatures
             for Referring Image Segmentation},
  author  = {Savioli, Nicol\`{o}},
  journal = {arXiv preprint arXiv:2606.22546},
  year    = {2026},
  note    = {OdaxAI Research},
}
```

---

## License

MIT License. © 2026 OdaxAI Research. All research conducted by Nicolò Savioli, Ph.D.