---
license: mit
language:
- en
tags:
- referring-image-segmentation
- re-ranking
- transformer
- computer-vision
- segmentation
- pytorch
pipeline_tag: image-segmentation
library_name: pytorch
---

<div align="center">

<img src="https://raw.githubusercontent.com/odaxai/Venice-H1/main/assets/odaxai_spiral.png" width="60"/>

# Venice-H1

### Failure-Aware Query Re-Ranking with Multi-Scale Grid Signatures<br>for Referring Image Segmentation

[![arXiv](https://img.shields.io/badge/arXiv-2506.XXXXX-b31b1b.svg)](https://arxiv.org/abs/2506.XXXXX)
[![GitHub](https://img.shields.io/badge/GitHub-odaxai/Venice--H1-black?logo=github)](https://github.com/odaxai/Venice-H1)
[![License: MIT](https://img.shields.io/badge/License-MIT-blue.svg)](https://github.com/odaxai/Venice-H1/blob/main/LICENSE)

**Nicolò Savioli, Ph.D.**  
[OdaxAI Research](https://odaxai.com) · nicolo.savioli@odaxai.com

</div>

---

## Model Description

Venice-H1 is a **lightweight, backbone-decoupled post-hoc re-ranking module** for Referring Image Segmentation (RIS).

Modern RIS systems (e.g. DeRIS) generate N candidate masks per expression but rely on a detection-score heuristic to select the final one. In **7–18% of samples** this choice is wrong — and these failures drive **40–68% of total segmentation error**.

Venice-H1 detects and corrects these failures using:
- **Multi-Scale Grid Signatures** — 675-dim spatial descriptors (4×4, 8×8, 16×16 pooling)
- **Failure Gate** — predicts P(Query-0 is suboptimal), AUC 0.78–0.82
- **Gain Predictor** — estimates IoU improvement per alternative query
- **Gated selection** — intervenes only on predicted failures

### Architecture

```
Input features f_i = [q_i; s_i; μ_i; p̂_i; a_i; σ_i; g_i] ∈ R^936
    ↓
QueryEncoder: 2-layer MLP → R^512 (Hd=512)
    ↓
Transformer: L=3, A=8 heads, pre-norm GELU
    ├── Failure Gate  → P_fail ∈ [0,1]
    └── Gain Predictor → ĝ_i ∈ R per query
    ↓
if P_fail > τ: select argmax_i ĝ_i
else:          retain Query-0
```

**~11.3M parameters · <1ms overhead · ~3min training on RTX 3090**

---

## Results

### RefCOCO / RefCOCO+ / RefCOCOg

| Backbone | Failure Rate | ∆_fail (mIoU) | Harmful-Switch | Gate AUC |
|---|---|---|---|---|
| DeRIS-L | 12.18% | **+1.402** | 0.343% | 0.800 |
| DeRIS-B | 20.68% | **+0.891** | 0.528% | 0.763 |

Bootstrap 95% CI strictly positive on **16/16** (split × backbone) pairs.

### Zero-Shot Medical Transfer

| Dataset | Default mIoU | Venice-H1 | ∆ |
|---|---|---|---|
| MS-CXR | 10.71 | 11.87 | **+1.16** |
| M3D-RefSeg-2D | 6.65 | 7.16 | **+0.51** |

---

## Usage

```python
import torch
from huggingface_hub import hf_hub_download
from venice_h1.model.reranker import VeniceH1Reranker

# Load checkpoint
ckpt_path = hf_hub_download(repo_id="OdaxAI/venice-h1", filename="venice_h1_deris_l.pt")
ckpt = torch.load(ckpt_path, map_location="cpu")

# Load model
model = VeniceH1Reranker(
    query_feat_dim=256,
    hidden_dim=512,
    n_layers=3,
    n_heads=8,
    tau=0.05
)
model.load_state_dict(ckpt["model"])
model.eval()

# Inference
# features: [B, N=10, 936] — from scripts/extract_features.py
with torch.no_grad():
    selected_query_idx = model.rerank(features, tau=0.05)
```

---

## Installation

```bash
git clone https://github.com/odaxai/Venice-H1
cd Venice-H1
pip install -r requirements.txt
```

---

## Files

| File | Description |
|---|---|
| `venice_h1_deris_l.pt` | Checkpoint trained on DeRIS-L (RefCOCO/+/g) |
| `config.yaml` | Training configuration |

---

## Citation

```bibtex
@article{savioli2026veniceh1,
  title   = {Venice-H1: Failure-Aware Query Re-Ranking with Multi-Scale
             Grid Signatures for Referring Image Segmentation},
  author  = {Savioli, Nicol{\`o}},
  journal = {arXiv preprint arXiv:2506.XXXXX},
  year    = {2026}
}
```