---
library_name: und3rstand
base_model:
- naver/DUSt3R_ViTLarge_BaseDecoder_224_linear
pipeline_tag: image-to-3d
tags:
- 3d-vision
- dust3r
- probing
- model_hub_mixin
- pytorch_model_hub_mixin
---

# Understanding Multi-View Transformers — DUSt3R 224 Pointmap Probes

This model provides **pretrained probes** for analyzing the internal representations of **multi-view transformers**, specifically **DUSt3R**
(checkpoint trained at resolution 224 with a linear output head).  
The probes decode **3D pointmaps** from intermediate transformer features, enabling **layer-wise study of geometric reasoning**.

This work accompanies the paper:

**Understanding Multi-View Transformers**  
ICCV 2025 E2E3D Workshop

- **Code:** https://github.com/JulienGaubil/und3rstand
- **Paper:** https://arxiv.org/abs/2510.24907
- **Other pretrained probes:** https://huggingface.co/jgaubil/und3rstand-dust3r-512-dpt


---

## Model Description

- **Backbone:** DUSt3R (ViT-Large, frozen)
- **Probe type:** 5-layer MLP, one per probed transformer layer
- **Task:** Decode per-pixel 3D pointmaps from transformer features
- **Input:** Two RGB images `(B, 3, H, W)` normalized to `[-1, 1]`
- **Output:** One prediction per probed transformer layer
  - `pts3d`: `(B, H, W, 3)` 3D pointmap
  - `conf`: `(B, H, W)` confidence map

## Usage

```python
import requests
from PIL import Image
import torchvision.transforms as T
from src.models.probes import PointmapProbes

model, probes = PointmapProbes.load_backbone_and_probe(
    "jgaubil/und3rstand-dust3r-224-linear"
)
model.eval()
probes.eval()

view1_path = "https://raw.githubusercontent.com/JulienGaubil/und3rstand/main/assets/samples/example_view1.jpg"
view2_path = "https://raw.githubusercontent.com/JulienGaubil/und3rstand/main/assets/samples/example_view2.jpg"
transform = T.Compose([
    T.Resize(224),
    T.CenterCrop(224),
    T.ToTensor(),
    T.Normalize(mean=[0.5, 0.5, 0.5], std=[0.5, 0.5, 0.5]),
])
view1_images = transform(
    Image.open(requests.get(view1_path, stream=True).raw).convert("RGB")
).unsqueeze(0)
view2_images = transform(
    Image.open(requests.get(view2_path, stream=True).raw).convert("RGB")
).unsqueeze(0)

feat_list = model(view1_images, view2_images)
outputs = probes(feat_list)

for layer_id, (pred1, pred2) in zip(model.probed_layers.layer_ids, outputs):
    print(f"{layer_id}: pts3d={pred1['pts3d'].shape}, conf={pred1['conf'].shape}")
```

---


## Citation

```bibtex
@inproceedings{stary2025understanding,
  title={{Understanding Multi-View Transformers}},
  author={Star{\'y}, Michal and Gaubil, Julien and Tewari, Ayush and Sitzmann, Vincent},
  booktitle={ICCV 2025 E2E3D Workshop},
  year={2025}
}