---
license: cc-by-nc-4.0
library_name: moe_vie
pipeline_tag: zero-shot-image-classification
---

# Model Details

[\[📃 Paper\]](https://arxiv.org/abs/2608.17402)
[\[📂 GitHub\]](https://github.com/facebookresearch/moe_vie)

MoE-ViE is a family of **Mixture-of-Experts vision encoders** for image and video
understanding, trained with a contrastive vision-language recipe. Each MoE block keeps a
shared always-on expert alongside a routed pool, so only a small fraction of the weights is
used for any given token.

**Model Developer**: Meta

**This checkpoint**: `MoEViE-H14-448` — H/14 at 448px.

## Model Configurations

MoE-ViE comes in three sizes. Only `active` experts run per token, so the activated
parameter count is far below the total.

| Scale | Width | Depth | Experts (active/total) | Total params | Activated params | Resolution |
|:--:|:--:|:--:|:--:|:--:|:--:|:--:|
| **B/16** | 768 | 12 | 4 / 32 | 0.5B | 0.1B | 224px |
| **L/16** | 1024 | 24 | 4 / 32 | 1.7B | 0.3B | 384px |
| **H/14** | 1280 | 32 | 8 / 32 | 3.5B | 1.1B | 448px |

Every MoE block has **1 shared expert that is always active** plus a routed pool; the
`active/total` column counts the shared expert. Routing is per-token top-k on a sigmoid gate.

## Model Performance

Zero-shot results. Top-1 accuracy (%) for classification, recall@1 (%) for retrieval.

| Model | Checkpoint | IN-1k | ObjectNet | COCO-T2I | Kinetics-400 | MSR-VTT-T2V |
|:--:|:--:|:--:|:--:|:--:|:--:|:--:|
| **B/16** 224px | [MoEViE-B16-224](https://huggingface.co/facebook/MoEViE-B16-224) | 79.3 | 74.4 | 52.1 | 68.3 | 47.9 |
| **L/16** 384px | [MoEViE-L16-384](https://huggingface.co/facebook/MoEViE-L16-384) | 83.6 | 85.0 | 57.2 | 74.5 | 50.5 |
| **H/14** 448px | [MoEViE-H14-448](https://huggingface.co/facebook/MoEViE-H14-448) | 85.1 | 87.0 | 56.8 | 76.9 | 51.6 |

# How to use

## Install

```bash
git clone https://github.com/facebookresearch/moe_vie
cd moe_vie
pip install -r requirements.txt
```

Requires a **CUDA GPU** — the Mixture-of-Experts kernels are compiled with Triton at runtime.

## Image and text feature extraction

```python
import torch
from PIL import Image
from open_clip import create_model_and_transforms, get_tokenizer, image_to_device

MEAN, STD = (0.5, 0.5, 0.5), (0.5, 0.5, 0.5)

model, _, preprocess = create_model_and_transforms(
    "MoEViE-H14-448",
    pretrained=True,   # downloads from the Hub
    force_preprocess_cfg=dict(
        patch_size=14, size_range=(448, 448), center_crop=True, window_size=1
    ),
    image_mean=MEAN, image_std=STD,
)
model = model.cuda().eval()
tokenizer = get_tokenizer("MoEViE-H14-448")

labels = ["a diagram", "a dog", "a cat"]
packed, _ = preprocess.collate_fn([(preprocess(Image.open("cat.png").convert("RGB")), 0)])
packed = image_to_device(packed, "cuda", torch.float32, mean=MEAN, std=STD)
text = tokenizer(labels).cuda()

with torch.no_grad(), torch.autocast("cuda"):
    image_features = model.encode_image(packed, normalize=True)
    text_features = model.encode_text(text, normalize=True)
    probs = (model.logit_scale.exp() * image_features @ text_features.T).softmax(dim=-1)

print("Label probs:", probs)
```

See [`demo/demo.py`](https://github.com/facebookresearch/moe_vie/blob/main/demo/demo.py) and the repository README for the
zero-shot evaluation suite.

# License

Released under **CC BY-NC 4.0** — non-commercial research use.

# Citation

If you find this work useful, please cite:

```bibtex
@article{zhang2026moevie,
  title={MoE-ViE: Mixture of Experts Vision Encoder for Efficient Image and Video Understanding},
  author={Bonan Zhang and Shiyu Dong and Quan Hung Tran and Katharina Gschwind and Shuqi Yang and Sijia Chen and Adel Ahmadyan and Seungwhan Moon and Lu Zhang and Ahmed Kirmani and Babak Damavandi and Anuj Kumar},
  journal={arXiv preprint arXiv:2608.17402},
  year={2026}
}
```