Initial release

Co-authored-by: meta-bot <meta-bot@users.noreply.huggingface.co>

.gitattributes

@@ -0,0 +1,35 @@
+*.7z filter=lfs diff=lfs merge=lfs -text
+*.arrow filter=lfs diff=lfs merge=lfs -text
+*.bin filter=lfs diff=lfs merge=lfs -text
+*.bz2 filter=lfs diff=lfs merge=lfs -text
+*.ckpt filter=lfs diff=lfs merge=lfs -text
+*.ftz filter=lfs diff=lfs merge=lfs -text
+*.gz filter=lfs diff=lfs merge=lfs -text
+*.h5 filter=lfs diff=lfs merge=lfs -text
+*.joblib filter=lfs diff=lfs merge=lfs -text
+*.lfs.* filter=lfs diff=lfs merge=lfs -text
+*.mlmodel filter=lfs diff=lfs merge=lfs -text
+*.model filter=lfs diff=lfs merge=lfs -text
+*.msgpack filter=lfs diff=lfs merge=lfs -text
+*.npy filter=lfs diff=lfs merge=lfs -text
+*.npz filter=lfs diff=lfs merge=lfs -text
+*.onnx filter=lfs diff=lfs merge=lfs -text
+*.ot filter=lfs diff=lfs merge=lfs -text
+*.parquet filter=lfs diff=lfs merge=lfs -text
+*.pb filter=lfs diff=lfs merge=lfs -text
+*.pickle filter=lfs diff=lfs merge=lfs -text
+*.pkl filter=lfs diff=lfs merge=lfs -text
+*.pt filter=lfs diff=lfs merge=lfs -text
+*.pth filter=lfs diff=lfs merge=lfs -text
+*.rar filter=lfs diff=lfs merge=lfs -text
+*.safetensors filter=lfs diff=lfs merge=lfs -text
+saved_model/**/* filter=lfs diff=lfs merge=lfs -text
+*.tar.* filter=lfs diff=lfs merge=lfs -text
+*.tar filter=lfs diff=lfs merge=lfs -text
+*.tflite filter=lfs diff=lfs merge=lfs -text
+*.tgz filter=lfs diff=lfs merge=lfs -text
+*.wasm filter=lfs diff=lfs merge=lfs -text
+*.xz filter=lfs diff=lfs merge=lfs -text
+*.zip filter=lfs diff=lfs merge=lfs -text
+*.zst filter=lfs diff=lfs merge=lfs -text
+*tfevents* filter=lfs diff=lfs merge=lfs -text

MoEViE-H14-448.pt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:43ed30ed9e53c981a5a67157e0b34a1449ed70912a143fec3f8ac28d0e1d328c
+size 17520115545

README.md

@@ -0,0 +1,108 @@
+---
+license: cc-by-nc-4.0
+library_name: moe_vie
+pipeline_tag: zero-shot-image-classification
+---
+
+# Model Details
+
+[\[📃 Paper\]](https://arxiv.org/abs/2608.17402)
+[\[📂 GitHub\]](https://github.com/facebookresearch/moe_vie)
+
+MoE-ViE is a family of **Mixture-of-Experts vision encoders** for image and video
+understanding, trained with a contrastive vision-language recipe. Each MoE block keeps a
+shared always-on expert alongside a routed pool, so only a small fraction of the weights is
+used for any given token.
+
+**Model Developer**: Meta
+
+**This checkpoint**: `MoEViE-H14-448` — H/14 at 448px.
+
+## Model Configurations
+
+MoE-ViE comes in three sizes. Only `active` experts run per token, so the activated
+parameter count is far below the total.
+
+| Scale | Width | Depth | Experts (active/total) | Total params | Activated params | Resolution |
+|:--:|:--:|:--:|:--:|:--:|:--:|:--:|
+| **B/16** | 768 | 12 | 4 / 32 | 0.5B | 0.1B | 224px |
+| **L/16** | 1024 | 24 | 4 / 32 | 1.7B | 0.3B | 384px |
+| **H/14** | 1280 | 32 | 8 / 32 | 3.5B | 1.1B | 448px |
+
+Every MoE block has **1 shared expert that is always active** plus a routed pool; the
+`active/total` column counts the shared expert. Routing is per-token top-k on a sigmoid gate.
+
+## Model Performance
+
+Zero-shot results. Top-1 accuracy (%) for classification, recall@1 (%) for retrieval.
+
+| Model | Checkpoint | IN-1k | ObjectNet | COCO-T2I | Kinetics-400 | MSR-VTT-T2V |
+|:--:|:--:|:--:|:--:|:--:|:--:|:--:|
+| **B/16** 224px | [MoEViE-B16-224](https://huggingface.co/facebook/MoEViE-B16-224) | 79.3 | 74.4 | 52.1 | 68.3 | 47.9 |
+| **L/16** 384px | [MoEViE-L16-384](https://huggingface.co/facebook/MoEViE-L16-384) | 83.6 | 85.0 | 57.2 | 74.5 | 50.5 |
+| **H/14** 448px | [MoEViE-H14-448](https://huggingface.co/facebook/MoEViE-H14-448) | 85.1 | 87.0 | 56.8 | 76.9 | 51.6 |
+
+# How to use
+
+## Install
+
+```bash
+git clone https://github.com/facebookresearch/moe_vie
+cd moe_vie
+pip install -r requirements.txt
+```
+
+Requires a **CUDA GPU** — the Mixture-of-Experts kernels are compiled with Triton at runtime.
+
+## Image and text feature extraction
+
+```python
+import torch
+from PIL import Image
+from open_clip import create_model_and_transforms, get_tokenizer, image_to_device
+
+MEAN, STD = (0.5, 0.5, 0.5), (0.5, 0.5, 0.5)
+
+model, _, preprocess = create_model_and_transforms(
+    "MoEViE-H14-448",
+    pretrained=True,   # downloads from the Hub
+    force_preprocess_cfg=dict(
+        patch_size=14, size_range=(448, 448), center_crop=True, window_size=1
+    ),
+    image_mean=MEAN, image_std=STD,
+)
+model = model.cuda().eval()
+tokenizer = get_tokenizer("MoEViE-H14-448")
+
+labels = ["a diagram", "a dog", "a cat"]
+packed, _ = preprocess.collate_fn([(preprocess(Image.open("cat.png").convert("RGB")), 0)])
+packed = image_to_device(packed, "cuda", torch.float32, mean=MEAN, std=STD)
+text = tokenizer(labels).cuda()
+
+with torch.no_grad(), torch.autocast("cuda"):
+    image_features = model.encode_image(packed, normalize=True)
+    text_features = model.encode_text(text, normalize=True)
+    probs = (model.logit_scale.exp() * image_features @ text_features.T).softmax(dim=-1)
+
+print("Label probs:", probs)
+```
+
+See [`demo/demo.py`](https://github.com/facebookresearch/moe_vie/blob/main/demo/demo.py) and the repository README for the
+zero-shot evaluation suite.
+
+# License
+
+Released under **CC BY-NC 4.0** — non-commercial research use.
+
+# Citation
+
+If you find this work useful, please cite:
+
+```bibtex
+@article{zhang2026moevie,
+  title={MoE-ViE: Mixture of Experts Vision Encoder for Efficient Image and Video Understanding},
+  author={Bonan Zhang and Shiyu Dong and Quan Hung Tran and Katharina Gschwind and Shuqi Yang and Sijia Chen and Adel Ahmadyan and Seungwhan Moon and Lu Zhang and Ahmed Kirmani and Babak Damavandi and Anuj Kumar},
+  journal={arXiv preprint arXiv:2608.17402},
+  year={2026}
+}
+```

config.yaml

@@ -0,0 +1,24 @@
+embed_dim: 1024
+quick_gelu: false
+vision_cfg:
+  global_layers: -1
+  relative_pos_embed_type: rope_2d
+  attn_pooler_heads: 16
+  use_ln_post: false
+  layers: 32
+  width: 1280
+  head_width: 80
+  patch_size: 14
+  mixture_of_expert: true
+  num_of_expert: 8
+  split: 4
+  k: 2
+  moe_layers: [1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31]
+text_cfg:
+  context_length: 144
+  vocab_size: 49408
+  width: 1408
+  mlp_ratio: 4
+  heads: 16
+  layers: 34
+  use_ln_post: false