# v2_loaders — Differentiable Vision Tower Loaders (spark2)

Environment: `source ~/workspace/vpy/bin/activate`, `export HF_HOME=~/workspace/hf-cache`, torch 2.12.1+cu130, transformers 5.13.0, aarch64 (no flash_attn).

Probe command: `cd ~/workspace/v2_loaders && python probe.py`

## Summary

| Tower | Status | Feature dim | Peak VRAM (probe) | Loader |
|-------|--------|-------------|-------------------|--------|
| MoonViT | **PASS** | 1152 | 3.50 GB | `moonvit_loader.py` |
| InternViT | **PASS** | 1024 | 1.94 GB | `internvit_loader.py` |
| MiniMax-M3 vision | **PASS** | 1280 | 3.07 GB | `minimax_loader.py` |

---

## 1. MoonViT (`moonshotai/MoonViT-SO-400M`)

**Status: PASS** — grad_l2 ≈ 7218, feat `(1, 1152)`.

**Failure:** `AutoModel.from_pretrained(..., trust_remote_code=True)` on transformers 5.13 raises `AttributeError: ... has no attribute all_tied_weights_keys` during `_finalize_model_loading` (custom `MoonVitPretrainedModel` predates the 5.13 tied-weights API).

**Fix (approach a — shim, preferred):** `_shim.py` monkey-patches `PreTrainedModel._finalize_model_loading` to inject an empty `all_tied_weights_keys` dict when missing. No transformers version pin required.

**Load recipe:**
```python
from _shim import apply_pretrained_shims
apply_pretrained_shims()
model = AutoModel.from_pretrained(
    "moonshotai/MoonViT-SO-400M",
    trust_remote_code=True,
    attn_implementation="sdpa",  # no flash_attn on aarch64; native code has sdpa_attention path
    dtype=torch.bfloat16,
)
```

**Preprocessing:** bicubic resize to 448×448, unfold into `(N,3,14,14)` native-resolution patches; `grid_hws=[[32,32]]`. No extra normalization (pixels stay in caller `[0,1]` space). Output merged tokens mean-pooled to `(1, 1152)`.

**Deviations:** None vs reference architecture; uses upstream HF custom code with SDPA attention path.

---

## 2. InternViT (`OpenGVLab/InternViT-300M-448px-V2_5`)

**Status: PASS** — grad_l2 ≈ 5144, feat `(1, 1024)`.

**Failure:** `ImportError: flash_attn` — transformers `check_imports` scans vendored `flash_attention.py` which hard-imports `flash_attn` even though `modeling_intern_vit.py` has a try/except fallback.

**Fix:** Before load, overwrite `flash_attention.py` in the HF hub cache (and transformers_modules copy) with a no-op stub from `vendored/internvit/flash_attention.py`. Set `config.use_flash_attn = False` so `_naive_attn` (manual matmul softmax) runs — fully differentiable, no flash_attn.

**Load recipe:**
```python
from _shim import apply_pretrained_shims
apply_pretrained_shims()
config = AutoConfig.from_pretrained(REPO, trust_remote_code=True)
config.use_flash_attn = False
model = AutoModel.from_pretrained(REPO, config=config, trust_remote_code=True, dtype=torch.bfloat16)
```

**Preprocessing:** bicubic resize to 448×448, ImageNet normalize mean/std, CLS `pooler_output` → `(1, 1024)`.

**Deviations:** Attention uses eager `_naive_attn` instead of flash_attn (numerically equivalent path, required on aarch64).

---

## 3. MiniMax-M3 vision tower (staged weights)

**Status: PASS** — grad_l2 ≈ 73.5, feat `(1, 1280)`.

**Weights:** staged as `vision_tower.safetensors` (+ config/processor files). No `modeling_*.py` in the bundle.

**Fix:** Reuse transformers 5.13 built-in `MiniMaxM3VLVisionModel` / `MiniMaxM3VL3DRotaryEmbedding` (config-faithful 3D RoPE). Load safetensors with key remap:
- strip `vision_tower.vision_model.` prefix
- `embeddings.patch_embedding` → `embeddings.proj`
- `encoder.layers` → `layers`

`load_state_dict(..., strict=True)` succeeds (515 tensors).

**Load recipe:** see `minimax_loader.py` — builds `MiniMaxM3VLVisionConfig` from `~/workspace/mmx_m3/config.json`, loads remapped weights.

**Preprocessing:** bicubic resize to 672×672, OpenAI CLIP mean/std, patchify per bundled `image_processor.py` (temporal dim padded to 2 by repeating frame), `grid_thw=[[1,48,48]]`, mean-pool sequence → `(1, 1280)`.

**Deviations:** None — uses official transformers 5.13 3D RoPE implementation (`rope_mode: 3d`, `rope_theta: 10000`). Multimodal projector / patch-merger MLP weights are **not** in the staged safetensors (vision encoder only).

---

## Shared shim (`_shim.py`)

Both MoonViT and InternViT custom `PreTrainedModel` subclasses hit the missing `all_tied_weights_keys` issue on transformers 5.13. The single shim patch is applied by all loaders.