---
license: apache-2.0
base_model:
- bytedance-research/Lance
- Qwen/Qwen2.5-VL-3B-Instruct
pipeline_tag: text-to-image
library_name: mlx
tags:
- multimodal
- mlx
- apple-silicon
- text-to-image
- image-generation
- video-generation
- diffusion
- flow-matching
- moe
- qwen2_5_vl
- wan
- port
---

# Lance-3B-MLX

First native [MLX](https://github.com/ml-explore/mlx) port of [ByteDance Research's Lance](https://huggingface.co/bytedance-research/Lance) — a 3B-parameter unified multimodal model for image/video generation, editing, and understanding. Runs natively on Apple Silicon, no CUDA required.

The architecture is **Qwen2.5-VL-3B + parallel MoE-gen experts + Wan 2.2 VAE**. Lance uses a "Mixture-of-Tokens" routing: every attention block and MLP has a parallel `*_moe_gen` branch. Text tokens go through normal weights; VAE-latent (generation) tokens go through the `_moe_gen` weights, in the same forward pass.

## What works

| Capability | Status |
|---|---|
| Text-to-image (T2I), single image, CFG | ✅ Working, verified |
| Strict load of all 1021 LLM/adapter tensors | ✅ Working |
| Wan 2.2 VAE encode/decode (T=1) | ✅ Working (uses [RockTalk/Wan2.2-VAE-MLX](https://huggingface.co/RockTalk/Wan2.2-VAE-MLX)) |
| Flow-matching denoising loop | ✅ Working |
| Classifier-free guidance | ✅ Working |
| 3D mrope position embeddings | ✅ Working |
| MoE-gen routing (per-token attention + MLP + layernorm) | ✅ Working |
| Text-to-video (T2V) | ⏳ Needs VAE T>1 streaming cache + larger pos embed |
| Image/video editing (TI2I, TIV2V) | ⏳ Phase 2 — needs ViT integration |
| X→T (image/video understanding) | ⏳ Phase 2 — needs AR sampling loop + KV cache |

## Sample generations

Verified on M4 Studio (128 GB). 30 steps, CFG=4, 512×512:

| Prompt | Output |
|---|---|
| *"a photo of a sunset over mountains"* | ![sunset](samples/sunset_mountains.png) |
| *"a fluffy orange cat sitting on a wooden chair, photorealistic"* | ![cat](samples/orange_cat_chair.png) |
| *"a majestic snowy mountain peak with a dramatic blue sky and clouds"* | ![mountain](samples/snowy_peak.png) |

## Performance

Measured on M4 Studio (128 GB) at CFG=4 (one conditional + one unconditional forward per step):

| Resolution | Steps | Per-step | Total sample | VAE decode |
|---|---|---|---|---|
| 256×256 | 24 | ~400 ms | ~9.6 s | ~0.1 s |
| 512×512 | 30 | ~1.2 s  | ~36 s | ~0.5 s |

First-call kernel-compile penalty: ~few seconds per new resolution.

## Files

| File | Size | Description |
|---|---|---|
| `model.safetensors` | 23 GB | LLM (Qwen2.5-VL with MoE-gen) + Lance adapters, 1021 tensors |
| `vit.safetensors` | 1.25 GB | Qwen2.5-VL ViT (for understanding mode — Phase 2) |
| `vae.safetensors` | 2.62 GB | Wan 2.2 VAE (older keying — for compatibility; the standalone [RockTalk/Wan2.2-VAE-MLX](https://huggingface.co/RockTalk/Wan2.2-VAE-MLX) uses cleaner keys and is recommended) |
| `config.json` | — | Distilled architecture config |
| `tokenizer.json`, `vocab.json`, `merges.txt` | — | Qwen2.5-VL tokenizer, verbatim |
| `samples/*.png` | — | Verified T2I outputs from this checkpoint |

## Usage

Requires `mlx >= 0.29`, `mlx-vlm >= 0.3`, `numpy`, `einops`, `transformers`, `pillow`, and the [`lance-mlx`](https://github.com/RockTalk/Lance-MLX) companion repo for the `Lance` Python class.

```bash
pip install mlx mlx-vlm numpy einops transformers pillow
```

```python
import mlx.core as mx
from lance_mlx.lance import Lance, LanceConfig
from lance_mlx.vae_wan22 import Wan2_2_VAE

# Build + strict-load (see tools/lance_t2i.py in the companion repo for the
# full builder; LanceConfig takes a Qwen2.5-VL ModelConfig built from
# config.json).
model = Lance(lance_cfg)
model.load_weights(list(mx.load("model.safetensors").items()), strict=True)

vae = Wan2_2_VAE(z_dim=48, c_dim=160, dim_mult=(1, 2, 4, 4),
                 temperal_downsample=(False, True, True))
vae.model.load_weights(list(mx.load("vae.safetensors").items()), strict=True)

# Sample
latent = model.sample_t2i(
    prompt_token_ids=text_ids,                  # (P,) int32 from tokenizer (no specials)
    latent_shape=(1, 32, 32),                   # (T_lat, H_lat, W_lat) for 512×512 image
    special_token_ids={"bos": 151644, "eos": 151645,
                       "start_of_image": 151652, "end_of_image": 151653,
                       "image_token_id": 151655},
    num_steps=30, timestep_shift=3.5, cfg_scale=4.0, seed=0,
)
img = vae.decode(latent)                        # (1, 1, 512, 512, 3) in [-1, 1]
```

End-to-end script: `tools/lance_t2i.py` in the [companion repo](https://github.com/RockTalk/Lance-MLX).

## How the MoE-gen routing is implemented in MLX

Lance's checkpoint contains *two* sets of weights per Qwen2 block:

```
self_attn.{q,k,v,o}_proj         self_attn.{q,k,v,o}_proj_moe_gen
self_attn.{q,k}_norm             self_attn.{q,k}_norm_moe_gen
mlp.{gate,down,up}_proj          mlp_moe_gen.{gate,down,up}_proj
input_layernorm                  input_layernorm_moe_gen
post_attention_layernorm         post_attention_layernorm_moe_gen
```

For T2I/T2V the sequence layout is:

```
<|im_start|> [prompt tokens] <|im_end|> <|vision_start|> [N latent placeholders] <|vision_end|>
                                                          └──── routed through moe_gen ────┘
                                                          ↑ everything else: normal weights
```

The MLX port (`qwen2_navit_mlx.py`) routes by slicing the sequence into the latent slab vs the surrounding text, applying the appropriate expert to each slab, and concatenating. mrope position ids continue to flow normally across both slabs (with axis-T/H/W coordinates only varying inside the latent slab).

## Conversion source

Converted from `bytedance-research/Lance/Lance_3B/*` using the open-source pipeline at https://github.com/RockTalk/Lance-MLX (`tools/convert_weights.py`). Layout transforms:

- Conv weights: PT `(O, I, [T,] H, W)` → MLX `(O, [T,] H, W, I)`
- Embedding weights: shape preserved
- `lm_head.weight` tied to `embed_tokens.weight` (Qwen default)
- All `*_moe_gen.*` keys copied verbatim under the same names

## License

Apache 2.0, inherited from upstream `bytedance-research/Lance`. The Wan 2.2 VAE component is also Apache 2.0 from Alibaba's Wan team.

## Acknowledgements

- **ByteDance Research** — original Lance training + PT release
- **Qwen team** — Qwen2.5-VL-3B-Instruct backbone
- **Alibaba Wan team** — Wan 2.2 VAE training
- **Apple `mlx` and `mlx-vlm` teams** — the underlying frameworks
- **This MLX port** — RockTalk

## Citation

```bibtex
@misc{lance_mlx,
  title  = {Lance-3B-MLX — First MLX port of ByteDance's Lance},
  author = {RockTalk},
  year   = {2026},
  url    = {https://huggingface.co/RockTalk/Lance-3B-MLX}
}
```