---
license: apache-2.0
base_model:
- bytedance-research/Lance
- Qwen/Qwen2.5-VL-3B-Instruct
pipeline_tag: any-to-any
library_name: mlx
tags:
- multimodal
- mlx
- apple-silicon
- image-generation
- video-generation
- image-editing
- video-understanding
- any-to-any
- port
---

# Lance-3B-MLX

A native [MLX](https://github.com/ml-explore/mlx) port of [ByteDance's Lance](https://huggingface.co/bytedance-research/Lance) — a 3B-parameter unified multimodal model for image and video generation, editing, and understanding.

Built on top of the Qwen2.5-VL-3B-Instruct backbone, with Lance's custom multi-task adapters and a Wan 2.2 VAE.

## Status

This is a **work-in-progress port** focused on parity with the original PyTorch checkpoint:

| Component | Status |
|---|---|
| Weight conversion (PT → MLX safetensors, with conv layout + name remaps) | ✅ DONE |
| `modeling_utils` (TimestepEmbedder, PositionEmbedding3D, MLP, sincos tables) | ✅ DONE (5/5 unit tests pass) |
| `vae_wan22` — image-mode encode/decode | ✅ DONE |
| `vae_wan22` — video streaming feat-cache | ⏳ PENDING (image-only currently; short clips work via single-pass) |
| `lance.py` adapters (vae2llm, llm2vae, latent patching, time/pos embed) | ✅ DONE |
| Flow-matching sampler (`validation_gen` — T2I/T2V/edit) | ⏳ STUB — primitives wired, denoising loop + CFG porting in progress |
| X→T (understanding) autoregressive loop | ⏳ Phase 2 |
| NaViT variable-resolution image packing | ⏳ Phase 2 |

## Files

- `model.safetensors` — Lance 3B image variant LLM + adapters (Qwen2.5-VL language model, vae2llm/llm2vae, time_embedder, latent_pos_embed), MLX-layout, **22.9 GB** (1021 tensors, ~6.19B params incl. embed table)
- `vit.safetensors` — Qwen2.5-VL ViT visual encoder, MLX-layout (NTHWC conv weights), **1.25 GB** (390 tensors, ~668M params, fp16 — bundled here for offline use; the source ships it as a separate shard)
- `vae.safetensors` — Wan 2.2 VAE, MLX-layout, **2.62 GB** (196 tensors, ~705M params). Converted from the upstream `Wan2.2_VAE.pth` pickle.
- `config.json` — distilled architecture config + embedded Qwen2.5-VL sub-config
- `vit_config.json` — Qwen2.5-VL ViT sub-config
- `tokenizer.json`, `tokenizer_config.json`, `vocab.json`, `merges.txt`, `generation_config.json` — copied verbatim from upstream

## Hardware

Targets Apple Silicon with unified memory. Verified on M3 Ultra (512 GB). Lower-RAM Macs may need to run the LLM forward only (no joint backbone + VAE).

## Loading

```python
import mlx.core as mx
import mlx.nn as nn
from lance_mlx.lance import Lance, LanceConfig
from mlx_vlm.models.qwen2_5_vl.config import ModelConfig as Qwen25VLConfig
import json

cfg = json.load(open("config.json"))
qcfg = Qwen25VLConfig.from_dict(cfg["qwen2_5_vl_config"])
model = Lance(LanceConfig(qwen_config=qcfg, **{k: v for k, v in cfg.items() if k != "qwen2_5_vl_config"}))

# Bundle the three weight files (model + ViT + VAE).
weights = {}
for f in ("model.safetensors", "vit.safetensors", "vae.safetensors"):
    weights.update(mx.load(f))
model.load_weights(list(weights.items()))
```

## Citation

```bibtex
@article{lance2026,
  title   = {Lance: Unified Multimodal Modeling by Multi-Task Synergy},
  author  = {Fu, Fengyi and Huang, Mengqi and Wu, Shaojin and Jiang, Yunsheng and Huo, Yufei and Guo, Jianzhu and others},
  journal = {arXiv preprint arXiv:2605.18678},
  year    = {2026},
  url     = {http://arxiv.org/abs/2605.18678}
}
```

## License

Apache-2.0, inherited from upstream `bytedance-research/Lance`.

## Acknowledgments

- ByteDance Research for the original Lance training and PyTorch release
- The `mlx` and `mlx-vlm` teams at Apple
- Qwen team for Qwen2.5-VL-3B-Instruct

---

**Port status reporting honestly:** this repo currently provides MLX-format weights with verified-loading scaffolding. Inference sampling (T2I/T2V) is a follow-up release; the building blocks are in place but the diffusion loop has not been parity-validated end-to-end yet. Pull requests welcome.