---
license: apache-2.0
language:
- en
- zh
library_name: mlx
pipeline_tag: image-to-image
tags:
- mlx
- apple-silicon
- lance
- bytedance
- multimodal
- text-to-image
- image-editing
- vqa
- qwen2.5-vl
base_model: bytedance-research/Lance
---

> 📂 Part of the **[Lance MLX collection](https://huggingface.co/collections/mlx-community/lance-mlx-6a0f3cd5648a74f8283fc8a4)** on mlx-community.


# Lance-3B-bf16 (MLX, image specialist)

MLX port of **ByteDance Intelligent Creation Lab's Lance** unified multimodal model — the image-specialist `Lance_3B` checkpoint, converted to bf16 for Apple Silicon. ~6.19 B LLM parameters in MoT (Mixture-of-Transformer-Experts) layout, plus the Qwen2.5-VL ViT (~669 M) and Lance's bundled Wan2.2 VAE (~705 M) for full image-task coverage.

> **Lance** is ByteDance's 3B-active unified multimodal model ([paper](https://arxiv.org/abs/2605.18678), [code](https://github.com/bytedance/Lance), [HF original](https://huggingface.co/bytedance-research/Lance)). This is **not** [Lance/LanceDB](https://github.com/lancedb/lance), the columnar data format.

## Status

🟢 **Production-ready for image tasks on Apple Silicon as of 2026-05-21.**

| Capability | Status |
|---|---|
| t2i (text → image) | ✅ Photorealistic, prompt-aligned. 768² output at ~6.7 s/step. |
| image_edit (instruction-based editing) | ✅ Identity + style + signature preservation verified. ~6.7 s/step. |
| x2t_image (image understanding / VQA) | ✅ Content-correct across all 6 oracle cases. |
| KV cache for autoregressive decode | ✅ 1.7×–2.8× speedup over no-cache baseline. |

For video tasks (t2v, video_edit, x2t_video), see [`mlx-community/Lance-3B-Video-bf16`](https://huggingface.co/mlx-community/Lance-3B-Video-bf16). All six Lance task families are now validated end-to-end on Apple Silicon as of 2026-05-21.

The 48-channel Wan2.2 VAE is bundled here for convenience but also published standalone at [`mlx-community/Wan2.2-VAE-Lance-bf16`](https://huggingface.co/mlx-community/Wan2.2-VAE-Lance-bf16) — both image_edit and the video pipelines need it.

### Hardware envelope (`memory_mode`, 2026-06-02)

The `lance-mlx` source repo's [`memory_mode`](https://github.com/xocialize/lance-mlx) knob (auto / parallel / relay) brings bf16 image generation within reach of 8–16 GB Apple Silicon Macs:

| RAM | Mode | t2i / image_edit | Notes |
|---|---|---|---|
| **8–16 GB** | `relay` (auto-resolved) | ✅ 768² bf16, no swap | Single-shot per pipeline load — re-prefill reloads the UND tower. ~5 min/image at 768² on a base-RAM M-series Mac. |
| **24 GB+** | `parallel` (auto-resolved) | ✅ 768² bf16, reusable | Pipeline stays resident across calls. |

`relay` produces **byte-identical output** to `parallel` (MD5-verified on real Lance-3B-bf16) — it sheds the UND tower after prefill and frees the GEN tower before VAE decode, so peak memory ≈ heaviest single phase rather than the sum of all three. Default `auto` resolves by `mx.device_info()`'s recommended working-set size with the split at ~18 GiB. The same envelope and modes apply to [`mlx-community/Lance-3B-Video-bf16`](https://huggingface.co/mlx-community/Lance-3B-Video-bf16) for video tasks.

### Lossless streaming VAE decode (`lossless_decode`, 2026-06-05)

`generate(..., lossless_decode=True)` (default since 2026-06-05) uses a **bit-identical** streaming VAE decode that's lighter than the naive whole `dec(z)` in every measured config — via temporal causal-cache streaming (flat in frame count) and spatial halo-tile + crop (no blend ⇒ exact). 50-case bit-identity test in [`lance-mlx`](https://github.com/xocialize/lance-mlx) with a negative control verifies the guarantee. `lossless_decode=False` keeps the lossy trapezoidal-blend tiling from the prior path (~1.5–4.8 / 255 off the reference) for the one config where lossless exceeds 16 GB (768² video; lossless on 1024² image fits 16 GB cleanly at ~12.2 GB).

Full envelope and `ri_phys`-measured numbers (true OS-committed footprint, supersedes the older `mx.get_peak_memory()` quotes which under-report ~2×) in the source repo's [`LIMITS.md`](https://github.com/xocialize/lance-mlx/blob/main/LIMITS.md).

For VQA-only use cases on small Macs, the compressed [`mlx-community/Lance-3B-AWQ-INT4`](https://huggingface.co/mlx-community/Lance-3B-AWQ-INT4) variant is also available (3.3 GB LLM, 6-9× faster long-form decode; VQA only, not for image generation).

## Quickstart

Install from the source repo (will be on PyPI in a follow-up release):

```bash
git clone https://github.com/xocialize/lance-mlx
cd lance-mlx && uv sync
```

Download this checkpoint:

```python
from huggingface_hub import snapshot_download
weights = snapshot_download("mlx-community/Lance-3B-bf16")
```

### Text-to-image

```python
from lance_mlx.pipeline.t2i import TextToImagePipeline

pipe = TextToImagePipeline.from_pretrained(
    lance_weights_dir=weights,
    vae_safetensors=f"{weights}/vae.safetensors",
)
image = pipe.generate(
    "A photorealistic tabby cat holding up a colorful STOP sign on a sunlit street.",
    height=768, width=768, num_steps=30, cfg_scale=4.0, seed=42,
)
image.save("cat_with_stop.png")
```

### Image editing

```python
from lance_mlx.pipeline.image_edit import ImageEditPipeline

pipe = ImageEditPipeline.from_pretrained(
    lance_weights_dir=weights,
    vae_safetensors=f"{weights}/vae.safetensors",
)
edited = pipe.generate(
    input_image="portrait.jpg",
    instruction="Remove the hat from the painting.",
    height=768, width=768, num_steps=30, cfg_scale=4.0, seed=42,
)
edited.save("portrait_no_hat.png")
```

### Image VQA / understanding

```python
from lance_mlx.pipeline.understanding import UnderstandingPipeline
from PIL import Image

pipe = UnderstandingPipeline.from_pretrained(
    lance_weights_dir=weights,
    vit_safetensors=f"{weights}/vit.safetensors",
)
answer = pipe.generate(
    Image.open("license_plate.png"),
    "What is the license plate number visible in this image?",
    max_new_tokens=64, prompt_style="lance",
)
print(answer)
```

## Performance (M5 Max 128 GB, macOS 26.2, MLX bf16)

| Task | Configuration | Wall-clock |
|---|---|---|
| t2i | 768² × 30 steps × CFG=4.0 | ~201 s |
| image_edit | 768² × 30 steps × CFG=4.0 | ~201 s |
| x2t_image | 6 oracle cases (5–100 token answers), KV-cached | ~34 s combined |

KV cache scales with answer length: 1.7× speedup on a 5-token answer, 2.8× on a ~100-token answer.

## Architecture

- **Two expert towers** (`LLM_UND`, `LLM_GEN`), each initialized from Qwen2.5-VL-3B-Instruct, with per-expert FFN, output projection, and QK-norm.
- **Modality-deterministic routing:** text + Qwen2.5-VL ViT semantic tokens → `LLM_UND` (autoregressive next-token); Wan2.2 VAE latent tokens → `LLM_GEN` (flow-matching velocity prediction). No learned gate.
- **MaPE** — modality-aware RoPE with per-modality temporal anchor (image-gen tokens re-anchored to t=1000).
- **Wan2.2 3D causal VAE** (16× spatial / 4× temporal compression, **48-channel** latent — Lance bundles its own VAE; do NOT use the public 16-ch `wan2.2_vae.safetensors`).
- **Bidirectional attention within latent block** — `causal_mask OR full_and_noise_mask` per upstream `data/data_utils.py::create_sparse_mask`. Without this, the noisy-VAE position 0 of a 2304-token image grid can only see itself + text, producing blurry outputs across all prompts.
- **Untied LM head**.

## Files in this repo

| File | Size | Purpose |
|---|---|---|
| `model.safetensors` | 12.37 GB | LLM weights (1021 tensors, both UND + GEN towers) |
| `vit.safetensors` | 1.34 GB | Qwen2.5-VL ViT (semantic encoder for x2t_image + image_edit) |
| `vae.safetensors` | 1.41 GB | Lance's bundled Wan2.2 VAE (encoder + decoder, 48-ch) |
| `config.json` | – | `Qwen2_5_VLForConditionalGeneration` config with `tie_word_embeddings=false` |
| `conversion_report.json` | – | Provenance of safetensors conversion (PyTorch → MLX bf16) |
| `tokenizer.json` / `vocab.json` | – | Qwen2.5-VL vocabulary (151,936 tokens) |

## Provenance

Source: `bytedance-research/Lance/Lance_3B/model.safetensors` (1021 tensors, 6.185 B params).
Conversion script: [`scripts/02_convert.py`](https://github.com/xocialize/lance-mlx/blob/main/scripts/02_convert.py) in the lance-mlx repo. The script:

- Loads original PyTorch safetensors, keeps F32 for normalization scales (per [Phase 1b notes](https://github.com/xocialize/lance-mlx/blob/main/notes/phase1b_converter_design.md)).
- Strips the `language_model.` prefix; the MLX `LanceModel` is the root, not nested.
- Splits the bundled ViT (`vit_model.*` keys) into a sibling `vit.safetensors` for parity with the `Lance_3B` distribution shape.
- Re-keys `llm2vae.weight/bias` and `time_embedder.mlp.{0,2}.{weight,bias}` to match scaffolded MLX modules.

Wan2.2 VAE source: `bytedance-research/Lance/Wan2.2_VAE.pth` → `scripts/06_convert_wan_vae.py`. Roundtrip MAD on a real photo at 768² is ~7/255 in u8 domain.

## Limitations

- **bf16 only.** 4-bit + 8-bit quantization in progress. Naive INT4 has been observed to degrade the GEN expert (per Reza2kn/lance-quant's findings); quantization needs per-tower calibration.
- **English + Chinese prompts work; other languages are training-distribution-limited** (Qwen2.5-VL was trained primarily on en + zh).
- **No streaming / batching API yet.** Single-image, single-prompt generation only.
- **CFG runs the LLM twice per step.** A future KV-cache for the text + clean-ref prefix would save ~30% on image_edit.

## Documented divergences from upstream PyTorch

- Outputs differ in low-level pixel detail from a CUDA reference run on the same seed/prompt (~1–5% per-pixel deviation expected from bf16 vs fp32, MLX RoPE vs PyTorch RoPE rounding, and a small number of intermediate-norm precision steps). Semantic correctness preserved across the 6 x2t_image oracle cases and all visually-verified t2i + image_edit prompts.
- x2t_image answers differ stylistically from Phase 0 oracle (PyTorch) — consistent across all 6 cases. Tracked as a Phase 5 parity follow-up; does not affect content correctness.

## License

This MLX port: **Apache 2.0**.

Underlying weights:
- Lance: Apache 2.0 (ByteDance Intelligent Creation Lab).
- Wan2.2 VAE: Apache 2.0 (Alibaba).
- Qwen2.5-VL: Apache 2.0 (Alibaba).

See [`NOTICE`](https://github.com/xocialize/lance-mlx/blob/main/NOTICE) for full attribution.

## Citation

```bibtex
@article{fu2026lance,
  title={Lance: Unified Multimodal Modeling by Multi-Task Synergy},
  author={Fu, Fengyi and Huang, Mengqi and Wu, Shaojin and others},
  journal={arXiv preprint arXiv:2605.18678},
  year={2026}
}
```

## Links

- **MLX port code + phase notes:** [`github.com/xocialize/lance-mlx`](https://github.com/xocialize/lance-mlx)
- **Original PyTorch model:** [`bytedance-research/Lance`](https://huggingface.co/bytedance-research/Lance)
- **Wan2.2 VAE (standalone):** [`mlx-community/Wan2.2-VAE-Lance-bf16`](https://huggingface.co/mlx-community/Wan2.2-VAE-Lance-bf16)
- **Video specialist (alpha):** [`mlx-community/Lance-3B-Video-bf16`](https://huggingface.co/mlx-community/Lance-3B-Video-bf16)