Initial MLX port of Lance 3B (image variant) + Wan 2.2 VAE

README.md

@@ -0,0 +1,98 @@
+---
+license: apache-2.0
+base_model:
+- bytedance-research/Lance
+- Qwen/Qwen2.5-VL-3B-Instruct
+pipeline_tag: any-to-any
+library_name: mlx
+tags:
+- multimodal
+- mlx
+- apple-silicon
+- image-generation
+- video-generation
+- image-editing
+- video-understanding
+- any-to-any
+- port
+---
+
+# Lance-3B-MLX
+
+A native [MLX](https://github.com/ml-explore/mlx) port of [ByteDance's Lance](https://huggingface.co/bytedance-research/Lance) — a 3B-parameter unified multimodal model for image and video generation, editing, and understanding.
+
+Built on top of the Qwen2.5-VL-3B-Instruct backbone, with Lance's custom multi-task adapters and a Wan 2.2 VAE.
+
+## Status
+
+This is a **work-in-progress port** focused on parity with the original PyTorch checkpoint:
+
+| Component | Status |
+|---|---|
+| Weight conversion (PT → MLX safetensors, with conv layout + name remaps) | ✅ DONE |
+| `modeling_utils` (TimestepEmbedder, PositionEmbedding3D, MLP, sincos tables) | ✅ DONE (5/5 unit tests pass) |
+| `vae_wan22` — image-mode encode/decode | ✅ DONE |
+| `vae_wan22` — video streaming feat-cache | ⏳ PENDING (image-only currently; short clips work via single-pass) |
+| `lance.py` adapters (vae2llm, llm2vae, latent patching, time/pos embed) | ✅ DONE |
+| Flow-matching sampler (`validation_gen` — T2I/T2V/edit) | ⏳ STUB — primitives wired, denoising loop + CFG porting in progress |
+| X→T (understanding) autoregressive loop | ⏳ Phase 2 |
+| NaViT variable-resolution image packing | ⏳ Phase 2 |
+
+## Files
+
+- `model.safetensors` — Lance 3B image variant LLM + adapters (Qwen2.5-VL language model, vae2llm/llm2vae, time_embedder, latent_pos_embed), MLX-layout, **22.9 GB** (1021 tensors, ~6.19B params incl. embed table)
+- `vit.safetensors` — Qwen2.5-VL ViT visual encoder, MLX-layout (NTHWC conv weights), **1.25 GB** (390 tensors, ~668M params, fp16 — bundled here for offline use; the source ships it as a separate shard)
+- `vae.safetensors` — Wan 2.2 VAE, MLX-layout, **2.62 GB** (196 tensors, ~705M params). Converted from the upstream `Wan2.2_VAE.pth` pickle.
+- `config.json` — distilled architecture config + embedded Qwen2.5-VL sub-config
+- `vit_config.json` — Qwen2.5-VL ViT sub-config
+- `tokenizer.json`, `tokenizer_config.json`, `vocab.json`, `merges.txt`, `generation_config.json` — copied verbatim from upstream
+
+## Hardware
+
+Targets Apple Silicon with unified memory. Verified on M3 Ultra (512 GB). Lower-RAM Macs may need to run the LLM forward only (no joint backbone + VAE).
+
+## Loading
+
+```python
+import mlx.core as mx
+import mlx.nn as nn
+from lance_mlx.lance import Lance, LanceConfig
+from mlx_vlm.models.qwen2_5_vl.config import ModelConfig as Qwen25VLConfig
+import json
+
+cfg = json.load(open("config.json"))
+qcfg = Qwen25VLConfig.from_dict(cfg["qwen2_5_vl_config"])
+model = Lance(LanceConfig(qwen_config=qcfg, **{k: v for k, v in cfg.items() if k != "qwen2_5_vl_config"}))
+
+# Bundle the three weight files (model + ViT + VAE).
+weights = {}
+for f in ("model.safetensors", "vit.safetensors", "vae.safetensors"):
+    weights.update(mx.load(f))
+model.load_weights(list(weights.items()))
+```
+
+## Citation
+
+```bibtex
+@article{lance2026,
+  title   = {Lance: Unified Multimodal Modeling by Multi-Task Synergy},
+  author  = {Fu, Fengyi and Huang, Mengqi and Wu, Shaojin and Jiang, Yunsheng and Huo, Yufei and Guo, Jianzhu and others},
+  journal = {arXiv preprint arXiv:2605.18678},
+  year    = {2026},
+  url     = {http://arxiv.org/abs/2605.18678}
+}
+```
+
+## License
+
+Apache-2.0, inherited from upstream `bytedance-research/Lance`.
+
+## Acknowledgments
+
+- ByteDance Research for the original Lance training and PyTorch release
+- The `mlx` and `mlx-vlm` teams at Apple
+- Qwen team for Qwen2.5-VL-3B-Instruct
+
+---
+
+**Port status reporting honestly:** this repo currently provides MLX-format weights with verified-loading scaffolding. Inference sampling (T2I/T2V) is a follow-up release; the building blocks are in place but the diffusion loop has not been parity-validated end-to-end yet. Pull requests welcome.

config.json

@@ -0,0 +1,80 @@
+{
+  "model_type": "lance",
+  "variant": "image",
+  "license": "apache-2.0",
+  "source": "bytedance-research/Lance",
+  "mlx_port_version": "0.0.1",
+  "qwen2_5_vl_config": {
+    "architectures": [
+      "Qwen2_5_VLForConditionalGeneration"
+    ],
+    "attention_dropout": 0.0,
+    "bos_token_id": 151643,
+    "eos_token_id": 151645,
+    "vision_start_token_id": 151652,
+    "vision_end_token_id": 151653,
+    "vision_token_id": 151654,
+    "image_token_id": 151655,
+    "video_token_id": 151656,
+    "hidden_act": "silu",
+    "hidden_size": 2048,
+    "initializer_range": 0.02,
+    "intermediate_size": 11008,
+    "max_position_embeddings": 128000,
+    "max_window_layers": 70,
+    "model_type": "qwen2_5_vl",
+    "num_attention_heads": 16,
+    "num_hidden_layers": 36,
+    "num_key_value_heads": 2,
+    "rms_norm_eps": 1e-06,
+    "rope_theta": 1000000.0,
+    "sliding_window": 32768,
+    "tie_word_embeddings": true,
+    "torch_dtype": "bfloat16",
+    "transformers_version": "4.41.2",
+    "use_cache": true,
+    "use_sliding_window": false,
+    "vision_config": {
+      "depth": 32,
+      "hidden_act": "silu",
+      "hidden_size": 1280,
+      "intermediate_size": 3420,
+      "num_heads": 16,
+      "in_chans": 3,
+      "out_hidden_size": 2048,
+      "patch_size": 14,
+      "spatial_merge_size": 2,
+      "spatial_patch_size": 14,
+      "window_size": 112,
+      "fullatt_block_indexes": [
+        7,
+        15,
+        23,
+        31
+      ],
+      "tokens_per_second": 2,
+      "temporal_patch_size": 2
+    },
+    "rope_scaling": {
+      "type": "mrope",
+      "mrope_section": [
+        16,
+        24,
+        24
+      ]
+    },
+    "vocab_size": 151936
+  },
+  "latent_patch_size": [
+    1,
+    2,
+    2
+  ],
+  "max_latent_size": 32,
+  "max_num_frames": 25,
+  "latent_channel": 48,
+  "vae_downsample_spatial": 16,
+  "vae_downsample_temporal": 4,
+  "connector_act": "gelu_pytorch_tanh",
+  "timestep_shift": 3.5
+}

generation_config.json

@@ -0,0 +1,12 @@
+{
+  "bos_token_id": 151643,
+  "pad_token_id": 151643,
+  "do_sample": true,
+  "eos_token_id": [
+    151645,
+    151643
+  ],
+  "repetition_penalty": 1.05,
+  "temperature": 0.000001,
+  "transformers_version": "4.49.0"
+}

model.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:5ede2f0a376096d51004dad85705938abbfe3cedb64e212631527e76b9547817
+size 24740958849

tokenizer_config.json

@@ -0,0 +1,207 @@
+{
+  "add_prefix_space": false,
+  "added_tokens_decoder": {
+    "151643": {
+      "content": "<|endoftext|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151644": {
+      "content": "<|im_start|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151645": {
+      "content": "<|im_end|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151646": {
+      "content": "<|object_ref_start|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151647": {
+      "content": "<|object_ref_end|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151648": {
+      "content": "<|box_start|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151649": {
+      "content": "<|box_end|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151650": {
+      "content": "<|quad_start|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151651": {
+      "content": "<|quad_end|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151652": {
+      "content": "<|vision_start|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151653": {
+      "content": "<|vision_end|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151654": {
+      "content": "<|vision_pad|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151655": {
+      "content": "<|image_pad|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151656": {
+      "content": "<|video_pad|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151657": {
+      "content": "<tool_call>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "151658": {
+      "content": "</tool_call>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "151659": {
+      "content": "<|fim_prefix|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "151660": {
+      "content": "<|fim_middle|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "151661": {
+      "content": "<|fim_suffix|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "151662": {
+      "content": "<|fim_pad|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "151663": {
+      "content": "<|repo_name|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "151664": {
+      "content": "<|file_sep|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    }
+  },
+  "additional_special_tokens": [
+    "<|im_start|>",
+    "<|im_end|>",
+    "<|object_ref_start|>",
+    "<|object_ref_end|>",
+    "<|box_start|>",
+    "<|box_end|>",
+    "<|quad_start|>",
+    "<|quad_end|>",
+    "<|vision_start|>",
+    "<|vision_end|>",
+    "<|vision_pad|>",
+    "<|image_pad|>",
+    "<|video_pad|>"
+  ],
+  "bos_token": null,
+  "chat_template": "{% set image_count = namespace(value=0) %}{% set video_count = namespace(value=0) %}{% for message in messages %}{% if loop.first and message['role'] != 'system' %}<|im_start|>system\nYou are a helpful assistant.<|im_end|>\n{% endif %}<|im_start|>{{ message['role'] }}\n{% if message['content'] is string %}{{ message['content'] }}<|im_end|>\n{% else %}{% for content in message['content'] %}{% if content['type'] == 'image' or 'image' in content or 'image_url' in content %}{% set image_count.value = image_count.value + 1 %}{% if add_vision_id %}Picture {{ image_count.value }}: {% endif %}<|vision_start|><|image_pad|><|vision_end|>{% elif content['type'] == 'video' or 'video' in content %}{% set video_count.value = video_count.value + 1 %}{% if add_vision_id %}Video {{ video_count.value }}: {% endif %}<|vision_start|><|video_pad|><|vision_end|>{% elif 'text' in content %}{{ content['text'] }}{% endif %}{% endfor %}<|im_end|>\n{% endif %}{% endfor %}{% if add_generation_prompt %}<|im_start|>assistant\n{% endif %}",
+  "clean_up_tokenization_spaces": false,
+  "eos_token": "<|im_end|>",
+  "errors": "replace",
+  "model_max_length": 131072,
+  "pad_token": "<|endoftext|>",
+  "split_special_tokens": false,
+  "tokenizer_class": "Qwen2Tokenizer",
+  "unk_token": null,
+  "add_bos_token": false
+}

vae.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:c2674b01dce2c027fa33fd1a212f858d65d7d2a99b9559363d8e1d35629faf37
+size 2818780182