Upload README.md with huggingface_hub
Browse files
README.md
ADDED
|
@@ -0,0 +1,79 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
---
|
| 2 |
+
license: mit
|
| 3 |
+
library_name: mlx
|
| 4 |
+
pipeline_tag: text-to-video
|
| 5 |
+
tags:
|
| 6 |
+
- mlx
|
| 7 |
+
- apple-silicon
|
| 8 |
+
- video-generation
|
| 9 |
+
- audio-driven-video
|
| 10 |
+
- longcat
|
| 11 |
+
- distilled
|
| 12 |
+
- quantized
|
| 13 |
+
- 4-bit
|
| 14 |
+
base_model:
|
| 15 |
+
- mlx-community/LongCat-Video-Avatar-1.5-bf16-dmd-merged
|
| 16 |
+
language:
|
| 17 |
+
- en
|
| 18 |
+
- zh
|
| 19 |
+
---
|
| 20 |
+
|
| 21 |
+
Part of the [LongCat-Video-Avatar 1.5 — MLX](https://huggingface.co/collections/mlx-community/longcat-video-avatar-15-mlx-6a185d1af4a43074d882e375) collection.
|
| 22 |
+
|
| 23 |
+
# LongCat-Video-Avatar-1.5-q4-dmd-merged (MLX)
|
| 24 |
+
|
| 25 |
+
4-bit quantized variant of [mlx-community/LongCat-Video-Avatar-1.5-bf16-dmd-merged](https://huggingface.co/mlx-community/LongCat-Video-Avatar-1.5-bf16-dmd-merged).
|
| 26 |
+
Same model, same DMD pre-merge, same 8-step inference path — just with the
|
| 27 |
+
DiT Linears quantized to 4-bit via `mlx.nn.quantize` for smaller-RAM Macs.
|
| 28 |
+
|
| 29 |
+
| | |
|
| 30 |
+
|---|---|
|
| 31 |
+
| **DiT** | 4-bit quantized (`group_size=64`, skip `final_layer.linear` + embedders + AdaLN) |
|
| 32 |
+
| **DiT shards** | ~11 GB (3 shards) |
|
| 33 |
+
| **umT5 / Whisper / VAE** | bf16 (unchanged from the bf16-dmd-merged variant) |
|
| 34 |
+
| **Total disk** | ~24 GB |
|
| 35 |
+
| **Min unified memory** | ~24 GB |
|
| 36 |
+
| **Inference** | 8-step DMD distilled (unchanged) |
|
| 37 |
+
| **License** | MIT |
|
| 38 |
+
|
| 39 |
+
## Performance
|
| 40 |
+
|
| 41 |
+
Measured on Apple M5 Max (128 GB unified memory), 256 × 432 × 29 frames,
|
| 42 |
+
8-step DMD sampling:
|
| 43 |
+
|
| 44 |
+
| Variant | Wall clock | ms/frame |
|
| 45 |
+
|---|---|---|
|
| 46 |
+
| bf16-dmd-merged | ~105 s | ~3.6 s |
|
| 47 |
+
| **q4-dmd-merged** | ~102 s | ~3.5 s |
|
| 48 |
+
| **q8-dmd-merged** | ~151 s | ~5.2 s |
|
| 49 |
+
|
| 50 |
+
q4 is bandwidth-bound (matches bf16 throughput); q8 currently runs slower
|
| 51 |
+
on M5's quantized matmul kernels but uses ~half the DiT disk vs bf16. Pick
|
| 52 |
+
the variant by RAM budget, not speed.
|
| 53 |
+
|
| 54 |
+
## Loading
|
| 55 |
+
|
| 56 |
+
The runtime pipeline (`longcat_video_avatar.pipeline_mlx.LongCatAvatarPipeline`)
|
| 57 |
+
auto-detects the `quantization` block in `dit/config.json` and applies
|
| 58 |
+
`mlx.nn.quantize` before loading the quantized weights. No user-facing API
|
| 59 |
+
change vs. the bf16 variant.
|
| 60 |
+
|
| 61 |
+
```bash
|
| 62 |
+
hf download mlx-community/LongCat-Video-Avatar-1.5-q4-dmd-merged \
|
| 63 |
+
--local-dir ./weights
|
| 64 |
+
.venv/bin/python scripts/run_inference.py \
|
| 65 |
+
--weights ./weights/.. \
|
| 66 |
+
--variant q4-merged \
|
| 67 |
+
--num-frames 93 \
|
| 68 |
+
--out output.mp4
|
| 69 |
+
```
|
| 70 |
+
|
| 71 |
+
## Source
|
| 72 |
+
|
| 73 |
+
Quantized from the bf16-dmd-merged variant via
|
| 74 |
+
[`recipes/convert_longcat_avatar.py`](https://github.com/xocialize/longcat-avatar-mlx/blob/main/recipes/convert_longcat_avatar.py).
|
| 75 |
+
Run with `--variant q4-merged --out <dir>` to reproduce from Meituan's
|
| 76 |
+
PT sources.
|
| 77 |
+
|
| 78 |
+
See the [bf16-dmd-merged card](https://huggingface.co/mlx-community/LongCat-Video-Avatar-1.5-bf16-dmd-merged)
|
| 79 |
+
for full architecture details, citation, and the non-quantized variant.
|