RockTalk commited on
Commit
c0541b7
Β·
verified Β·
1 Parent(s): c3930a1

TI2I edit: Lance chat template + 3-component CFG verified. Semantic edits working (bow tie, panther).

Browse files
.gitattributes CHANGED
@@ -36,3 +36,4 @@ saved_model/**/* filter=lfs diff=lfs merge=lfs -text
36
  samples/orange_cat_chair.png filter=lfs diff=lfs merge=lfs -text
37
  samples/snowy_peak.png filter=lfs diff=lfs merge=lfs -text
38
  samples/sunset_mountains.png filter=lfs diff=lfs merge=lfs -text
 
 
36
  samples/orange_cat_chair.png filter=lfs diff=lfs merge=lfs -text
37
  samples/snowy_peak.png filter=lfs diff=lfs merge=lfs -text
38
  samples/sunset_mountains.png filter=lfs diff=lfs merge=lfs -text
39
+ samples/edit_bowtie.png filter=lfs diff=lfs merge=lfs -text
README.md CHANGED
@@ -39,7 +39,7 @@ The architecture is **Qwen2.5-VL-3B + parallel MoE-gen experts + Wan 2.2 VAE**.
39
  | MoE-gen routing (per-token attention + MLP + layernorm) | βœ… Working |
40
  | Text-to-video (T2V) | βœ… Working on [Lance-3B-Video-MLX](https://huggingface.co/RockTalk/Lance-3B-Video-MLX) (verified at T_lat=3, 9 frames @ 256Γ—256) |
41
  | Xβ†’T (image understanding) | βœ… Working β€” accurate captioning at ~29 tok/s with KV cache |
42
- | Image editing (TI2I) | ⚠ Pipeline working β€” ViT + VAE conditioning, flow-matching loop, CFG, full strict-load. Output coherence requires further calibration matching PT's training-time chat template. |
43
 
44
  ## Sample generations
45
 
@@ -53,15 +53,16 @@ Verified on M4 Studio (128 GB). 30 steps, CFG=4, 512Γ—512:
53
  | *"a fluffy orange cat sitting on a wooden chair, photorealistic"* | ![cat](samples/orange_cat_chair.png) |
54
  | *"a majestic snowy mountain peak with a dramatic blue sky and clouds"* | ![mountain](samples/snowy_peak.png) |
55
 
56
- ### TI2I β€” image editing (architectural deliverable)
57
 
58
- End-to-end pipeline working: input image β†’ ViT (visual tokens) + VAE encode (condition latent) β†’ text instruction + ViT block + condition latent block + target noise β†’ flow-matching denoise with CFG β†’ VAE decode.
59
 
60
- Pipeline runs at ~960 ms/step at 256Β² (24 steps β‰ˆ 23 s). All loads strict, dual-branch attention routing places ViT/cond tokens through normal weights and target noise through `moe_gen`. The output is influenced by the input image (mean shifts, structural fragments visible) but coherent semantic editing requires matching PT's exact chat template and modality tagging at training time β€” a calibration follow-up.
61
 
62
  | Input | Instruction | Output |
63
  |---|---|---|
64
- | ![cat](samples/orange_cat_chair.png) | *"Change the cat color to blue, keep the chair the same"* | ![edit](samples/edit_blue_cat_attempt.png) |
 
65
 
66
  ### Xβ†’T β€” image understanding
67
 
 
39
  | MoE-gen routing (per-token attention + MLP + layernorm) | βœ… Working |
40
  | Text-to-video (T2V) | βœ… Working on [Lance-3B-Video-MLX](https://huggingface.co/RockTalk/Lance-3B-Video-MLX) (verified at T_lat=3, 9 frames @ 256Γ—256) |
41
  | Xβ†’T (image understanding) | βœ… Working β€” accurate captioning at ~29 tok/s with KV cache |
42
+ | Image editing (TI2I) | βœ… Working β€” ViT + VAE dual conditioning, Lance chat template, three-component CFG (cfg_text + cfg_vit). Semantic edits verified (color change, object addition). |
43
 
44
  ## Sample generations
45
 
 
53
  | *"a fluffy orange cat sitting on a wooden chair, photorealistic"* | ![cat](samples/orange_cat_chair.png) |
54
  | *"a majestic snowy mountain peak with a dramatic blue sky and clouds"* | ![mountain](samples/snowy_peak.png) |
55
 
56
+ ### TI2I β€” image editing
57
 
58
+ End-to-end edit pipeline: input image β†’ ViT (UND tokens) + VAE-encode (cond latent) β†’ Lance edit-mode chat template β†’ three-component CFG flow-matching β†’ VAE decode.
59
 
60
+ Three-component CFG (mirrors PT Lance): `v_final = v_tv_uncond + cfg_text * (v_full - v_t_uncond) + cfg_vit * (v_t_uncond - v_tv_uncond)`. CFG settings: `cfg_text=3.0, cfg_vit=1.0`. ~1.5 s/step at 256Β² (three forward passes per step), 24 steps β‰ˆ 37 s.
61
 
62
  | Input | Instruction | Output |
63
  |---|---|---|
64
+ | ![cat](samples/orange_cat_chair.png) | *"Add a small red bow tie to the cat."* | ![bowtie](samples/edit_bowtie.png) |
65
+ | ![cat](samples/orange_cat_chair.png) | *"Make the cat completely black, like a panther."* | ![panther](samples/edit_black_panther.png) |
66
 
67
  ### Xβ†’T β€” image understanding
68
 
samples/edit_black_panther.png ADDED
samples/edit_blue_cat_attempt.png DELETED
Binary file (91.6 kB)
 
samples/edit_bowtie.png ADDED

Git LFS Details

  • SHA256: 4a9a1ed5949caa09fa2417414d384202a1085161a99c4f985addec91b39908a8
  • Pointer size: 131 Bytes
  • Size of remote file: 107 kB