Add files using upload-large-folder tool

.gitattributes

@@ -33,3 +33,15 @@ saved_model/**/* filter=lfs diff=lfs merge=lfs -text
 *.zip filter=lfs diff=lfs merge=lfs -text
 *.zst filter=lfs diff=lfs merge=lfs -text
 *tfevents* filter=lfs diff=lfs merge=lfs -text
+examples/nvidia_example_caption_bf16.png filter=lfs diff=lfs merge=lfs -text
+examples/nvidia_example_caption_sdnq_uint4_svd.png filter=lfs diff=lfs merge=lfs -text
+examples/01_metro_archive_reading_room_sdnq_uint4_svd.png filter=lfs diff=lfs merge=lfs -text
+examples/02_arctic_greenhouse_night_shift_sdnq_uint4_svd.png filter=lfs diff=lfs merge=lfs -text
+examples/03_control_room_restoration_sdnq_uint4_svd.png filter=lfs diff=lfs merge=lfs -text
+examples/04_rain_market_cross_section_sdnq_uint4_svd.png filter=lfs diff=lfs merge=lfs -text
+examples/05_manuscript_restoration_table_sdnq_uint4_svd.png filter=lfs diff=lfs merge=lfs -text
+examples/06_robotic_assembly_line_signage_sdnq_uint4_svd.png filter=lfs diff=lfs merge=lfs -text
+examples/07_kitchen_storm_chess_table_sdnq_uint4_svd.png filter=lfs diff=lfs merge=lfs -text
+examples/08_orbital_cockpit_cyrillic_ui_sdnq_uint4_svd.png filter=lfs diff=lfs merge=lfs -text
+examples/09_flood_command_center_sdnq_uint4_svd.png filter=lfs diff=lfs merge=lfs -text
+examples/10_cyrillic_newspaper_press_sdnq_uint4_svd.png filter=lfs diff=lfs merge=lfs -text

README.md

@@ -0,0 +1,150 @@
+---
+base_model: nvidia/Cosmos3-Super-Text2Image
+library_name: diffusers
+pipeline_tag: text-to-image
+tags:
+  - cosmos3
+  - diffusers
+  - sdnq
+  - text-to-image
+  - uint4
+license: other
+license_name: openmdw1.1-license
+license_link: https://openmdw.ai/license/1-1/
+---
+
+# Cosmos3-Super-Text2Image SDNQ UINT4 SVD Transformer
+
+This repository contains a transformer-only SDNQ quantization for [nvidia/Cosmos3-Super-Text2Image](https://huggingface.co/nvidia/Cosmos3-Super-Text2Image).
+
+It does not repeat the original model card. Read NVIDIA's model card, prompt-format guidance, license, and safety notes here:
+[nvidia/Cosmos3-Super-Text2Image](https://huggingface.co/nvidia/Cosmos3-Super-Text2Image).
+
+Only `transformer/` is provided as a weight artifact. The VAE, scheduler, tokenizers, safety checker, and other components are loaded from the base model.
+
+The quantization format comes from [Disty0/sdnq](https://github.com/Disty0/sdnq). SD.Next's quantization overview is here:
+[vladmandic/sdnext Quantization](https://github.com/vladmandic/sdnext/wiki/Quantization).
+
+## Recipe
+
+| Setting | Value |
+| --- | --- |
+| Weights dtype | `uint4` |
+| Static quantization | `True` |
+| Dynamic quantization | `False` |
+| SVD | `True` |
+| SVD rank / steps | `32` / `8` |
+| Quantized matmul | `False` |
+| Dequantize FP32 | `False` |
+| Quantized conv / embedding | `False` / `False` |
+
+Quantization run: 33.74s; save time: 42.32s; transformer safetensors: 34.89 GiB.
+
+## Assemble The Pipeline
+
+```python
+import json
+import torch
+from diffusers import Cosmos3OmniPipeline, Cosmos3OmniTransformer
+from diffusers.schedulers.scheduling_unipc_multistep import UniPCMultistepScheduler
+from huggingface_hub import snapshot_download
+from sdnq.loader import load_sdnq_model
+
+snapshot_path = snapshot_download("WaveCut/Cosmos3-Super-Text2Image-SDNQ-Uint4-SVD-Transformer")
+transformer = load_sdnq_model(
+    f"{snapshot_path}/transformer",
+    model_cls=Cosmos3OmniTransformer,
+    dtype=torch.bfloat16,
+    device=torch.device("cuda"),
+    dequantize_fp32=False,
+    use_quantized_matmul=False,
+)
+
+pipe = Cosmos3OmniPipeline.from_pretrained(
+    "nvidia/Cosmos3-Super-Text2Image",
+    transformer=transformer,
+    torch_dtype=torch.bfloat16,
+    device_map="cuda",
+    enable_safety_checker=True,
+)
+pipe.scheduler = UniPCMultistepScheduler.from_config(pipe.scheduler.config, flow_shift=3.0)
+pipe.to("cuda")
+
+json_caption = {
+    "subjects": [],
+    "background_setting": "A concise scene description.",
+    "comprehensive_t2i_caption": "A detailed natural-language caption.",
+    "resolution": {"H": 1024, "W": 1024},
+    "aspect_ratio": "1,1",
+}
+
+result = pipe(
+    prompt=json.dumps(json_caption),
+    negative_prompt="",
+    num_frames=1,
+    height=1024,
+    width=1024,
+    num_inference_steps=50,
+    guidance_scale=4.0,
+    generator=torch.Generator(device="cuda").manual_seed(1143),
+)
+result.video[0].save("cosmos3_sdnq_uint4_svd.png")
+```
+
+`load_sdnq_model` expects a local path. Download this repository first, or use `huggingface_hub.snapshot_download("WaveCut/Cosmos3-Super-Text2Image-SDNQ-Uint4-SVD-Transformer")` and pass `snapshot_path + "/transformer"`.
+
+## Benchmarks
+
+Measured on one RunPod NVIDIA B200 instance with local container storage, cached model files, PyTorch `2.9.1+cu130`, 1024x1024 image generation, 50 inference steps, guidance scale 4.0, `flow_shift=3.0`, system prompt enabled.
+
+### Transformer Component Load
+
+| Variant | Load to CUDA | VRAM after load | Torch allocated | Torch reserved | Transformer safetensors |
+| --- | ---: | ---: | ---: | ---: | ---: |
+| BF16 base transformer | 23.91s | 122,760 MiB | 122,121 MiB | 122,132 MiB | 119.21 GiB |
+| SDNQ UINT4 SVD transformer | 10.74s | 37,096 MiB | 35,760 MiB | 36,376 MiB | 34.89 GiB |
+
+### Full Pipeline Generation
+
+The stress set is ten handwritten JSON-caption prompts designed to stress Cyrillic text, reflections, multi-object composition, anatomy, small details, and scene-following.
+
+| Variant | Full pipeline load | VRAM after load | Torch allocated after load | Avg generation time | Min / max generation time | Peak sampled VRAM | Images |
+| --- | ---: | ---: | ---: | ---: | ---: | ---: | ---: |
+| BF16 base pipeline | 31.31s | 125,134 MiB | 124,386 MiB | 16.05s | 15.51s / 17.97s | 141,104 MiB | 10 |
+| SDNQ UINT4 SVD pipeline | 23.42s | 40,590 MiB | 39,269 MiB | 38.28s | 37.74s / 40.68s | 56,538 MiB | 10 |
+
+### Original NVIDIA Example Caption
+
+The original model repository provides [`assets/example_caption.json`](https://huggingface.co/nvidia/Cosmos3-Super-Text2Image/blob/main/assets/example_caption.json). The images below are generated locally with the same JSON-caption, seed 1143, 1024x1024, 50 steps, guidance scale 4.0.
+
+| Variant | Pipeline load | Generation time | Peak sampled VRAM |
+| --- | ---: | ---: | ---: |
+| BF16 base pipeline | 35.41s | 18.01s | 141,098 MiB |
+| SDNQ UINT4 SVD pipeline | 24.06s | 74.37s | 56,576 MiB |
+
+BF16 reference output:
+
+![BF16 output for NVIDIA example caption](examples/nvidia_example_caption_bf16.png)
+
+SDNQ UINT4 SVD output:
+
+![SDNQ UINT4 SVD output for NVIDIA example caption](examples/nvidia_example_caption_sdnq_uint4_svd.png)
+
+## Stress Prompt Examples
+
+The following ten images use the same handwritten stress prompt set and seeds as the benchmark table.
+
+![01 metro archive reading room](examples/01_metro_archive_reading_room_sdnq_uint4_svd.png)
+![02 arctic greenhouse night shift](examples/02_arctic_greenhouse_night_shift_sdnq_uint4_svd.png)
+![03 control room restoration](examples/03_control_room_restoration_sdnq_uint4_svd.png)
+![04 rain market cross section](examples/04_rain_market_cross_section_sdnq_uint4_svd.png)
+![05 manuscript restoration table](examples/05_manuscript_restoration_table_sdnq_uint4_svd.png)
+![06 robotic assembly line signage](examples/06_robotic_assembly_line_signage_sdnq_uint4_svd.png)
+![07 kitchen storm chess table](examples/07_kitchen_storm_chess_table_sdnq_uint4_svd.png)
+![08 orbital cockpit cyrillic ui](examples/08_orbital_cockpit_cyrillic_ui_sdnq_uint4_svd.png)
+![09 flood command center](examples/09_flood_command_center_sdnq_uint4_svd.png)
+![10 cyrillic newspaper press](examples/10_cyrillic_newspaper_press_sdnq_uint4_svd.png)
+
+## Notes
+
+This repository is an independent transformer-only quantization artifact. NVIDIA's original card states that Cosmos3-Super-Text2Image was tested in BF16; this SDNQ artifact should be treated as an experimental deployment variant and evaluated for each workload.

transformer/config.json

@@ -0,0 +1,633 @@
+{
+  "_class_name": "Cosmos3OmniTransformer",
+  "_diffusers_version": "0.39.0.dev0",
+  "_name_or_path": "nvidia/Cosmos3-Super-Text2Image",
+  "action_dim": 32,
+  "action_gen": false,
+  "attention_bias": false,
+  "attention_dropout": 0.0,
+  "base_fps": 24,
+  "dtype": "bfloat16",
+  "enable_fps_modulation": true,
+  "freeze_und": false,
+  "head_dim": 128,
+  "hidden_act": "silu",
+  "hidden_size": 5120,
+  "initializer_range": 0.02,
+  "intermediate_size": 25600,
+  "joint_attn_implementation": "two_way",
+  "latent_channel": 48,
+  "latent_patch_size": 2,
+  "max_action_dim": 32,
+  "max_position_embeddings": 262144,
+  "model_type": "qwen3_vl_text",
+  "num_attention_heads": 64,
+  "num_embodiment_domains": 32,
+  "num_hidden_layers": 64,
+  "num_key_value_heads": 8,
+  "patch_latent_dim": 192,
+  "position_embedding_type": "unified_3d_mrope",
+  "qk_norm": false,
+  "qk_norm_for_diffusion": true,
+  "qk_norm_for_text": true,
+  "quantization_config": {
+    "add_skip_keys": false,
+    "dequantize_fp32": false,
+    "dynamic_loss_threshold": null,
+    "group_size": 0,
+    "hadamard_group_size": 128,
+    "is_integer": true,
+    "is_training": false,
+    "modules_dtype_dict": {},
+    "modules_quant_config": {},
+    "modules_to_not_convert": [
+      "time_text_embed",
+      ".x_embedder",
+      "lm_head",
+      ".txt_out",
+      ".txt_in",
+      ".vid_in",
+      ".y_embedder",
+      ".context_embedder",
+      "patch_embed",
+      ".emb_in",
+      ".condition_embedder",
+      ".img_in",
+      ".final_layer",
+      "patch_emb",
+      "multi_modal_projector",
+      ".proj_out",
+      "patch_embedding",
+      ".vid_out",
+      "wte",
+      ".img_out",
+      ".time_embed",
+      "norm",
+      ".norm_out",
+      ".t_embedder",
+      "embed_tokens",
+      ".emb_out",
+      "time_embedder",
+      "layers.0.self_attn.norm_q.weight",
+      "layers.0.self_attn.norm_k.weight",
+      "layers.0.self_attn.norm_added_q.weight",
+      "layers.0.self_attn.norm_added_k.weight",
+      "layers.0.input_layernorm.weight",
+      "layers.0.input_layernorm_moe_gen.weight",
+      "layers.0.post_attention_layernorm.weight",
+      "layers.0.post_attention_layernorm_moe_gen.weight",
+      "layers.1.self_attn.norm_q.weight",
+      "layers.1.self_attn.norm_k.weight",
+      "layers.1.self_attn.norm_added_q.weight",
+      "layers.1.self_attn.norm_added_k.weight",
+      "layers.1.input_layernorm.weight",
+      "layers.1.input_layernorm_moe_gen.weight",
+      "layers.1.post_attention_layernorm.weight",
+      "layers.1.post_attention_layernorm_moe_gen.weight",
+      "layers.2.self_attn.norm_q.weight",
+      "layers.2.self_attn.norm_k.weight",
+      "layers.2.self_attn.norm_added_q.weight",
+      "layers.2.self_attn.norm_added_k.weight",
+      "layers.2.input_layernorm.weight",
+      "layers.2.input_layernorm_moe_gen.weight",
+      "layers.2.post_attention_layernorm.weight",
+      "layers.2.post_attention_layernorm_moe_gen.weight",
+      "layers.3.self_attn.norm_q.weight",
+      "layers.3.self_attn.norm_k.weight",
+      "layers.3.self_attn.norm_added_q.weight",
+      "layers.3.self_attn.norm_added_k.weight",
+      "layers.3.input_layernorm.weight",
+      "layers.3.input_layernorm_moe_gen.weight",
+      "layers.3.post_attention_layernorm.weight",
+      "layers.3.post_attention_layernorm_moe_gen.weight",
+      "layers.4.self_attn.norm_q.weight",
+      "layers.4.self_attn.norm_k.weight",
+      "layers.4.self_attn.norm_added_q.weight",
+      "layers.4.self_attn.norm_added_k.weight",
+      "layers.4.input_layernorm.weight",
+      "layers.4.input_layernorm_moe_gen.weight",
+      "layers.4.post_attention_layernorm.weight",
+      "layers.4.post_attention_layernorm_moe_gen.weight",
+      "layers.5.self_attn.norm_q.weight",
+      "layers.5.self_attn.norm_k.weight",
+      "layers.5.self_attn.norm_added_q.weight",
+      "layers.5.self_attn.norm_added_k.weight",
+      "layers.5.input_layernorm.weight",
+      "layers.5.input_layernorm_moe_gen.weight",
+      "layers.5.post_attention_layernorm.weight",
+      "layers.5.post_attention_layernorm_moe_gen.weight",
+      "layers.6.self_attn.norm_q.weight",
+      "layers.6.self_attn.norm_k.weight",
+      "layers.6.self_attn.norm_added_q.weight",
+      "layers.6.self_attn.norm_added_k.weight",
+      "layers.6.input_layernorm.weight",
+      "layers.6.input_layernorm_moe_gen.weight",
+      "layers.6.post_attention_layernorm.weight",
+      "layers.6.post_attention_layernorm_moe_gen.weight",
+      "layers.7.self_attn.norm_q.weight",
+      "layers.7.self_attn.norm_k.weight",
+      "layers.7.self_attn.norm_added_q.weight",
+      "layers.7.self_attn.norm_added_k.weight",
+      "layers.7.input_layernorm.weight",
+      "layers.7.input_layernorm_moe_gen.weight",
+      "layers.7.post_attention_layernorm.weight",
+      "layers.7.post_attention_layernorm_moe_gen.weight",
+      "layers.8.self_attn.norm_q.weight",
+      "layers.8.self_attn.norm_k.weight",
+      "layers.8.self_attn.norm_added_q.weight",
+      "layers.8.self_attn.norm_added_k.weight",
+      "layers.8.input_layernorm.weight",
+      "layers.8.input_layernorm_moe_gen.weight",
+      "layers.8.post_attention_layernorm.weight",
+      "layers.8.post_attention_layernorm_moe_gen.weight",
+      "layers.9.self_attn.norm_q.weight",
+      "layers.9.self_attn.norm_k.weight",
+      "layers.9.self_attn.norm_added_q.weight",
+      "layers.9.self_attn.norm_added_k.weight",
+      "layers.9.input_layernorm.weight",
+      "layers.9.input_layernorm_moe_gen.weight",
+      "layers.9.post_attention_layernorm.weight",
+      "layers.9.post_attention_layernorm_moe_gen.weight",
+      "layers.10.self_attn.norm_q.weight",
+      "layers.10.self_attn.norm_k.weight",
+      "layers.10.self_attn.norm_added_q.weight",
+      "layers.10.self_attn.norm_added_k.weight",
+      "layers.10.input_layernorm.weight",
+      "layers.10.input_layernorm_moe_gen.weight",
+      "layers.10.post_attention_layernorm.weight",
+      "layers.10.post_attention_layernorm_moe_gen.weight",
+      "layers.11.self_attn.norm_q.weight",
+      "layers.11.self_attn.norm_k.weight",
+      "layers.11.self_attn.norm_added_q.weight",
+      "layers.11.self_attn.norm_added_k.weight",
+      "layers.11.input_layernorm.weight",
+      "layers.11.input_layernorm_moe_gen.weight",
+      "layers.11.post_attention_layernorm.weight",
+      "layers.11.post_attention_layernorm_moe_gen.weight",
+      "layers.12.self_attn.norm_q.weight",
+      "layers.12.self_attn.norm_k.weight",
+      "layers.12.self_attn.norm_added_q.weight",
+      "layers.12.self_attn.norm_added_k.weight",
+      "layers.12.input_layernorm.weight",
+      "layers.12.input_layernorm_moe_gen.weight",
+      "layers.12.post_attention_layernorm.weight",
+      "layers.12.post_attention_layernorm_moe_gen.weight",
+      "layers.13.self_attn.norm_q.weight",
+      "layers.13.self_attn.norm_k.weight",
+      "layers.13.self_attn.norm_added_q.weight",
+      "layers.13.self_attn.norm_added_k.weight",
+      "layers.13.input_layernorm.weight",
+      "layers.13.input_layernorm_moe_gen.weight",
+      "layers.13.post_attention_layernorm.weight",
+      "layers.13.post_attention_layernorm_moe_gen.weight",
+      "layers.14.self_attn.norm_q.weight",
+      "layers.14.self_attn.norm_k.weight",
+      "layers.14.self_attn.norm_added_q.weight",
+      "layers.14.self_attn.norm_added_k.weight",
+      "layers.14.input_layernorm.weight",
+      "layers.14.input_layernorm_moe_gen.weight",
+      "layers.14.post_attention_layernorm.weight",
+      "layers.14.post_attention_layernorm_moe_gen.weight",
+      "layers.15.self_attn.norm_q.weight",
+      "layers.15.self_attn.norm_k.weight",
+      "layers.15.self_attn.norm_added_q.weight",
+      "layers.15.self_attn.norm_added_k.weight",
+      "layers.15.input_layernorm.weight",
+      "layers.15.input_layernorm_moe_gen.weight",
+      "layers.15.post_attention_layernorm.weight",
+      "layers.15.post_attention_layernorm_moe_gen.weight",
+      "layers.16.self_attn.norm_q.weight",
+      "layers.16.self_attn.norm_k.weight",
+      "layers.16.self_attn.norm_added_q.weight",
+      "layers.16.self_attn.norm_added_k.weight",
+      "layers.16.input_layernorm.weight",
+      "layers.16.input_layernorm_moe_gen.weight",
+      "layers.16.post_attention_layernorm.weight",
+      "layers.16.post_attention_layernorm_moe_gen.weight",
+      "layers.17.self_attn.norm_q.weight",
+      "layers.17.self_attn.norm_k.weight",
+      "layers.17.self_attn.norm_added_q.weight",
+      "layers.17.self_attn.norm_added_k.weight",
+      "layers.17.input_layernorm.weight",
+      "layers.17.input_layernorm_moe_gen.weight",
+      "layers.17.post_attention_layernorm.weight",
+      "layers.17.post_attention_layernorm_moe_gen.weight",
+      "layers.18.self_attn.norm_q.weight",
+      "layers.18.self_attn.norm_k.weight",
+      "layers.18.self_attn.norm_added_q.weight",
+      "layers.18.self_attn.norm_added_k.weight",
+      "layers.18.input_layernorm.weight",
+      "layers.18.input_layernorm_moe_gen.weight",
+      "layers.18.post_attention_layernorm.weight",
+      "layers.18.post_attention_layernorm_moe_gen.weight",
+      "layers.19.self_attn.norm_q.weight",
+      "layers.19.self_attn.norm_k.weight",
+      "layers.19.self_attn.norm_added_q.weight",
+      "layers.19.self_attn.norm_added_k.weight",
+      "layers.19.input_layernorm.weight",
+      "layers.19.input_layernorm_moe_gen.weight",
+      "layers.19.post_attention_layernorm.weight",
+      "layers.19.post_attention_layernorm_moe_gen.weight",
+      "layers.20.self_attn.norm_q.weight",
+      "layers.20.self_attn.norm_k.weight",
+      "layers.20.self_attn.norm_added_q.weight",
+      "layers.20.self_attn.norm_added_k.weight",
+      "layers.20.input_layernorm.weight",
+      "layers.20.input_layernorm_moe_gen.weight",
+      "layers.20.post_attention_layernorm.weight",
+      "layers.20.post_attention_layernorm_moe_gen.weight",
+      "layers.21.self_attn.norm_q.weight",
+      "layers.21.self_attn.norm_k.weight",
+      "layers.21.self_attn.norm_added_q.weight",
+      "layers.21.self_attn.norm_added_k.weight",
+      "layers.21.input_layernorm.weight",
+      "layers.21.input_layernorm_moe_gen.weight",
+      "layers.21.post_attention_layernorm.weight",
+      "layers.21.post_attention_layernorm_moe_gen.weight",
+      "layers.22.self_attn.norm_q.weight",
+      "layers.22.self_attn.norm_k.weight",
+      "layers.22.self_attn.norm_added_q.weight",
+      "layers.22.self_attn.norm_added_k.weight",
+      "layers.22.input_layernorm.weight",
+      "layers.22.input_layernorm_moe_gen.weight",
+      "layers.22.post_attention_layernorm.weight",
+      "layers.22.post_attention_layernorm_moe_gen.weight",
+      "layers.23.self_attn.norm_q.weight",
+      "layers.23.self_attn.norm_k.weight",
+      "layers.23.self_attn.norm_added_q.weight",
+      "layers.23.self_attn.norm_added_k.weight",
+      "layers.23.input_layernorm.weight",
+      "layers.23.input_layernorm_moe_gen.weight",
+      "layers.23.post_attention_layernorm.weight",
+      "layers.23.post_attention_layernorm_moe_gen.weight",
+      "layers.24.self_attn.norm_q.weight",
+      "layers.24.self_attn.norm_k.weight",
+      "layers.24.self_attn.norm_added_q.weight",
+      "layers.24.self_attn.norm_added_k.weight",
+      "layers.24.input_layernorm.weight",
+      "layers.24.input_layernorm_moe_gen.weight",
+      "layers.24.post_attention_layernorm.weight",
+      "layers.24.post_attention_layernorm_moe_gen.weight",
+      "layers.25.self_attn.norm_q.weight",
+      "layers.25.self_attn.norm_k.weight",
+      "layers.25.self_attn.norm_added_q.weight",
+      "layers.25.self_attn.norm_added_k.weight",
+      "layers.25.input_layernorm.weight",
+      "layers.25.input_layernorm_moe_gen.weight",
+      "layers.25.post_attention_layernorm.weight",
+      "layers.25.post_attention_layernorm_moe_gen.weight",
+      "layers.26.self_attn.norm_q.weight",
+      "layers.26.self_attn.norm_k.weight",
+      "layers.26.self_attn.norm_added_q.weight",
+      "layers.26.self_attn.norm_added_k.weight",
+      "layers.26.input_layernorm.weight",
+      "layers.26.input_layernorm_moe_gen.weight",
+      "layers.26.post_attention_layernorm.weight",
+      "layers.26.post_attention_layernorm_moe_gen.weight",
+      "layers.27.self_attn.norm_q.weight",
+      "layers.27.self_attn.norm_k.weight",
+      "layers.27.self_attn.norm_added_q.weight",
+      "layers.27.self_attn.norm_added_k.weight",
+      "layers.27.input_layernorm.weight",
+      "layers.27.input_layernorm_moe_gen.weight",
+      "layers.27.post_attention_layernorm.weight",
+      "layers.27.post_attention_layernorm_moe_gen.weight",
+      "layers.28.self_attn.norm_q.weight",
+      "layers.28.self_attn.norm_k.weight",
+      "layers.28.self_attn.norm_added_q.weight",
+      "layers.28.self_attn.norm_added_k.weight",
+      "layers.28.input_layernorm.weight",
+      "layers.28.input_layernorm_moe_gen.weight",
+      "layers.28.post_attention_layernorm.weight",
+      "layers.28.post_attention_layernorm_moe_gen.weight",
+      "layers.29.self_attn.norm_q.weight",
+      "layers.29.self_attn.norm_k.weight",
+      "layers.29.self_attn.norm_added_q.weight",
+      "layers.29.self_attn.norm_added_k.weight",
+      "layers.29.input_layernorm.weight",
+      "layers.29.input_layernorm_moe_gen.weight",
+      "layers.29.post_attention_layernorm.weight",
+      "layers.29.post_attention_layernorm_moe_gen.weight",
+      "layers.30.self_attn.norm_q.weight",
+      "layers.30.self_attn.norm_k.weight",
+      "layers.30.self_attn.norm_added_q.weight",
+      "layers.30.self_attn.norm_added_k.weight",
+      "layers.30.input_layernorm.weight",
+      "layers.30.input_layernorm_moe_gen.weight",
+      "layers.30.post_attention_layernorm.weight",
+      "layers.30.post_attention_layernorm_moe_gen.weight",
+      "layers.31.self_attn.norm_q.weight",
+      "layers.31.self_attn.norm_k.weight",
+      "layers.31.self_attn.norm_added_q.weight",
+      "layers.31.self_attn.norm_added_k.weight",
+      "layers.31.input_layernorm.weight",
+      "layers.31.input_layernorm_moe_gen.weight",
+      "layers.31.post_attention_layernorm.weight",
+      "layers.31.post_attention_layernorm_moe_gen.weight",
+      "layers.32.self_attn.norm_q.weight",
+      "layers.32.self_attn.norm_k.weight",
+      "layers.32.self_attn.norm_added_q.weight",
+      "layers.32.self_attn.norm_added_k.weight",
+      "layers.32.input_layernorm.weight",
+      "layers.32.input_layernorm_moe_gen.weight",
+      "layers.32.post_attention_layernorm.weight",
+      "layers.32.post_attention_layernorm_moe_gen.weight",
+      "layers.33.self_attn.norm_q.weight",
+      "layers.33.self_attn.norm_k.weight",
+      "layers.33.self_attn.norm_added_q.weight",
+      "layers.33.self_attn.norm_added_k.weight",
+      "layers.33.input_layernorm.weight",
+      "layers.33.input_layernorm_moe_gen.weight",
+      "layers.33.post_attention_layernorm.weight",
+      "layers.33.post_attention_layernorm_moe_gen.weight",
+      "layers.34.self_attn.norm_q.weight",
+      "layers.34.self_attn.norm_k.weight",
+      "layers.34.self_attn.norm_added_q.weight",
+      "layers.34.self_attn.norm_added_k.weight",
+      "layers.34.input_layernorm.weight",
+      "layers.34.input_layernorm_moe_gen.weight",
+      "layers.34.post_attention_layernorm.weight",
+      "layers.34.post_attention_layernorm_moe_gen.weight",
+      "layers.35.self_attn.norm_q.weight",
+      "layers.35.self_attn.norm_k.weight",
+      "layers.35.self_attn.norm_added_q.weight",
+      "layers.35.self_attn.norm_added_k.weight",
+      "layers.35.input_layernorm.weight",
+      "layers.35.input_layernorm_moe_gen.weight",
+      "layers.35.post_attention_layernorm.weight",
+      "layers.35.post_attention_layernorm_moe_gen.weight",
+      "layers.36.self_attn.norm_q.weight",
+      "layers.36.self_attn.norm_k.weight",
+      "layers.36.self_attn.norm_added_q.weight",
+      "layers.36.self_attn.norm_added_k.weight",
+      "layers.36.input_layernorm.weight",
+      "layers.36.input_layernorm_moe_gen.weight",
+      "layers.36.post_attention_layernorm.weight",
+      "layers.36.post_attention_layernorm_moe_gen.weight",
+      "layers.37.self_attn.norm_q.weight",
+      "layers.37.self_attn.norm_k.weight",
+      "layers.37.self_attn.norm_added_q.weight",
+      "layers.37.self_attn.norm_added_k.weight",
+      "layers.37.input_layernorm.weight",
+      "layers.37.input_layernorm_moe_gen.weight",
+      "layers.37.post_attention_layernorm.weight",
+      "layers.37.post_attention_layernorm_moe_gen.weight",
+      "layers.38.self_attn.norm_q.weight",
+      "layers.38.self_attn.norm_k.weight",
+      "layers.38.self_attn.norm_added_q.weight",
+      "layers.38.self_attn.norm_added_k.weight",
+      "layers.38.input_layernorm.weight",
+      "layers.38.input_layernorm_moe_gen.weight",
+      "layers.38.post_attention_layernorm.weight",
+      "layers.38.post_attention_layernorm_moe_gen.weight",
+      "layers.39.self_attn.norm_q.weight",
+      "layers.39.self_attn.norm_k.weight",
+      "layers.39.self_attn.norm_added_q.weight",
+      "layers.39.self_attn.norm_added_k.weight",
+      "layers.39.input_layernorm.weight",
+      "layers.39.input_layernorm_moe_gen.weight",
+      "layers.39.post_attention_layernorm.weight",
+      "layers.39.post_attention_layernorm_moe_gen.weight",
+      "layers.40.self_attn.norm_q.weight",
+      "layers.40.self_attn.norm_k.weight",
+      "layers.40.self_attn.norm_added_q.weight",
+      "layers.40.self_attn.norm_added_k.weight",
+      "layers.40.input_layernorm.weight",
+      "layers.40.input_layernorm_moe_gen.weight",
+      "layers.40.post_attention_layernorm.weight",
+      "layers.40.post_attention_layernorm_moe_gen.weight",
+      "layers.41.self_attn.norm_q.weight",
+      "layers.41.self_attn.norm_k.weight",
+      "layers.41.self_attn.norm_added_q.weight",
+      "layers.41.self_attn.norm_added_k.weight",
+      "layers.41.input_layernorm.weight",
+      "layers.41.input_layernorm_moe_gen.weight",
+      "layers.41.post_attention_layernorm.weight",
+      "layers.41.post_attention_layernorm_moe_gen.weight",
+      "layers.42.self_attn.norm_q.weight",
+      "layers.42.self_attn.norm_k.weight",
+      "layers.42.self_attn.norm_added_q.weight",
+      "layers.42.self_attn.norm_added_k.weight",
+      "layers.42.input_layernorm.weight",
+      "layers.42.input_layernorm_moe_gen.weight",
+      "layers.42.post_attention_layernorm.weight",
+      "layers.42.post_attention_layernorm_moe_gen.weight",
+      "layers.43.self_attn.norm_q.weight",
+      "layers.43.self_attn.norm_k.weight",
+      "layers.43.self_attn.norm_added_q.weight",
+      "layers.43.self_attn.norm_added_k.weight",
+      "layers.43.input_layernorm.weight",
+      "layers.43.input_layernorm_moe_gen.weight",
+      "layers.43.post_attention_layernorm.weight",
+      "layers.43.post_attention_layernorm_moe_gen.weight",
+      "layers.44.self_attn.norm_q.weight",
+      "layers.44.self_attn.norm_k.weight",
+      "layers.44.self_attn.norm_added_q.weight",
+      "layers.44.self_attn.norm_added_k.weight",
+      "layers.44.input_layernorm.weight",
+      "layers.44.input_layernorm_moe_gen.weight",
+      "layers.44.post_attention_layernorm.weight",
+      "layers.44.post_attention_layernorm_moe_gen.weight",
+      "layers.45.self_attn.norm_q.weight",
+      "layers.45.self_attn.norm_k.weight",
+      "layers.45.self_attn.norm_added_q.weight",
+      "layers.45.self_attn.norm_added_k.weight",
+      "layers.45.input_layernorm.weight",
+      "layers.45.input_layernorm_moe_gen.weight",
+      "layers.45.post_attention_layernorm.weight",
+      "layers.45.post_attention_layernorm_moe_gen.weight",
+      "layers.46.self_attn.norm_q.weight",
+      "layers.46.self_attn.norm_k.weight",
+      "layers.46.self_attn.norm_added_q.weight",
+      "layers.46.self_attn.norm_added_k.weight",
+      "layers.46.input_layernorm.weight",
+      "layers.46.input_layernorm_moe_gen.weight",
+      "layers.46.post_attention_layernorm.weight",
+      "layers.46.post_attention_layernorm_moe_gen.weight",
+      "layers.47.self_attn.norm_q.weight",
+      "layers.47.self_attn.norm_k.weight",
+      "layers.47.self_attn.norm_added_q.weight",
+      "layers.47.self_attn.norm_added_k.weight",
+      "layers.47.input_layernorm.weight",
+      "layers.47.input_layernorm_moe_gen.weight",
+      "layers.47.post_attention_layernorm.weight",
+      "layers.47.post_attention_layernorm_moe_gen.weight",
+      "layers.48.self_attn.norm_q.weight",
+      "layers.48.self_attn.norm_k.weight",
+      "layers.48.self_attn.norm_added_q.weight",
+      "layers.48.self_attn.norm_added_k.weight",
+      "layers.48.input_layernorm.weight",
+      "layers.48.input_layernorm_moe_gen.weight",
+      "layers.48.post_attention_layernorm.weight",
+      "layers.48.post_attention_layernorm_moe_gen.weight",
+      "layers.49.self_attn.norm_q.weight",
+      "layers.49.self_attn.norm_k.weight",
+      "layers.49.self_attn.norm_added_q.weight",
+      "layers.49.self_attn.norm_added_k.weight",
+      "layers.49.input_layernorm.weight",
+      "layers.49.input_layernorm_moe_gen.weight",
+      "layers.49.post_attention_layernorm.weight",
+      "layers.49.post_attention_layernorm_moe_gen.weight",
+      "layers.50.self_attn.norm_q.weight",
+      "layers.50.self_attn.norm_k.weight",
+      "layers.50.self_attn.norm_added_q.weight",
+      "layers.50.self_attn.norm_added_k.weight",
+      "layers.50.input_layernorm.weight",
+      "layers.50.input_layernorm_moe_gen.weight",
+      "layers.50.post_attention_layernorm.weight",
+      "layers.50.post_attention_layernorm_moe_gen.weight",
+      "layers.51.self_attn.norm_q.weight",
+      "layers.51.self_attn.norm_k.weight",
+      "layers.51.self_attn.norm_added_q.weight",
+      "layers.51.self_attn.norm_added_k.weight",
+      "layers.51.input_layernorm.weight",
+      "layers.51.input_layernorm_moe_gen.weight",
+      "layers.51.post_attention_layernorm.weight",
+      "layers.51.post_attention_layernorm_moe_gen.weight",
+      "layers.52.self_attn.norm_q.weight",
+      "layers.52.self_attn.norm_k.weight",
+      "layers.52.self_attn.norm_added_q.weight",
+      "layers.52.self_attn.norm_added_k.weight",
+      "layers.52.input_layernorm.weight",
+      "layers.52.input_layernorm_moe_gen.weight",
+      "layers.52.post_attention_layernorm.weight",
+      "layers.52.post_attention_layernorm_moe_gen.weight",
+      "layers.53.self_attn.norm_q.weight",
+      "layers.53.self_attn.norm_k.weight",
+      "layers.53.self_attn.norm_added_q.weight",
+      "layers.53.self_attn.norm_added_k.weight",
+      "layers.53.input_layernorm.weight",
+      "layers.53.input_layernorm_moe_gen.weight",
+      "layers.53.post_attention_layernorm.weight",
+      "layers.53.post_attention_layernorm_moe_gen.weight",
+      "layers.54.self_attn.norm_q.weight",
+      "layers.54.self_attn.norm_k.weight",
+      "layers.54.self_attn.norm_added_q.weight",
+      "layers.54.self_attn.norm_added_k.weight",
+      "layers.54.input_layernorm.weight",
+      "layers.54.input_layernorm_moe_gen.weight",
+      "layers.54.post_attention_layernorm.weight",
+      "layers.54.post_attention_layernorm_moe_gen.weight",
+      "layers.55.self_attn.norm_q.weight",
+      "layers.55.self_attn.norm_k.weight",
+      "layers.55.self_attn.norm_added_q.weight",
+      "layers.55.self_attn.norm_added_k.weight",
+      "layers.55.input_layernorm.weight",
+      "layers.55.input_layernorm_moe_gen.weight",
+      "layers.55.post_attention_layernorm.weight",
+      "layers.55.post_attention_layernorm_moe_gen.weight",
+      "layers.56.self_attn.norm_q.weight",
+      "layers.56.self_attn.norm_k.weight",
+      "layers.56.self_attn.norm_added_q.weight",
+      "layers.56.self_attn.norm_added_k.weight",
+      "layers.56.input_layernorm.weight",
+      "layers.56.input_layernorm_moe_gen.weight",
+      "layers.56.post_attention_layernorm.weight",
+      "layers.56.post_attention_layernorm_moe_gen.weight",
+      "layers.57.self_attn.norm_q.weight",
+      "layers.57.self_attn.norm_k.weight",
+      "layers.57.self_attn.norm_added_q.weight",
+      "layers.57.self_attn.norm_added_k.weight",
+      "layers.57.input_layernorm.weight",
+      "layers.57.input_layernorm_moe_gen.weight",
+      "layers.57.post_attention_layernorm.weight",
+      "layers.57.post_attention_layernorm_moe_gen.weight",
+      "layers.58.self_attn.norm_q.weight",
+      "layers.58.self_attn.norm_k.weight",
+      "layers.58.self_attn.norm_added_q.weight",
+      "layers.58.self_attn.norm_added_k.weight",
+      "layers.58.input_layernorm.weight",
+      "layers.58.input_layernorm_moe_gen.weight",
+      "layers.58.post_attention_layernorm.weight",
+      "layers.58.post_attention_layernorm_moe_gen.weight",
+      "layers.59.self_attn.norm_q.weight",
+      "layers.59.self_attn.norm_k.weight",
+      "layers.59.self_attn.norm_added_q.weight",
+      "layers.59.self_attn.norm_added_k.weight",
+      "layers.59.input_layernorm.weight",
+      "layers.59.input_layernorm_moe_gen.weight",
+      "layers.59.post_attention_layernorm.weight",
+      "layers.59.post_attention_layernorm_moe_gen.weight",
+      "layers.60.self_attn.norm_q.weight",
+      "layers.60.self_attn.norm_k.weight",
+      "layers.60.self_attn.norm_added_q.weight",
+      "layers.60.self_attn.norm_added_k.weight",
+      "layers.60.input_layernorm.weight",
+      "layers.60.input_layernorm_moe_gen.weight",
+      "layers.60.post_attention_layernorm.weight",
+      "layers.60.post_attention_layernorm_moe_gen.weight",
+      "layers.61.self_attn.norm_q.weight",
+      "layers.61.self_attn.norm_k.weight",
+      "layers.61.self_attn.norm_added_q.weight",
+      "layers.61.self_attn.norm_added_k.weight",
+      "layers.61.input_layernorm.weight",
+      "layers.61.input_layernorm_moe_gen.weight",
+      "layers.61.post_attention_layernorm.weight",
+      "layers.61.post_attention_layernorm_moe_gen.weight",
+      "layers.62.self_attn.norm_q.weight",
+      "layers.62.self_attn.norm_k.weight",
+      "layers.62.self_attn.norm_added_q.weight",
+      "layers.62.self_attn.norm_added_k.weight",
+      "layers.62.input_layernorm.weight",
+      "layers.62.input_layernorm_moe_gen.weight",
+      "layers.62.post_attention_layernorm.weight",
+      "layers.62.post_attention_layernorm_moe_gen.weight",
+      "layers.63.self_attn.norm_q.weight",
+      "layers.63.self_attn.norm_k.weight",
+      "layers.63.self_attn.norm_added_q.weight",
+      "layers.63.self_attn.norm_added_k.weight",
+      "layers.63.input_layernorm.weight",
+      "layers.63.input_layernorm_moe_gen.weight",
+      "layers.63.post_attention_layernorm.weight",
+      "layers.63.post_attention_layernorm_moe_gen.weight",
+      "norm_moe_gen.weight"
+    ],
+    "modules_to_not_use_matmul": [],
+    "non_blocking": false,
+    "quant_conv": false,
+    "quant_embedding": false,
+    "quant_method": "sdnq",
+    "quantization_device": null,
+    "quantized_matmul_dtype": null,
+    "return_device": null,
+    "sdnq_version": "0.1.9",
+    "svd_rank": 32,
+    "svd_steps": 8,
+    "use_dynamic_quantization": false,
+    "use_grad_ckpt": true,
+    "use_hadamard": false,
+    "use_quantized_matmul": false,
+    "use_quantized_matmul_conv": false,
+    "use_static_quantization": true,
+    "use_stochastic_rounding": false,
+    "use_svd": true,
+    "weights_dtype": "uint4"
+  },
+  "rms_norm_eps": 1e-06,
+  "rope_axes_dim": [
+    24,
+    20,
+    20
+  ],
+  "rope_scaling": {
+    "mrope_interleaved": true,
+    "mrope_section": [
+      24,
+      20,
+      20
+    ],
+    "rope_type": "default"
+  },
+  "rope_theta": 5000000,
+  "sound_dim": 64,
+  "sound_gen": true,
+  "sound_latent_fps": 25,
+  "temporal_compression_factor_sound": 1,
+  "timestep_scale": 0.001,
+  "unified_3d_mrope_reset_spatial_ids": true,
+  "unified_3d_mrope_temporal_modality_margin": 15000,
+  "use_cache": true,
+  "use_moe": true,
+  "video_temporal_causal": false,
+  "vocab_size": 151936
+}

transformer/diffusion_pytorch_model-00001-of-00008.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:c138003f0b0b27a2228c864b62eace75145999aefc404963e64a404e973316e4
+size 4942030352

transformer/diffusion_pytorch_model-00002-of-00008.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:5d96ba4d6f4d7f7102c04bc340144ca704112536cb1b57252fb78725cf98c776
+size 4957206064

transformer/diffusion_pytorch_model-00003-of-00008.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:e43c3c3984e1a5893c99615cd5ee52870e4c2ad869a411ba3d5d8eba9a000336
+size 4957206480

transformer/diffusion_pytorch_model-00004-of-00008.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:075f0578c37257b6f789e9adbd3b7456d8a7847603e4c8e6731bef09444adf52
+size 4990905880

transformer/diffusion_pytorch_model-00005-of-00008.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:5531d1943a3ac2fb24c33346dc88e1f625cbf6a0581fc6173849dcfe215c8032
+size 4957206400

transformer/diffusion_pytorch_model-00006-of-00008.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:aea9f5386bcaa04f09e5adbce3142ef7ff514ca75321f6c250c4cbbe3c7d8b67
+size 4957206448

transformer/diffusion_pytorch_model-00007-of-00008.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:f0d0affbcf98e49eb936d22787bde4e344fd1b2b866b37af830a7ddc8858d044
+size 4986740672

transformer/diffusion_pytorch_model-00008-of-00008.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:64059b2950ac343d60ec9757321f8c4929d83fef4e9071ab7a2226dd010a0f2f
+size 2710137152

transformer/quantization_config.json

@@ -0,0 +1,573 @@
+{
+  "add_skip_keys": false,
+  "dequantize_fp32": false,
+  "dynamic_loss_threshold": null,
+  "group_size": 0,
+  "hadamard_group_size": 128,
+  "is_integer": true,
+  "is_training": false,
+  "modules_dtype_dict": {},
+  "modules_quant_config": {},
+  "modules_to_not_convert": [
+    "time_text_embed",
+    ".x_embedder",
+    "lm_head",
+    ".txt_out",
+    ".txt_in",
+    ".vid_in",
+    ".y_embedder",
+    ".context_embedder",
+    "patch_embed",
+    ".emb_in",
+    ".condition_embedder",
+    ".img_in",
+    ".final_layer",
+    "patch_emb",
+    "multi_modal_projector",
+    ".proj_out",
+    "patch_embedding",
+    ".vid_out",
+    "wte",
+    ".img_out",
+    ".time_embed",
+    "norm",
+    ".norm_out",
+    ".t_embedder",
+    "embed_tokens",
+    ".emb_out",
+    "time_embedder",
+    "layers.0.self_attn.norm_q.weight",
+    "layers.0.self_attn.norm_k.weight",
+    "layers.0.self_attn.norm_added_q.weight",
+    "layers.0.self_attn.norm_added_k.weight",
+    "layers.0.input_layernorm.weight",
+    "layers.0.input_layernorm_moe_gen.weight",
+    "layers.0.post_attention_layernorm.weight",
+    "layers.0.post_attention_layernorm_moe_gen.weight",
+    "layers.1.self_attn.norm_q.weight",
+    "layers.1.self_attn.norm_k.weight",
+    "layers.1.self_attn.norm_added_q.weight",
+    "layers.1.self_attn.norm_added_k.weight",
+    "layers.1.input_layernorm.weight",
+    "layers.1.input_layernorm_moe_gen.weight",
+    "layers.1.post_attention_layernorm.weight",
+    "layers.1.post_attention_layernorm_moe_gen.weight",
+    "layers.2.self_attn.norm_q.weight",
+    "layers.2.self_attn.norm_k.weight",
+    "layers.2.self_attn.norm_added_q.weight",
+    "layers.2.self_attn.norm_added_k.weight",
+    "layers.2.input_layernorm.weight",
+    "layers.2.input_layernorm_moe_gen.weight",
+    "layers.2.post_attention_layernorm.weight",
+    "layers.2.post_attention_layernorm_moe_gen.weight",
+    "layers.3.self_attn.norm_q.weight",
+    "layers.3.self_attn.norm_k.weight",
+    "layers.3.self_attn.norm_added_q.weight",
+    "layers.3.self_attn.norm_added_k.weight",
+    "layers.3.input_layernorm.weight",
+    "layers.3.input_layernorm_moe_gen.weight",
+    "layers.3.post_attention_layernorm.weight",
+    "layers.3.post_attention_layernorm_moe_gen.weight",
+    "layers.4.self_attn.norm_q.weight",
+    "layers.4.self_attn.norm_k.weight",
+    "layers.4.self_attn.norm_added_q.weight",
+    "layers.4.self_attn.norm_added_k.weight",
+    "layers.4.input_layernorm.weight",
+    "layers.4.input_layernorm_moe_gen.weight",
+    "layers.4.post_attention_layernorm.weight",
+    "layers.4.post_attention_layernorm_moe_gen.weight",
+    "layers.5.self_attn.norm_q.weight",
+    "layers.5.self_attn.norm_k.weight",
+    "layers.5.self_attn.norm_added_q.weight",
+    "layers.5.self_attn.norm_added_k.weight",
+    "layers.5.input_layernorm.weight",
+    "layers.5.input_layernorm_moe_gen.weight",
+    "layers.5.post_attention_layernorm.weight",
+    "layers.5.post_attention_layernorm_moe_gen.weight",
+    "layers.6.self_attn.norm_q.weight",
+    "layers.6.self_attn.norm_k.weight",
+    "layers.6.self_attn.norm_added_q.weight",
+    "layers.6.self_attn.norm_added_k.weight",
+    "layers.6.input_layernorm.weight",
+    "layers.6.input_layernorm_moe_gen.weight",
+    "layers.6.post_attention_layernorm.weight",
+    "layers.6.post_attention_layernorm_moe_gen.weight",
+    "layers.7.self_attn.norm_q.weight",
+    "layers.7.self_attn.norm_k.weight",
+    "layers.7.self_attn.norm_added_q.weight",
+    "layers.7.self_attn.norm_added_k.weight",
+    "layers.7.input_layernorm.weight",
+    "layers.7.input_layernorm_moe_gen.weight",
+    "layers.7.post_attention_layernorm.weight",
+    "layers.7.post_attention_layernorm_moe_gen.weight",
+    "layers.8.self_attn.norm_q.weight",
+    "layers.8.self_attn.norm_k.weight",
+    "layers.8.self_attn.norm_added_q.weight",
+    "layers.8.self_attn.norm_added_k.weight",
+    "layers.8.input_layernorm.weight",
+    "layers.8.input_layernorm_moe_gen.weight",
+    "layers.8.post_attention_layernorm.weight",
+    "layers.8.post_attention_layernorm_moe_gen.weight",
+    "layers.9.self_attn.norm_q.weight",
+    "layers.9.self_attn.norm_k.weight",
+    "layers.9.self_attn.norm_added_q.weight",
+    "layers.9.self_attn.norm_added_k.weight",
+    "layers.9.input_layernorm.weight",
+    "layers.9.input_layernorm_moe_gen.weight",
+    "layers.9.post_attention_layernorm.weight",
+    "layers.9.post_attention_layernorm_moe_gen.weight",
+    "layers.10.self_attn.norm_q.weight",
+    "layers.10.self_attn.norm_k.weight",
+    "layers.10.self_attn.norm_added_q.weight",
+    "layers.10.self_attn.norm_added_k.weight",
+    "layers.10.input_layernorm.weight",
+    "layers.10.input_layernorm_moe_gen.weight",
+    "layers.10.post_attention_layernorm.weight",
+    "layers.10.post_attention_layernorm_moe_gen.weight",
+    "layers.11.self_attn.norm_q.weight",
+    "layers.11.self_attn.norm_k.weight",
+    "layers.11.self_attn.norm_added_q.weight",
+    "layers.11.self_attn.norm_added_k.weight",
+    "layers.11.input_layernorm.weight",
+    "layers.11.input_layernorm_moe_gen.weight",
+    "layers.11.post_attention_layernorm.weight",
+    "layers.11.post_attention_layernorm_moe_gen.weight",
+    "layers.12.self_attn.norm_q.weight",
+    "layers.12.self_attn.norm_k.weight",
+    "layers.12.self_attn.norm_added_q.weight",
+    "layers.12.self_attn.norm_added_k.weight",
+    "layers.12.input_layernorm.weight",
+    "layers.12.input_layernorm_moe_gen.weight",
+    "layers.12.post_attention_layernorm.weight",
+    "layers.12.post_attention_layernorm_moe_gen.weight",
+    "layers.13.self_attn.norm_q.weight",
+    "layers.13.self_attn.norm_k.weight",
+    "layers.13.self_attn.norm_added_q.weight",
+    "layers.13.self_attn.norm_added_k.weight",
+    "layers.13.input_layernorm.weight",
+    "layers.13.input_layernorm_moe_gen.weight",
+    "layers.13.post_attention_layernorm.weight",
+    "layers.13.post_attention_layernorm_moe_gen.weight",
+    "layers.14.self_attn.norm_q.weight",
+    "layers.14.self_attn.norm_k.weight",
+    "layers.14.self_attn.norm_added_q.weight",
+    "layers.14.self_attn.norm_added_k.weight",
+    "layers.14.input_layernorm.weight",
+    "layers.14.input_layernorm_moe_gen.weight",
+    "layers.14.post_attention_layernorm.weight",
+    "layers.14.post_attention_layernorm_moe_gen.weight",
+    "layers.15.self_attn.norm_q.weight",
+    "layers.15.self_attn.norm_k.weight",
+    "layers.15.self_attn.norm_added_q.weight",
+    "layers.15.self_attn.norm_added_k.weight",
+    "layers.15.input_layernorm.weight",
+    "layers.15.input_layernorm_moe_gen.weight",
+    "layers.15.post_attention_layernorm.weight",
+    "layers.15.post_attention_layernorm_moe_gen.weight",
+    "layers.16.self_attn.norm_q.weight",
+    "layers.16.self_attn.norm_k.weight",
+    "layers.16.self_attn.norm_added_q.weight",
+    "layers.16.self_attn.norm_added_k.weight",
+    "layers.16.input_layernorm.weight",
+    "layers.16.input_layernorm_moe_gen.weight",
+    "layers.16.post_attention_layernorm.weight",
+    "layers.16.post_attention_layernorm_moe_gen.weight",
+    "layers.17.self_attn.norm_q.weight",
+    "layers.17.self_attn.norm_k.weight",
+    "layers.17.self_attn.norm_added_q.weight",
+    "layers.17.self_attn.norm_added_k.weight",
+    "layers.17.input_layernorm.weight",
+    "layers.17.input_layernorm_moe_gen.weight",
+    "layers.17.post_attention_layernorm.weight",
+    "layers.17.post_attention_layernorm_moe_gen.weight",
+    "layers.18.self_attn.norm_q.weight",
+    "layers.18.self_attn.norm_k.weight",
+    "layers.18.self_attn.norm_added_q.weight",
+    "layers.18.self_attn.norm_added_k.weight",
+    "layers.18.input_layernorm.weight",
+    "layers.18.input_layernorm_moe_gen.weight",
+    "layers.18.post_attention_layernorm.weight",
+    "layers.18.post_attention_layernorm_moe_gen.weight",
+    "layers.19.self_attn.norm_q.weight",
+    "layers.19.self_attn.norm_k.weight",
+    "layers.19.self_attn.norm_added_q.weight",
+    "layers.19.self_attn.norm_added_k.weight",
+    "layers.19.input_layernorm.weight",
+    "layers.19.input_layernorm_moe_gen.weight",
+    "layers.19.post_attention_layernorm.weight",
+    "layers.19.post_attention_layernorm_moe_gen.weight",
+    "layers.20.self_attn.norm_q.weight",
+    "layers.20.self_attn.norm_k.weight",
+    "layers.20.self_attn.norm_added_q.weight",
+    "layers.20.self_attn.norm_added_k.weight",
+    "layers.20.input_layernorm.weight",
+    "layers.20.input_layernorm_moe_gen.weight",
+    "layers.20.post_attention_layernorm.weight",
+    "layers.20.post_attention_layernorm_moe_gen.weight",
+    "layers.21.self_attn.norm_q.weight",
+    "layers.21.self_attn.norm_k.weight",
+    "layers.21.self_attn.norm_added_q.weight",
+    "layers.21.self_attn.norm_added_k.weight",
+    "layers.21.input_layernorm.weight",
+    "layers.21.input_layernorm_moe_gen.weight",
+    "layers.21.post_attention_layernorm.weight",
+    "layers.21.post_attention_layernorm_moe_gen.weight",
+    "layers.22.self_attn.norm_q.weight",
+    "layers.22.self_attn.norm_k.weight",
+    "layers.22.self_attn.norm_added_q.weight",
+    "layers.22.self_attn.norm_added_k.weight",
+    "layers.22.input_layernorm.weight",
+    "layers.22.input_layernorm_moe_gen.weight",
+    "layers.22.post_attention_layernorm.weight",
+    "layers.22.post_attention_layernorm_moe_gen.weight",
+    "layers.23.self_attn.norm_q.weight",
+    "layers.23.self_attn.norm_k.weight",
+    "layers.23.self_attn.norm_added_q.weight",
+    "layers.23.self_attn.norm_added_k.weight",
+    "layers.23.input_layernorm.weight",
+    "layers.23.input_layernorm_moe_gen.weight",
+    "layers.23.post_attention_layernorm.weight",
+    "layers.23.post_attention_layernorm_moe_gen.weight",
+    "layers.24.self_attn.norm_q.weight",
+    "layers.24.self_attn.norm_k.weight",
+    "layers.24.self_attn.norm_added_q.weight",
+    "layers.24.self_attn.norm_added_k.weight",
+    "layers.24.input_layernorm.weight",
+    "layers.24.input_layernorm_moe_gen.weight",
+    "layers.24.post_attention_layernorm.weight",
+    "layers.24.post_attention_layernorm_moe_gen.weight",
+    "layers.25.self_attn.norm_q.weight",
+    "layers.25.self_attn.norm_k.weight",
+    "layers.25.self_attn.norm_added_q.weight",
+    "layers.25.self_attn.norm_added_k.weight",
+    "layers.25.input_layernorm.weight",
+    "layers.25.input_layernorm_moe_gen.weight",
+    "layers.25.post_attention_layernorm.weight",
+    "layers.25.post_attention_layernorm_moe_gen.weight",
+    "layers.26.self_attn.norm_q.weight",
+    "layers.26.self_attn.norm_k.weight",
+    "layers.26.self_attn.norm_added_q.weight",
+    "layers.26.self_attn.norm_added_k.weight",
+    "layers.26.input_layernorm.weight",
+    "layers.26.input_layernorm_moe_gen.weight",
+    "layers.26.post_attention_layernorm.weight",
+    "layers.26.post_attention_layernorm_moe_gen.weight",
+    "layers.27.self_attn.norm_q.weight",
+    "layers.27.self_attn.norm_k.weight",
+    "layers.27.self_attn.norm_added_q.weight",
+    "layers.27.self_attn.norm_added_k.weight",
+    "layers.27.input_layernorm.weight",
+    "layers.27.input_layernorm_moe_gen.weight",
+    "layers.27.post_attention_layernorm.weight",
+    "layers.27.post_attention_layernorm_moe_gen.weight",
+    "layers.28.self_attn.norm_q.weight",
+    "layers.28.self_attn.norm_k.weight",
+    "layers.28.self_attn.norm_added_q.weight",
+    "layers.28.self_attn.norm_added_k.weight",
+    "layers.28.input_layernorm.weight",
+    "layers.28.input_layernorm_moe_gen.weight",
+    "layers.28.post_attention_layernorm.weight",
+    "layers.28.post_attention_layernorm_moe_gen.weight",
+    "layers.29.self_attn.norm_q.weight",
+    "layers.29.self_attn.norm_k.weight",
+    "layers.29.self_attn.norm_added_q.weight",
+    "layers.29.self_attn.norm_added_k.weight",
+    "layers.29.input_layernorm.weight",
+    "layers.29.input_layernorm_moe_gen.weight",
+    "layers.29.post_attention_layernorm.weight",
+    "layers.29.post_attention_layernorm_moe_gen.weight",
+    "layers.30.self_attn.norm_q.weight",
+    "layers.30.self_attn.norm_k.weight",
+    "layers.30.self_attn.norm_added_q.weight",
+    "layers.30.self_attn.norm_added_k.weight",
+    "layers.30.input_layernorm.weight",
+    "layers.30.input_layernorm_moe_gen.weight",
+    "layers.30.post_attention_layernorm.weight",
+    "layers.30.post_attention_layernorm_moe_gen.weight",
+    "layers.31.self_attn.norm_q.weight",
+    "layers.31.self_attn.norm_k.weight",
+    "layers.31.self_attn.norm_added_q.weight",
+    "layers.31.self_attn.norm_added_k.weight",
+    "layers.31.input_layernorm.weight",
+    "layers.31.input_layernorm_moe_gen.weight",
+    "layers.31.post_attention_layernorm.weight",
+    "layers.31.post_attention_layernorm_moe_gen.weight",
+    "layers.32.self_attn.norm_q.weight",
+    "layers.32.self_attn.norm_k.weight",
+    "layers.32.self_attn.norm_added_q.weight",
+    "layers.32.self_attn.norm_added_k.weight",
+    "layers.32.input_layernorm.weight",
+    "layers.32.input_layernorm_moe_gen.weight",
+    "layers.32.post_attention_layernorm.weight",
+    "layers.32.post_attention_layernorm_moe_gen.weight",
+    "layers.33.self_attn.norm_q.weight",
+    "layers.33.self_attn.norm_k.weight",
+    "layers.33.self_attn.norm_added_q.weight",
+    "layers.33.self_attn.norm_added_k.weight",
+    "layers.33.input_layernorm.weight",
+    "layers.33.input_layernorm_moe_gen.weight",
+    "layers.33.post_attention_layernorm.weight",
+    "layers.33.post_attention_layernorm_moe_gen.weight",
+    "layers.34.self_attn.norm_q.weight",
+    "layers.34.self_attn.norm_k.weight",
+    "layers.34.self_attn.norm_added_q.weight",
+    "layers.34.self_attn.norm_added_k.weight",
+    "layers.34.input_layernorm.weight",
+    "layers.34.input_layernorm_moe_gen.weight",
+    "layers.34.post_attention_layernorm.weight",
+    "layers.34.post_attention_layernorm_moe_gen.weight",
+    "layers.35.self_attn.norm_q.weight",
+    "layers.35.self_attn.norm_k.weight",
+    "layers.35.self_attn.norm_added_q.weight",
+    "layers.35.self_attn.norm_added_k.weight",
+    "layers.35.input_layernorm.weight",
+    "layers.35.input_layernorm_moe_gen.weight",
+    "layers.35.post_attention_layernorm.weight",
+    "layers.35.post_attention_layernorm_moe_gen.weight",
+    "layers.36.self_attn.norm_q.weight",
+    "layers.36.self_attn.norm_k.weight",
+    "layers.36.self_attn.norm_added_q.weight",
+    "layers.36.self_attn.norm_added_k.weight",
+    "layers.36.input_layernorm.weight",
+    "layers.36.input_layernorm_moe_gen.weight",
+    "layers.36.post_attention_layernorm.weight",
+    "layers.36.post_attention_layernorm_moe_gen.weight",
+    "layers.37.self_attn.norm_q.weight",
+    "layers.37.self_attn.norm_k.weight",
+    "layers.37.self_attn.norm_added_q.weight",
+    "layers.37.self_attn.norm_added_k.weight",
+    "layers.37.input_layernorm.weight",
+    "layers.37.input_layernorm_moe_gen.weight",
+    "layers.37.post_attention_layernorm.weight",
+    "layers.37.post_attention_layernorm_moe_gen.weight",
+    "layers.38.self_attn.norm_q.weight",
+    "layers.38.self_attn.norm_k.weight",
+    "layers.38.self_attn.norm_added_q.weight",
+    "layers.38.self_attn.norm_added_k.weight",
+    "layers.38.input_layernorm.weight",
+    "layers.38.input_layernorm_moe_gen.weight",
+    "layers.38.post_attention_layernorm.weight",
+    "layers.38.post_attention_layernorm_moe_gen.weight",
+    "layers.39.self_attn.norm_q.weight",
+    "layers.39.self_attn.norm_k.weight",
+    "layers.39.self_attn.norm_added_q.weight",
+    "layers.39.self_attn.norm_added_k.weight",
+    "layers.39.input_layernorm.weight",
+    "layers.39.input_layernorm_moe_gen.weight",
+    "layers.39.post_attention_layernorm.weight",
+    "layers.39.post_attention_layernorm_moe_gen.weight",
+    "layers.40.self_attn.norm_q.weight",
+    "layers.40.self_attn.norm_k.weight",
+    "layers.40.self_attn.norm_added_q.weight",
+    "layers.40.self_attn.norm_added_k.weight",
+    "layers.40.input_layernorm.weight",
+    "layers.40.input_layernorm_moe_gen.weight",
+    "layers.40.post_attention_layernorm.weight",
+    "layers.40.post_attention_layernorm_moe_gen.weight",
+    "layers.41.self_attn.norm_q.weight",
+    "layers.41.self_attn.norm_k.weight",
+    "layers.41.self_attn.norm_added_q.weight",
+    "layers.41.self_attn.norm_added_k.weight",
+    "layers.41.input_layernorm.weight",
+    "layers.41.input_layernorm_moe_gen.weight",
+    "layers.41.post_attention_layernorm.weight",
+    "layers.41.post_attention_layernorm_moe_gen.weight",
+    "layers.42.self_attn.norm_q.weight",
+    "layers.42.self_attn.norm_k.weight",
+    "layers.42.self_attn.norm_added_q.weight",
+    "layers.42.self_attn.norm_added_k.weight",
+    "layers.42.input_layernorm.weight",
+    "layers.42.input_layernorm_moe_gen.weight",
+    "layers.42.post_attention_layernorm.weight",
+    "layers.42.post_attention_layernorm_moe_gen.weight",
+    "layers.43.self_attn.norm_q.weight",
+    "layers.43.self_attn.norm_k.weight",
+    "layers.43.self_attn.norm_added_q.weight",
+    "layers.43.self_attn.norm_added_k.weight",
+    "layers.43.input_layernorm.weight",
+    "layers.43.input_layernorm_moe_gen.weight",
+    "layers.43.post_attention_layernorm.weight",
+    "layers.43.post_attention_layernorm_moe_gen.weight",
+    "layers.44.self_attn.norm_q.weight",
+    "layers.44.self_attn.norm_k.weight",
+    "layers.44.self_attn.norm_added_q.weight",
+    "layers.44.self_attn.norm_added_k.weight",
+    "layers.44.input_layernorm.weight",
+    "layers.44.input_layernorm_moe_gen.weight",
+    "layers.44.post_attention_layernorm.weight",
+    "layers.44.post_attention_layernorm_moe_gen.weight",
+    "layers.45.self_attn.norm_q.weight",
+    "layers.45.self_attn.norm_k.weight",
+    "layers.45.self_attn.norm_added_q.weight",
+    "layers.45.self_attn.norm_added_k.weight",
+    "layers.45.input_layernorm.weight",
+    "layers.45.input_layernorm_moe_gen.weight",
+    "layers.45.post_attention_layernorm.weight",
+    "layers.45.post_attention_layernorm_moe_gen.weight",
+    "layers.46.self_attn.norm_q.weight",
+    "layers.46.self_attn.norm_k.weight",
+    "layers.46.self_attn.norm_added_q.weight",
+    "layers.46.self_attn.norm_added_k.weight",
+    "layers.46.input_layernorm.weight",
+    "layers.46.input_layernorm_moe_gen.weight",
+    "layers.46.post_attention_layernorm.weight",
+    "layers.46.post_attention_layernorm_moe_gen.weight",
+    "layers.47.self_attn.norm_q.weight",
+    "layers.47.self_attn.norm_k.weight",
+    "layers.47.self_attn.norm_added_q.weight",
+    "layers.47.self_attn.norm_added_k.weight",
+    "layers.47.input_layernorm.weight",
+    "layers.47.input_layernorm_moe_gen.weight",
+    "layers.47.post_attention_layernorm.weight",
+    "layers.47.post_attention_layernorm_moe_gen.weight",
+    "layers.48.self_attn.norm_q.weight",
+    "layers.48.self_attn.norm_k.weight",
+    "layers.48.self_attn.norm_added_q.weight",
+    "layers.48.self_attn.norm_added_k.weight",
+    "layers.48.input_layernorm.weight",
+    "layers.48.input_layernorm_moe_gen.weight",
+    "layers.48.post_attention_layernorm.weight",
+    "layers.48.post_attention_layernorm_moe_gen.weight",
+    "layers.49.self_attn.norm_q.weight",
+    "layers.49.self_attn.norm_k.weight",
+    "layers.49.self_attn.norm_added_q.weight",
+    "layers.49.self_attn.norm_added_k.weight",
+    "layers.49.input_layernorm.weight",
+    "layers.49.input_layernorm_moe_gen.weight",
+    "layers.49.post_attention_layernorm.weight",
+    "layers.49.post_attention_layernorm_moe_gen.weight",
+    "layers.50.self_attn.norm_q.weight",
+    "layers.50.self_attn.norm_k.weight",
+    "layers.50.self_attn.norm_added_q.weight",
+    "layers.50.self_attn.norm_added_k.weight",
+    "layers.50.input_layernorm.weight",
+    "layers.50.input_layernorm_moe_gen.weight",
+    "layers.50.post_attention_layernorm.weight",
+    "layers.50.post_attention_layernorm_moe_gen.weight",
+    "layers.51.self_attn.norm_q.weight",
+    "layers.51.self_attn.norm_k.weight",
+    "layers.51.self_attn.norm_added_q.weight",
+    "layers.51.self_attn.norm_added_k.weight",
+    "layers.51.input_layernorm.weight",
+    "layers.51.input_layernorm_moe_gen.weight",
+    "layers.51.post_attention_layernorm.weight",
+    "layers.51.post_attention_layernorm_moe_gen.weight",
+    "layers.52.self_attn.norm_q.weight",
+    "layers.52.self_attn.norm_k.weight",
+    "layers.52.self_attn.norm_added_q.weight",
+    "layers.52.self_attn.norm_added_k.weight",
+    "layers.52.input_layernorm.weight",
+    "layers.52.input_layernorm_moe_gen.weight",
+    "layers.52.post_attention_layernorm.weight",
+    "layers.52.post_attention_layernorm_moe_gen.weight",
+    "layers.53.self_attn.norm_q.weight",
+    "layers.53.self_attn.norm_k.weight",
+    "layers.53.self_attn.norm_added_q.weight",
+    "layers.53.self_attn.norm_added_k.weight",
+    "layers.53.input_layernorm.weight",
+    "layers.53.input_layernorm_moe_gen.weight",
+    "layers.53.post_attention_layernorm.weight",
+    "layers.53.post_attention_layernorm_moe_gen.weight",
+    "layers.54.self_attn.norm_q.weight",
+    "layers.54.self_attn.norm_k.weight",
+    "layers.54.self_attn.norm_added_q.weight",
+    "layers.54.self_attn.norm_added_k.weight",
+    "layers.54.input_layernorm.weight",
+    "layers.54.input_layernorm_moe_gen.weight",
+    "layers.54.post_attention_layernorm.weight",
+    "layers.54.post_attention_layernorm_moe_gen.weight",
+    "layers.55.self_attn.norm_q.weight",
+    "layers.55.self_attn.norm_k.weight",
+    "layers.55.self_attn.norm_added_q.weight",
+    "layers.55.self_attn.norm_added_k.weight",
+    "layers.55.input_layernorm.weight",
+    "layers.55.input_layernorm_moe_gen.weight",
+    "layers.55.post_attention_layernorm.weight",
+    "layers.55.post_attention_layernorm_moe_gen.weight",
+    "layers.56.self_attn.norm_q.weight",
+    "layers.56.self_attn.norm_k.weight",
+    "layers.56.self_attn.norm_added_q.weight",
+    "layers.56.self_attn.norm_added_k.weight",
+    "layers.56.input_layernorm.weight",
+    "layers.56.input_layernorm_moe_gen.weight",
+    "layers.56.post_attention_layernorm.weight",
+    "layers.56.post_attention_layernorm_moe_gen.weight",
+    "layers.57.self_attn.norm_q.weight",
+    "layers.57.self_attn.norm_k.weight",
+    "layers.57.self_attn.norm_added_q.weight",
+    "layers.57.self_attn.norm_added_k.weight",
+    "layers.57.input_layernorm.weight",
+    "layers.57.input_layernorm_moe_gen.weight",
+    "layers.57.post_attention_layernorm.weight",
+    "layers.57.post_attention_layernorm_moe_gen.weight",
+    "layers.58.self_attn.norm_q.weight",
+    "layers.58.self_attn.norm_k.weight",
+    "layers.58.self_attn.norm_added_q.weight",
+    "layers.58.self_attn.norm_added_k.weight",
+    "layers.58.input_layernorm.weight",
+    "layers.58.input_layernorm_moe_gen.weight",
+    "layers.58.post_attention_layernorm.weight",
+    "layers.58.post_attention_layernorm_moe_gen.weight",
+    "layers.59.self_attn.norm_q.weight",
+    "layers.59.self_attn.norm_k.weight",
+    "layers.59.self_attn.norm_added_q.weight",
+    "layers.59.self_attn.norm_added_k.weight",
+    "layers.59.input_layernorm.weight",
+    "layers.59.input_layernorm_moe_gen.weight",
+    "layers.59.post_attention_layernorm.weight",
+    "layers.59.post_attention_layernorm_moe_gen.weight",
+    "layers.60.self_attn.norm_q.weight",
+    "layers.60.self_attn.norm_k.weight",
+    "layers.60.self_attn.norm_added_q.weight",
+    "layers.60.self_attn.norm_added_k.weight",
+    "layers.60.input_layernorm.weight",
+    "layers.60.input_layernorm_moe_gen.weight",
+    "layers.60.post_attention_layernorm.weight",
+    "layers.60.post_attention_layernorm_moe_gen.weight",
+    "layers.61.self_attn.norm_q.weight",
+    "layers.61.self_attn.norm_k.weight",
+    "layers.61.self_attn.norm_added_q.weight",
+    "layers.61.self_attn.norm_added_k.weight",
+    "layers.61.input_layernorm.weight",
+    "layers.61.input_layernorm_moe_gen.weight",
+    "layers.61.post_attention_layernorm.weight",
+    "layers.61.post_attention_layernorm_moe_gen.weight",
+    "layers.62.self_attn.norm_q.weight",
+    "layers.62.self_attn.norm_k.weight",
+    "layers.62.self_attn.norm_added_q.weight",
+    "layers.62.self_attn.norm_added_k.weight",
+    "layers.62.input_layernorm.weight",
+    "layers.62.input_layernorm_moe_gen.weight",
+    "layers.62.post_attention_layernorm.weight",
+    "layers.62.post_attention_layernorm_moe_gen.weight",
+    "layers.63.self_attn.norm_q.weight",
+    "layers.63.self_attn.norm_k.weight",
+    "layers.63.self_attn.norm_added_q.weight",
+    "layers.63.self_attn.norm_added_k.weight",
+    "layers.63.input_layernorm.weight",
+    "layers.63.input_layernorm_moe_gen.weight",
+    "layers.63.post_attention_layernorm.weight",
+    "layers.63.post_attention_layernorm_moe_gen.weight",
+    "norm_moe_gen.weight"
+  ],
+  "modules_to_not_use_matmul": [],
+  "non_blocking": false,
+  "quant_conv": false,
+  "quant_embedding": false,
+  "quant_method": "sdnq",
+  "quantization_device": null,
+  "quantized_matmul_dtype": null,
+  "return_device": null,
+  "sdnq_version": "0.1.9",
+  "svd_rank": 32,
+  "svd_steps": 8,
+  "use_dynamic_quantization": false,
+  "use_grad_ckpt": true,
+  "use_hadamard": false,
+  "use_quantized_matmul": false,
+  "use_quantized_matmul_conv": false,
+  "use_static_quantization": true,
+  "use_stochastic_rounding": false,
+  "use_svd": true,
+  "weights_dtype": "uint4"
+}