Duplicate from drbaph/LongCat-AudioDiT-3.5B-fp8

Co-authored-by: DRBAPH <drbaph@users.noreply.huggingface.co>

.gitattributes

@@ -0,0 +1,35 @@
+*.7z filter=lfs diff=lfs merge=lfs -text
+*.arrow filter=lfs diff=lfs merge=lfs -text
+*.bin filter=lfs diff=lfs merge=lfs -text
+*.bz2 filter=lfs diff=lfs merge=lfs -text
+*.ckpt filter=lfs diff=lfs merge=lfs -text
+*.ftz filter=lfs diff=lfs merge=lfs -text
+*.gz filter=lfs diff=lfs merge=lfs -text
+*.h5 filter=lfs diff=lfs merge=lfs -text
+*.joblib filter=lfs diff=lfs merge=lfs -text
+*.lfs.* filter=lfs diff=lfs merge=lfs -text
+*.mlmodel filter=lfs diff=lfs merge=lfs -text
+*.model filter=lfs diff=lfs merge=lfs -text
+*.msgpack filter=lfs diff=lfs merge=lfs -text
+*.npy filter=lfs diff=lfs merge=lfs -text
+*.npz filter=lfs diff=lfs merge=lfs -text
+*.onnx filter=lfs diff=lfs merge=lfs -text
+*.ot filter=lfs diff=lfs merge=lfs -text
+*.parquet filter=lfs diff=lfs merge=lfs -text
+*.pb filter=lfs diff=lfs merge=lfs -text
+*.pickle filter=lfs diff=lfs merge=lfs -text
+*.pkl filter=lfs diff=lfs merge=lfs -text
+*.pt filter=lfs diff=lfs merge=lfs -text
+*.pth filter=lfs diff=lfs merge=lfs -text
+*.rar filter=lfs diff=lfs merge=lfs -text
+*.safetensors filter=lfs diff=lfs merge=lfs -text
+saved_model/**/* filter=lfs diff=lfs merge=lfs -text
+*.tar.* filter=lfs diff=lfs merge=lfs -text
+*.tar filter=lfs diff=lfs merge=lfs -text
+*.tflite filter=lfs diff=lfs merge=lfs -text
+*.tgz filter=lfs diff=lfs merge=lfs -text
+*.wasm filter=lfs diff=lfs merge=lfs -text
+*.xz filter=lfs diff=lfs merge=lfs -text
+*.zip filter=lfs diff=lfs merge=lfs -text
+*.zst filter=lfs diff=lfs merge=lfs -text
+*tfevents* filter=lfs diff=lfs merge=lfs -text

README.md

@@ -0,0 +1,115 @@
+---
+license: mit
+language:
+- zh
+- en
+pipeline_tag: text-to-speech
+tags:
+- comfyui
+- audiodit
+- transformers
+---
+
+# LongCat-AudioDiT-3.5B — FP8 Weight-Only Quantized (ComfyUI)
+
+**FP8 quantized version of [meituan-longcat/LongCat-AudioDiT-3.5B](https://huggingface.co/meituan-longcat/LongCat-AudioDiT-3.5B).**
+
+[**Original Model**](https://huggingface.co/meituan-longcat/LongCat-AudioDiT-3.5B) | [**Paper**](https://github.com/meituan-longcat/LongCat-AudioDiT/blob/main/LongCat-AudioDiT.pdf) | [**GitHub (Original)**](https://github.com/meituan-longcat/LongCat-AudioDiT) | [**ComfyUI Node**](https://github.com/Saganaki22/ComfyUI-LongCat-AudioDIT-TTS)
+
+---
+
+
+![Screenshot 2026-03-30 210100](https://cdn-uploads.huggingface.co/production/uploads/63473b59e5c0717e6737b872/oU6qjLAgmxPXEjsCVCXyg.png)
+
+## What is this?
+
+This is a weight-only FP8 quantization of LongCat-AudioDiT-3.5B — a state-of-the-art diffusion-based zero-shot TTS model by Meituan that operates directly in the waveform latent space. The quantization cuts the on-disk size roughly in half and reduces VRAM usage from ~14 GB to ~4 GB, with no perceptible quality loss in practice.
+
+| | Original (3.5B FP32) | This (3.5B FP8) |
+|---|---|---|
+| **Weight dtype** | float32 | float8_e4m3fn |
+| **Activation dtype** | bfloat16 | bfloat16 |
+| **Scale** | — | per-tensor float32 |
+| **File size** | ~14 GB | ~4 GB |
+| **VRAM (inference)** | ~20 GB | ~8 GB |
+| **Extra dependencies** | none | none |
+
+---
+
+## Quantization Details
+
+**What is quantized:** All linear weight matrices in the DiT transformer backbone. Non-linear weights (VAE, embeddings, layer norms, text encoder) remain in bfloat16.
+
+**Method: Per-tensor symmetric FP8**
+
+Each weight matrix is quantized using a per-tensor scale factor stored in `fp8_scales.json`:
+````
+scale = max(abs(W)) / FP8_MAX        # FP8_MAX = 448.0 for float8_e4m3fn
+W_fp8 = round(W_fp32 / scale)        # quantize
+W_bf16 = W_fp8.to(bfloat16) * scale  # dequantize at inference
+````
+
+**No external quantization library required.** Dequantization is handled automatically by the [ComfyUI-LongCat-AudioDIT-TTS](https://github.com/Saganaki22/ComfyUI-LongCat-AudioDIT-TTS) node pack at load time. The model is fully compatible with the original `audiodit` inference code once dequantized.
+
+**File layout:**
+- `model.safetensors` — FP8 weight tensors
+- `fp8_scales.json` — per-tensor float32 scale factors for dequantization
+
+---
+
+## Hardware Requirements
+
+- **GPU:** NVIDIA GPU with CUDA support
+- **VRAM:** ~8 GB
+- **Native FP8 tensor cores:** Ada Lovelace or Blackwell (RTX 4090, RTX 5090, H100, etc.) — recommended for full speed
+- **Older GPUs (Ampere and below):** Will load and run correctly. Dequantization to bfloat16 happens on all hardware, so you still get the reduced VRAM footprint even without native FP8 cores.
+
+---
+
+## Usage — ComfyUI (Recommended)
+
+The easiest way to use this model is with **[ComfyUI-LongCat-AudioDIT-TTS](https://github.com/Saganaki22/ComfyUI-LongCat-AudioDIT-TTS)**, which has native support for this FP8 model with zero extra setup.
+
+### Installation
+
+1. Install the ComfyUI node via **ComfyUI Manager** (search `LongCat-AudioDiT`) or manually:
+```bash
+   cd ComfyUI/custom_nodes
+   git clone https://github.com/Saganaki22/ComfyUI-LongCat-AudioDIT-TTS.git
+```
+
+2. The model **auto-downloads on first use** — select `LongCat-AudioDiT-3.5B-fp8` from the model dropdown in any LongCat node.
+
+3. Or download manually:
+```bash
+   huggingface-cli download drbaph/LongCat-AudioDiT-3.5B-fp8 --local-dir ComfyUI/models/audiodit/LongCat-AudioDiT-3.5B-fp8
+```
+
+### Available Nodes
+
+- **LongCat AudioDiT TTS** — Zero-shot text-to-speech
+- **LongCat AudioDiT Voice Clone TTS** — Voice cloning from reference audio
+- **LongCat AudioDiT Multi-Speaker TTS** — Multi-speaker conversation synthesis
+
+### Recommended Settings
+
+- `dtype`: `auto` or `bf16` — FP8 weights are dequantized to BF16 at load time
+- `guidance_method`: `cfg` for TTS, `apg` for voice cloning
+- `steps`: `16` (balanced), `32` (higher quality)
+- `keep_model_loaded`: `True` for repeated use
+
+---
+
+## About LongCat-AudioDiT
+
+LongCat-AudioDiT is a non-autoregressive diffusion-based TTS model from Meituan that achieves state-of-the-art zero-shot voice cloning performance on the Seed benchmark. Unlike previous methods relying on mel-spectrograms, it operates directly in the waveform latent space using only a Wav-VAE and a DiT backbone.
+
+The 3.5B variant achieves **0.818 SIM on Seed-ZH** and **0.797 SIM on Seed-Hard**, surpassing both open-source and closed-source competitors.
+
+---
+
+## License
+
+This model inherits the [MIT License](https://github.com/meituan-longcat/LongCat-AudioDiT/blob/main/LICENSE) from [meituan-longcat/LongCat-AudioDiT-3.5B](https://huggingface.co/meituan-longcat/LongCat-AudioDiT-3.5B).
+
+The FP8 quantization was produced by [drbaph](https://huggingface.co/drbaph) and is released under the same license.

config.json

@@ -0,0 +1,106 @@
+{
+  "dit_adaln_type": "global",
+  "dit_adaln_use_text_cond": true,
+  "dit_bias": true,
+  "dit_cross_attn": true,
+  "dit_cross_attn_norm": false,
+  "dit_depth": 32,
+  "dit_dim": 2560,
+  "dit_dropout": 0.0,
+  "dit_eps": 1e-06,
+  "dit_ff_mult": 3.6,
+  "dit_heads": 32,
+  "dit_long_skip": true,
+  "dit_qk_norm": true,
+  "dit_text_conv": true,
+  "dit_text_dim": 768,
+  "dit_use_latent_condition": true,
+  "latent_dim": 64,
+  "latent_hop": 2048,
+  "max_wav_duration": 60,
+  "model_type": "audiodit",
+  "repa_dit_layer": 8,
+  "sampling_rate": 24000,
+  "sigma": 0.0,
+  "text_add_embed": true,
+  "text_encoder_config": {
+    "_name_or_path": "ArthurZ/umt5-base",
+    "architectures": [
+      "UMT5ForConditionalGeneration"
+    ],
+    "chunk_size_feed_forward": 0,
+    "classifier_dropout": 0.0,
+    "d_ff": 2048,
+    "d_kv": 64,
+    "d_model": 768,
+    "decoder_start_token_id": 0,
+    "dense_act_fn": "gelu_new",
+    "dropout_rate": 0.1,
+    "dtype": "float32",
+    "eos_token_id": 1,
+    "feed_forward_proj": "gated-gelu",
+    "id2label": {
+      "0": "LABEL_0",
+      "1": "LABEL_1"
+    },
+    "initializer_factor": 1.0,
+    "is_decoder": false,
+    "is_encoder_decoder": true,
+    "is_gated_act": true,
+    "label2id": {
+      "LABEL_0": 0,
+      "LABEL_1": 1
+    },
+    "layer_norm_epsilon": 1e-06,
+    "model_type": "umt5",
+    "num_decoder_layers": 12,
+    "num_heads": 12,
+    "num_layers": 12,
+    "output_attentions": false,
+    "output_hidden_states": false,
+    "output_past": true,
+    "pad_token_id": 0,
+    "problem_type": null,
+    "relative_attention_max_distance": 128,
+    "relative_attention_num_buckets": 32,
+    "return_dict": true,
+    "scalable_attention": true,
+    "tie_word_embeddings": true,
+    "tokenizer_class": "T5Tokenizer",
+    "use_cache": true,
+    "vocab_size": 256384
+  },
+  "text_encoder_model": "google/umt5-base",
+  "text_norm_feat": true,
+  "transformers_version": "5.3.0",
+  "vae_config": {
+    "c_mults": [
+      1,
+      2,
+      4,
+      8,
+      16
+    ],
+    "channels": 128,
+    "downsample_shortcut": "averaging",
+    "downsampling_ratio": 2048,
+    "encoder_latent_dim": 128,
+    "final_tanh": false,
+    "in_channels": 1,
+    "in_shortcut": "duplicating",
+    "latent_dim": 64,
+    "model_type": "audiodit_vae",
+    "out_shortcut": "averaging",
+    "sample_rate": 24000,
+    "scale": 0.71,
+    "strides": [
+      2,
+      4,
+      4,
+      8,
+      8
+    ],
+    "upsample_shortcut": "duplicating",
+    "use_snake": true
+  }
+}

fp8_scales.json

@@ -0,0 +1,600 @@
+{
+  "text_encoder.encoder.block.0.layer.0.SelfAttention.k.weight": 0.003016880580357143,
+  "text_encoder.encoder.block.0.layer.0.SelfAttention.o.weight": 0.015904017857142856,
+  "text_encoder.encoder.block.0.layer.0.SelfAttention.q.weight": 0.0004359654017857143,
+  "text_encoder.encoder.block.0.layer.0.SelfAttention.relative_attention_bias.weight": 0.05915178571428571,
+  "text_encoder.encoder.block.0.layer.0.SelfAttention.v.weight": 0.0019967215401785715,
+  "text_encoder.encoder.block.0.layer.1.DenseReluDense.wi_0.weight": 0.0025460379464285715,
+  "text_encoder.encoder.block.0.layer.1.DenseReluDense.wi_1.weight": 0.002894810267857143,
+  "text_encoder.encoder.block.0.layer.1.DenseReluDense.wo.weight": 0.010044642857142858,
+  "text_encoder.encoder.block.1.layer.0.SelfAttention.k.weight": 0.005580357142857143,
+  "text_encoder.encoder.block.1.layer.0.SelfAttention.o.weight": 0.01708984375,
+  "text_encoder.encoder.block.1.layer.0.SelfAttention.q.weight": 0.0003901890345982143,
+  "text_encoder.encoder.block.1.layer.0.SelfAttention.relative_attention_bias.weight": 3.6785714285714284,
+  "text_encoder.encoder.block.1.layer.0.SelfAttention.v.weight": 0.0029820033482142855,
+  "text_encoder.encoder.block.1.layer.1.DenseReluDense.wi_0.weight": 0.004377092633928571,
+  "text_encoder.encoder.block.1.layer.1.DenseReluDense.wi_1.weight": 0.0038364955357142855,
+  "text_encoder.encoder.block.1.layer.1.DenseReluDense.wo.weight": 0.007742745535714286,
+  "text_encoder.encoder.block.10.layer.0.SelfAttention.k.weight": 0.005824497767857143,
+  "text_encoder.encoder.block.10.layer.0.SelfAttention.o.weight": 0.03431919642857143,
+  "text_encoder.encoder.block.10.layer.0.SelfAttention.q.weight": 0.0006626674107142857,
+  "text_encoder.encoder.block.10.layer.0.SelfAttention.relative_attention_bias.weight": 0.06277901785714286,
+  "text_encoder.encoder.block.10.layer.0.SelfAttention.v.weight": 0.007359095982142857,
+  "text_encoder.encoder.block.10.layer.1.DenseReluDense.wi_0.weight": 0.007045200892857143,
+  "text_encoder.encoder.block.10.layer.1.DenseReluDense.wi_1.weight": 0.016043526785714284,
+  "text_encoder.encoder.block.10.layer.1.DenseReluDense.wo.weight": 0.018694196428571428,
+  "text_encoder.encoder.block.11.layer.0.SelfAttention.k.weight": 0.00439453125,
+  "text_encoder.encoder.block.11.layer.0.SelfAttention.o.weight": 0.018973214285714284,
+  "text_encoder.encoder.block.11.layer.0.SelfAttention.q.weight": 0.0005711146763392857,
+  "text_encoder.encoder.block.11.layer.0.SelfAttention.relative_attention_bias.weight": 0.06501116071428571,
+  "text_encoder.encoder.block.11.layer.0.SelfAttention.v.weight": 0.009347098214285714,
+  "text_encoder.encoder.block.11.layer.1.DenseReluDense.wi_0.weight": 0.007882254464285714,
+  "text_encoder.encoder.block.11.layer.1.DenseReluDense.wi_1.weight": 0.025111607142857144,
+  "text_encoder.encoder.block.11.layer.1.DenseReluDense.wo.weight": 0.018973214285714284,
+  "text_encoder.encoder.block.2.layer.0.SelfAttention.k.weight": 0.003574916294642857,
+  "text_encoder.encoder.block.2.layer.0.SelfAttention.o.weight": 0.006940569196428571,
+  "text_encoder.encoder.block.2.layer.0.SelfAttention.q.weight": 0.0005384172712053571,
+  "text_encoder.encoder.block.2.layer.0.SelfAttention.relative_attention_bias.weight": 0.07142857142857142,
+  "text_encoder.encoder.block.2.layer.0.SelfAttention.v.weight": 0.0027901785714285715,
+  "text_encoder.encoder.block.2.layer.1.DenseReluDense.wi_0.weight": 0.004813058035714286,
+  "text_encoder.encoder.block.2.layer.1.DenseReluDense.wi_1.weight": 0.006626674107142857,
+  "text_encoder.encoder.block.2.layer.1.DenseReluDense.wo.weight": 0.008091517857142858,
+  "text_encoder.encoder.block.3.layer.0.SelfAttention.k.weight": 0.0034877232142857145,
+  "text_encoder.encoder.block.3.layer.0.SelfAttention.o.weight": 0.01025390625,
+  "text_encoder.encoder.block.3.layer.0.SelfAttention.q.weight": 0.00054931640625,
+  "text_encoder.encoder.block.3.layer.0.SelfAttention.relative_attention_bias.weight": 0.06277901785714286,
+  "text_encoder.encoder.block.3.layer.0.SelfAttention.v.weight": 0.0030343191964285715,
+  "text_encoder.encoder.block.3.layer.1.DenseReluDense.wi_0.weight": 0.005510602678571429,
+  "text_encoder.encoder.block.3.layer.1.DenseReluDense.wi_1.weight": 0.006033761160714286,
+  "text_encoder.encoder.block.3.layer.1.DenseReluDense.wo.weight": 0.011509486607142858,
+  "text_encoder.encoder.block.4.layer.0.SelfAttention.k.weight": 0.004987444196428571,
+  "text_encoder.encoder.block.4.layer.0.SelfAttention.o.weight": 0.012625558035714286,
+  "text_encoder.encoder.block.4.layer.0.SelfAttention.q.weight": 0.0005623953683035714,
+  "text_encoder.encoder.block.4.layer.0.SelfAttention.relative_attention_bias.weight": 0.07421875,
+  "text_encoder.encoder.block.4.layer.0.SelfAttention.v.weight": 0.003958565848214286,
+  "text_encoder.encoder.block.4.layer.1.DenseReluDense.wi_0.weight": 0.007184709821428571,
+  "text_encoder.encoder.block.4.layer.1.DenseReluDense.wi_1.weight": 0.011509486607142858,
+  "text_encoder.encoder.block.4.layer.1.DenseReluDense.wo.weight": 0.00927734375,
+  "text_encoder.encoder.block.5.layer.0.SelfAttention.k.weight": 0.004010881696428571,
+  "text_encoder.encoder.block.5.layer.0.SelfAttention.o.weight": 0.008614676339285714,
+  "text_encoder.encoder.block.5.layer.0.SelfAttention.q.weight": 0.0005841936383928571,
+  "text_encoder.encoder.block.5.layer.0.SelfAttention.relative_attention_bias.weight": 0.06529017857142858,
+  "text_encoder.encoder.block.5.layer.0.SelfAttention.v.weight": 0.0035574776785714285,
+  "text_encoder.encoder.block.5.layer.1.DenseReluDense.wi_0.weight": 0.004743303571428571,
+  "text_encoder.encoder.block.5.layer.1.DenseReluDense.wi_1.weight": 0.007882254464285714,
+  "text_encoder.encoder.block.5.layer.1.DenseReluDense.wo.weight": 0.008126395089285714,
+  "text_encoder.encoder.block.6.layer.0.SelfAttention.k.weight": 0.004063197544642857,
+  "text_encoder.encoder.block.6.layer.0.SelfAttention.o.weight": 0.016322544642857144,
+  "text_encoder.encoder.block.6.layer.0.SelfAttention.q.weight": 0.0005841936383928571,
+  "text_encoder.encoder.block.6.layer.0.SelfAttention.relative_attention_bias.weight": 0.06975446428571429,
+  "text_encoder.encoder.block.6.layer.0.SelfAttention.v.weight": 0.004220145089285714,
+  "text_encoder.encoder.block.6.layer.1.DenseReluDense.wi_0.weight": 0.004464285714285714,
+  "text_encoder.encoder.block.6.layer.1.DenseReluDense.wi_1.weight": 0.009974888392857142,
+  "text_encoder.encoder.block.6.layer.1.DenseReluDense.wo.weight": 0.009347098214285714,
+  "text_encoder.encoder.block.7.layer.0.SelfAttention.k.weight": 0.005022321428571429,
+  "text_encoder.encoder.block.7.layer.0.SelfAttention.o.weight": 0.01123046875,
+  "text_encoder.encoder.block.7.layer.0.SelfAttention.q.weight": 0.0005035400390625,
+  "text_encoder.encoder.block.7.layer.0.SelfAttention.relative_attention_bias.weight": 0.05998883928571429,
+  "text_encoder.encoder.block.7.layer.0.SelfAttention.v.weight": 0.005057198660714286,
+  "text_encoder.encoder.block.7.layer.1.DenseReluDense.wi_0.weight": 0.009765625,
+  "text_encoder.encoder.block.7.layer.1.DenseReluDense.wi_1.weight": 0.009974888392857142,
+  "text_encoder.encoder.block.7.layer.1.DenseReluDense.wo.weight": 0.0146484375,
+  "text_encoder.encoder.block.8.layer.0.SelfAttention.k.weight": 0.005824497767857143,
+  "text_encoder.encoder.block.8.layer.0.SelfAttention.o.weight": 0.0224609375,
+  "text_encoder.encoder.block.8.layer.0.SelfAttention.q.weight": 0.0005100795200892857,
+  "text_encoder.encoder.block.8.layer.0.SelfAttention.relative_attention_bias.weight": 0.06668526785714286,
+  "text_encoder.encoder.block.8.layer.0.SelfAttention.v.weight": 0.005859375,
+  "text_encoder.encoder.block.8.layer.1.DenseReluDense.wi_0.weight": 0.005510602678571429,
+  "text_encoder.encoder.block.8.layer.1.DenseReluDense.wi_1.weight": 0.01513671875,
+  "text_encoder.encoder.block.8.layer.1.DenseReluDense.wo.weight": 0.017229352678571428,
+  "text_encoder.encoder.block.9.layer.0.SelfAttention.k.weight": 0.0033656529017857145,
+  "text_encoder.encoder.block.9.layer.0.SelfAttention.o.weight": 0.023995535714285716,
+  "text_encoder.encoder.block.9.layer.0.SelfAttention.q.weight": 0.0005405970982142857,
+  "text_encoder.encoder.block.9.layer.0.SelfAttention.relative_attention_bias.weight": 0.06194196428571429,
+  "text_encoder.encoder.block.9.layer.0.SelfAttention.v.weight": 0.005894252232142857,
+  "text_encoder.encoder.block.9.layer.1.DenseReluDense.wi_0.weight": 0.007184709821428571,
+  "text_encoder.encoder.block.9.layer.1.DenseReluDense.wi_1.weight": 0.012346540178571428,
+  "text_encoder.encoder.block.9.layer.1.DenseReluDense.wo.weight": 0.018275669642857144,
+  "text_encoder.encoder.embed_tokens.weight": 0.33035714285714285,
+  "transformer.adaln_global_mlp.mlp.1.weight": 0.004167829241071429,
+  "transformer.blocks.0.cross_attn.to_k.weight": 0.0013689313616071428,
+  "transformer.blocks.0.cross_attn.to_out.0.weight": 0.0009591238839285714,
+  "transformer.blocks.0.cross_attn.to_q.weight": 0.00091552734375,
+  "transformer.blocks.0.cross_attn.to_v.weight": 0.0015171595982142857,
+  "transformer.blocks.0.ffn.ff.0.weight": 0.00115966796875,
+  "transformer.blocks.0.ffn.ff.3.weight": 0.0012991768973214285,
+  "transformer.blocks.0.self_attn.to_k.weight": 0.0024588448660714285,
+  "transformer.blocks.0.self_attn.to_out.0.weight": 0.0012991768973214285,
+  "transformer.blocks.0.self_attn.to_q.weight": 0.0023716517857142855,
+  "transformer.blocks.0.self_attn.to_v.weight": 0.0013427734375,
+  "transformer.blocks.1.cross_attn.to_k.weight": 0.0011029924665178572,
+  "transformer.blocks.1.cross_attn.to_out.0.weight": 0.0010899135044642857,
+  "transformer.blocks.1.cross_attn.to_q.weight": 0.0011771065848214285,
+  "transformer.blocks.1.cross_attn.to_v.weight": 0.0007847377232142857,
+  "transformer.blocks.1.ffn.ff.0.weight": 0.0012730189732142857,
+  "transformer.blocks.1.ffn.ff.3.weight": 0.0010157993861607143,
+  "transformer.blocks.1.self_attn.to_k.weight": 0.0013776506696428572,
+  "transformer.blocks.1.self_attn.to_out.0.weight": 0.0010157993861607143,
+  "transformer.blocks.1.self_attn.to_q.weight": 0.0014561244419642857,
+  "transformer.blocks.1.self_attn.to_v.weight": 0.0009373256138392857,
+  "transformer.blocks.10.cross_attn.to_k.weight": 0.0012730189732142857,
+  "transformer.blocks.10.cross_attn.to_out.0.weight": 0.0009678431919642857,
+  "transformer.blocks.10.cross_attn.to_q.weight": 0.0011422293526785715,
+  "transformer.blocks.10.cross_attn.to_v.weight": 0.0008108956473214286,
+  "transformer.blocks.10.ffn.ff.0.weight": 0.0019269670758928572,
+  "transformer.blocks.10.ffn.ff.3.weight": 0.004150390625,
+  "transformer.blocks.10.self_attn.to_k.weight": 0.0012032645089285715,
+  "transformer.blocks.10.self_attn.to_out.0.weight": 0.0008457728794642857,
+  "transformer.blocks.10.self_attn.to_q.weight": 0.00164794921875,
+  "transformer.blocks.10.self_attn.to_v.weight": 0.0011117117745535715,
+  "transformer.blocks.11.cross_attn.to_k.weight": 0.0006365094866071429,
+  "transformer.blocks.11.cross_attn.to_out.0.weight": 0.006173270089285714,
+  "transformer.blocks.11.cross_attn.to_q.weight": 0.0008501325334821429,
+  "transformer.blocks.11.cross_attn.to_v.weight": 0.0006103515625,
+  "transformer.blocks.11.ffn.ff.0.weight": 0.0017874581473214285,
+  "transformer.blocks.11.ffn.ff.3.weight": 0.0038539341517857145,
+  "transformer.blocks.11.self_attn.to_k.weight": 0.0018484933035714285,
+  "transformer.blocks.11.self_attn.to_out.0.weight": 0.0015694754464285715,
+  "transformer.blocks.11.self_attn.to_q.weight": 0.0020054408482142855,
+  "transformer.blocks.11.self_attn.to_v.weight": 0.0013776506696428572,
+  "transformer.blocks.12.cross_attn.to_k.weight": 0.0006495884486607143,
+  "transformer.blocks.12.cross_attn.to_out.0.weight": 0.0025809151785714285,
+  "transformer.blocks.12.cross_attn.to_q.weight": 0.0008719308035714286,
+  "transformer.blocks.12.cross_attn.to_v.weight": 0.0005754743303571429,
+  "transformer.blocks.12.ffn.ff.0.weight": 0.0016392299107142857,
+  "transformer.blocks.12.ffn.ff.3.weight": 0.0013253348214285715,
+  "transformer.blocks.12.self_attn.to_k.weight": 0.0016392299107142857,
+  "transformer.blocks.12.self_attn.to_out.0.weight": 0.0006801060267857143,
+  "transformer.blocks.12.self_attn.to_q.weight": 0.002720424107142857,
+  "transformer.blocks.12.self_attn.to_v.weight": 0.0008152553013392857,
+  "transformer.blocks.13.cross_attn.to_k.weight": 0.0009722028459821429,
+  "transformer.blocks.13.cross_attn.to_out.0.weight": 0.003976004464285714,
+  "transformer.blocks.13.cross_attn.to_q.weight": 0.000885009765625,
+  "transformer.blocks.13.cross_attn.to_v.weight": 0.0005449567522321429,
+  "transformer.blocks.13.ffn.ff.0.weight": 0.0012468610491071428,
+  "transformer.blocks.13.ffn.ff.3.weight": 0.004377092633928571,
+  "transformer.blocks.13.self_attn.to_k.weight": 0.0013689313616071428,
+  "transformer.blocks.13.self_attn.to_out.0.weight": 0.0009940011160714285,
+  "transformer.blocks.13.self_attn.to_q.weight": 0.0021885463169642855,
+  "transformer.blocks.13.self_attn.to_v.weight": 0.0007367815290178571,
+  "transformer.blocks.14.cross_attn.to_k.weight": 0.0011771065848214285,
+  "transformer.blocks.14.cross_attn.to_out.0.weight": 0.0023193359375,
+  "transformer.blocks.14.cross_attn.to_q.weight": 0.0009678431919642857,
+  "transformer.blocks.14.cross_attn.to_v.weight": 0.0010201590401785715,
+  "transformer.blocks.14.ffn.ff.0.weight": 0.0015520368303571428,
+  "transformer.blocks.14.ffn.ff.3.weight": 0.0026681082589285715,
+  "transformer.blocks.14.self_attn.to_k.weight": 0.00164794921875,
+  "transformer.blocks.14.self_attn.to_out.0.weight": 0.0014386858258928572,
+  "transformer.blocks.14.self_attn.to_q.weight": 0.0021536690848214285,
+  "transformer.blocks.14.self_attn.to_v.weight": 0.0007455008370535714,
+  "transformer.blocks.15.cross_attn.to_k.weight": 0.0007890973772321429,
+  "transformer.blocks.15.cross_attn.to_out.0.weight": 0.002406529017857143,
+  "transformer.blocks.15.cross_attn.to_q.weight": 0.0008326939174107143,
+  "transformer.blocks.15.cross_attn.to_v.weight": 0.0006583077566964286,
+  "transformer.blocks.15.ffn.ff.0.weight": 0.0014561244419642857,
+  "transformer.blocks.15.ffn.ff.3.weight": 0.002101353236607143,
+  "transformer.blocks.15.self_attn.to_k.weight": 0.0018484933035714285,
+  "transformer.blocks.15.self_attn.to_out.0.weight": 0.0015869140625,
+  "transformer.blocks.15.self_attn.to_q.weight": 0.002284458705357143,
+  "transformer.blocks.15.self_attn.to_v.weight": 0.0009722028459821429,
+  "transformer.blocks.16.cross_attn.to_k.weight": 0.0008806501116071429,
+  "transformer.blocks.16.cross_attn.to_out.0.weight": 0.0028599330357142855,
+  "transformer.blocks.16.cross_attn.to_q.weight": 0.0010419573102678572,
+  "transformer.blocks.16.cross_attn.to_v.weight": 0.0009852818080357143,
+  "transformer.blocks.16.ffn.ff.0.weight": 0.0013253348214285715,
+  "transformer.blocks.16.ffn.ff.3.weight": 0.0017787388392857143,
+  "transformer.blocks.16.self_attn.to_k.weight": 0.0019444056919642857,
+  "transformer.blocks.16.self_attn.to_out.0.weight": 0.0009111676897321429,
+  "transformer.blocks.16.self_attn.to_q.weight": 0.002284458705357143,
+  "transformer.blocks.16.self_attn.to_v.weight": 0.0007455008370535714,
+  "transformer.blocks.17.cross_attn.to_k.weight": 0.0008588518415178571,
+  "transformer.blocks.17.cross_attn.to_out.0.weight": 0.0023890904017857145,
+  "transformer.blocks.17.cross_attn.to_q.weight": 0.0009329659598214286,
+  "transformer.blocks.17.cross_attn.to_v.weight": 0.0011771065848214285,
+  "transformer.blocks.17.ffn.ff.0.weight": 0.0010811941964285715,
+  "transformer.blocks.17.ffn.ff.3.weight": 0.004185267857142857,
+  "transformer.blocks.17.self_attn.to_k.weight": 0.0018136160714285715,
+  "transformer.blocks.17.self_attn.to_out.0.weight": 0.0013776506696428572,
+  "transformer.blocks.17.self_attn.to_q.weight": 0.0021885463169642855,
+  "transformer.blocks.17.self_attn.to_v.weight": 0.0007978166852678571,
+  "transformer.blocks.18.cross_attn.to_k.weight": 0.000885009765625,
+  "transformer.blocks.18.cross_attn.to_out.0.weight": 0.0020839146205357145,
+  "transformer.blocks.18.cross_attn.to_q.weight": 0.0012119838169642857,
+  "transformer.blocks.18.cross_attn.to_v.weight": 0.0007542201450892857,
+  "transformer.blocks.18.ffn.ff.0.weight": 0.0018223353794642857,
+  "transformer.blocks.18.ffn.ff.3.weight": 0.00634765625,
+  "transformer.blocks.18.self_attn.to_k.weight": 0.0021275111607142855,
+  "transformer.blocks.18.self_attn.to_out.0.weight": 0.0017613002232142857,
+  "transformer.blocks.18.self_attn.to_q.weight": 0.002406529017857143,
+  "transformer.blocks.18.self_attn.to_v.weight": 0.00091552734375,
+  "transformer.blocks.19.cross_attn.to_k.weight": 0.0008893694196428571,
+  "transformer.blocks.19.cross_attn.to_out.0.weight": 0.0026681082589285715,
+  "transformer.blocks.19.cross_attn.to_q.weight": 0.0010419573102678572,
+  "transformer.blocks.19.cross_attn.to_v.weight": 0.000762939453125,
+  "transformer.blocks.19.ffn.ff.0.weight": 0.0013166155133928572,
+  "transformer.blocks.19.ffn.ff.3.weight": 0.0013514927455357143,
+  "transformer.blocks.19.self_attn.to_k.weight": 0.0016653878348214285,
+  "transformer.blocks.19.self_attn.to_out.0.weight": 0.0019182477678571428,
+  "transformer.blocks.19.self_attn.to_q.weight": 0.0028250558035714285,
+  "transformer.blocks.19.self_attn.to_v.weight": 0.0008893694196428571,
+  "transformer.blocks.2.cross_attn.to_k.weight": 0.0009722028459821429,
+  "transformer.blocks.2.cross_attn.to_out.0.weight": 0.0015171595982142857,
+  "transformer.blocks.2.cross_attn.to_q.weight": 0.0012555803571428572,
+  "transformer.blocks.2.cross_attn.to_v.weight": 0.0009111676897321429,
+  "transformer.blocks.2.ffn.ff.0.weight": 0.0011160714285714285,
+  "transformer.blocks.2.ffn.ff.3.weight": 0.0012991768973214285,
+  "transformer.blocks.2.self_attn.to_k.weight": 0.0011945452008928572,
+  "transformer.blocks.2.self_attn.to_out.0.weight": 0.0011945452008928572,
+  "transformer.blocks.2.self_attn.to_q.weight": 0.0011509486607142857,
+  "transformer.blocks.2.self_attn.to_v.weight": 0.0006234305245535714,
+  "transformer.blocks.20.cross_attn.to_k.weight": 0.0009809221540178572,
+  "transformer.blocks.20.cross_attn.to_out.0.weight": 0.009626116071428572,
+  "transformer.blocks.20.cross_attn.to_q.weight": 0.0012642996651785715,
+  "transformer.blocks.20.cross_attn.to_v.weight": 0.0011945452008928572,
+  "transformer.blocks.20.ffn.ff.0.weight": 0.0018136160714285715,
+  "transformer.blocks.20.ffn.ff.3.weight": 0.0023193359375,
+  "transformer.blocks.20.self_attn.to_k.weight": 0.0015781947544642857,
+  "transformer.blocks.20.self_attn.to_out.0.weight": 0.0021275111607142855,
+  "transformer.blocks.20.self_attn.to_q.weight": 0.0021275111607142855,
+  "transformer.blocks.20.self_attn.to_v.weight": 0.0006626674107142857,
+  "transformer.blocks.21.cross_attn.to_k.weight": 0.00115966796875,
+  "transformer.blocks.21.cross_attn.to_out.0.weight": 0.010184151785714286,
+  "transformer.blocks.21.cross_attn.to_q.weight": 0.0009809221540178572,
+  "transformer.blocks.21.cross_attn.to_v.weight": 0.0008893694196428571,
+  "transformer.blocks.21.ffn.ff.0.weight": 0.0021275111607142855,
+  "transformer.blocks.21.ffn.ff.3.weight": 0.003016880580357143,
+  "transformer.blocks.21.self_attn.to_k.weight": 0.0018484933035714285,
+  "transformer.blocks.21.self_attn.to_out.0.weight": 0.002110072544642857,
+  "transformer.blocks.21.self_attn.to_q.weight": 0.002040318080357143,
+  "transformer.blocks.21.self_attn.to_v.weight": 0.0007716587611607143,
+  "transformer.blocks.22.cross_attn.to_k.weight": 0.0010114397321428572,
+  "transformer.blocks.22.cross_attn.to_out.0.weight": 0.010463169642857142,
+  "transformer.blocks.22.cross_attn.to_q.weight": 0.0011073521205357143,
+  "transformer.blocks.22.cross_attn.to_v.weight": 0.000823974609375,
+  "transformer.blocks.22.ffn.ff.0.weight": 0.0018397739955357143,
+  "transformer.blocks.22.ffn.ff.3.weight": 0.0033133370535714285,
+  "transformer.blocks.22.self_attn.to_k.weight": 0.00201416015625,
+  "transformer.blocks.22.self_attn.to_out.0.weight": 0.0019967215401785715,
+  "transformer.blocks.22.self_attn.to_q.weight": 0.0023367745535714285,
+  "transformer.blocks.22.self_attn.to_v.weight": 0.0009286063058035714,
+  "transformer.blocks.23.cross_attn.to_k.weight": 0.0010463169642857143,
+  "transformer.blocks.23.cross_attn.to_out.0.weight": 0.004446847098214286,
+  "transformer.blocks.23.cross_attn.to_q.weight": 0.0014299665178571428,
+  "transformer.blocks.23.cross_attn.to_v.weight": 0.0006801060267857143,
+  "transformer.blocks.23.ffn.ff.0.weight": 0.0014299665178571428,
+  "transformer.blocks.23.ffn.ff.3.weight": 0.0013863699776785715,
+  "transformer.blocks.23.self_attn.to_k.weight": 0.0016828264508928572,
+  "transformer.blocks.23.self_attn.to_out.0.weight": 0.002040318080357143,
+  "transformer.blocks.23.self_attn.to_q.weight": 0.0017351422991071428,
+  "transformer.blocks.23.self_attn.to_v.weight": 0.0009198869977678571,
+  "transformer.blocks.24.cross_attn.to_k.weight": 0.0011335100446428572,
+  "transformer.blocks.24.cross_attn.to_out.0.weight": 0.0025634765625,
+  "transformer.blocks.24.cross_attn.to_q.weight": 0.002284458705357143,
+  "transformer.blocks.24.cross_attn.to_v.weight": 0.0005754743303571429,
+  "transformer.blocks.24.ffn.ff.0.weight": 0.0015694754464285715,
+  "transformer.blocks.24.ffn.ff.3.weight": 0.00189208984375,
+  "transformer.blocks.24.self_attn.to_k.weight": 0.0016130719866071428,
+  "transformer.blocks.24.self_attn.to_out.0.weight": 0.0015520368303571428,
+  "transformer.blocks.24.self_attn.to_q.weight": 0.001708984375,
+  "transformer.blocks.24.self_attn.to_v.weight": 0.0008544921875,
+  "transformer.blocks.25.cross_attn.to_k.weight": 0.0013602120535714285,
+  "transformer.blocks.25.cross_attn.to_out.0.weight": 0.003574916294642857,
+  "transformer.blocks.25.cross_attn.to_q.weight": 0.0014735630580357143,
+  "transformer.blocks.25.cross_attn.to_v.weight": 0.00079345703125,
+  "transformer.blocks.25.ffn.ff.0.weight": 0.0013166155133928572,
+  "transformer.blocks.25.ffn.ff.3.weight": 0.0014212472098214285,
+  "transformer.blocks.25.self_attn.to_k.weight": 0.0021798270089285715,
+  "transformer.blocks.25.self_attn.to_out.0.weight": 0.0014386858258928572,
+  "transformer.blocks.25.self_attn.to_q.weight": 0.0017874581473214285,
+  "transformer.blocks.25.self_attn.to_v.weight": 0.00115966796875,
+  "transformer.blocks.26.cross_attn.to_k.weight": 0.0009242466517857143,
+  "transformer.blocks.26.cross_attn.to_out.0.weight": 0.0011683872767857143,
+  "transformer.blocks.26.cross_attn.to_q.weight": 0.0012991768973214285,
+  "transformer.blocks.26.cross_attn.to_v.weight": 0.0012032645089285715,
+  "transformer.blocks.26.ffn.ff.0.weight": 0.0016653878348214285,
+  "transformer.blocks.26.ffn.ff.3.weight": 0.0029994419642857145,
+  "transformer.blocks.26.self_attn.to_k.weight": 0.0017525809151785715,
+  "transformer.blocks.26.self_attn.to_out.0.weight": 0.0012904575892857143,
+  "transformer.blocks.26.self_attn.to_q.weight": 0.0015694754464285715,
+  "transformer.blocks.26.self_attn.to_v.weight": 0.0013776506696428572,
+  "transformer.blocks.27.cross_attn.to_k.weight": 0.0010811941964285715,
+  "transformer.blocks.27.cross_attn.to_out.0.weight": 0.008231026785714286,
+  "transformer.blocks.27.cross_attn.to_q.weight": 0.0016217912946428572,
+  "transformer.blocks.27.cross_attn.to_v.weight": 0.0007978166852678571,
+  "transformer.blocks.27.ffn.ff.0.weight": 0.0011858258928571428,
+  "transformer.blocks.27.ffn.ff.3.weight": 0.002894810267857143,
+  "transformer.blocks.27.self_attn.to_k.weight": 0.0011247907366071428,
+  "transformer.blocks.27.self_attn.to_out.0.weight": 0.002197265625,
+  "transformer.blocks.27.self_attn.to_q.weight": 0.0010899135044642857,
+  "transformer.blocks.27.self_attn.to_v.weight": 0.0006365094866071429,
+  "transformer.blocks.28.cross_attn.to_k.weight": 0.0009591238839285714,
+  "transformer.blocks.28.cross_attn.to_out.0.weight": 0.010602678571428572,
+  "transformer.blocks.28.cross_attn.to_q.weight": 0.0015956333705357143,
+  "transformer.blocks.28.cross_attn.to_v.weight": 0.0007062639508928571,
+  "transformer.blocks.28.ffn.ff.0.weight": 0.0017438616071428572,
+  "transformer.blocks.28.ffn.ff.3.weight": 0.004150390625,
+  "transformer.blocks.28.self_attn.to_k.weight": 0.0016915457589285715,
+  "transformer.blocks.28.self_attn.to_out.0.weight": 0.0020751953125,
+  "transformer.blocks.28.self_attn.to_q.weight": 0.0027029854910714285,
+  "transformer.blocks.28.self_attn.to_v.weight": 0.001007080078125,
+  "transformer.blocks.29.cross_attn.to_k.weight": 0.0009765625,
+  "transformer.blocks.29.cross_attn.to_out.0.weight": 0.012276785714285714,
+  "transformer.blocks.29.cross_attn.to_q.weight": 0.0008021763392857143,
+  "transformer.blocks.29.cross_attn.to_v.weight": 0.0010986328125,
+  "transformer.blocks.29.ffn.ff.0.weight": 0.0015607561383928572,
+  "transformer.blocks.29.ffn.ff.3.weight": 0.003871372767857143,
+  "transformer.blocks.29.self_attn.to_k.weight": 0.0019182477678571428,
+  "transformer.blocks.29.self_attn.to_out.0.weight": 0.0027901785714285715,
+  "transformer.blocks.29.self_attn.to_q.weight": 0.0018048967633928572,
+  "transformer.blocks.29.self_attn.to_v.weight": 0.0008544921875,
+  "transformer.blocks.3.cross_attn.to_k.weight": 0.0007978166852678571,
+  "transformer.blocks.3.cross_attn.to_out.0.weight": 0.0010332380022321428,
+  "transformer.blocks.3.cross_attn.to_q.weight": 0.0013602120535714285,
+  "transformer.blocks.3.cross_attn.to_v.weight": 0.0008021763392857143,
+  "transformer.blocks.3.ffn.ff.0.weight": 0.0010288783482142857,
+  "transformer.blocks.3.ffn.ff.3.weight": 0.001708984375,
+  "transformer.blocks.3.self_attn.to_k.weight": 0.0013950892857142857,
+  "transformer.blocks.3.self_attn.to_out.0.weight": 0.00115966796875,
+  "transformer.blocks.3.self_attn.to_q.weight": 0.0012730189732142857,
+  "transformer.blocks.3.self_attn.to_v.weight": 0.000762939453125,
+  "transformer.blocks.30.cross_attn.to_k.weight": 0.0008196149553571429,
+  "transformer.blocks.30.cross_attn.to_out.0.weight": 0.007952008928571428,
+  "transformer.blocks.30.cross_attn.to_q.weight": 0.0011422293526785715,
+  "transformer.blocks.30.cross_attn.to_v.weight": 0.0009809221540178572,
+  "transformer.blocks.30.ffn.ff.0.weight": 0.0016043526785714285,
+  "transformer.blocks.30.ffn.ff.3.weight": 0.002528599330357143,
+  "transformer.blocks.30.self_attn.to_k.weight": 0.0019356863839285715,
+  "transformer.blocks.30.self_attn.to_out.0.weight": 0.0017351422991071428,
+  "transformer.blocks.30.self_attn.to_q.weight": 0.0017002650669642857,
+  "transformer.blocks.30.self_attn.to_v.weight": 0.0013950892857142857,
+  "transformer.blocks.31.cross_attn.to_k.weight": 0.0006583077566964286,
+  "transformer.blocks.31.cross_attn.to_out.0.weight": 0.006870814732142857,
+  "transformer.blocks.31.cross_attn.to_q.weight": 0.0009809221540178572,
+  "transformer.blocks.31.cross_attn.to_v.weight": 0.0011683872767857143,
+  "transformer.blocks.31.ffn.ff.0.weight": 0.00152587890625,
+  "transformer.blocks.31.ffn.ff.3.weight": 0.0021187918526785715,
+  "transformer.blocks.31.self_attn.to_k.weight": 0.0019705636160714285,
+  "transformer.blocks.31.self_attn.to_out.0.weight": 0.0013340541294642857,
+  "transformer.blocks.31.self_attn.to_q.weight": 0.0018659319196428572,
+  "transformer.blocks.31.self_attn.to_v.weight": 0.0014997209821428572,
+  "transformer.blocks.4.cross_attn.to_k.weight": 0.000885009765625,
+  "transformer.blocks.4.cross_attn.to_out.0.weight": 0.0011683872767857143,
+  "transformer.blocks.4.cross_attn.to_q.weight": 0.0014474051339285715,
+  "transformer.blocks.4.cross_attn.to_v.weight": 0.0006626674107142857,
+  "transformer.blocks.4.ffn.ff.0.weight": 0.0012381417410714285,
+  "transformer.blocks.4.ffn.ff.3.weight": 0.0011247907366071428,
+  "transformer.blocks.4.self_attn.to_k.weight": 0.0016217912946428572,
+  "transformer.blocks.4.self_attn.to_out.0.weight": 0.000762939453125,
+  "transformer.blocks.4.self_attn.to_q.weight": 0.00164794921875,
+  "transformer.blocks.4.self_attn.to_v.weight": 0.0008675711495535714,
+  "transformer.blocks.5.cross_attn.to_k.weight": 0.0009547642299107143,
+  "transformer.blocks.5.cross_attn.to_out.0.weight": 0.0014561244419642857,
+  "transformer.blocks.5.cross_attn.to_q.weight": 0.0009111676897321429,
+  "transformer.blocks.5.cross_attn.to_v.weight": 0.0008675711495535714,
+  "transformer.blocks.5.ffn.ff.0.weight": 0.0012032645089285715,
+  "transformer.blocks.5.ffn.ff.3.weight": 0.0016566685267857143,
+  "transformer.blocks.5.self_attn.to_k.weight": 0.0014474051339285715,
+  "transformer.blocks.5.self_attn.to_out.0.weight": 0.0008501325334821429,
+  "transformer.blocks.5.self_attn.to_q.weight": 0.0015869140625,
+  "transformer.blocks.5.self_attn.to_v.weight": 0.0007760184151785714,
+  "transformer.blocks.6.cross_attn.to_k.weight": 0.0008588518415178571,
+  "transformer.blocks.6.cross_attn.to_out.0.weight": 0.0010114397321428572,
+  "transformer.blocks.6.cross_attn.to_q.weight": 0.0011335100446428572,
+  "transformer.blocks.6.cross_attn.to_v.weight": 0.001068115234375,
+  "transformer.blocks.6.ffn.ff.0.weight": 0.0012381417410714285,
+  "transformer.blocks.6.ffn.ff.3.weight": 0.0020926339285714285,
+  "transformer.blocks.6.self_attn.to_k.weight": 0.0011029924665178572,
+  "transformer.blocks.6.self_attn.to_out.0.weight": 0.000762939453125,
+  "transformer.blocks.6.self_attn.to_q.weight": 0.0016741071428571428,
+  "transformer.blocks.6.self_attn.to_v.weight": 0.0008501325334821429,
+  "transformer.blocks.7.cross_attn.to_k.weight": 0.0008283342633928571,
+  "transformer.blocks.7.cross_attn.to_out.0.weight": 0.0013602120535714285,
+  "transformer.blocks.7.cross_attn.to_q.weight": 0.0007760184151785714,
+  "transformer.blocks.7.cross_attn.to_v.weight": 0.0009329659598214286,
+  "transformer.blocks.7.ffn.ff.0.weight": 0.0016392299107142857,
+  "transformer.blocks.7.ffn.ff.3.weight": 0.00244140625,
+  "transformer.blocks.7.self_attn.to_k.weight": 0.0017525809151785715,
+  "transformer.blocks.7.self_attn.to_out.0.weight": 0.0008675711495535714,
+  "transformer.blocks.7.self_attn.to_q.weight": 0.001708984375,
+  "transformer.blocks.7.self_attn.to_v.weight": 0.0010724748883928572,
+  "transformer.blocks.8.cross_attn.to_k.weight": 0.0007716587611607143,
+  "transformer.blocks.8.cross_attn.to_out.0.weight": 0.0013078962053571428,
+  "transformer.blocks.8.cross_attn.to_q.weight": 0.0008632114955357143,
+  "transformer.blocks.8.cross_attn.to_v.weight": 0.0009416852678571429,
+  "transformer.blocks.8.ffn.ff.0.weight": 0.0012381417410714285,
+  "transformer.blocks.8.ffn.ff.3.weight": 0.001953125,
+  "transformer.blocks.8.self_attn.to_k.weight": 0.0011422293526785715,
+  "transformer.blocks.8.self_attn.to_out.0.weight": 0.0008021763392857143,
+  "transformer.blocks.8.self_attn.to_q.weight": 0.0015171595982142857,
+  "transformer.blocks.8.self_attn.to_v.weight": 0.0012119838169642857,
+  "transformer.blocks.9.cross_attn.to_k.weight": 0.0008326939174107143,
+  "transformer.blocks.9.cross_attn.to_out.0.weight": 0.0008588518415178571,
+  "transformer.blocks.9.cross_attn.to_q.weight": 0.0007803780691964286,
+  "transformer.blocks.9.cross_attn.to_v.weight": 0.0006321498325892857,
+  "transformer.blocks.9.ffn.ff.0.weight": 0.0016130719866071428,
+  "transformer.blocks.9.ffn.ff.3.weight": 0.0023018973214285715,
+  "transformer.blocks.9.self_attn.to_k.weight": 0.00128173828125,
+  "transformer.blocks.9.self_attn.to_out.0.weight": 0.0006495884486607143,
+  "transformer.blocks.9.self_attn.to_q.weight": 0.0014212472098214285,
+  "transformer.blocks.9.self_attn.to_v.weight": 0.0007716587611607143,
+  "transformer.input_embed.proj.0.weight": 0.00029427664620535713,
+  "transformer.input_embed.proj.2.weight": 0.00032479422433035713,
+  "transformer.latent_cond_embedder.proj.0.weight": 0.000732421875,
+  "transformer.latent_cond_embedder.proj.2.weight": 0.0024239676339285715,
+  "transformer.latent_embed.proj.0.weight": 0.0004730224609375,
+  "transformer.latent_embed.proj.2.weight": 0.00037057059151785713,
+  "transformer.norm_out.linear.weight": 0.0009286063058035714,
+  "transformer.proj_out.weight": 0.0003880092075892857,
+  "transformer.text_conv_layer.0.dwconv.weight": 0.0015171595982142857,
+  "transformer.text_conv_layer.0.grn.beta": 0.0008632114955357143,
+  "transformer.text_conv_layer.0.grn.gamma": 0.0010986328125,
+  "transformer.text_conv_layer.0.pwconv1.weight": 0.0011509486607142857,
+  "transformer.text_conv_layer.0.pwconv2.weight": 0.00164794921875,
+  "transformer.text_conv_layer.1.dwconv.weight": 0.0023018973214285715,
+  "transformer.text_conv_layer.1.grn.beta": 0.0006321498325892857,
+  "transformer.text_conv_layer.1.grn.gamma": 0.0011858258928571428,
+  "transformer.text_conv_layer.1.pwconv1.weight": 0.0012119838169642857,
+  "transformer.text_conv_layer.1.pwconv2.weight": 0.0017787388392857143,
+  "transformer.text_conv_layer.2.dwconv.weight": 0.0026157924107142855,
+  "transformer.text_conv_layer.2.grn.beta": 0.000579833984375,
+  "transformer.text_conv_layer.2.grn.gamma": 0.0012642996651785715,
+  "transformer.text_conv_layer.2.pwconv1.weight": 0.0013253348214285715,
+  "transformer.text_conv_layer.2.pwconv2.weight": 0.0021449497767857145,
+  "transformer.text_conv_layer.3.dwconv.weight": 0.002685546875,
+  "transformer.text_conv_layer.3.grn.beta": 0.00040762765066964287,
+  "transformer.text_conv_layer.3.grn.gamma": 0.0016043526785714285,
+  "transformer.text_conv_layer.3.pwconv1.weight": 0.0015869140625,
+  "transformer.text_conv_layer.3.pwconv2.weight": 0.0027901785714285715,
+  "transformer.text_embed.proj.0.weight": 0.0004795619419642857,
+  "transformer.text_embed.proj.2.weight": 0.0015869140625,
+  "transformer.time_embed.time_mlp.0.weight": 0.0006277901785714286,
+  "transformer.time_embed.time_mlp.2.weight": 0.0014822823660714285,
+  "vae.decoder.layers.0.weight_g": 0.0021536690848214285,
+  "vae.decoder.layers.0.weight_v": 0.002528599330357143,
+  "vae.decoder.layers.1.layers.1.weight_g": 0.003208705357142857,
+  "vae.decoder.layers.1.layers.1.weight_v": 0.0023890904017857145,
+  "vae.decoder.layers.1.layers.2.layers.1.weight_g": 0.0029820033482142855,
+  "vae.decoder.layers.1.layers.2.layers.1.weight_v": 0.0017264229910714285,
+  "vae.decoder.layers.1.layers.2.layers.3.weight_g": 0.0027901785714285715,
+  "vae.decoder.layers.1.layers.2.layers.3.weight_v": 0.0016392299107142857,
+  "vae.decoder.layers.1.layers.3.layers.1.weight_g": 0.0032784598214285715,
+  "vae.decoder.layers.1.layers.3.layers.1.weight_v": 0.0024239676339285715,
+  "vae.decoder.layers.1.layers.3.layers.3.weight_g": 0.0031215122767857145,
+  "vae.decoder.layers.1.layers.3.layers.3.weight_v": 0.0011422293526785715,
+  "vae.decoder.layers.1.layers.4.layers.1.weight_g": 0.0020751953125,
+  "vae.decoder.layers.1.layers.4.layers.1.weight_v": 0.0009373256138392857,
+  "vae.decoder.layers.1.layers.4.layers.3.weight_g": 0.0029994419642857145,
+  "vae.decoder.layers.1.layers.4.layers.3.weight_v": 0.0011117117745535715,
+  "vae.decoder.layers.2.layers.1.weight_g": 0.003923688616071429,
+  "vae.decoder.layers.2.layers.1.weight_v": 0.002197265625,
+  "vae.decoder.layers.2.layers.2.layers.1.weight_g": 0.0029820033482142855,
+  "vae.decoder.layers.2.layers.2.layers.1.weight_v": 0.0011683872767857143,
+  "vae.decoder.layers.2.layers.2.layers.3.weight_g": 0.0028773716517857145,
+  "vae.decoder.layers.2.layers.2.layers.3.weight_v": 0.0010855538504464285,
+  "vae.decoder.layers.2.layers.3.layers.1.weight_g": 0.002720424107142857,
+  "vae.decoder.layers.2.layers.3.layers.1.weight_v": 0.0008501325334821429,
+  "vae.decoder.layers.2.layers.3.layers.3.weight_g": 0.0031040736607142855,
+  "vae.decoder.layers.2.layers.3.layers.3.weight_v": 0.0013078962053571428,
+  "vae.decoder.layers.2.layers.4.layers.1.weight_g": 0.002528599330357143,
+  "vae.decoder.layers.2.layers.4.layers.1.weight_v": 0.0012991768973214285,
+  "vae.decoder.layers.2.layers.4.layers.3.weight_g": 0.003208705357142857,
+  "vae.decoder.layers.2.layers.4.layers.3.weight_v": 0.0010768345424107143,
+  "vae.decoder.layers.3.layers.1.weight_g": 0.0029471261160714285,
+  "vae.decoder.layers.3.layers.1.weight_v": 0.0018833705357142857,
+  "vae.decoder.layers.3.layers.2.layers.1.weight_g": 0.002162388392857143,
+  "vae.decoder.layers.3.layers.2.layers.1.weight_v": 0.0007498604910714286,
+  "vae.decoder.layers.3.layers.2.layers.3.weight_g": 0.002528599330357143,
+  "vae.decoder.layers.3.layers.2.layers.3.weight_v": 0.0007237025669642857,
+  "vae.decoder.layers.3.layers.3.layers.1.weight_g": 0.0032784598214285715,
+  "vae.decoder.layers.3.layers.3.layers.1.weight_v": 0.0007542201450892857,
+  "vae.decoder.layers.3.layers.3.layers.3.weight_g": 0.0037144252232142855,
+  "vae.decoder.layers.3.layers.3.layers.3.weight_v": 0.0007847377232142857,
+  "vae.decoder.layers.3.layers.4.layers.1.weight_g": 0.002720424107142857,
+  "vae.decoder.layers.3.layers.4.layers.1.weight_v": 0.0010855538504464285,
+  "vae.decoder.layers.3.layers.4.layers.3.weight_g": 0.0026681082589285715,
+  "vae.decoder.layers.3.layers.4.layers.3.weight_v": 0.0009504045758928571,
+  "vae.decoder.layers.4.layers.1.weight_g": 0.0020926339285714285,
+  "vae.decoder.layers.4.layers.1.weight_v": 0.0008980887276785714,
+  "vae.decoder.layers.4.layers.2.layers.1.weight_g": 0.002406529017857143,
+  "vae.decoder.layers.4.layers.2.layers.1.weight_v": 0.0006059919084821429,
+  "vae.decoder.layers.4.layers.2.layers.3.weight_g": 0.0019444056919642857,
+  "vae.decoder.layers.4.layers.2.layers.3.weight_v": 0.0008980887276785714,
+  "vae.decoder.layers.4.layers.3.layers.1.weight_g": 0.0022495814732142855,
+  "vae.decoder.layers.4.layers.3.layers.1.weight_v": 0.0008021763392857143,
+  "vae.decoder.layers.4.layers.3.layers.3.weight_g": 0.0020751953125,
+  "vae.decoder.layers.4.layers.3.layers.3.weight_v": 0.0007803780691964286,
+  "vae.decoder.layers.4.layers.4.layers.1.weight_g": 0.002528599330357143,
+  "vae.decoder.layers.4.layers.4.layers.1.weight_v": 0.0006059919084821429,
+  "vae.decoder.layers.4.layers.4.layers.3.weight_g": 0.003086635044642857,
+  "vae.decoder.layers.4.layers.4.layers.3.weight_v": 0.0009940011160714285,
+  "vae.decoder.layers.5.layers.1.weight_g": 0.0013689313616071428,
+  "vae.decoder.layers.5.layers.1.weight_v": 0.0005558558872767857,
+  "vae.decoder.layers.5.layers.2.layers.1.weight_g": 0.0025634765625,
+  "vae.decoder.layers.5.layers.2.layers.1.weight_v": 0.0005449567522321429,
+  "vae.decoder.layers.5.layers.2.layers.3.weight_g": 0.0023193359375,
+  "vae.decoder.layers.5.layers.2.layers.3.weight_v": 0.0006931849888392857,
+  "vae.decoder.layers.5.layers.3.layers.1.weight_g": 0.0020839146205357145,
+  "vae.decoder.layers.5.layers.3.layers.1.weight_v": 0.0005362374441964286,
+  "vae.decoder.layers.5.layers.3.layers.3.weight_g": 0.0020054408482142855,
+  "vae.decoder.layers.5.layers.3.layers.3.weight_v": 0.0009068080357142857,
+  "vae.decoder.layers.5.layers.4.layers.1.weight_g": 0.002040318080357143,
+  "vae.decoder.layers.5.layers.4.layers.1.weight_v": 0.0008152553013392857,
+  "vae.decoder.layers.5.layers.4.layers.3.weight_g": 0.0016653878348214285,
+  "vae.decoder.layers.5.layers.4.layers.3.weight_v": 0.0013950892857142857,
+  "vae.decoder.layers.7.weight_g": 1.0132789611816406e-06,
+  "vae.decoder.layers.7.weight_v": 0.00015803745814732144,
+  "vae.encoder.layers.0.weight_g": 0.0031215122767857145,
+  "vae.encoder.layers.0.weight_v": 0.0011683872767857143,
+  "vae.encoder.layers.1.layers.0.layers.1.weight_g": 0.002354213169642857,
+  "vae.encoder.layers.1.layers.0.layers.1.weight_v": 0.00067138671875,
+  "vae.encoder.layers.1.layers.0.layers.3.weight_g": 0.0014299665178571428,
+  "vae.encoder.layers.1.layers.0.layers.3.weight_v": 0.0005754743303571429,
+  "vae.encoder.layers.1.layers.1.layers.1.weight_g": 0.0014822823660714285,
+  "vae.encoder.layers.1.layers.1.layers.1.weight_v": 0.0004795619419642857,
+  "vae.encoder.layers.1.layers.1.layers.3.weight_g": 0.0019444056919642857,
+  "vae.encoder.layers.1.layers.1.layers.3.weight_v": 0.0004664829799107143,
+  "vae.encoder.layers.1.layers.2.layers.1.weight_g": 0.00164794921875,
+  "vae.encoder.layers.1.layers.2.layers.1.weight_v": 0.0004991803850446429,
+  "vae.encoder.layers.1.layers.2.layers.3.weight_g": 0.0015781947544642857,
+  "vae.encoder.layers.1.layers.2.layers.3.weight_v": 0.00047520228794642856,
+  "vae.encoder.layers.1.layers.4.weight_g": 0.0013950892857142857,
+  "vae.encoder.layers.1.layers.4.weight_v": 0.00034659249441964287,
+  "vae.encoder.layers.2.layers.0.layers.1.weight_g": 0.0015433175223214285,
+  "vae.encoder.layers.2.layers.0.layers.1.weight_v": 0.0004991803850446429,
+  "vae.encoder.layers.2.layers.0.layers.3.weight_g": 0.0018397739955357143,
+  "vae.encoder.layers.2.layers.0.layers.3.weight_v": 0.0004926409040178571,
+  "vae.encoder.layers.2.layers.1.layers.1.weight_g": 0.0015607561383928572,
+  "vae.encoder.layers.2.layers.1.layers.1.weight_v": 0.0007890973772321429,
+  "vae.encoder.layers.2.layers.1.layers.3.weight_g": 0.0021449497767857145,
+  "vae.encoder.layers.2.layers.1.layers.3.weight_v": 0.0005623953683035714,
+  "vae.encoder.layers.2.layers.2.layers.1.weight_g": 0.0018484933035714285,
+  "vae.encoder.layers.2.layers.2.layers.1.weight_v": 0.0006365094866071429,
+  "vae.encoder.layers.2.layers.2.layers.3.weight_g": 0.002354213169642857,
+  "vae.encoder.layers.2.layers.2.layers.3.weight_v": 0.0005536760602678571,
+  "vae.encoder.layers.2.layers.4.weight_g": 0.0014299665178571428,
+  "vae.encoder.layers.2.layers.4.weight_v": 0.0005166190011160714,
+  "vae.encoder.layers.3.layers.0.layers.1.weight_g": 0.001708984375,
+  "vae.encoder.layers.3.layers.0.layers.1.weight_v": 0.0007455008370535714,
+  "vae.encoder.layers.3.layers.0.layers.3.weight_g": 0.0028773716517857145,
+  "vae.encoder.layers.3.layers.0.layers.3.weight_v": 0.0005711146763392857,
+  "vae.encoder.layers.3.layers.1.layers.1.weight_g": 0.0016305106026785715,
+  "vae.encoder.layers.3.layers.1.layers.1.weight_v": 0.0009242466517857143,
+  "vae.encoder.layers.3.layers.1.layers.3.weight_g": 0.0025634765625,
+  "vae.encoder.layers.3.layers.1.layers.3.weight_v": 0.0006277901785714286,
+  "vae.encoder.layers.3.layers.2.layers.1.weight_g": 0.0017438616071428572,
+  "vae.encoder.layers.3.layers.2.layers.1.weight_v": 0.000946044921875,
+  "vae.encoder.layers.3.layers.2.layers.3.weight_g": 0.0022147042410714285,
+  "vae.encoder.layers.3.layers.2.layers.3.weight_v": 0.0005318777901785714,
+  "vae.encoder.layers.3.layers.4.weight_g": 0.0017002650669642857,
+  "vae.encoder.layers.3.layers.4.weight_v": 0.0010811941964285715,
+  "vae.encoder.layers.4.layers.0.layers.1.weight_g": 0.0017787388392857143,
+  "vae.encoder.layers.4.layers.0.layers.1.weight_v": 0.0010855538504464285,
+  "vae.encoder.layers.4.layers.0.layers.3.weight_g": 0.0025460379464285715,
+  "vae.encoder.layers.4.layers.0.layers.3.weight_v": 0.0010114397321428572,
+  "vae.encoder.layers.4.layers.1.layers.1.weight_g": 0.0018310546875,
+  "vae.encoder.layers.4.layers.1.layers.1.weight_v": 0.0010288783482142857,
+  "vae.encoder.layers.4.layers.1.layers.3.weight_g": 0.0022670200892857145,
+  "vae.encoder.layers.4.layers.1.layers.3.weight_v": 0.0006059919084821429,
+  "vae.encoder.layers.4.layers.2.layers.1.weight_g": 0.002110072544642857,
+  "vae.encoder.layers.4.layers.2.layers.1.weight_v": 0.0011335100446428572,
+  "vae.encoder.layers.4.layers.2.layers.3.weight_g": 0.002842494419642857,
+  "vae.encoder.layers.4.layers.2.layers.3.weight_v": 0.0006321498325892857,
+  "vae.encoder.layers.4.layers.4.weight_g": 0.0015781947544642857,
+  "vae.encoder.layers.4.layers.4.weight_v": 0.0019705636160714285,
+  "vae.encoder.layers.5.layers.0.layers.1.weight_g": 0.0022059849330357145,
+  "vae.encoder.layers.5.layers.0.layers.1.weight_v": 0.0010899135044642857,
+  "vae.encoder.layers.5.layers.0.layers.3.weight_g": 0.002685546875,
+  "vae.encoder.layers.5.layers.0.layers.3.weight_v": 0.0010332380022321428,
+  "vae.encoder.layers.5.layers.1.layers.1.weight_g": 0.002528599330357143,
+  "vae.encoder.layers.5.layers.1.layers.1.weight_v": 0.0016915457589285715,
+  "vae.encoder.layers.5.layers.1.layers.3.weight_g": 0.0028076171875,
+  "vae.encoder.layers.5.layers.1.layers.3.weight_v": 0.0010506766183035715,
+  "vae.encoder.layers.5.layers.2.layers.1.weight_g": 0.003138950892857143,
+  "vae.encoder.layers.5.layers.2.layers.1.weight_v": 0.0013689313616071428,
+  "vae.encoder.layers.5.layers.2.layers.3.weight_g": 0.0027553013392857145,
+  "vae.encoder.layers.5.layers.2.layers.3.weight_v": 0.0011073521205357143,
+  "vae.encoder.layers.5.layers.4.weight_g": 0.0012381417410714285,
+  "vae.encoder.layers.5.layers.4.weight_v": 0.00128173828125,
+  "vae.encoder.layers.6.weight_g": 0.0019356863839285715,
+  "vae.encoder.layers.6.weight_v": 0.0009024483816964286
+}

model.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:07dbc962ceae00333505c1daf649e4bd15709168f7295cc01bd750813c193085
+size 3836215977