Upload folder using huggingface_hub

.gitattributes

@@ -33,3 +33,4 @@ saved_model/**/* filter=lfs diff=lfs merge=lfs -text
 *.zip filter=lfs diff=lfs merge=lfs -text
 *.zst filter=lfs diff=lfs merge=lfs -text
 *tfevents* filter=lfs diff=lfs merge=lfs -text
+tokenizer.json filter=lfs diff=lfs merge=lfs -text

README.md

@@ -0,0 +1,724 @@
+---
+license: mit
+base_model:
+- zerofata/GLM-4.5-Iceblink-v2-106B-A12B
+datasets:
+- zerofata/Instruct-Anime
+- zerofata/Roleplay-Anime-Characters
+- zerofata/Instruct-Anime-CreativeWriting
+- zerofata/Summaries-Anime-FandomPages
+pipeline_tag: text-generation
+tags:
+- text adventure
+- roleplay
+- rpg
+- creative writing
+- conversational
+- vllm
+---
+# GLM-4.5-Iceblink-v2-106B-A12B (W8A8 FP8 with 2D-block quantization)
+
+This repo contains GLM-4.5-Iceblink-v2-106B-A12B quantized with mixed FP8/BF16 precision following state-of-the-art Mixture-Of-Expert quantization.
+
+- Original Model:
+  - [zerofata/GLM-4.5-Iceblink-v2-106B-A12B](https://huggingface.co/zerofata/GLM-4.5-Iceblink-v2-106B-A12B)
+
+The model requires Ada (4000 series), Hopper (H100) or Blackwell (5000 series) GPUs for hardware FP8 support.
+
+## 📥 Usage & Running Instructions
+
+The model was tested with vLLM + 2x RTX Pro 6000, here is a script suitable for such configuration with 131072 context length.
+
+### Recommendations
+
+It is however recommended to use only 65K context to avoid significant degradation (https://fiction.live/stories/Fiction-liveBench-Sept-29-2025/oQdzQvKHw8JyXbN87)
+
+The recommended sampler is "min-p" sampling, this sampling is available through
+both the oldest Text completions API and the Chat completions API (and there is a new Response API),
+however most LLM frontends only support modifying min-p when using Text completions.
+You can however use `--override-generation-config "${SAMPLER_JSONCONFIG}"` to override the sampler (which is a merge of generation_config.json and vLLM defaults)
+
+### Running script
+
+```bash
+# Model configuration (Mandatory)
+MODEL="mratsim/GLM-4.5-Iceblink-v2-106B-A12B-FP8"
+MODELNAME="GLM-4.5-Iceblink-v2"
+GPU_UTIL=0.75
+
+# Sampling configuration (Optional, if departing from `generation_config.json`)
+# Note that top_p=0.95 seems to lead to a serious paragraph repetition issue
+SAMPLER_OVERRIDE='{"temperature": 0.8, "min_p": 0.05, "top_p": 1}'
+
+# Prevent memory fragmentation
+export PYTORCH_ALLOC_CONF=expandable_segments:True,max_split_size_mb:512
+
+# Prevent vLLM from using 100% CPU when idle (Very Recommended)
+export VLLM_SLEEP_WHEN_IDLE=1
+
+# Use FlashInfer backend (fastest, recommended, "instant" context reprocessing)
+# export VLLM_ATTENTION_BACKEND=FLASHINFER
+
+vllm serve "${MODEL}" \
+  --tensor-parallel 2 \
+  --served-model-name "${MODELNAME}" \
+  --gpu-memory-utilization ${GPU_UTIL} \
+  --override-generation-config "${SAMPLER_OVERRIDE}"
+```
+
+> ℹ️ The FlashInfer backend may fail with an error similar to
+> `Failed to allocate memory for batch_prefill_tmp_v with size XYZ and alignment 16 in AlignedAllocator`.
+>
+> A workaround is running a sed replacement command within vllm install to increase buffer space
+> ```bash
+> sed -i 's/FLASHINFER_WORKSPACE_BUFFER_SIZE = 256 \* 1024 \* 1024/FLASHINFER_WORKSPACE_BUFFER_SIZE = 768 \* 1024 \* 1024/g' vllm/v1/attention/backends/flashinfer.py
+> ```
+> This will be fixed by PR https://github.com/vllm-project/vllm/pull/25344 or https://github.com/vllm-project/vllm/pull/28269
+
+## 🔬 Quantization method
+
+The llmcompressor library was used with the following recipe:
+
+```
+    scheme=QuantizationScheme(
+        targets=["Linear"],
+        weights=QuantizationArgs(
+            num_bits=8,
+            type=QuantizationType.FLOAT,
+            dynamic=False,
+            symmetric=True,
+            strategy=QuantizationStrategy.BLOCK,
+            block_structure=[32, 32],
+        ),
+        input_activations=QuantizationArgs(
+            num_bits=8,
+            type=QuantizationType.FLOAT,
+            strategy=QuantizationStrategy.GROUP,
+            symmetric=True,
+            dynamic=True,
+            observer=None,
+            group_size=128,
+        ),
+    ),
+    ignore=[
+        "lm_head",
+        "model.embed_tokens",
+        "model.norm",
+        "re:.*input_layernorm$",
+        "re:.*post_attention_layernorm$",
+        "re:.*self_attn.*",
+        "re:.*shared_experts.*",
+        "re:.*mlp\\.gate$",             # MoE router
+        "re:model.layers.0.*",          # Keep first block, (GLM-4.5-Air first_k_dense_replace = 1), also weird loading here https://github.com/vllm-project/vllm/blob/v0.11.0/vllm/model_executor/models/glm4_moe.py#L525-L547
+        "re:model.layers.46.*"          # MTP layer (Multi-token prediction, cannot be loaded by huggingface/transformers)
+    ],
+```
+
+FP8 quantization does not require calibration
+
+### Deep-dive
+
+Quantization should be focused on Linear layer (also called Dense or Fully-Connected layers i.e. MatMu+Bias)
+In particular quantizing LayerNorm/RMSnorm layer is strongly discouraged, see [1]
+> LayerNorm in Quantization. Kovaleva et al. (2021); Wei et al. (2022) find that outliers in the
+> LayerNorm parameters of BERT (Devlin et al., 2019) cause difficulties in model compression.
+> Given the importance of LayerNorm, all the quantization methods we discuss above leave LayerNorm unquantized.
+
+_Note: Experts layers might not be stored as a `Linear` layer, meaning they might be skipped if using `llmcompressor` with a `Linear` target._
+
+Some layers have a higher impact on LLM performance.
+According to [2], spending more bits in attention layers results in large gain compared to spending them in FFN layers.
+According to [3] on 2-bit quantization:
+- quantizing expert FFN layers do not seriously impact model quality
+- quantizing cross-attention has some impact
+- quantizing self-attention has a large impact
+- quantizing dense FFN has a very significant impact
+
+Hence to preserve model quality we choose not to quantize dense FFN layers (i.e. shared experts) and self-attention layers.
+
+We notice that:
+- official MXFP4 weights of gpt-oss-120b from OpenAI keep self-attention in BF16:
+  - https://huggingface.co/openai/gpt-oss-120b/blob/main/model.safetensors.index.json
+- NVFP4 weights of DeepSeek-R1 quantized by Nvidia also keep self-attention in BF16:
+  - https://huggingface.co/nvidia/DeepSeek-R1-0528-FP4/blob/main/model.safetensors.index.json
+
+According to [2], giving more bits to the first `k` blocks have a significantly higher impact on model quality than for the same last `k` blocks.
+In this case, we keep the first layer unquantized as `"first_k_dense_replace": 1` in [config.json](config.json)
+
+### References
+
+1. Why Do Some Inputs Break Low-Bit LLM Quantization? (2025)\
+  Ting-Yun Chang, Muru Zhang, Jesse Thomason, Robin Jia\
+  https://arxiv.org/pdf/2506.12044
+
+2. Examining Post-Training Quantization for Mixture-of-Experts: A Benchmark (2024)\
+  Pingzhi Li, Xiaolong Jin, Yu Cheng, Tianlong Chen\
+  https://arxiv.org/pdf/2406.08155v1
+
+3. Mixture of Quantized Experts (MoQE): Complementary Effect of Low-bit Quantization and Robustness (2023)\
+  Young Jin Kim, Raffy Fahim, Hany Hassan Awadalla\
+  https://arxiv.org/pdf/2310.02410
+
+***
+
+<style>
+.container {
+  --primary-accent: #87CEEB;
+  --secondary-accent: #B0E5F5;
+  --tertiary-accent: #5FA8D3;
+  --ice-accent: #E0F4FF;
+  --silver-accent: #C8D8E4;
+  --glow-primary: rgba(135, 206, 235, 0.6);
+  --glow-secondary: rgba(176, 229, 245, 0.7);
+  
+  --bg-main: #0a1628;
+  --bg-container: #0f1e35;
+  --bg-card: rgba(15, 30, 53, 0.95);
+  --bg-elevated: #162840;
+  
+  --text-main: #E8F4F8;
+  --text-muted: #9BC4E2;
+  --text-bright: #FFFFFF;
+  --white: #FFFFFF;
+  --border-color: #2B4F76;
+  --border-ice: #B0E5F5;
+  
+  --font-title: 'Inter', sans-serif;
+  --font-body: 'Source Sans Pro', sans-serif;
+  --font-code: 'JetBrains Mono', monospace;
+
+  font-family: var(--font-body);
+  color: var(--text-main);
+  line-height: 1.6;
+  font-weight: 400;
+  
+  max-width: 1100px;
+  margin: 20px auto;
+  padding: 60px;
+  background: 
+    linear-gradient(135deg, #0a1628 0%, #0f1e35 50%, #0a1628 100%);
+  min-height: calc(100vh - 40px);
+  position: relative;
+  
+  border: 1px solid var(--border-ice);
+  box-shadow: 
+    0 0 0 3px var(--border-color),
+    0 0 0 5px var(--border-ice),
+    0 0 0 8px var(--border-color),
+    0 0 60px rgba(135, 206, 235, 0.4),
+    inset 0 0 100px rgba(135, 206, 235, 0.15);
+}
+
+.container .title-container {
+  background: linear-gradient(135deg, var(--bg-elevated), var(--bg-card));
+  margin-bottom: 50px;
+  border: 2px solid var(--border-ice);
+  padding: 50px;
+  text-align: center;
+  position: relative;
+  box-shadow: 
+    0 0 0 1px var(--border-color),
+    0 0 0 4px var(--border-ice),
+    0 0 0 6px var(--border-color),
+    0 0 40px var(--glow-primary),
+    inset 0 0 60px rgba(135, 206, 235, 0.2);
+  overflow: visible;
+}
+
+.container .title-container::before {
+  content: '';
+  position: absolute;
+  top: 0;
+  left: 0;
+  right: 0;
+  height: 3px;
+  background: linear-gradient(90deg, transparent, var(--ice-accent), transparent);
+  box-shadow: 0 0 10px var(--ice-accent);
+}
+
+.container .title-container::after {
+  content: '';
+  position: absolute;
+  bottom: 0;
+  left: 0;
+  right: 0;
+  height: 3px;
+  background: linear-gradient(90deg, transparent, var(--ice-accent), transparent);
+  box-shadow: 0 0 10px var(--ice-accent);
+}
+
+
+
+.container .title-container .title-wrapper {
+  position: relative;
+  z-index: 2;
+}
+
+.container .title-main {
+  color: var(--text-bright);
+  font-size: 3rem;
+  font-weight: 900;
+  margin: 0;
+  letter-spacing: 4px;
+  display: block;
+  text-transform: uppercase;
+  background: linear-gradient(90deg, var(--ice-accent), var(--text-bright), var(--ice-accent));
+  background-clip: text;
+  -webkit-background-clip: text;
+  -webkit-text-fill-color: transparent;
+  font-family: var(--font-title);
+  text-shadow: 
+    0 0 30px var(--ice-accent),
+    0 0 60px rgba(176, 229, 245, 0.5),
+    0 4px 8px rgba(135, 206, 235, 0.6);
+  position: relative;
+}
+
+.container .lemonade-text {
+    background: linear-gradient(135deg, var(--silver-accent), var(--ice-accent));
+    background-clip: text;
+    -webkit-background-clip: text;
+    -webkit-text-fill-color: transparent;
+}
+
+.container .version-indicator {
+  color: var(--text-muted);
+  font-size: 0.85rem;
+  font-weight: 600;
+  letter-spacing: 3px;
+  margin-top: 15px;
+  text-transform: uppercase;
+  font-family: var(--font-title);
+  opacity: 0.7;
+}
+
+.container .title-subtitle {
+  padding: 20px;
+  margin-top: 25px;
+  border: 1px solid var(--border-ice);
+  box-shadow: 0 0 20px rgba(135, 206, 235, 0.3);
+}
+
+.container .subtitle-text {
+  color: var(--text-muted);
+  font-size: 1.3rem;
+  font-family: var(--font-body);
+  font-style: italic;
+  font-weight: 400;
+  letter-spacing: 3px;
+  text-transform: uppercase;
+}
+
+.container img {
+  max-width: 100%;
+  border: 3px solid var(--border-ice);
+  margin-bottom: 40px;
+  box-shadow: 
+    0 0 0 1px var(--border-color),
+    0 0 0 5px var(--border-ice),
+    0 12px 24px rgba(135, 206, 235, 0.3);
+}
+
+.container .section-container {
+  margin-bottom: 40px;
+  padding: 40px;
+  background: linear-gradient(135deg, var(--bg-card), var(--bg-elevated));
+  border: 2px solid var(--border-ice);
+  box-shadow: 
+    0 0 0 1px var(--border-color),
+    0 0 0 5px var(--border-ice),
+    0 8px 24px rgba(135, 206, 235, 0.3),
+    inset 0 0 40px rgba(135, 206, 235, 0.1);
+}
+.container .section-container:last-of-type {
+    margin-bottom: 0;
+}
+
+.container .section-header {
+  display: flex;
+  align-items: center;
+  padding: 20px;
+  border: 1px solid var(--border-ice);
+  margin-bottom: 30px;
+  background: rgba(43, 79, 118, 0.2);
+  box-shadow: 0 0 20px rgba(135, 206, 235, 0.2);
+}
+
+.container .section-title {
+  font-family: var(--font-title);
+  background: linear-gradient(90deg, var(--ice-accent), var(--text-bright), var(--ice-accent));
+  background-clip: text;
+  -webkit-background-clip: text;
+  -webkit-text-fill-color: transparent;
+  font-size: 1.8rem;
+  margin: 0 !important;
+  padding: 0 !important;
+  letter-spacing: 4px;
+  font-weight: 800;
+  text-transform: uppercase;
+  border: none !important;
+  display: inline-block;
+  text-shadow: 0 0 20px var(--ice-accent);
+}
+
+.container .section-content {
+    padding: 0;
+}
+
+.container .subheading {
+  color: var(--text-bright);
+  font-size: 1.4rem;
+  margin-top: 30px;
+  margin-bottom: 20px;
+  font-weight: 700;
+  display: block;
+  text-transform: uppercase;
+  letter-spacing: 3px;
+  font-family: var(--font-title);
+  border-bottom: 2px solid var(--border-ice);
+  padding-bottom: 12px;
+  text-shadow: 0 0 15px var(--ice-accent);
+}
+
+.container .data-box {
+  background: linear-gradient(135deg, var(--bg-card), rgba(22, 40, 64, 0.8));
+  padding: 25px;
+  border: 2px solid var(--border-ice);
+  border-left: 5px solid var(--primary-accent);
+  margin-bottom: 25px;
+  box-shadow: 
+    0 0 20px rgba(135, 206, 235, 0.3),
+    inset 0 0 20px rgba(135, 206, 235, 0.1);
+  font-size: 1rem;
+}
+
+.container .data-row {
+  display: flex;
+  align-items: center;
+  margin-bottom: 12px;
+  padding: 10px 0;
+  border-bottom: 1px solid rgba(176, 229, 245, 0.2);
+}
+
+.container .data-row:last-child {
+  margin-bottom: 0;
+}
+
+.container .data-arrow {
+  color: var(--ice-accent);
+  font-weight: bold;
+  margin-right: 15px;
+  font-family: var(--font-code);
+  font-size: 1.2rem;
+  text-shadow: 0 0 10px var(--ice-accent);
+}
+
+.container .data-label {
+  color: var(--text-muted);
+  font-weight: 700;
+  font-family: var(--font-body);
+  margin-right: 15px;
+  min-width: 120px;
+  text-transform: uppercase;
+  letter-spacing: 1px;
+}
+
+.container a {
+  color: var(--text-bright);
+  text-decoration: none;
+  font-weight: 600;
+  transition: all .2s;
+}
+
+.container .data-row a {
+  border-bottom: 1px dotted var(--ice-accent);
+}
+
+.container a:hover {
+  color: var(--ice-accent);
+  text-shadow: 0 0 10px var(--ice-accent);
+}
+
+.container .data-row a:hover {
+  border-bottom-style: solid;
+}
+
+.container .dropdown-container {
+  margin-top: 30px;
+}
+
+.container .dropdown-summary {
+  cursor: pointer;
+  padding: 15px 20px;
+  color: var(--text-muted);
+  font-size: 1.2rem;
+  font-weight: 700;
+  text-transform: uppercase;
+  font-family: var(--font-title);
+  letter-spacing: 2px;
+  list-style: none;
+  transition: all 0.2s ease;
+  border: 1px solid var(--border-ice);
+  background: rgba(43, 79, 118, 0.2);
+  box-shadow: 0 0 15px rgba(135, 206, 235, 0.2);
+}
+.container .dropdown-summary:hover {
+    color: var(--ice-accent);
+    background: rgba(43, 79, 118, 0.3);
+    box-shadow: 0 0 25px rgba(135, 206, 235, 0.3);
+}
+
+.container .dropdown-summary::-webkit-details-marker {
+  display: none;
+}
+
+.container .dropdown-arrow {
+  color: var(--ice-accent);
+  margin-right: 15px;
+  transition: transform 0.2s ease;
+  text-shadow: 0 0 10px var(--ice-accent);
+}
+
+.container details[open] .dropdown-arrow {
+  transform: rotate(90deg);
+}
+
+.container .dropdown-content {
+  margin-top: 20px;
+  padding: 20px 15px;
+  background: linear-gradient(135deg, var(--bg-card), rgba(22, 40, 64, 0.9));
+  border: 2px solid var(--border-ice);
+  box-shadow: 
+    0 0 20px rgba(135, 206, 235, 0.3),
+    inset 0 0 30px rgba(135, 206, 235, 0.1);
+}
+
+.container .config-title {
+  color: var(--text-bright);
+  font-size: 1.1rem;
+  margin-bottom: 10px;
+  font-family: var(--font-body);
+  text-transform: uppercase;
+  letter-spacing: 2px;
+  font-weight: 700;
+  text-shadow: 0 0 10px var(--ice-accent);
+}
+
+.container pre {
+  background: #050a14;
+  padding: 8px 16px;
+  margin: 0;
+  border: 2px solid var(--border-ice);
+  white-space: pre;
+  overflow-x: auto;
+  color: var(--text-main);
+  box-shadow: 
+    0 0 20px rgba(135, 206, 235, 0.2),
+    inset 0 0 20px rgba(135, 206, 235, 0.15);
+}
+
+.container pre code {
+  background: none;
+  color: inherit;
+  padding: 0;
+  margin: 0;
+  display: block;
+  border: none;
+  outline: none;
+}
+
+.container code {
+  font-family: var(--font-code);
+  color: var(--ice-accent);
+  background: rgba(176, 229, 245, 0.15);
+  padding: 4px 8px;
+  border: 1px solid rgba(135, 206, 235, 0.3);
+}
+</style>
+<html lang="en">
+<head>
+  <meta charset="UTF-8">
+  <meta name="viewport" content="width=device-width, initial-scale=1.0">
+  <title>Iceblink</title>
+  <link rel="preconnect" href="https://fonts.googleapis.com">
+  <link rel="preconnect" href="https://fonts.gstatic.com" crossorigin>
+  <link href="https://fonts.googleapis.com/css2?family=Inter:wght@400;600;700;800&family=Source+Sans+Pro:ital,wght@0,400;0,600;1,400&family=JetBrains+Mono:wght@400;500&display=swap" rel="stylesheet">
+</head>
+<body>
+
+<div class="container">
+  <div class="title-container">
+    <div class="glitchy-overlay"></div>
+    <div class="title-wrapper">
+      <h1 class="title-main">
+        <span class="title-prefix">ICEBLINK</span>
+      </h1>
+      <div class="version-indicator">VERSION 2</div>
+    </div>
+  </div>
+
+![image](https://cdn-uploads.huggingface.co/production/uploads/65b19c6c638328850e12d38c/zA2QsjdJNOC-62-bZiOun.png)
+
+  <div class="section-container">
+    <div class="section-header">
+      <div class="section-indicator"></div>
+      <h2 class="section-title">Overview</h2>
+    </div>
+    <div class="section-content">
+      <p></p>
+      <p>Another re-attempt at GLM 4.5 Air. This time using a different training framework, some updated data and better hyperparameters.</p>
+      <p>This model is a creative writing and RP model. It's pretty verbose. The intent is to keep the behavior of the original model, but to improve writing, dialogue & creativity.</p>
+      <p>Compared to the original Iceblink, the effect on this one is more pronounced, with hopefully minimal impact on the intelligence.</p>
+    </div>
+  </div>
+
+  <div class="section-container">
+    <div class="section-header">
+      <div class="section-indicator"></div>
+      <h2 class="section-title">SillyTavern Settings</h2>
+    </div>
+    <div class="section-content">
+      <h3 class="subheading">Recommended Roleplay Format</h3>
+      <div class="data-box">
+        <div class="data-row">
+            <span class="data-arrow">></span>
+            <span class="data-label">Actions:</span>
+            <span>In plaintext</span>
+        </div>
+      <div class="data-row">
+            <span class="data-arrow">></span>
+            <span class="data-label">Dialogue:</span>
+            <span>"In quotes"</span>
+      </div>
+      <div class="data-row">
+            <span class="data-arrow">></span>
+            <span class="data-label">Thoughts:</span>
+            <span>*In asterisks*</span>
+      </div>
+      </div>
+      <h3 class="subheading">Recommended Samplers</h3>
+      <div class="data-box">
+        <div class="data-row">
+          <span class="data-arrow">></span>
+          <span class="data-label">Temp:</span>
+          <span>0.8 - 0.9</span>
+        </div>
+        <div class="data-row">
+          <span class="data-arrow">></span>
+          <span class="data-label">MinP:</span>
+          <span>0.05</span>
+        </div>
+        <div class="data-row">
+          <span class="data-arrow">></span>
+          <span class="data-label">TopP:</span>
+          <span>0.95 - 1.00</span>
+        </div>
+      </div>
+      <h3 class="subheading">Instruct</h3>
+      <div class="data-box">
+        <p style="margin: 0;">GLM4.5 (no thinking): <a href="https://huggingface.co/zerofata/GLM-4.5-Iceblink-106B-A12B/raw/main/GLM45-NoThink-SillyTavern-Preset.json">SillyTavern Preset</a></p>
+      </div>
+    </div>
+  </div>
+
+  <div class="section-container">
+    <div class="section-header">
+      <div class="section-indicator"></div>
+      <h2 class="section-title">Quantizations</h2>
+    </div>
+    <div class="section-content">
+      <div style="margin-bottom: 20px;">
+        <h3 class="subheading">GGUF</h3>
+        <div class="data-box">
+          <div class="data-row">
+            <span class="data-arrow">></span>
+            <a href="https://huggingface.co/ddh0/Iceblink-v3-SFT-3-GGUF">iMatrix (ddh0)</a>
+          </div>
+        </div>
+      </div>
+    </div>
+  </div>
+
+  <div class="section-container">
+    <div class="section-header">
+      <div class="section-indicator"></div>
+      <h2 class="section-title">Creation Process</h2>
+    </div>
+    <div class="section-content">
+      <p>Creation Process: SFT</p>
+      <p>SFT on approx 13 million tokens, SFW / NSFW RP, stories, creative instruct & chat data. Some of the SFW datasets are public and can be found in the model datasets list.</p>
+      <p>I've switched over from Axolotl to MS-Swift w/ Megatron to train MoE models now. There's a roughly 5-10x speedup in training the models, thanks to escaping the naive MoE implementation in TRL. The training time for this run took only 40 minutes, excluding environment setup time.</p>
+      <p>A low LR for GLM Air appears to be king. Going any higher, I've found it extremely easy to begin overcooking the model.</p>
+        <div class="dropdown-container">
+          <details>
+            <summary class="dropdown-summary">
+              <span class="dropdown-arrow">></span>
+              MS-Swift config
+            </summary>
+          <div class="dropdown-content">
+            <p>Not optimized for cost / performance efficiency, YMMV.</p>
+            <div class="config-title">SFT (8*H200)</div>
+            <pre><code>PYTORCH_CUDA_ALLOC_CONF='expandable_segments:True' \
+NPROC_PER_NODE=8 \
+WANDB_API_KEY=wandb_key \
+CUDA_VISIBLE_DEVICES=0,1,2,3,4,5,6,7 \
+megatron sft \
+    --load '/workspace/glm-4.5-air-mcore' \
+    --dataset '/workspace/joined_dataset_cleaned_modified.jsonl' \
+    --load_from_cache_file true \
+    --train_type lora \
+    --lora_rank 256 \
+    --lora_alpha 16 \
+    --use-rslora true \
+    --target_modules all-linear \
+    --split_dataset_ratio 0.01 \
+    --moe_permute_fusion true \
+    --tensor_model_parallel_size 8 \
+    --expert_tensor_parallel_size 1 \
+    --expert_model_parallel_size 8 \
+    --moe_grouped_gemm true \
+    --moe_shared_expert_overlap true \
+    --moe_aux_loss_coeff 6e-5 \
+    --micro_batch_size 4 \
+    --global_batch_size 32 \
+    --recompute_granularity full \
+    --recompute_method uniform \
+    --recompute_num_layers 1 \
+    --max_epochs 2 \
+    --cross_entropy_loss_fusion true \
+    --lr 6e-6 \
+    --lr_warmup_fraction 0.05 \
+    --min_lr 6e-7 \
+    --save megatron_output/Iceblink-v3-SFT-3 \
+    --eval_interval 20 \
+    --save_interval 25 \
+    --finetune true \
+    --packing true \
+    --max_length 10280 \
+    --num_workers 8 \
+    --dataset_num_proc 8 \
+    --no_save_optim true \
+    --no_save_rng true \
+    --sequence_parallel true \
+    --wandb_project Megatron-Air-SFT \
+    --wandb_exp_name Iceblink-v3-SFT-3 \
+    --attention_backend flash</code></pre>
+          </div>
+        </details>
+      </div>
+    </div>
+  </div>
+
+  <div class="section-container">
+    <div class="section-header">
+      <div class="section-indicator"></div>
+      <h2 class="section-title">Special Thanks</h2>
+    </div>
+    <div class="section-content">
+      <p>A shoutout to the people in BeaverAI discord that helped me test this model and my intermediate versions.</p>
+      <p>ddh0 (Madison), Ambius, Dysfunctional & my dude.</p>
+    </div>
+  </div>
+</div>
+</body>
+</html>

args.json

@@ -0,0 +1,467 @@
+{
+  "use_ray": false,
+  "ray_exp_name": null,
+  "device_groups": null,
+  "model": "zai-org/GLM-4.5-Air",
+  "model_type": "glm4_5",
+  "model_revision": null,
+  "task_type": "causal_lm",
+  "torch_dtype": "bfloat16",
+  "attn_impl": null,
+  "new_special_tokens": [],
+  "num_labels": null,
+  "problem_type": null,
+  "rope_scaling": null,
+  "device_map": null,
+  "max_memory": {},
+  "max_model_len": null,
+  "local_repo_path": null,
+  "init_strategy": null,
+  "template": "glm4_5",
+  "system": null,
+  "max_length": 10280,
+  "truncation_strategy": "delete",
+  "max_pixels": null,
+  "agent_template": null,
+  "norm_bbox": null,
+  "use_chat_template": true,
+  "padding_free": true,
+  "padding_side": "right",
+  "loss_scale": "default",
+  "sequence_parallel_size": 1,
+  "response_prefix": null,
+  "template_backend": "swift",
+  "dataset": [
+    "/workspace/joined_dataset_cleaned_modified.jsonl"
+  ],
+  "val_dataset": [],
+  "split_dataset_ratio": 0.01,
+  "data_seed": 42,
+  "dataset_num_proc": 8,
+  "load_from_cache_file": true,
+  "dataset_shuffle": true,
+  "val_dataset_shuffle": false,
+  "streaming": false,
+  "interleave_prob": null,
+  "stopping_strategy": "first_exhausted",
+  "shuffle_buffer_size": 1000,
+  "download_mode": "reuse_dataset_if_exists",
+  "columns": {},
+  "strict": false,
+  "remove_unused_columns": true,
+  "model_name": null,
+  "model_author": null,
+  "custom_dataset_info": [],
+  "quant_method": null,
+  "quant_bits": null,
+  "hqq_axis": null,
+  "bnb_4bit_compute_dtype": "bfloat16",
+  "bnb_4bit_quant_type": "nf4",
+  "bnb_4bit_use_double_quant": true,
+  "bnb_4bit_quant_storage": null,
+  "max_new_tokens": null,
+  "temperature": null,
+  "top_k": null,
+  "top_p": null,
+  "repetition_penalty": null,
+  "num_beams": 1,
+  "stream": false,
+  "stop_words": [],
+  "logprobs": false,
+  "top_logprobs": null,
+  "ckpt_dir": "/workspace/glm-4.5-air-mcore",
+  "lora_modules": [],
+  "tuner_backend": "peft",
+  "train_type": "lora",
+  "adapters": [],
+  "external_plugins": [],
+  "seed": 42,
+  "model_kwargs": {},
+  "load_args": false,
+  "load_data_args": false,
+  "packing": true,
+  "packing_length": 10280,
+  "lazy_tokenize": false,
+  "cached_dataset": [],
+  "custom_register_path": [],
+  "use_hf": false,
+  "hub_token": null,
+  "ddp_timeout": 18000000,
+  "ddp_backend": null,
+  "ignore_args_error": false,
+  "use_swift_lora": false,
+  "freeze_llm": false,
+  "freeze_vit": true,
+  "freeze_aligner": true,
+  "freeze_parameters": [],
+  "freeze_parameters_regex": null,
+  "freeze_parameters_ratio": 0.0,
+  "trainable_parameters": [],
+  "trainable_parameters_regex": null,
+  "adapter_load": null,
+  "target_modules": [
+    "all-linear"
+  ],
+  "target_regex": null,
+  "modules_to_save": [],
+  "lora_rank": 256,
+  "lora_alpha": 16,
+  "lora_dropout": 0.05,
+  "lora_bias": "none",
+  "lora_dtype": null,
+  "use_rslora": true,
+  "rlhf_type": null,
+  "ref_load": null,
+  "ref_adapter_load": null,
+  "beta": 0.1,
+  "rpo_alpha": null,
+  "reference_free": false,
+  "label_smoothing": 0.0,
+  "f_divergence_type": "reverse_kl",
+  "loss_type": null,
+  "desirable_weight": 1.0,
+  "undesirable_weight": 1.0,
+  "calculate_KL": null,
+  "center_rewards_coefficient": null,
+  "padded_vocab_size": 151552,
+  "initialize_embedding": false,
+  "mlp_padding_free": false,
+  "dataloader_persistent_workers": true,
+  "dataloader_prefetch_factor": 10,
+  "architectures": "Glm4MoeForCausalLM",
+  "llm_architectures": null,
+  "max_epochs": 2,
+  "enable_dft_loss": false,
+  "enable_channel_loss": false,
+  "original_max_position_embeddings": null,
+  "partial_rotary_factor": 0.5,
+  "use_shared_expert_gate": false,
+  "vit_gradient_checkpointing": true,
+  "gradient_checkpointing_kwargs": null,
+  "linear_num_value_heads": null,
+  "linear_num_key_heads": null,
+  "linear_key_head_dim": null,
+  "linear_value_head_dim": null,
+  "linear_conv_kernel_dim": null,
+  "layer_types": null,
+  "mrope_interleaved": false,
+  "micro_batch_size": 4,
+  "global_batch_size": 32,
+  "recompute_granularity": "full",
+  "recompute_method": "uniform",
+  "recompute_num_layers": 1,
+  "recompute_modules": [
+    "core_attn"
+  ],
+  "use_cpu_initialization": false,
+  "deterministic_mode": false,
+  "train_iters": null,
+  "log_interval": 5,
+  "tensorboard_dir": "/workspace/megatron_output/Iceblink-v3-SFT-3/v0-20251101-214719/runs",
+  "no_masked_softmax_fusion": false,
+  "no_bias_dropout_fusion": false,
+  "no_bias_swiglu_fusion": false,
+  "no_rope_fusion": false,
+  "no_gradient_accumulation_fusion": false,
+  "cross_entropy_loss_fusion": true,
+  "cross_entropy_fusion_impl": "native",
+  "calculate_per_token_loss": true,
+  "use_flash_attn": false,
+  "attention_backend": "flash",
+  "optimizer": "adam",
+  "optimizer_cpu_offload": false,
+  "optimizer_offload_fraction": 1.0,
+  "use_precision_aware_optimizer": false,
+  "main_grads_dtype": "fp32",
+  "main_params_dtype": "fp32",
+  "exp_avg_dtype": "fp32",
+  "exp_avg_sq_dtype": "fp32",
+  "dataloader_type": "cyclic",
+  "manual_gc": false,
+  "manual_gc_interval": 0,
+  "lr": 6e-06,
+  "lr_decay_style": "cosine",
+  "lr_decay_iters": null,
+  "lr_warmup_iters": 0,
+  "lr_warmup_fraction": 0.05,
+  "min_lr": 6e-07,
+  "weight_decay": 0.1,
+  "clip_grad": 1.0,
+  "adam_beta1": 0.9,
+  "adam_beta2": 0.95,
+  "adam_eps": 1e-08,
+  "sgd_momentum": 0.9,
+  "save": "/workspace/megatron_output/Iceblink-v3-SFT-3/v0-20251101-214719",
+  "save_interval": 25,
+  "save_retain_interval": null,
+  "no_save_optim": true,
+  "no_save_rng": true,
+  "load": "/workspace/glm-4.5-air-mcore",
+  "no_load_optim": false,
+  "no_load_rng": false,
+  "finetune": true,
+  "ckpt_format": "torch_dist",
+  "no_initialization": true,
+  "auto_detect_ckpt_format": true,
+  "exit_on_missing_checkpoint": true,
+  "async_save": false,
+  "use_persistent_ckpt_worker": false,
+  "ckpt_fully_parallel_load": false,
+  "ckpt_assume_constant_structure": false,
+  "distributed_backend": "nccl",
+  "local_rank": 0,
+  "use_distributed_optimizer": true,
+  "tensor_model_parallel_size": 8,
+  "pipeline_model_parallel_size": 1,
+  "decoder_first_pipeline_num_layers": null,
+  "decoder_last_pipeline_num_layers": null,
+  "sequence_parallel": true,
+  "context_parallel_size": 1,
+  "tp_comm_overlap": false,
+  "overlap_grad_reduce": false,
+  "overlap_param_gather": false,
+  "distributed_timeout_minutes": 300000,
+  "num_layers_per_virtual_pipeline_stage": null,
+  "num_virtual_stages_per_pipeline_rank": null,
+  "microbatch_group_size_per_virtual_pipeline_stage": null,
+  "pipeline_model_parallel_layout": null,
+  "num_layers": 46,
+  "hidden_size": 4096,
+  "ffn_hidden_size": 10944,
+  "num_attention_heads": 96,
+  "group_query_attention": true,
+  "num_query_groups": 8,
+  "max_position_embeddings": 131072,
+  "position_embedding_type": "rope",
+  "mrope_section": null,
+  "rotary_base": 1000000,
+  "rotary_percent": 1.0,
+  "rotary_interleaved": false,
+  "normalization": "RMSNorm",
+  "norm_epsilon": 1e-05,
+  "swiglu": true,
+  "untie_embeddings_and_output_weights": true,
+  "disable_bias_linear": true,
+  "add_qkv_bias": true,
+  "attention_dropout": 0.0,
+  "hidden_dropout": 0.0,
+  "kv_channels": 128,
+  "qk_layernorm": false,
+  "transformer_impl": "transformer_engine",
+  "num_experts": 128,
+  "moe_layer_freq": "[0]*1+[1]*45",
+  "moe_ffn_hidden_size": 1408,
+  "moe_shared_expert_intermediate_size": 1408,
+  "moe_router_topk": 8,
+  "moe_router_pre_softmax": false,
+  "moe_router_dtype": "fp32",
+  "moe_router_score_function": "sigmoid",
+  "moe_router_bias_update_rate": 0.001,
+  "moe_router_enable_expert_bias": true,
+  "moe_router_topk_scaling_factor": 1.0,
+  "moe_router_load_balancing_type": "aux_loss",
+  "expert_model_parallel_size": 8,
+  "expert_tensor_parallel_size": 1,
+  "moe_token_dispatcher_type": null,
+  "moe_enable_deepep": false,
+  "moe_grouped_gemm": true,
+  "moe_permute_fusion": true,
+  "moe_aux_loss_coeff": 6e-05,
+  "moe_z_loss_coeff": null,
+  "moe_shared_expert_overlap": true,
+  "moe_layer_recompute": false,
+  "moe_expert_capacity_factor": null,
+  "moe_pad_expert_input_to_capacity": false,
+  "moe_token_drop_policy": null,
+  "multi_latent_attention": false,
+  "q_lora_rank": null,
+  "kv_lora_rank": 32,
+  "qk_head_dim": 128,
+  "qk_pos_emb_head_dim": 64,
+  "fp8_format": null,
+  "fp8_recipe": "delayed",
+  "fp8_amax_history_len": 1024,
+  "fp8_amax_compute_algo": "max",
+  "fp8_param_gather": false,
+  "fp16": false,
+  "bf16": true,
+  "apply_query_key_layer_scaling": false,
+  "attention_softmax_in_fp32": true,
+  "log_params_norm": false,
+  "log_throughput": false,
+  "tensorboard_log_interval": 1,
+  "tensorboard_queue_size": 50,
+  "log_timers_to_tensorboard": true,
+  "no_log_learning_rate_to_tensorboard": false,
+  "log_validation_ppl_to_tensorboard": true,
+  "log_memory_to_tensorboard": true,
+  "logging_level": null,
+  "wandb_project": "Megatron-Air-SFT",
+  "wandb_exp_name": "Iceblink-v3-SFT-3",
+  "wandb_save_dir": null,
+  "eval_iters": -1,
+  "eval_interval": 20,
+  "seq_length": 10280,
+  "num_workers": 8,
+  "megatron_extra_kwargs": {},
+  "add_version": true,
+  "rank": 0,
+  "global_world_size": 8,
+  "local_world_size": 8,
+  "model_suffix": "GLM-4.5-Air",
+  "model_info": "ModelInfo(model_type='glm4_5', model_dir='/root/.cache/modelscope/hub/models/ZhipuAI/GLM-4___5-Air', torch_dtype=torch.bfloat16, max_model_len=131072, quant_method=None, quant_bits=None, rope_scaling=None, is_moe_model=True, config=None, task_type='causal_lm', num_labels=None)",
+  "model_meta": "ModelMeta(model_type='glm4_5', model_groups=[ModelGroup(models=[Model(ms_model_id='ZhipuAI/GLM-4.5-Air-Base', hf_model_id='zai-org/GLM-4.5-Air-Base', model_path=None, ms_revision=None, hf_revision=None), Model(ms_model_id='ZhipuAI/GLM-4.5-Air', hf_model_id='zai-org/GLM-4.5-Air', model_path=None, ms_revision=None, hf_revision=None), Model(ms_model_id='ZhipuAI/GLM-4.5-Air-FP8', hf_model_id='zai-org/GLM-4.5-Air-FP8', model_path=None, ms_revision=None, hf_revision=None), Model(ms_model_id='ZhipuAI/GLM-4.5-Base', hf_model_id='zai-org/GLM-4.5-Base', model_path=None, ms_revision=None, hf_revision=None), Model(ms_model_id='ZhipuAI/GLM-4.5', hf_model_id='zai-org/GLM-4.5', model_path=None, ms_revision=None, hf_revision=None), Model(ms_model_id='ZhipuAI/GLM-4.5-FP8', hf_model_id='zai-org/GLM-4.5-FP8', model_path=None, ms_revision=None, hf_revision=None)], ignore_patterns=None, requires=None, tags=[]), ModelGroup(models=[Model(ms_model_id='ZhipuAI/GLM-4.6', hf_model_id='zai-org/GLM-4.6', model_path=None, ms_revision=None, hf_revision=None)], ignore_patterns=None, requires=None, tags=[])], template='glm4_5', get_function=<function get_model_tokenizer_with_flash_attn at 0x76c9ab052520>, model_arch=None, architectures=['Glm4MoeForCausalLM'], additional_saved_files=[], torch_dtype=None, is_multimodal=False, is_reward=False, is_reranker=False, task_type=None, ignore_patterns=None, requires=['transformers>=4.54'], tags=[])",
+  "model_dir": "/root/.cache/modelscope/hub/models/ZhipuAI/GLM-4___5-Air",
+  "hub": "<class 'swift.hub.hub.MSHub'>",
+  "megatron_model_meta": "MegatronModelMeta(megatron_model_type='gpt', model_types=['qwen2', 'qwen2_5', 'qwq', 'qwq_preview', 'qwen2_5_math', 'llama', 'llama3', 'llama3_1', 'llama3_2', 'longwriter_llama3_1', 'codefuse_codellama', 'marco_o1', 'deepseek', 'deepseek_r1_distill', 'yi', 'yi_coder', 'sus', 'skywork_o1', 'openbuddy_llama', 'openbuddy_llama3', 'megrez', 'reflection', 'numina', 'ziya', 'mengzi3', 'qwen3', 'qwen3_thinking', 'qwen3_nothinking', 'qwen2_moe', 'qwen3_moe', 'qwen3_moe_thinking', 'qwen3_coder', 'internlm3', 'mimo', 'mimo_rl', 'moonlight', 'deepseek_moe', 'deepseek_v2', 'deepseek_v2_5', 'deepseek_r1', 'dots1', 'ernie', 'glm4_5', 'deepseek_v3_1', 'ernie_thinking'], convert_mcore2hf=<function convert_mcore2hf at 0x76c91dcb0fe0>, convert_hf2mcore=<function convert_hf2mcore at 0x76c91dcb0b80>, model_cls=<class 'swift.megatron.model.gpt_model.GPTModel'>, convert_hf_config=<function convert_gpt_hf_config at 0x76c91de6c9a0>, get_transformer_layer_spec=None, model_provider=<function model_provider at 0x76c91dddb880>, visual_cls=None, extra_args_provider=None)",
+  "extra_args": {
+    "use_ray": false,
+    "ray_exp_name": null,
+    "device_groups": null,
+    "model": "ZhipuAI/GLM-4.5-Air",
+    "model_type": "glm4_5",
+    "model_revision": null,
+    "task_type": "causal_lm",
+    "torch_dtype": "bfloat16",
+    "attn_impl": null,
+    "new_special_tokens": [],
+    "num_labels": null,
+    "problem_type": null,
+    "rope_scaling": null,
+    "device_map": null,
+    "max_memory": {},
+    "max_model_len": null,
+    "local_repo_path": null,
+    "init_strategy": null,
+    "template": "glm4_5",
+    "system": null,
+    "max_length": 10280,
+    "truncation_strategy": "delete",
+    "max_pixels": null,
+    "agent_template": null,
+    "norm_bbox": null,
+    "use_chat_template": true,
+    "padding_free": true,
+    "padding_side": "right",
+    "sequence_parallel_size": 1,
+    "response_prefix": null,
+    "template_backend": "swift",
+    "dataset": [
+      "/workspace/joined_dataset_cleaned_modified.jsonl"
+    ],
+    "val_dataset": [],
+    "split_dataset_ratio": 0.01,
+    "data_seed": 42,
+    "dataset_num_proc": 8,
+    "load_from_cache_file": true,
+    "dataset_shuffle": true,
+    "val_dataset_shuffle": false,
+    "streaming": false,
+    "interleave_prob": null,
+    "stopping_strategy": "first_exhausted",
+    "shuffle_buffer_size": 1000,
+    "download_mode": "reuse_dataset_if_exists",
+    "columns": {},
+    "strict": false,
+    "remove_unused_columns": true,
+    "model_name": null,
+    "model_author": null,
+    "custom_dataset_info": [],
+    "quant_method": null,
+    "quant_bits": null,
+    "hqq_axis": null,
+    "bnb_4bit_compute_dtype": "bfloat16",
+    "bnb_4bit_quant_type": "nf4",
+    "bnb_4bit_use_double_quant": true,
+    "bnb_4bit_quant_storage": null,
+    "max_new_tokens": null,
+    "temperature": null,
+    "top_k": null,
+    "top_p": null,
+    "repetition_penalty": null,
+    "num_beams": 1,
+    "stream": false,
+    "stop_words": [],
+    "logprobs": false,
+    "top_logprobs": null,
+    "ckpt_dir": "/workspace/glm-4.5-air-mcore",
+    "lora_modules": [],
+    "tuner_backend": "peft",
+    "train_type": "lora",
+    "adapters": [],
+    "external_plugins": [],
+    "model_kwargs": {},
+    "load_args": false,
+    "load_data_args": false,
+    "packing": true,
+    "packing_length": 10280,
+    "lazy_tokenize": false,
+    "cached_dataset": [],
+    "custom_register_path": [],
+    "use_hf": false,
+    "hub_token": null,
+    "ddp_timeout": 18000000,
+    "ddp_backend": null,
+    "ignore_args_error": false,
+    "use_swift_lora": false,
+    "freeze_llm": false,
+    "freeze_vit": true,
+    "freeze_aligner": true,
+    "freeze_parameters": [],
+    "freeze_parameters_regex": null,
+    "freeze_parameters_ratio": 0.0,
+    "trainable_parameters": [],
+    "trainable_parameters_regex": null,
+    "adapter_load": null,
+    "target_modules": [
+      "all-linear"
+    ],
+    "target_regex": null,
+    "modules_to_save": [],
+    "lora_rank": 256,
+    "lora_alpha": 16,
+    "lora_dropout": 0.05,
+    "lora_bias": "none",
+    "lora_dtype": null,
+    "use_rslora": true,
+    "rlhf_type": null,
+    "ref_load": null,
+    "ref_adapter_load": null,
+    "beta": 0.1,
+    "rpo_alpha": null,
+    "reference_free": false,
+    "label_smoothing": 0.0,
+    "f_divergence_type": "reverse_kl",
+    "loss_type": null,
+    "desirable_weight": 1.0,
+    "undesirable_weight": 1.0,
+    "calculate_KL": null,
+    "center_rewards_coefficient": null,
+    "padded_vocab_size": 151552,
+    "initialize_embedding": false,
+    "mlp_padding_free": false,
+    "dataloader_persistent_workers": true,
+    "dataloader_prefetch_factor": 10,
+    "architectures": "Glm4MoeForCausalLM",
+    "llm_architectures": null,
+    "max_epochs": 2,
+    "enable_dft_loss": false,
+    "enable_channel_loss": false,
+    "original_max_position_embeddings": null,
+    "partial_rotary_factor": 0.5,
+    "use_shared_expert_gate": false,
+    "vit_gradient_checkpointing": true,
+    "gradient_checkpointing_kwargs": null,
+    "linear_num_value_heads": null,
+    "linear_num_key_heads": null,
+    "linear_key_head_dim": null,
+    "linear_value_head_dim": null,
+    "linear_conv_kernel_dim": null,
+    "layer_types": null,
+    "mrope_interleaved": false,
+    "add_version": true,
+    "model_info": "ModelInfo(model_type='glm4_5', model_dir='/root/.cache/modelscope/hub/models/ZhipuAI/GLM-4___5-Air', torch_dtype=torch.bfloat16, max_model_len=131072, quant_method=None, quant_bits=None, rope_scaling=None, is_moe_model=True, config=None, task_type='causal_lm', num_labels=None)",
+    "model_meta": "ModelMeta(model_type='glm4_5', model_groups=[ModelGroup(models=[Model(ms_model_id='ZhipuAI/GLM-4.5-Air-Base', hf_model_id='zai-org/GLM-4.5-Air-Base', model_path=None, ms_revision=None, hf_revision=None), Model(ms_model_id='ZhipuAI/GLM-4.5-Air', hf_model_id='zai-org/GLM-4.5-Air', model_path=None, ms_revision=None, hf_revision=None), Model(ms_model_id='ZhipuAI/GLM-4.5-Air-FP8', hf_model_id='zai-org/GLM-4.5-Air-FP8', model_path=None, ms_revision=None, hf_revision=None), Model(ms_model_id='ZhipuAI/GLM-4.5-Base', hf_model_id='zai-org/GLM-4.5-Base', model_path=None, ms_revision=None, hf_revision=None), Model(ms_model_id='ZhipuAI/GLM-4.5', hf_model_id='zai-org/GLM-4.5', model_path=None, ms_revision=None, hf_revision=None), Model(ms_model_id='ZhipuAI/GLM-4.5-FP8', hf_model_id='zai-org/GLM-4.5-FP8', model_path=None, ms_revision=None, hf_revision=None)], ignore_patterns=None, requires=None, tags=[]), ModelGroup(models=[Model(ms_model_id='ZhipuAI/GLM-4.6', hf_model_id='zai-org/GLM-4.6', model_path=None, ms_revision=None, hf_revision=None)], ignore_patterns=None, requires=None, tags=[])], template='glm4_5', get_function=<function get_model_tokenizer_with_flash_attn at 0x76c9ab052520>, model_arch=None, architectures=['Glm4MoeForCausalLM'], additional_saved_files=[], torch_dtype=None, is_multimodal=False, is_reward=False, is_reranker=False, task_type=None, ignore_patterns=None, requires=['transformers>=4.54'], tags=[])",
+    "megatron_model_meta": "MegatronModelMeta(megatron_model_type='gpt', model_types=['qwen2', 'qwen2_5', 'qwq', 'qwq_preview', 'qwen2_5_math', 'llama', 'llama3', 'llama3_1', 'llama3_2', 'longwriter_llama3_1', 'codefuse_codellama', 'marco_o1', 'deepseek', 'deepseek_r1_distill', 'yi', 'yi_coder', 'sus', 'skywork_o1', 'openbuddy_llama', 'openbuddy_llama3', 'megrez', 'reflection', 'numina', 'ziya', 'mengzi3', 'qwen3', 'qwen3_thinking', 'qwen3_nothinking', 'qwen2_moe', 'qwen3_moe', 'qwen3_moe_thinking', 'qwen3_coder', 'internlm3', 'mimo', 'mimo_rl', 'moonlight', 'deepseek_moe', 'deepseek_v2', 'deepseek_v2_5', 'deepseek_r1', 'dots1', 'ernie', 'glm4_5', 'deepseek_v3_1', 'ernie_thinking'], convert_mcore2hf=<function convert_mcore2hf at 0x76c91dcb0fe0>, convert_hf2mcore=<function convert_hf2mcore at 0x76c91dcb0b80>, model_cls=<class 'swift.megatron.model.gpt_model.GPTModel'>, convert_hf_config=<function convert_gpt_hf_config at 0x76c91de6c9a0>, get_transformer_layer_spec=None, model_provider=<function model_provider at 0x76c91dddb880>, visual_cls=None, extra_args_provider=None)"
+  }
+}

chat_template.jinja

@@ -0,0 +1,103 @@
+[gMASK]<sop>
+{%- if tools -%}
+<|system|>
+# Tools
+
+You may call one or more functions to assist with the user query.
+
+You are provided with function signatures within <tools></tools> XML tags:
+<tools>
+{% for tool in tools %}
+{{ tool | tojson(ensure_ascii=False) }}
+{% endfor %}
+</tools>
+
+For each function call, output the function name and arguments within the following XML format:
+<tool_call>{function-name}
+<arg_key>{arg-key-1}</arg_key>
+<arg_value>{arg-value-1}</arg_value>
+<arg_key>{arg-key-2}</arg_key>
+<arg_value>{arg-value-2}</arg_value>
+...
+</tool_call>{%- endif -%}
+{%- macro visible_text(content) -%}
+    {%- if content is string -%}
+        {{- content }}
+    {%- elif content is iterable and content is not mapping -%}
+        {%- for item in content -%}
+            {%- if item is mapping and item.type == 'text' -%}
+                {{- item.text }}
+            {%- elif item is string -%}
+                {{- item }}
+            {%- endif -%}
+        {%- endfor -%}
+    {%- else -%}
+        {{- content }}
+    {%- endif -%}
+{%- endmacro -%}
+{%- set ns = namespace(last_user_index=-1) %}
+{%- for m in messages %}
+    {%- if m.role == 'user' %}
+        {% set ns.last_user_index = loop.index0 -%}
+    {%- endif %}
+{%- endfor %}
+{% for m in messages %}
+{%- if m.role == 'user' -%}<|user|>
+{{ visible_text(m.content) }}
+{{- '/nothink' if (enable_thinking is defined and not enable_thinking and not visible_text(m.content).endswith("/nothink")) else '' -}}
+{%- elif m.role == 'assistant' -%}
+<|assistant|>
+{%- set reasoning_content = '' %}
+{%- set content = visible_text(m.content) %}
+{%- if m.reasoning_content is string %}
+    {%- set reasoning_content = m.reasoning_content %}
+{%- else %}
+    {%- if '</think>' in content %}
+        {%- set reasoning_content = content.split('</think>')[0].rstrip('\n').split('<think>')[-1].lstrip('\n') %}
+        {%- set content = content.split('</think>')[-1].lstrip('\n') %}
+    {%- endif %}
+{%- endif %}
+{%- if loop.index0 > ns.last_user_index and reasoning_content -%}
+{{ '\n<think>' + reasoning_content.strip() +  '</think>'}}
+{%- else -%}
+{{ '\n<think></think>' }}
+{%- endif -%}
+{%- if content.strip() -%}
+{{ '\n' + content.strip() }}
+{%- endif -%}
+{% if m.tool_calls %}
+{% for tc in m.tool_calls %}
+{%- if tc.function %}
+    {%- set tc = tc.function %}
+{%- endif %}
+{{ '\n<tool_call>' + tc.name }}
+{% set _args = tc.arguments %}
+{% for k, v in _args.items() %}
+<arg_key>{{ k }}</arg_key>
+<arg_value>{{ v | tojson(ensure_ascii=False) if v is not string else v }}</arg_value>
+{% endfor %}
+</tool_call>{% endfor %}
+{% endif %}
+{%- elif m.role == 'tool' -%}
+{%- if m.content is string -%}
+{%- if loop.first or (messages[loop.index0 - 1].role != "tool") %}
+    {{- '<|observation|>' }}
+{%- endif %}
+{{- '\n<tool_response>\n' }}
+{{- m.content }}
+{{- '\n</tool_response>' }}
+{%- else -%}
+<|observation|>{% for tr in m.content %}
+
+<tool_response>
+{{ tr.output if tr.output is defined else tr }}
+</tool_response>{% endfor -%}
+{% endif -%}
+{%- elif m.role == 'system' -%}
+<|system|>
+{{ visible_text(m.content) }}
+{%- endif -%}
+{%- endfor -%}
+{%- if add_generation_prompt -%}
+    <|assistant|>{{- '\n<think></think>' if (enable_thinking is defined and not enable_thinking) else '' -}}
+{%- endif -%}

config.json

@@ -0,0 +1,101 @@
+{
+  "architectures": [
+    "Glm4MoeForCausalLM"
+  ],
+  "attention_bias": true,
+  "attention_dropout": 0.0,
+  "dtype": "bfloat16",
+  "eos_token_id": [
+    151329,
+    151336,
+    151338
+  ],
+  "first_k_dense_replace": 1,
+  "head_dim": 128,
+  "hidden_act": "silu",
+  "hidden_size": 4096,
+  "initializer_range": 0.02,
+  "intermediate_size": 10944,
+  "max_position_embeddings": 131072,
+  "model_type": "glm4_moe",
+  "moe_intermediate_size": 1408,
+  "n_group": 1,
+  "n_routed_experts": 128,
+  "n_shared_experts": 1,
+  "norm_topk_prob": true,
+  "num_attention_heads": 96,
+  "num_experts_per_tok": 8,
+  "num_hidden_layers": 46,
+  "num_key_value_heads": 8,
+  "num_nextn_predict_layers": 1,
+  "pad_token_id": 151329,
+  "partial_rotary_factor": 0.5,
+  "quantization_config": {
+    "config_groups": {
+      "config_group_0": {
+        "format": "float-quantized",
+        "input_activations": {
+          "actorder": null,
+          "block_structure": null,
+          "dynamic": true,
+          "group_size": 32,
+          "num_bits": 8,
+          "observer": null,
+          "observer_kwargs": {},
+          "strategy": "group",
+          "symmetric": true,
+          "type": "float"
+        },
+        "output_activations": null,
+        "targets": [
+          "Linear"
+        ],
+        "weights": {
+          "actorder": null,
+          "block_structure": [
+            32,
+            32
+          ],
+          "dynamic": false,
+          "group_size": null,
+          "num_bits": 8,
+          "observer": "static_minmax",
+          "observer_kwargs": {},
+          "strategy": "block",
+          "symmetric": true,
+          "type": "float"
+        }
+      }
+    },
+    "format": "float-quantized",
+    "global_compression_ratio": null,
+    "ignore": [
+      "lm_head",
+      "model.embed_tokens",
+      "model.norm",
+      "re:.*input_layernorm$",
+      "re:.*post_attention_layernorm$",
+      "re:.*self_attn.*",
+      "re:.*shared_experts.*",
+      "re:.*mlp\\.gate$",
+      "re:model.layers.0.*",
+      "re:model.layers.46.*"
+    ],
+    "kv_cache_scheme": null,
+    "quant_method": "compressed-tensors",
+    "quantization_status": "compressed",
+    "sparsity_config": {},
+    "transform_config": {},
+    "version": "0.12.3.a20251114"
+  },
+  "rms_norm_eps": 1e-05,
+  "rope_scaling": null,
+  "rope_theta": 1000000,
+  "routed_scaling_factor": 1.0,
+  "tie_word_embeddings": false,
+  "topk_group": 1,
+  "transformers_version": "4.57.1",
+  "use_cache": true,
+  "use_qk_norm": false,
+  "vocab_size": 151552
+}

generation_config.json

@@ -0,0 +1,13 @@
+{
+  "_from_model_config": true,
+  "eos_token_id": [
+    151329,
+    151336,
+    151338
+  ],
+  "pad_token_id": 151329,
+  "transformers_version": "4.57.1",
+  "min_p": 0.05,
+  "temperature": 0.8,
+  "top_p": 1,
+}

model-00001-of-00043.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:e98647c7b2193bc5cdf6d8f5d6bb6914135198e1ba6ff8bfdccbcfcc561055e6
+size 3472330576

model-00002-of-00043.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:b58406b1302ca309ca8427d28b32487df3fdbf5f22f6e578e41af96c3f7f06e8
+size 2628840536

model-00003-of-00043.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:06a9729f66e275675cd36543ef33e76e39770bb0cfc4ee2b2420aba63172d7f8
+size 2628840624

model-00004-of-00043.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:f016add1951f3d8c344870bf5f2900d54e67bf56699da589fc7bfb7f5290e8ce
+size 2628840744

model-00005-of-00043.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:7bdd6fe32208fd3396e66c1b54574691506ec71d387d36565c0bfa461697160c
+size 2628840848

model-00006-of-00043.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:30f765974a1e6bb689de58799ea41f233737dc454c5e46b527a00f30c903f6a0
+size 2695299880

model-00007-of-00043.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:5f6fef216e39915c38bb077b2a9d2f0d18431cdfe7149c9af686fef30f0116d2
+size 2677495312

model-00008-of-00043.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:7aac3f587512917f9aa96daa835988baf895ec19a6db45a58495c89a0f6296df
+size 2628840024

model-00009-of-00043.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:e018a272942ffa6dc4ac9b870e4ddc35b96152ee9db91f8a452e718b8a753384
+size 2628841512

model-00010-of-00043.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:abb95fd623ab827541fe0159cc4b1b9396e4b521197ecc5c8de8cef9157d1345
+size 2628840992

model-00011-of-00043.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:42d2d82f32b4dd090ca8f71fe55ee37304f628eb01f39b447300514f29fedc27
+size 2628841040

model-00012-of-00043.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:48fe23cc37310aeec38c81f5ebbe5391854305dbc01fbb7cb48d0fa4a4682e2a
+size 2628841096

model-00013-of-00043.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:5d52c18565718163dd45dc5de161a3b22183bc2e2a3c7e4d7b82c4a74c642c6d
+size 2628841152

model-00014-of-00043.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:e8bb0e7a2322844ceefe50624c71255991fff4b8006a49173d92aaa380e017bd
+size 2628841208

model-00015-of-00043.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:edfe328c041b156392ec8b32695d759f07a944a3c3320d4f802d1463bd623797
+size 2628841256

model-00016-of-00043.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:83c78490d906ae156d0c3971b886e719889a11aa340becd36ff0b370fb547a10
+size 2628841312

model-00017-of-00043.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:5e59efcd6e2f2088ca253ee34cd51dccb593e085b8b85ef20f303d55d38a30d3
+size 2628841368

model-00018-of-00043.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:45aaf47ee18955bc90adcb2573776667785493a177b1c9a012eb26351c82a365
+size 2628841448

model-00019-of-00043.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:90daacb2ba240342dcda680e5b0208a90a58ae4c0b69f9a449207b53589126a5
+size 2628841568

model-00020-of-00043.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:4fa9c37b00b51eb33b1889c7e6c708f6ef5d85d4cafd95ca29b231ce995cdf65
+size 2628841672

model-00021-of-00043.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:3966becdc946ff26d6a0e3597608a0e31fa6982adecbe7970fd17c8d2d56fbba
+size 2695854984

model-00022-of-00043.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:feceda6a5ac4778e8b89a05c839bf0b1e0aa74c93987bde22903782fa661b326
+size 2688499216

model-00023-of-00043.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:307ce7b469368ddcb32b0a43c08a11c1225886034988c881fac661c2088e0d3f
+size 2628840856

model-00024-of-00043.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:1fd15d2154bbc2435d469cdb5d86a9382e00729628eb5f246e45ad23e949cdc2
+size 2628840936

model-00025-of-00043.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:cb32956236edb0543c306adf5f1ff093fc06ca2f0ada3f77cf30acf4f5cee490
+size 2628840984

model-00026-of-00043.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:856a7387cc11f3e83d4f1cea1ea718cf871f12b8120424689e26f29d0fdc66a8
+size 2628841040

model-00027-of-00043.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:02f6177359393906e21cf4216884fc5193c2ee6dd186ec53d3bfa2ce6d676c02
+size 2628841096

model-00028-of-00043.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:2854a163bdc2d3cee25d570fde80eedde779d631f7c075b6f6e7d0a543e1ce3f
+size 2628841152

model-00029-of-00043.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:bafe58fc3438fdf2e7050ba93b1857062043b1e90c4c6bd5ae3aff6df0cba9d3
+size 2628841200

model-00030-of-00043.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:657f5401377233b26df954736f9a003d9ff85c026b6c2f08fbc8a012cfbecd5e
+size 2628841256

model-00031-of-00043.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:f207f7155038e676e10b7612319468bfc70a07acc1a758be29ddc7ebf13c4931
+size 2628841312

model-00032-of-00043.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:911a45cf09ce5949143746603570c17437b96eb9e30e2706df23e515eb183310
+size 2628841368

model-00033-of-00043.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:55c860c93c99214ecf51bbc3f11906f011c206840c90efd9e88e73a1a9db0a93
+size 2628841448

model-00034-of-00043.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:95c4d9e89a895bc72e9a3b1c5755fa9ffd4966614d41761777c481c0f63b46e8
+size 2628841560

model-00035-of-00043.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:dbf96c800abf6205a0e108be5865a6b4cfd9e878bd8b6c6608d221897b00ad36
+size 2628841672

model-00036-of-00043.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:525a5d907b0c4247d034f16fcabad0d4451dc7efc85cab149143bcb52fad4812
+size 2600945536

model-00037-of-00043.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:ac1d8862547b2dfcbce77ea1f8b6d8501e5b62050079478fddde666d308120a0
+size 2737179184

model-00038-of-00043.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:86b23dbc83691b73cff27ffca659d16fdae9ce1dbd6f47f7171814e6fb828888
+size 2628840816

model-00039-of-00043.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:21281dc97867745fd0d240f21d97e2eeed3ae235982fc889f5eda30b027f18aa
+size 2628840912

model-00040-of-00043.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:cb3e970f609132aaa085feb420c1565f5518f404fadd10b0639cf061110e9136
+size 2628840968

model-00041-of-00043.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:52f1f6ff5773ea423cbe9ef2d63a301cec4e748d695233cacf90f4801c0ed52b
+size 2628841024

model-00042-of-00043.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:b7195a323760cde8a0bb8323a425655c9882af978c25419e8816d40e2bcd8811
+size 2628841080

model-00043-of-00043.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:fab0bcb4bb9cf192e81dbd110bac3500456d3702f0c383539d2add0b82510186
+size 9903053704