model card (WIP banner), serving patches + SM120 tuned config, conversion/acceptance scripts, LICENSE

LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2026 Z.AI Co., Ltd
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

README.md

@@ -0,0 +1,427 @@
+---
+license: mit
+base_model: LibertAIDAI/GLM-5.3-Flash-NVFP4
+base_model_relation: quantized
+quantized_by: tacos4me
+tags:
+  - nvfp4
+  - fp8
+  - blackwell
+  - vllm
+  - glm
+  - glm5_next
+  - moe
+  - long-context
+  - modelopt
+language:
+  - en
+  - zh
+pipeline_tag: text-generation
+library_name: vllm
+---
+
+> [!WARNING]
+> **⚠️ WIP / preview — weights uploading and final pre-publish checks in flight (2026-08-30).**
+> The serve recipe and patches below are measured-working, the shipped checkpoint carries the
+> parent repo's recalibrated input scales (gated directly — see the provenance note), and
+> independent re-verification (quality re-validation, clean-room patch repro, claims-vs-logs
+> audit) has been folded in. Expect final edits over the next hours.
+
+# GLM-5.3-Flash · NVFP4 experts + block-FP8 attention* · 524,288 ctx in VRAM on 2× RTX PRO 6000
+
+**524,288-token max-model-len, entirely in VRAM (zero CPU offload) on 2× RTX PRO 6000
+Blackwell Workstation (96 GB each, TP2, PCIe only — no NVLink).** Measured: three consecutive
+real prompts of **503,374 / 503,441 / 503,461 tokens** all completed (prefill 75–87 s;
+re-confirmed on the shipped recalibrated-scales build: 503,374 tokens in 80.4 s), with a
+published teacher-forced quality gate (≥95% top-1 vs the BF16-attention baseline, n=271):
+**95.57%** on the shipped build, 96.68% on the pre-recalibration build. Single-stream decode
+**101.7 tok/s** at short context, no speculative decoding.
+
+To our knowledge this is the **first 192 GB setup to reach 512k-class context (524,288) with
+8-bit (FP8-e4m3) KV cache and a published quality gate**. It is *not* the first zero-offload
+GLM-5.3-Flash on this hardware: [brandonmusic/GLM-5.3-Flash-tr3-4bpw](https://huggingface.co/brandonmusic/GLM-5.3-Flash-tr3-4bpw)
+(2026-08-27) served measured ~498k-token needle retrievals at a 499,968-token ceiling on the
+same 2-GPU setup — using **4-bit NVFP4 KV** in that mode (a mode whose own card reports a
+failed quality gate, 3/10), while its FP8-KV mode caps at 262,144. See
+[Related work](#related-work) for the full field.
+
+The trades, up front: the 512k lane is **text-only** (vision tower not loaded), **MTP
+speculative decoding off**, and **single-sequence** (`--max-num-seqs 1`, batched 1024) —
+those three buy the memory that 512k needs; concurrent requests queue. Same checkpoint,
+alternative operating point: **129–146 tok/s** single-stream (content-dependent — MTP
+acceptance varies with content) at **262,144** context with MTP k=1 (clearly a different
+lane — do not mix its numbers with the 512k lane's).
+
+\* "FP8 attention" precisely: all attention *projections* are block-FP8 **except**
+`kv_b_proj`, `f_b/g_b`, and the sparse indexer, which stay BF16 — full map below.
+
+- **Base:** [zai-org/GLM-5.3-Flash](https://huggingface.co/zai-org/GLM-5.3-Flash) (MIT) — 320B
+  total / 18B active MoE, 34 KDA linear-attention layers + 11 DeepSeek-sparse-attention (MLA)
+  layers + 1 MTP draft layer, natively multimodal, 1M-token trained context.
+- **Parent:** [LibertAIDAI/GLM-5.3-Flash-NVFP4](https://huggingface.co/LibertAIDAI/GLM-5.3-Flash-NVFP4)
+  @ revision `caca4e6a` (head, with the **recalibrated** expert input scales) — the NVFP4
+  routed-expert quantization (~97% of parameters) is **their work**, inherited unchanged
+  here; everything else BF16; 181.3 GiB.
+- **This checkpoint (173.85 GiB):** the parent, further quantized — attention projections,
+  shared/dense MLP, and `lm_head` converted from BF16 to block-FP8 (E4M3 + FP32
+  `weight_scale_inv` dequant scales). That is what lets the whole model + 512k-class FP8 KV
+  fit in 2×96 GB with zero CPU offload: the BF16-attention parent cannot even *load*
+  offload-free on this hardware (OOM during weight loading at any context).
+
+Not affiliated with Z.ai / Zhipu or LibertAI.
+
+> [!IMPORTANT]
+> **🔒 This checkpoint does NOT load on stock vLLM — and not on the stock GLM-5.3-Flash vendor
+> images either.** `glm5_next` support currently ships in per-model images
+> ([vllm#53906](https://github.com/vllm-project/vllm/pull/53906) is still open), and those
+> images deliberately construct all attention layers *unquantized* (they strip `quant_config`
+> because the official checkpoints keep attention BF16) and cannot load an FP8 `lm_head`.
+> Serving this checkpoint requires the four files in [`serving/`](./serving) — applied on top
+> of the public `cstechdev/vllm:glm53-flash-nope-sm120-cu130-20260826-r1` image (Dockerfile
+> included). Checkpoint and image are a **matched pair**. Without the patches you will get
+> `KeyError` on attention `weight_scale_inv` tensors or a vocab-embedding shape assert.
+
+> [!NOTE]
+> **Input-scales provenance (2026-08-30).** The parent's expert `input_scale` tensors went
+> through three upstream revisions in one day: an `input_scale=1.0` placeholder
+> (`357b45cc`), its retraction
+> ([discussion #7](https://huggingface.co/LibertAIDAI/GLM-5.3-Flash-NVFP4/discussions/7) —
+> at 1.0, activation blocks with amax < ~0.1 underflow their fp8 block scale to zero,
+> causing input-dependent intermittent repetition), and a same-day recalibration
+> (`caca4e6a`, calibrated per-projection scales). **This repo pins `caca4e6a` — the
+> recalibrated scales — and gates them directly:** teacher-forced top-1 **95.57%** vs the
+> BF16-attention baseline (gate ≥95), long-generation repetition probes clean at 225k and
+> 259,632-token contexts (2,712-token greedy generations; worst 4-gram repeat 3 and 2 vs
+> fail threshold ≥12), and a 503,374-token round completed in **80.4 s** with **~122.7
+> tok/s** marginal decode. Raw gate dumps: `scripts/results-recal.json`,
+> `scripts/results-longrep-recal.json`, `scripts/results-longrep-prod-topup.json`.
+
+---
+
+## What exactly is quantized
+
+Rebuilt from the pristine parent checkpoint by `scripts/convert_fp8attn.py` followed by
+`scripts/convert_lmhead.py` (together they reproduce this checkpoint bit-for-bit —
+determinism experimentally verified), 425 tensors converted BF16 → `F8_E4M3` weight +
+`weight_scale_inv` FP32 block scales, dispatched per-layer through a `MIXED_PRECISION`
+ModelOpt-style manifest in `config.json`:
+
+| family | tensors | FP8 block | size before → after | round-trip err vs BF16 |
+|---|---:|---|---|---:|
+| routed experts (incl. MTP draft) | — | **NVFP4** (unchanged — LibertAIDAI's work) | 163.3 GiB | — |
+| KDA q/k/v_proj (34 layers) | 102 | [32,32] | 6.38 → 3.19 GiB | ~2.4% |
+| KDA b/f_a/g_a_proj (34 layers) | 102 | [32,32] | 0.08 → 0.04 GiB | ~2.3% |
+| o_proj, all 46 layers (KDA + MLA + MTP) | 46 | [128,128] | 3.63 → 1.81 GiB | ~2.3% |
+| MLA q_a/kv_a/q_b (12 layers incl. MTP) | 36 | [128,128] | 0.75 → 0.38 GiB | **0.12–0.16%** |
+| shared experts + dense MLP (L0–2) | 138 | [128,128] | 2.86 → 1.43 GiB | **0.16%** |
+| lm_head | 1 | [128,128] | 1.18 → 0.59 GiB | 2.64% |
+
+Kept **BF16** on purpose: `kv_b_proj` (absorbed to BF16 by the MLA runtime anyway; FP8 would
+leak error into the decode path for ~0.09 GiB/rank), KDA `f_b/g_b` gate projections (the
+upstream authors are most explicit about delta-rule precision there), the sparse-attention
+indexer, `embed_tokens`, the **entire vision tower** (present and unquantized), norms,
+routers, convs.
+
+The MLA q-LoRA family and the MLP family measure at 0.12–0.16% round-trip because the base
+GLM-5.3 model is natively block-FP8 — those BF16 tensors are dequantized copies, and block-128
+re-quantization is a near-bit-level re-derivation. The KDA projections and `o_proj` sit at the
+~2.3% E4M3 floor: that is a real numerics change on the precision-sensitive KDA delta-rule
+path, which is why the quality gate below was run. We do not call the attention conversion
+lossless or near-lossless — it passes our ≥95% teacher-forced gate; the protocol and n are
+published below and the raw dumps ship in `scripts/`.
+
+Why block [32,32] for the KDA in_proj: vLLM fuses q,k,v,b,f_a,g_a into one
+`in_proj_qkvbfg_a` module with a 32-row `b` shard at a non-128-aligned offset, and the
+activation group derives from the block's first dim — the block must be square and divide 32.
+
+Integrity: full index↔safetensors-header cross-check over all 121 shards — 150,651 tensors,
+zero mismatches; `total_size` exact (186,674,168,184 bytes = 173.85 GiB of tensors).
+
+## Measured results
+
+All numbers below are **measured** on this box (2× RTX PRO 6000 Blackwell Workstation
+Edition, 96 GB each, TP2, PCIe, no NVLink; vLLM `0.1.dev20051+g487ecf187` in the patched
+image). Anything inferred is labeled. Full raw quality dumps: `scripts/results-512k.json`
+(this build), `scripts/results-base.json` (BF16-attention baseline),
+`scripts/results-fp8attn-r2.json` (pre-lm-head/pre-tune intermediate).
+
+### Operating points
+
+| | previous prod lane¹ | **512k lane (this card's headline)** | **262k MTP lane** |
+|---|---:|---:|---:|
+| max-model-len | 262,144 | **524,288** | 262,144 |
+| largest measured prompt | — | **503,374 / 503,441 / 503,461 tok (3/3 completed)** | 34k (16.3 s prefill) |
+| CPU offload | 6 GiB/rank experts | **none** | **none** |
+| MTP speculative decode | k=1 | off | k=1 |
+| single-stream decode (short ctx) | 87 tok/s | **101.7 tok/s** | **129–146 tok/s** (content-dependent) |
+| marginal decode @503k ctx | — | ~120–130 tok/s² (~122.7 on the shipped build's gate) | — |
+| ~503k-token prefill | — | 75–87 s (~6.3k tok/s, batched 1024); 80.4 s on the shipped build | — |
+| multimodal | yes | text-only (tower not loaded) | text-only (tower not loaded) |
+| teacher-forced top-1 vs BF16-attn baseline | reference | **95.57%** shipped scales (96.68% pre-recal) | **95.57%** shipped scales (96.68% pre-recal) |
+
+¹ Our previous production configuration: BF16-attention parent checkpoint, CPU-offloaded
+experts, multimodal, MTP k=1 — a **different feature set**, quoted for context, not as an
+apples-to-apples baseline.
+² Prefix-cached differential (61 tokens over a 1.6 s window on top of a cached 503k
+context) — an estimate of marginal decode rate, not a sustained-throughput measurement.
+
+For reference, the BF16-attention parent checkpoint **cannot** operate offload-free on this
+hardware at all: it OOMs during weight loading even at `--max-model-len 32768`
+(~92 GiB/rank of weights alone). Block-FP8 attention is what unlocks offload-free operation.
+
+### Memory (per rank, from vLLM's own ledger)
+
+At identical flags to the old offloaded prod config, FP8 attention+MLP moved the KV pool from
+**340,163 → 645,997 tokens (+89.9%)** and max *loadable* context from **315,392 → 524,288**.
+Offload-free at 524,288: weights+non-torch **84.77 GiB/rank** (vs 87.7 baseline — MTP-off
+−2.01, vision-skip −0.53, FP8 lm_head −0.29, each landing to the decimal), KV pool pinned
+3.95 GiB = **526,825 tokens** at fp8 KV (~7.84 KiB/token/rank).
+
+### The 512k fit ladder (how the final flags were found)
+
+| # | attempt | result |
+|---|---|---|
+| A | 524,288, util 0.985, seqs 2, batched 2048, MTP off, mm off, FP8 lm_head | FAIL — avail 3.08 vs 3.92 GiB needed; profile peak grows ~O(max-model-len) (sparse-MLA prefill workspace) |
+| B | util 0.99, warmups off, graphs [1,2] | FAIL — **on this GPU the init free-memory reading is always ~93.89/94.97 GiB (the worker's own CUDA context ≈ 1.05 GiB), so `--gpu-memory-utilization` ≥ 0.989 can never boot.** Use `--kv-cache-memory` instead |
+| B3 | **`--kv-cache-memory` 4.15 GiB pin** (skips profiling + the util gate), util 0.95 as formality | BOOTS + SERVES, pool 552,543 tok |
+| C1 | + tuned kernels image, seqs 2, batched 2048 | quality gate PASSED, 101.7 tok/s — but FAIL at a real 503k prefill: torch OOM from an O(context) sparse-MLA indexer transient (~4+ GiB at 500k) |
+| C2 | **seqs 1, batched 1024, kv pin 3.95 GiB, graphs [1], JIT warmups off** | **ACCEPTANCE PASSED** — 3× consecutive 503,374 / 503,441 / 503,461-token prompts all completed (`finish_reason: stop`, coherent), min-free watermark 290 MiB, 64 recoverable allocator retries, zero hard OOMs, zero request failures |
+
+### Quality gates
+
+Protocol: teacher-forced comparison (identical context both sides, `prompt_logprobs`, n=271
+scored positions) plus 10 greedy 50-token generations, against the **unmodified NVFP4 parent
+under the unpatched vendor image** (BF16 attention everywhere). Raw dumps and the scorer
+(`compare.py`) ship in `scripts/` — every number below is reproducible from them.
+
+| metric | MLP-only FP8 (r1) | + attention FP8 (r2) | pre-recal build³ | **shipped build (recal scales)⁵** |
+|---|---:|---:|---:|---:|
+| teacher-forced top-1 agreement | 97.4% | 97.05% | 96.68% | **95.57%** (gate: ≥95) |
+| teacher-forced mean \|Δlogprob\| | 0.146 | 0.148 | 0.142 | — (0.142 / 95.94% vs the pre-recal dump) |
+| greedy agreement until first divergence | 94.7% | 90.5% | 60.0% (n=25)⁴ | 90.82% (89/98) |
+| long-generation degeneracy | none (2400 tok) | none (2400 tok) | none (2400 tok) | none — incl. 2,712-tok gens at 225k & 259,632-tok contexts |
+
+³ `scripts/results-512k.json` vs `scripts/results-base.json` under the shipped `compare.py` —
+run it yourself. (An intermediate tuned-kernel lane measured 0.159 / 91.7% at a different
+operating point; the dump's numbers are the ones printed here.)
+⁴ Greedy chains on a reasoning model diverge at the first flipped token, so
+until-divergence agreement is noisy at small n (25 agreeing positions across 10 prompts at
+this operating point); the teacher-forced row is the signal.
+⁵ `scripts/results-recal.json` vs `scripts/results-base.json`; repetition probes in
+`scripts/results-longrep-recal.json` (2× ~225k ctx) and
+`scripts/results-longrep-prod-topup.json` (259,632 ctx). The recalibrated input scales cost
+~1.1 pt of teacher-forced agreement vs the pre-recal build on this metric while fixing the
+upstream-documented fp8-underflow repetition mechanism — see the provenance note up top.
+
+The patched image itself is **bitwise neutral** for BF16-attention checkpoints (measured:
+100.00% teacher-forced agreement, |Δlogprob| = 0.00000 vs the vendor image on the parent
+checkpoint). MTP draft acceptance stays healthy (81% mixed suite / 98.5% on a counting
+bench, r2 measurement). Sanity: `17*23 → 391` correct on every configuration, including
+served at the 524,288 ceiling.
+
+### The honest detour: MLP-only FP8 was a net LOSS
+
+The first conversion round (shared+dense MLP only, −0.71 GiB/rank of weights) measured
+**negative**: KV pool 330,800 vs 340,163 baseline (−2.8%), max context 301,056 vs 315,392
+(−4.5%). vLLM permanently reserves the profiled activation peak, and activating the FP8
+block-GEMM path costs a fixed **+0.80 GiB/rank** in that profile — independent of how many
+layers are FP8. It outspent the saving. Only when attention (+2.81 GiB/rank) joined did the
+fixed cost amortize 4:1. If you are building a partial-FP8 derivative: go all-in or don't
+bother.
+
+### The kernel story (why the tuned config JSON ships in `serving/`)
+
+vLLM has **zero** tuned `w8a8` block-FP8 triton configs for SM120 in-tree (219 configs, none
+for this device name). The KDA fused in_proj (N=12576, K=4096, block [32,32]) therefore ran a
+default config at **313 µs/call — 43.8% of all decode GPU time** (torch-profiler measured).
+One autotuned JSON (`serving/configs/…block_shape=[32,32].json`, benched at 22.3 µs at M=1)
+plus dropping CPU offload took the same checkpoint from 66 → 142 tok/s at 114k ctx (and
+66 → 91.5 tok/s with offload kept). The tuned tiles keep the default's `BLOCK_SIZE_K=32`
+k-split, so the reduction order — and the numerics — are unchanged. The block-128 GEMMs run
+vendored DeepGEMM SM120 kernels at ~1.35 TB/s effective and were left alone.
+
+## Serving
+
+### 1. Build the patched image
+
+```bash
+docker build -t local/vllm-glm53:fp8attn-512k serving/
+```
+
+`serving/` = the public `cstechdev/vllm:glm53-flash-nope-sm120-cu130-20260826-r1` image
+(the [chriswritescode-dev/glm-5.3-flash-sm120](https://github.com/chriswritescode-dev/glm-5.3-flash-sm120)
+overlay of the official per-model vLLM image — our image lineage) plus exactly four files
+(see [`serving/README.md`](./serving/README.md) for the per-file breakdown):
+
+1. `kda.py` — remove the `quant_config` save/None/restore strip around KDA layer construction;
+2. `model.py` — `Glm5NextDecoderLayer`: pass the real `quant_config` instead of `None` to MLA
+   attention (covers the 11 MLA layers + the MTP draft layer);
+3. `modelopt.py` — `ModelOptMixedPrecisionConfig`: `FP8_BLOCK128/64/32` dispatch, fused-module
+   name resolution, MTP draft-prefix aliases, and a `ParallelLMHead` method whose
+   `weight_scale_inv` loader shards scale rows by vocab shard (the FP8 lm_head loader);
+4. `configs/*.json` — the SM120-tuned triton config above.
+
+These files are Apache-2.0 vLLM-tree files (SPDX headers retained), credit to the vLLM
+project and the per-model image authors; the base image is pulled from Docker Hub, not
+redistributed here. **Parallel-work credit:**
+[local-inference-lab](https://github.com/local-inference-lab/vllm) independently landed the
+equivalent MLA quant-config passthrough in their public vLLM fork
+([`dev/jovian-judgement@8590bf9c`](https://github.com/local-inference-lab/vllm/commit/8590bf9c),
+2026-08-29 — about half a day before our acceptance run), paired with MXFP8 attention via
+the b12x SM120 kernels; our patch does the same for the official per-model-image code path
+and adds the block-[32,32] FP8 dispatch for the fused KDA in_proj plus the FP8 lm_head
+loader.
+
+### 2. Run — 512k operating point (as measured in acceptance)
+
+```bash
+docker run --init --rm --gpus all --runtime nvidia --ipc=host --network host \
+  --shm-size=32g --ulimit memlock=-1 --ulimit stack=67108864 \
+  -e VLLM_ENGINE_READY_TIMEOUT_S=3600 \
+  -v /path/to/GLM-5.3-Flash-NVFP4-FP8ATTN-512K:/model:ro \
+  local/vllm-glm53:fp8attn-512k /model \
+  --served-model-name glm-5.3-flash \
+  --host 0.0.0.0 --port 8000 \
+  --tensor-parallel-size 2 \
+  --max-model-len 524288 \
+  --gpu-memory-utilization 0.95 \
+  --kv-cache-memory 4241026048 \
+  --kv-cache-dtype fp8 \
+  --max-num-seqs 1 \
+  --max-num-batched-tokens 1024 \
+  --limit-mm-per-prompt '{"image":0,"video":0}' \
+  --kernel-config '{"enable_jit_warmup":false,"enable_cutedsl_warmup":false}' \
+  --compilation-config '{"cudagraph_capture_sizes":[1]}' \
+  --enable-prefix-caching \
+  --no-enable-flashinfer-autotune \
+  --enable-auto-tool-choice --tool-call-parser glm47 --reasoning-parser glm45 \
+  --trust-remote-code
+```
+
+Every flag is load-bearing:
+
+- **`--kv-cache-memory 4241026048` (3.95 GiB/rank, pinned)** — the load-bearing trick, and a
+  disclosed trade: the pin **bypasses vLLM's memory-profiling gate**. Background: on this
+  GPU the boot-time free-memory reading is always ~93.89 of 94.97 GiB (the worker's own CUDA
+  context ≈ 1.05 GiB is counted as used), so `--gpu-memory-utilization` ≥ 0.989 can **never
+  boot** — the util gate compares against a number that includes your own overhead. The
+  explicit pin skips profiling *and* the util gate; `0.95` remains only as a formality.
+  Pool at this pin: 526,825 tokens. The headroom was then proven by the 3×503k soak, not by
+  a profiler.
+- **`--max-num-seqs 1 --max-num-batched-tokens 1024`** — the sparse-MLA prefill carries an
+  O(context) transient (~4+ GiB at 500k context, growing 306→517 MiB allocations in the
+  indexer scoring path). seqs 2 / batched 2048 boots and passes short-context quality but
+  **dies with a torch OOM at a real 503k prefill** (ladder row C1). Single-sequence is the
+  512k use case; concurrent requests queue.
+- **`--limit-mm-per-prompt '{"image":0,"video":0}'`** — builds the vision tower as a missing
+  stage (never allocated, weights skipped): −0.53 GiB/rank. This lane is **text-only**.
+- **No `--speculative-config`** — MTP off frees ~2.0 GiB/rank (draft layer); required to fit 512k.
+- **JIT warmups off, cudagraph sizes [1]** — trims the profile/graph reservations that the
+  512k fit needs back.
+
+Margin disclosure: the min-free VRAM watermark during a 503k prefill is **~290 MiB**, with
+recoverable PyTorch allocator retries by design (64 across the 3-prompt soak, zero request
+failures). **Do not co-locate other GPU tenants while a long prefill is in flight** — under
+external VRAM perturbation this configuration will OOM.
+
+### 2b. Run — 262k MTP speed lane (129–146 tok/s)
+
+Same command, changed flags: `--max-model-len 262144 --kv-cache-memory 3113851289
+--max-num-seqs 2` (drop `--compilation-config`), add
+`--speculative-config '{"method":"mtp","num_speculative_tokens":1}'`.
+Measured: pool 327,680 tokens (1.25× at 262k), single-stream decode **129–146 tok/s**
+depending on content (MTP acceptance varies with the text being generated; 146.4 was our
+original bench, independent re-measurement across mixed prompts spans the range), 34k-token
+prompt prefill in 16.3 s.
+
+## Related work
+
+The 192 GB long-context field is moving fast; credit where due, and plain statements about
+what is and is not first here. (Corrections welcome — open a discussion.)
+
+- **[brandonmusic/GLM-5.3-Flash-tr3-4bpw](https://huggingface.co/brandonmusic/GLM-5.3-Flash-tr3-4bpw)**
+  (2026-08-27) — EXL3-K4 experts on the same 2× RTX PRO 6000, TP2, **zero offload, before
+  us**, with MTP3 active and strong short-ctx decode (145–151 tok/s at 0–64k). Its
+  `long500k` profile reaches a 499,968-token ceiling with measured needle retrievals from
+  ~498k-token prompts (17/18 through 499K) — using **4-bit NVFP4 KV** in that mode, which
+  its own card reports failing its quality gate (LAVD 3/10); its 8-bit FP8-KV mode is
+  capped at 262,144. Respect: they were first to zero-offload long context on this
+  hardware. The delta this repo adds is 524,288 **with FP8 KV** and a passing published
+  quality gate. (Their repo is `license: other` — cited, nothing reused.)
+- **[local-inference-lab/GLM-5.3-Flash-NVFP4-4p67](https://huggingface.co/local-inference-lab/GLM-5.3-Flash-NVFP4-4p67)**
+  (2026-08-27) — calibrated NVFP4 experts (better-calibrated input scales than our parent's
+  revision) + **MXFP8 attention & shared experts**, KLD ~0.04 published, explicitly targeted
+  at 2× RTX 6000. Their public vLLM fork has had a TP2 offload-free launcher since
+  2026-08-29 (`8590bf9c`), including the same MLA quant-config passthrough we ship (see
+  Serving §1 credit). MXFP8-on-SM120 is code-complete and kernel-proven in their stack; no
+  published end-to-end TP2 context/throughput/eval measurements yet ("Evals to follow").
+- **[local-inference-lab/vllm PR #522](https://github.com/local-inference-lab/vllm/pull/522)**
+  (2026-08-30) — **4× RTX PRO 6000 (384 GB, not 192 GB)**: 1,048,576-token qualified,
+  benchmark rows at 512k (66.0/122.7 tok/s), ~200k-token retrieval passed. Bigger iron,
+  bigger numbers, already public.
+- **[3× DGX Spark, TP=3](https://forums.developer.nvidia.com/t/glm-5-3-flash-nvfp4-on-3x-dgx-spark-tp-3-512k-context-35-tok-s/381534)** —
+  512K context at 35 tok/s on 384 GB unified memory (SM121; not 192 GB, different hardware
+  class).
+- **[Infatoshi/GLM-5.3-Flash-NVFP4-FP8](https://huggingface.co/Infatoshi/GLM-5.3-Flash-NVFP4-FP8)**
+  (2026-08-29) — NVFP4+FP8 variant calibrated on 4× B200; vLLM explicitly not validated by
+  its card; 1k-context diagnostic only.
+
+**What this repo claims, precisely:** the first published 192 GB (2×96 GB) configuration
+reaching 524,288 max-model-len with 8-bit (FP8) KV cache *and* a published quality gate,
+with measured ≥503k-token prompts served and the exact recipe + raw evidence included.
+Nothing more. If you know of prior art that contests this, open a discussion and this
+section will be corrected.
+
+## Repo layout
+
+```
+├─ model-000{01..120}-of-00120.safetensors   # 173.9 GiB total: NVFP4 + block-FP8 + BF16
+├─ model.safetensors.index.json
+├─ model-input-scales.safetensors            # recalibrated scales from parent @caca4e6a — see provenance note
+├─ config.json                               # MIXED_PRECISION manifest (38k+ per-layer entries)
+├─ tokenizer.json / tokenizer_config.json / chat_template.jinja / generation_config.json
+├─ serving/                                  # the matched-pair runtime (Apache-2.0, vLLM-derived)
+│  ├─ Dockerfile  ├─ kda.py  ├─ model.py  ├─ modelopt.py  └─ configs/*.json
+└─ scripts/                                  # provenance + evidence (documents our exact env)
+   ├─ convert_fp8attn.py                     # parent → attention/MLP FP8 (CPU-only, deterministic)
+   ├─ convert_lmhead.py                      # + lm_head FP8, in place (reversible, --revert)
+   ├─ accept-512k.sh / longctx.py            # the 512k acceptance harness (3× ~503k prompts)
+   ├─ boot-512k-fit.sh / validate.py / bench_decode.py   # harness deps (boot, quality dump, decode bench)
+   ├─ compare.py                             # teacher-forced / greedy A/B scorer
+   └─ results-512k.json / results-base.json / results-fp8attn-r2.json   # raw dumps behind every quality number
+```
+
+## Provenance, license, attribution
+
+- Weights: **MIT**, © 2026 Z.AI Co., Ltd (LICENSE carried verbatim). Chain:
+  [zai-org/GLM-5.3-Flash](https://huggingface.co/zai-org/GLM-5.3-Flash) →
+  [LibertAIDAI/GLM-5.3-Flash-NVFP4](https://huggingface.co/LibertAIDAI/GLM-5.3-Flash-NVFP4)
+  (revision `caca4e6a`, the recalibrated-input-scales head — NVFP4 expert quantization and
+  input scales are LibertAI's work) →
+  this repo (block-FP8 attention/MLP/lm_head conversion). Not affiliated with Z.ai / Zhipu
+  or LibertAI.
+- `serving/*.py`: Apache-2.0, derived from the vLLM project (SPDX headers retained), via the
+  official per-model image as packaged by
+  [chriswritescode-dev/glm-5.3-flash-sm120](https://github.com/chriswritescode-dev/glm-5.3-flash-sm120);
+  parallel-work credit to local-inference-lab per Serving §1.
+- Conversion was CPU-only, deterministic, calibration-free (block-wise amax/448 scaling);
+  `scripts/convert_fp8attn.py` **+ `scripts/convert_lmhead.py`** reproduce this checkpoint
+  bit-for-bit from the parent (determinism experimentally verified during audit).
+
+## Known limitations
+
+- **Matched pair:** loads only under the `serving/` patched image. Stock vLLM has no
+  `glm5_next`; vendor images assert/KeyError on this checkpoint.
+- **Input-scales history:** the parent's scales were revised twice on release week (see the
+  provenance note up top); this repo ships the recalibrated `caca4e6a` scales, gated directly.
+- **Text-only as served:** both measured operating points skip the vision tower. The tower
+  weights are present and byte-identical to the parent, and the patched image leaves the
+  tower unquantized, but multimodal serving of this checkpoint is **untested**
+  (inferred-safe, not measured).
+- Single-stream orientation at 512k (`--max-num-seqs 1`); throughput under concurrency was
+  not the target and is unmeasured beyond a 4-way probe at smaller contexts.
+- ~290 MiB VRAM margin during a full 512k prefill — no other GPU tenants.
+- SM120 (RTX PRO 6000 Blackwell) is the only measured platform. The quant format itself is
+  hardware-agnostic block-FP8/NVFP4, but the kernel selection and the tuned config are
+  SM120.
+- Scripts in `scripts/` document our exact environment; paths and flags may need adaptation.

scripts/accept-512k.sh

@@ -0,0 +1,68 @@
+#!/bin/bash
+# Full acceptance under one gpu-lock window. Boots the candidate config,
+# runs: 391 check, validate.py (quality dump), compare vs r2 + base,
+# short-ctx decode bench, 3x >=500k-token rounds (longctx.py), teardown.
+# Usage: accept-512k.sh [MAXLEN] [IMAGE] [EXTRA vllm args...]   (env: SEQS BATCHED UTIL MM)
+set -u
+D="$(cd "$(dirname "$0")" && pwd)"
+LOG=/tmp/glm-512k-accept.log
+MODEL=glm-5.3-flash-fp8attn
+PORT=18996
+
+docker rm -f glm-512k-fit >/dev/null 2>&1
+# teardown race guard: another worker's dying container may still hold VRAM
+# for a few seconds after the lock frees; wait for both cards to drain.
+for i in $(seq 1 60); do
+  MAXUSED=$(nvidia-smi --query-gpu=memory.used --format=csv,noheader,nounits | sort -n | tail -1)
+  [ "$MAXUSED" -lt 200 ] && break
+  sleep 3
+done
+echo "GPU drain: max used ${MAXUSED} MiB before boot"
+setsid "$D/boot-512k-fit.sh" "$@" >"$LOG" 2>&1 &
+BOOT=$!
+READY=0
+for i in $(seq 1 240); do
+  sleep 5
+  if curl -sf -m 2 http://127.0.0.1:$PORT/health >/dev/null 2>&1; then READY=1; break; fi
+  kill -0 "$BOOT" 2>/dev/null || break
+done
+echo "=== LEDGER ==="
+grep -aiE "consumed|non-torch|activation|KV cache size|kv cache memory|Maximum concurrency|graph|needs|available|ERROR|OOM" "$LOG" | tail -25
+if [ "$READY" != "1" ]; then echo "BOOT FAILED"; docker stop glm-512k-fit >/dev/null 2>&1; exit 2; fi
+
+# VRAM watermark sampler (min free MiB seen during the run)
+( MINFREE=999999; while docker ps --format '{{.Names}}' | grep -q glm-512k-fit; do
+    F=$(nvidia-smi --query-gpu=memory.free --format=csv,noheader,nounits | sort -n | head -1)
+    [ -n "$F" ] && [ "$F" -lt "$MINFREE" ] && MINFREE=$F && echo "$MINFREE" > /tmp/glm-512k-minfree
+    sleep 4
+  done ) &
+WATCH=$!
+
+echo "=== 391 ==="
+curl -s -m 300 http://127.0.0.1:$PORT/v1/chat/completions -H 'Content-Type: application/json' \
+  -d "{\"model\":\"$MODEL\",\"messages\":[{\"role\":\"user\",\"content\":\"What is 17*23? Reply with just the number.\"}],\"max_tokens\":200,\"temperature\":0}" \
+  | python3 -c 'import json,sys; r=json.load(sys.stdin); m=r["choices"][0]["message"]; print((m.get("content") or "")[:100], "| reasoning:", (m.get("reasoning_content") or "")[-80:])'
+
+echo "=== validate.py ==="
+python3 "$D/validate.py" $PORT $MODEL "$D/results-512k.json" || echo "validate FAILED"
+
+echo "=== compare vs r2 ==="
+python3 "$D/compare.py" "$D/results-fp8attn-r2.json" "$D/results-512k.json" || true
+echo "=== compare vs base ==="
+python3 "$D/compare.py" "$D/results-base.json" "$D/results-512k.json" || true
+
+echo "=== decode short-ctx ==="
+python3 "$D/bench_decode.py" $PORT $MODEL 512 || true
+
+echo "=== longctx 3 rounds ==="
+python3 "$D/longctx.py" $PORT $MODEL 3 505000
+LC=$?
+
+echo "=== post-run VRAM ==="
+nvidia-smi --query-gpu=index,memory.used --format=csv,noheader
+kill "$WATCH" 2>/dev/null
+echo "=== min free watermark (MiB): $(cat /tmp/glm-512k-minfree 2>/dev/null || echo n/a) ==="
+
+if [ "${KEEP:-0}" != "1" ]; then docker stop glm-512k-fit >/dev/null 2>&1; wait "$BOOT" 2>/dev/null; fi
+echo "=== ACCEPT EXIT: longctx=$LC ==="
+exit $LC

scripts/bench_decode.py

@@ -0,0 +1,37 @@
+"""Single-stream decode throughput. Usage: bench_decode.py <port> <model> [n_tokens]"""
+import json
+import sys
+import time
+import urllib.request
+
+PORT, MODEL = int(sys.argv[1]), sys.argv[2]
+N = int(sys.argv[3]) if len(sys.argv) > 3 else 512
+
+payload = {
+    "model": MODEL,
+    "messages": [
+        {
+            "role": "user",
+            "content": "Count from 1 to 300 in words, one number per line (one, two, three, ...).",
+        }
+    ],
+    "temperature": 0,
+    "max_tokens": N,
+    "ignore_eos": True,
+}
+req = urllib.request.Request(
+    f"http://127.0.0.1:{PORT}/v1/chat/completions",
+    data=json.dumps(payload).encode(),
+    headers={"Content-Type": "application/json"},
+)
+# warmup (also primes prefix cache off: different prompt)
+warm = dict(payload, max_tokens=32)
+urllib.request.urlopen(
+    urllib.request.Request(req.full_url, data=json.dumps(warm).encode(), headers=req.headers),
+    timeout=600,
+).read()
+t0 = time.time()
+r = json.loads(urllib.request.urlopen(req, timeout=600).read())
+dt = time.time() - t0
+ct = r["usage"]["completion_tokens"]
+print(f"{ct} completion tokens in {dt:.2f}s = {ct / dt:.1f} tok/s (single stream, MTP on)")

scripts/boot-512k-fit.sh

@@ -0,0 +1,52 @@
+#!/bin/bash
+# Offload-free 512k lane boot. Container glm-512k-fit, port 18996.
+# Usage: boot-512k-fit.sh [MAXLEN] [IMAGE] [EXTRA vllm args...]
+# Env knobs: MODEL_DIR (checkpoint path), CACHE_DIR (vllm cache),
+#            SEQS (2), BATCHED (2048), UTIL (0.98), MM ("0" -> tower skipped)
+MODEL_DIR="${MODEL_DIR:-$PWD/GLM-5.3-Flash-NVFP4-FP8ATTN-512K}"
+CACHE_DIR="${CACHE_DIR:-$HOME/.cache/vllm-glm53}"
+set -e
+MAXLEN="${1:-524288}"
+IMAGE="${2:-local/vllm-glm53:fp8attn-r4}"
+shift 2 2>/dev/null || shift $# 2>/dev/null || true
+MM_ARGS=()
+if [ "${MM:-0}" = "0" ]; then
+  MM_ARGS=(--limit-mm-per-prompt '{"image":0,"video":0}')
+fi
+SPEC_ARGS=(--speculative-config '{"method":"mtp","num_speculative_tokens":1}')
+if [ "${SPEC:-1}" = "0" ]; then
+  SPEC_ARGS=()
+fi
+exec docker run --name glm-512k-fit --init --rm \
+  --gpus all \
+  --runtime nvidia \
+  --ipc=host \
+  --network host \
+  --shm-size=32g \
+  --ulimit memlock=-1 \
+  --ulimit stack=67108864 \
+  -e VLLM_ENGINE_READY_TIMEOUT_S=3600 \
+  -e HF_HUB_OFFLINE=1 \
+  -e VLLM_MEMORY_PROFILER_ESTIMATE_CUDAGRAPHS="${ESTG:-1}" \
+  -v "$MODEL_DIR":/model:ro \
+  -v "$CACHE_DIR":/root/.cache \
+  "$IMAGE" \
+  /model \
+  --served-model-name glm-5.3-flash-fp8attn \
+  --host 0.0.0.0 \
+  --port 18996 \
+  --tensor-parallel-size 2 \
+  --max-model-len "$MAXLEN" \
+  --gpu-memory-utilization "${UTIL:-0.98}" \
+  --kv-cache-dtype fp8 \
+  --max-num-seqs "${SEQS:-2}" \
+  --max-num-batched-tokens "${BATCHED:-2048}" \
+  --enable-prefix-caching \
+  --no-enable-flashinfer-autotune \
+  --enable-auto-tool-choice \
+  --tool-call-parser glm47 \
+  --reasoning-parser glm45 \
+  --trust-remote-code \
+  "${SPEC_ARGS[@]}" \
+  "${MM_ARGS[@]}" \
+  "$@"

scripts/compare.py

@@ -0,0 +1,79 @@
+"""Compare two validate.py result dumps (baseline vs converted).
+
+Usage: python3 compare.py baseline.json converted.json
+"""
+
+import json
+import sys
+
+a = json.load(open(sys.argv[1]))
+b = json.load(open(sys.argv[2]))
+
+print(f"baseline={a['model']}  converted={b['model']}")
+print(f"arith 17*23: base={a['arith_correct']} conv={b['arith_correct']}")
+print(
+    f"long gen: base {a['long_gen_tokens']} tok degenerate={a['long_gen_degenerate']} | "
+    f"conv {b['long_gen_tokens']} tok degenerate={b['long_gen_degenerate']}"
+)
+
+# greedy 50-token comparison: positionwise top-1 agreement until first
+# divergence (after divergence contexts differ), plus |dlogprob| on agreeing
+# positions.
+tot_pos = agree = 0
+dl = []
+full_match = 0
+for ga, gb in zip(a["greedy"], b["greedy"]):
+    ta, tb = ga["tokens"], gb["tokens"]
+    la, lb = ga["token_logprobs"], gb["token_logprobs"]
+    n = min(len(ta), len(tb))
+    diverged = False
+    prompt_match = 0
+    for i in range(n):
+        if diverged:
+            break
+        tot_pos += 1
+        if ta[i] == tb[i]:
+            agree += 1
+            prompt_match += 1
+            if la[i] is not None and lb[i] is not None:
+                dl.append(abs(la[i] - lb[i]))
+        else:
+            diverged = True
+    if not diverged and len(ta) == len(tb):
+        full_match += 1
+print(
+    f"greedy: {full_match}/{len(a['greedy'])} prompts identical for all 50 tok; "
+    f"agreement until divergence {agree}/{tot_pos} = {100 * agree / max(tot_pos, 1):.2f}%"
+)
+if dl:
+    print(
+        f"greedy |dlogprob| on agreeing tokens: mean {sum(dl) / len(dl):.5f} "
+        f"max {max(dl):.5f} (n={len(dl)})"
+    )
+
+# teacher-forced prompt logprobs: identical context by construction.
+tf_dl = []
+tf_pos = tf_agree = 0
+for sa, sb in zip(a["scored"], b["scored"]):
+    for pa, pb in zip(sa["prompt_logprobs"], sb["prompt_logprobs"]):
+        if pa is None or pb is None:
+            continue
+        # chosen prompt token = the key with rank field that matches; both dicts
+        # contain the actual token (rank r) and the top-1. Find common token id
+        # present in both (the prompt token id is the same on both sides).
+        common = set(pa) & set(pb)
+        if not common:
+            continue
+        # the prompt token appears in both dicts (it is always included)
+        for tid in common:
+            ra, rb = pa[tid].get("rank"), pb[tid].get("rank")
+            tf_dl.append(abs(pa[tid]["logprob"] - pb[tid]["logprob"]))
+            tf_pos += 1
+            if (ra == 1) == (rb == 1):
+                tf_agree += 1
+            break
+print(
+    f"teacher-forced: n={tf_pos} mean |dlogprob| {sum(tf_dl) / max(len(tf_dl), 1):.5f} "
+    f"max {max(tf_dl) if tf_dl else 0:.5f}; top-1-status agreement "
+    f"{100 * tf_agree / max(tf_pos, 1):.2f}%"
+)

scripts/convert_fp8attn.py

@@ -0,0 +1,345 @@
+"""Derive GLM-5.3-Flash-NVFP4-FP8ATTN (r2: MLP + ATTENTION FP8) from
+GLM-5.3-Flash-NVFP4 (rev caca4e6a; expert weight shards are byte-identical to
+357b45cc — only the input scales and card differ between those revisions).
+
+Targets the patched serving image that passes the real quant_config into the
+KDA/MLA attention constructors (kda.py strip removed, model.py:329
+None -> real). See the model card and serving/README.md.
+
+Run convert_lmhead.py afterward to reproduce the published checkpoint
+(block-FP8 lm_head) bit-for-bit.
+
+Converted surfaces and block sizes (weight F8_E4M3 + weight_scale_inv F32,
+dequant multiplier amax/448):
+
+  FP8_BLOCK128 ([128,128], the proven MLP path):
+    - *.mlp.shared_experts.{gate,up,down}_proj   layers 3..45 (incl. MTP 45)
+    - *.mlp.{gate,up,down}_proj                  dense layers 0..2
+    - *.self_attn.o_proj                         ALL 46 layers (KDA + MLA)
+    - *.self_attn.{q_a_proj,kv_a_proj_with_mqa,q_b_proj}   MLA layers
+      (q_a + kv_a fuse into fused_qkv_a_proj at runtime: offsets 0/1536 and
+       sizes 1536/512 all divide 128 because qk_rope_head_dim=0)
+
+  FP8_BLOCK32 ([32,32], new dispatch in the r2 image's modelopt.py):
+    - *.self_attn.{q,k,v,b,f_a,g_a}_proj         KDA layers
+      (they fuse into in_proj_qkvbfg_a whose per-rank TP2 layout is
+       q|k|v 4096 each | b 32 | f_a 128 | g_a 128 — block_n must divide 32;
+       square because Fp8LinearMethod's activation group = block_n)
+
+  Kept BF16 (deliberate):
+    - kv_b_proj      — MLA absorption stores BF16 W_UK/W_UV copies regardless
+                       (mla_attention.py:1022); net saving ~0.09 GiB/rank and
+                       _get_kv_b_proj_input_dtype has an untested fp8 branch.
+    - f_b_proj/g_b_proj — delta-rule gate projections, not in the tasking.
+    - indexer.*, convs, norms, embed_tokens, lm_head, visual.* — as before.
+
+Untouched shards are HARDLINKED to the source. Manifest carries both the
+main-model spelling (model.language_model.layers.N.*) and the MTP draft
+runtime spelling (model.layers.45.*), for MLP AND attention.
+"""
+
+import json
+import os
+import re
+import struct
+import time
+
+import ml_dtypes
+import numpy as np
+
+SRC = None  # set from --input in __main__
+DST = None  # set from --output in __main__
+FIRST_K_DENSE = 3
+NUM_LAYERS = 46  # 45 decoder + MTP layer 45
+NUM_EXPERTS = 288
+
+MLP_RE = re.compile(
+    r"^model\.language_model\.layers\.(\d+)\.mlp\."
+    r"(shared_experts\.)?(gate|up|down)_proj\.weight$"
+)
+ATTN128_RE = re.compile(
+    r"^model\.language_model\.layers\.(\d+)\.self_attn\."
+    r"(o_proj|q_a_proj|kv_a_proj_with_mqa|q_b_proj)\.weight$"
+)
+ATTN32_RE = re.compile(
+    r"^model\.language_model\.layers\.(\d+)\.self_attn\."
+    r"(q|k|v|b|f_a|g_a)_proj\.weight$"
+)
+
+# ignore-list patterns whose modules are now FP8-quantized
+UNIGNORE = {
+    # MLP family (as in convert_fp8mlp.py)
+    "*.mlp.shared_experts.gate_up_proj",
+    "*.mlp.shared_experts.down_proj",
+    "*.mlp.shared_experts.gate_proj",
+    "*.mlp.shared_experts.up_proj",
+    "*.mlp.gate_up_proj",
+    "*.mlp.down_proj",
+    "*.mlp.gate_proj",
+    "*.mlp.up_proj",
+    # attention: KDA fused in_proj shards + fused spellings
+    "*.self_attn.q_proj",
+    "*.self_attn.k_proj",
+    "*.self_attn.v_proj",
+    "*.self_attn.b_proj",
+    "*.self_attn.f_a_proj",
+    "*.self_attn.g_a_proj",
+    "*.self_attn.in_proj_qkvbfg_a",
+    "*.self_attn.fused_qkvbfg_a_proj",
+    # attention: o_proj (KDA + MLA) and the MLA q/lora side
+    "*.self_attn.o_proj",
+    "*.self_attn.q_a_proj",
+    "*.self_attn.kv_a_proj_with_mqa",
+    "*.self_attn.fused_qkv_a_proj",
+    "*.self_attn.fused_qkv_a_proj_with_mqa",
+    "*.self_attn.q_b_proj",
+}
+# NOTE deliberately NOT unignored: *.self_attn.kv_b_proj, *.self_attn.f_b_proj,
+# *.self_attn.g_b_proj, *.self_attn.fused_fg_b_proj, indexer/conv/norm entries.
+
+
+def read_header(path):
+    with open(path, "rb") as f:
+        n = struct.unpack("<Q", f.read(8))[0]
+        return json.loads(f.read(n)), 8 + n
+
+
+def block_for(name):
+    """Return the quant block size for a target tensor name, else None."""
+    m = MLP_RE.match(name)
+    if m:
+        layer, shared = int(m.group(1)), bool(m.group(2))
+        return 128 if (shared or layer < FIRST_K_DENSE) else None
+    if ATTN128_RE.match(name):
+        return 128
+    if ATTN32_RE.match(name):
+        return 32
+    return None
+
+
+def quant_block(w32, b):
+    n, k = w32.shape
+    assert n % b == 0 and k % b == 0, (n, k, b)
+    B = w32.reshape(n // b, b, k // b, b)
+    amax = np.abs(B).max(axis=(1, 3), keepdims=True)
+    s = np.where(amax == 0, np.float32(1), amax / np.float32(448)).astype(np.float32)
+    q = np.clip(B / s, -448, 448).astype(ml_dtypes.float8_e4m3fn)
+    return (
+        np.ascontiguousarray(q.reshape(n, k)),
+        np.ascontiguousarray(s.reshape(B.shape[0], B.shape[2])),
+    )
+
+
+def build_quantized_layers(mla_layers):
+    ql = {}
+    nvfp4 = {"quant_algo": "NVFP4", "group_size": 16}
+    fp8_128 = {"quant_algo": "FP8_BLOCK128"}
+    fp8_32 = {"quant_algo": "FP8_BLOCK32"}
+
+    def attn_entries(base, layer):
+        # base ends in ".self_attn"
+        ql[f"{base}.o_proj"] = fp8_128
+        if layer in mla_layers:
+            for p in ("q_a_proj", "kv_a_proj_with_mqa", "q_b_proj"):
+                ql[f"{base}.{p}"] = fp8_128
+        else:
+            for p in ("q_proj", "k_proj", "v_proj", "b_proj", "f_a_proj", "g_a_proj"):
+                ql[f"{base}.{p}"] = fp8_32
+
+    for layer in range(NUM_LAYERS):
+        base = f"model.language_model.layers.{layer}"
+        attn_entries(f"{base}.self_attn", layer)
+        if layer >= FIRST_K_DENSE:
+            for e in range(NUM_EXPERTS):
+                for p in ("gate", "up", "down"):
+                    ql[f"{base}.mlp.experts.{e}.{p}_proj"] = nvfp4
+            for p in ("gate", "up", "down"):
+                ql[f"{base}.mlp.shared_experts.{p}_proj"] = fp8_128
+        else:
+            for p in ("gate", "up", "down"):
+                ql[f"{base}.mlp.{p}_proj"] = fp8_128
+
+    # MTP draft-model runtime spelling: mtp.py builds the layer-45 decoder
+    # block with prefix "model.layers.45" (no ".mtp_block" in the quant-lookup
+    # prefix), so the manifest must carry these keys verbatim. Mirrors the MLP
+    # aliases that fixed the FP8MIX-v2 KeyError; extended here to attention.
+    mtp = "model.layers.45"
+    attn_entries(f"{mtp}.self_attn", 45)
+    for e in range(NUM_EXPERTS):
+        for p in ("gate", "up", "down"):
+            ql[f"{mtp}.mlp.experts.{e}.{p}_proj"] = nvfp4
+    ql[f"{mtp}.mlp.experts.routed_experts.0.up_proj"] = nvfp4
+    for p in ("gate", "up", "down"):
+        ql[f"{mtp}.mlp.shared_experts.{p}_proj"] = fp8_128
+    return ql
+
+
+def main():
+    os.makedirs(DST, exist_ok=True)
+    idx = json.load(open(f"{SRC}/model.safetensors.index.json"))
+    wmap = dict(idx["weight_map"])
+    shards = sorted(set(wmap.values()))
+
+    mla_layers = sorted(
+        {
+            int(re.match(r"model\.language_model\.layers\.(\d+)\.", n).group(1))
+            for n in wmap
+            if ".self_attn.q_a_proj." in n
+        }
+    )
+    assert mla_layers == [3, 7, 11, 15, 19, 23, 27, 31, 35, 39, 43, 45], mla_layers
+
+    todo = {}
+    n32 = n128 = 0
+    for name, sh in wmap.items():
+        b = block_for(name)
+        if b is not None:
+            todo.setdefault(sh, {})[name] = b
+            if b == 32:
+                n32 += 1
+            else:
+                n128 += 1
+    n_targets = n32 + n128
+    print(f"{n_targets} targets ({n32} block32 + {n128} block128) in {len(todo)}/{len(shards)} shards")
+    assert n32 == 204, n32          # 34 KDA layers x 6
+    assert n128 == 138 + 82, n128   # MLP 138 + attn (46 o + 12x3 MLA)
+
+    t0 = time.time()
+    linked = rewritten = 0
+    size_delta = 0
+    for si, sh in enumerate(shards, 1):
+        src, dst = f"{SRC}/{sh}", f"{DST}/{sh}"
+        if os.path.exists(dst):
+            os.unlink(dst)
+        if sh not in todo:
+            os.link(src, dst)
+            linked += 1
+            continue
+        hdr, base = read_header(src)
+        meta = hdr.get("__metadata__")
+        conv = todo[sh]
+        order = [k for k in hdr if k != "__metadata__"]
+        new_hdr, blobs, off = {}, [], 0
+        with open(src, "rb") as f:
+            for k in order:
+                m = hdr[k]
+                s, e = m["data_offsets"]
+                f.seek(base + s)
+                raw = f.read(e - s)
+                if k in conv:
+                    assert m["dtype"] == "BF16", (k, m["dtype"])
+                    w = (
+                        np.frombuffer(raw, dtype=ml_dtypes.bfloat16)
+                        .reshape(m["shape"])
+                        .astype(np.float32)
+                    )
+                    qw, sc = quant_block(w, conv[k])
+                    for nm, arr, dt in (
+                        (k, qw, "F8_E4M3"),
+                        (k[: -len(".weight")] + ".weight_scale_inv", sc, "F32"),
+                    ):
+                        b = arr.tobytes()
+                        new_hdr[nm] = {
+                            "dtype": dt,
+                            "shape": list(arr.shape),
+                            "data_offsets": [off, off + len(b)],
+                        }
+                        blobs.append(b)
+                        off += len(b)
+                        size_delta += len(b)
+                    size_delta -= len(raw)
+                else:
+                    new_hdr[k] = {
+                        "dtype": m["dtype"],
+                        "shape": m["shape"],
+                        "data_offsets": [off, off + len(raw)],
+                    }
+                    blobs.append(raw)
+                    off += len(raw)
+        if meta is not None:
+            new_hdr["__metadata__"] = meta
+        hb = json.dumps(new_hdr, separators=(",", ":")).encode()
+        pad = (-(8 + len(hb))) % 8
+        hb += b" " * pad
+        tmp = dst + ".tmp"
+        with open(tmp, "wb") as f:
+            f.write(struct.pack("<Q", len(hb)))
+            f.write(hb)
+            for b in blobs:
+                f.write(b)
+        os.replace(tmp, dst)
+        rewritten += 1
+        print(
+            f"[{si}/{len(shards)}] {sh} rewrote {len(conv)} tensors "
+            f"({time.time() - t0:.0f}s)",
+            flush=True,
+        )
+
+    for sh, names in todo.items():
+        for name in names:
+            wmap[name[: -len(".weight")] + ".weight_scale_inv"] = sh
+    idx["weight_map"] = wmap
+    if "metadata" in idx and "total_size" in idx["metadata"]:
+        idx["metadata"]["total_size"] += size_delta
+    json.dump(idx, open(f"{DST}/model.safetensors.index.json", "w"))
+
+    cfg = json.load(open(f"{SRC}/config.json"))
+    src_q = cfg["quantization_config"]
+    ignore = [p for p in src_q["ignore"] if p not in UNIGNORE]
+    cfg["quantization_config"] = {
+        "producer": src_q.get("producer", {"name": "modelopt", "version": "0.45.0"}),
+        "quant_method": "modelopt",
+        "quant_algo": "MIXED_PRECISION",
+        "kv_cache_scheme": src_q.get("kv_cache_scheme"),
+        "group_size": 16,
+        "ignore": ignore,
+        "quantized_layers": build_quantized_layers(set(mla_layers)),
+    }
+    json.dump(cfg, open(f"{DST}/config.json", "w"), indent=1)
+
+    for fn in os.listdir(SRC):
+        if fn.endswith(".safetensors") or fn in (
+            "config.json",
+            "model.safetensors.index.json",
+        ):
+            continue
+        s, d = f"{SRC}/{fn}", f"{DST}/{fn}"
+        if os.path.isfile(s) and not os.path.exists(d):
+            os.link(s, d)
+    print(
+        f"DONE linked={linked} rewritten={rewritten} "
+        f"size_delta={size_delta / 2**30:.3f} GiB in {time.time() - t0:.0f}s"
+    )
+
+
+if __name__ == "__main__":
+    import argparse
+
+    ap = argparse.ArgumentParser(
+        description="Derive the NVFP4 + block-FP8 (MLP + attention) checkpoint from a "
+        "pristine GLM-5.3-Flash-NVFP4 checkpoint (rev caca4e6a). Unchanged shards are "
+        "hardlinked (same filesystem required). Run convert_lmhead.py afterward to "
+        "reproduce the published checkpoint exactly."
+    )
+    ap.add_argument(
+        "--input",
+        required=True,
+        help="pristine GLM-5.3-Flash-NVFP4 checkpoint dir (only read)",
+    )
+    ap.add_argument(
+        "--output",
+        required=True,
+        help="output checkpoint dir (created if missing; EXISTING FILES IN IT ARE REPLACED)",
+    )
+    ap.add_argument("--yes", action="store_true", help="skip the confirmation prompt")
+    args = ap.parse_args()
+    SRC, DST = os.path.abspath(args.input), os.path.abspath(args.output)
+    if SRC == DST:
+        raise SystemExit("--input and --output must differ (conversion is not in-place)")
+    if not os.path.isfile(os.path.join(SRC, "model.safetensors.index.json")):
+        raise SystemExit(f"{SRC} has no model.safetensors.index.json — not a checkpoint dir")
+    if not args.yes:
+        reply = input(f"(Re)write shards in {DST} from {SRC}? [y/N] ")
+        if reply.strip().lower() not in ("y", "yes"):
+            raise SystemExit("aborted")
+    main()

scripts/convert_lmhead.py

@@ -0,0 +1,197 @@
+"""Post-step on GLM-5.3-Flash-NVFP4-FP8ATTN: quantize lm_head to block-FP8.
+
+Composable AFTER convert_fp8attn.py (r2 content). Targets the fp8attn-r4
+image, whose modelopt.py dispatches ParallelLMHead + FP8_BLOCK128 to
+_Fp8BlockLMHeadMethod (vocab-block-sharded scale loader).
+
+  lm_head.weight  BF16 [154880, 4096]  ->  F8_E4M3 + weight_scale_inv F32
+  [1210, 32], block [128,128], dequant multiplier amax/448 — identical
+  scheme/derivation to convert_fp8attn.py's quant_block.
+
+Geometry: 154880 % 128 == 0 and the TP2 vocab shard 77440 % 128 == 0, so
+weight and scale shard exactly on block boundaries.
+
+config.json: remove "lm_head" from quantization_config.ignore (its substring
+match would keep the head excluded) and add
+quantized_layers["lm_head"] = {"quant_algo": "FP8_BLOCK128"}.
+
+Reversible: the touched shard, config.json and the index are snapshotted as
+*.pre-lmhead first (shard via hardlink — zero cost). `--revert` puts all
+three back and removes the snapshots. embed_tokens stays BF16 (gather
+semantics: VocabParallelEmbedding never receives a quant method).
+
+Source NVFP4 dir untouched (all writes are new-file + os.replace).
+"""
+
+import json
+import os
+import struct
+import sys
+import time
+
+import ml_dtypes
+import numpy as np
+
+DST = None  # set from --dir in __main__
+TENSOR = "lm_head.weight"
+BLOCK = 128
+SUFFIX = ".pre-lmhead"
+
+
+def read_header(path):
+    with open(path, "rb") as f:
+        n = struct.unpack("<Q", f.read(8))[0]
+        return json.loads(f.read(n)), 8 + n
+
+
+def quant_block(w32, b):
+    n, k = w32.shape
+    assert n % b == 0 and k % b == 0, (n, k, b)
+    B = w32.reshape(n // b, b, k // b, b)
+    amax = np.abs(B).max(axis=(1, 3), keepdims=True)
+    s = np.where(amax == 0, np.float32(1), amax / np.float32(448)).astype(np.float32)
+    q = np.clip(B / s, -448, 448).astype(ml_dtypes.float8_e4m3fn)
+    return (
+        np.ascontiguousarray(q.reshape(n, k)),
+        np.ascontiguousarray(s.reshape(B.shape[0], B.shape[2])),
+    )
+
+
+def revert():
+    idx_path = f"{DST}/model.safetensors.index.json"
+    cfg_path = f"{DST}/config.json"
+    idx_bak, cfg_bak = idx_path + SUFFIX, cfg_path + SUFFIX
+    if not os.path.exists(idx_bak):
+        sys.exit("no .pre-lmhead snapshots found; nothing to revert")
+    shard = json.load(open(idx_bak))["weight_map"][TENSOR]
+    shard_path = f"{DST}/{shard}"
+    for bak, live in ((shard_path + SUFFIX, shard_path), (idx_bak, idx_path), (cfg_bak, cfg_path)):
+        os.replace(bak, live)
+    print(f"reverted: {shard}, index, config restored")
+
+
+def main():
+    idx_path = f"{DST}/model.safetensors.index.json"
+    cfg_path = f"{DST}/config.json"
+    idx = json.load(open(idx_path))
+    if TENSOR + "_scale_inv" in idx["weight_map"] or (
+        TENSOR[: -len(".weight")] + ".weight_scale_inv"
+    ) in idx["weight_map"]:
+        sys.exit("lm_head already converted (scale in index); use --revert first")
+    shard = idx["weight_map"][TENSOR]
+    src = f"{DST}/{shard}"
+
+    # snapshots (shard as hardlink: pre-conversion inode preserved for free)
+    os.link(src, src + SUFFIX)
+    for p in (idx_path, cfg_path):
+        with open(p) as f, open(p + SUFFIX, "w") as g:
+            g.write(f.read())
+
+    t0 = time.time()
+    hdr, base = read_header(src)
+    meta = hdr.get("__metadata__")
+    order = [k for k in hdr if k != "__metadata__"]
+    new_hdr, blobs, off, size_delta = {}, [], 0, 0
+    with open(src, "rb") as f:
+        for k in order:
+            m = hdr[k]
+            s, e = m["data_offsets"]
+            f.seek(base + s)
+            raw = f.read(e - s)
+            if k == TENSOR:
+                assert m["dtype"] == "BF16", m["dtype"]
+                w = (
+                    np.frombuffer(raw, dtype=ml_dtypes.bfloat16)
+                    .reshape(m["shape"])
+                    .astype(np.float32)
+                )
+                qw, sc = quant_block(w, BLOCK)
+                # round-trip check
+                deq = qw.astype(np.float32) * np.repeat(
+                    np.repeat(sc, BLOCK, 0), BLOCK, 1
+                )
+                err = np.linalg.norm(deq - w) / np.linalg.norm(w)
+                print(f"lm_head shape {m['shape']} round-trip rel err {err:.4%}")
+                for nm, arr, dt in (
+                    (k, qw, "F8_E4M3"),
+                    (k[: -len(".weight")] + ".weight_scale_inv", sc, "F32"),
+                ):
+                    b = arr.tobytes()
+                    new_hdr[nm] = {
+                        "dtype": dt,
+                        "shape": list(arr.shape),
+                        "data_offsets": [off, off + len(b)],
+                    }
+                    blobs.append(b)
+                    off += len(b)
+                    size_delta += len(b)
+                size_delta -= len(raw)
+            else:
+                new_hdr[k] = {
+                    "dtype": m["dtype"],
+                    "shape": m["shape"],
+                    "data_offsets": [off, off + len(raw)],
+                }
+                blobs.append(raw)
+                off += len(raw)
+    if meta is not None:
+        new_hdr["__metadata__"] = meta
+    hb = json.dumps(new_hdr, separators=(",", ":")).encode()
+    hb += b" " * ((-(8 + len(hb))) % 8)
+    tmp = src + ".tmp"
+    with open(tmp, "wb") as f:
+        f.write(struct.pack("<Q", len(hb)))
+        f.write(hb)
+        for b in blobs:
+            f.write(b)
+    os.replace(tmp, src)
+
+    wmap = dict(idx["weight_map"])
+    wmap[TENSOR[: -len(".weight")] + ".weight_scale_inv"] = shard
+    idx["weight_map"] = wmap
+    if "metadata" in idx and "total_size" in idx["metadata"]:
+        idx["metadata"]["total_size"] += size_delta
+    json.dump(idx, open(idx_path, "w"))
+
+    cfg = json.load(open(cfg_path))
+    q = cfg["quantization_config"]
+    q["ignore"] = [p for p in q["ignore"] if p != "lm_head"]
+    q["quantized_layers"]["lm_head"] = {"quant_algo": "FP8_BLOCK128"}
+    json.dump(cfg, open(cfg_path, "w"), indent=1)
+
+    print(
+        f"DONE {shard} rewritten, size_delta={size_delta / 2**30:+.3f} GiB, "
+        f"{time.time() - t0:.0f}s. Revert: python3 convert_lmhead.py --dir {DST} --revert"
+    )
+
+
+if __name__ == "__main__":
+    import argparse
+
+    ap = argparse.ArgumentParser(
+        description="Quantize lm_head to block-FP8 IN PLACE in a converted "
+        "GLM-5.3-Flash-NVFP4-FP8ATTN checkpoint dir. The touched shard, config.json and "
+        "index are snapshotted as *.pre-lmhead first; --revert restores them."
+    )
+    ap.add_argument(
+        "--dir", required=True, help="checkpoint dir to modify IN PLACE"
+    )
+    ap.add_argument(
+        "--revert", action="store_true", help="restore the *.pre-lmhead snapshots"
+    )
+    ap.add_argument("--yes", action="store_true", help="skip the confirmation prompt")
+    args = ap.parse_args()
+    DST = os.path.abspath(args.dir)
+    if not os.path.isfile(os.path.join(DST, "model.safetensors.index.json")):
+        raise SystemExit(f"{DST} has no model.safetensors.index.json — not a checkpoint dir")
+    if args.revert:
+        revert()
+    else:
+        if not args.yes:
+            reply = input(
+                f"Rewrite the lm_head shard in {DST} IN PLACE "
+                "(with .pre-lmhead snapshots)? [y/N] "
+            )
+            if reply.strip().lower() not in ("y", "yes"):
+                raise SystemExit("aborted")
+        main()

scripts/longctx.py

@@ -0,0 +1,140 @@
+"""512k acceptance: N consecutive >=500k-token prompts through a live server.
+
+Usage: python3 longctx.py <port> <model> [n_rounds=3] [target_tokens=505000]
+
+Each round builds a DIFFERENT ~target-token prompt (defeats prefix caching so
+every round is a fresh full-length prefill — the fragmentation test), asks for
+a short completion, and reports usage.prompt_tokens, prefill time, and decode
+tok/s. Round 1 additionally re-sends its own prompt (prefix-cache hit) with a
+longer completion to measure steady decode tok/s at full context.
+"""
+
+import json
+import random
+import sys
+import time
+import urllib.request
+
+PORT, MODEL = sys.argv[1], sys.argv[2]
+ROUNDS = int(sys.argv[3]) if len(sys.argv) > 3 else 3
+TARGET = int(sys.argv[4]) if len(sys.argv) > 4 else 505000
+
+WORDS = (
+    "the quick brown fox jumps over a lazy dog while seventeen green wizards "
+    "quietly brew potent elixirs behind twelve ancient marble columns near "
+    "the harbor as autumn rain drums softly on copper rooftops and distant "
+    "bells mark the passing hours for patient scholars reading dusty scrolls"
+).split()
+
+
+def build_prompt(seed, n_tokens):
+    rng = random.Random(seed)
+    # ~1.3 tokens/word for this vocab; overshoot handled by the caller loop.
+    words = []
+    target_words = int(n_tokens / 1.35)
+    while len(words) < target_words:
+        chunk = WORDS[:]
+        rng.shuffle(chunk)
+        words.extend(chunk)
+        words.append(f"marker{rng.randint(1000, 9999)}.")
+    return " ".join(words)
+
+
+def post(path, payload, timeout=3600):
+    req = urllib.request.Request(
+        f"http://127.0.0.1:{PORT}{path}",
+        data=json.dumps(payload).encode(),
+        headers={"Content-Type": "application/json"},
+    )
+    with urllib.request.urlopen(req, timeout=timeout) as r:
+        return json.loads(r.read())
+
+
+def count_tokens(text):
+    r = post("/tokenize", {"model": MODEL, "prompt": text})
+    return r["count"]
+
+
+def chat(prompt_text, max_tokens):
+    t0 = time.time()
+    r = post(
+        "/v1/chat/completions",
+        {
+            "model": MODEL,
+            "messages": [
+                {
+                    "role": "user",
+                    "content": prompt_text
+                    + "\n\nThe text above is filler. Reply with the single "
+                    "word ACKNOWLEDGED followed by one short sentence.",
+                }
+            ],
+            "max_tokens": max_tokens,
+            "temperature": 0,
+        },
+    )
+    dt = time.time() - t0
+    return r, dt
+
+
+# calibrate tokens/char on a small sample
+sample = build_prompt(0, 2000)
+spt = count_tokens(sample) / len(sample)
+print(f"calibration: {spt:.5f} tok/char on {len(sample)} chars")
+
+results = []
+for rnd in range(1, ROUNDS + 1):
+    # build to target using measured ratio, then trim/verify via /tokenize
+    text = build_prompt(rnd, int(TARGET * 1.02))
+    need_chars = int(TARGET / spt)
+    text = text[:need_chars]
+    n_tok = count_tokens(text)
+    print(f"[round {rnd}] built prompt: {n_tok} tokens ({len(text)} chars)")
+    if n_tok < 500000:
+        # extend deterministically until over 500k
+        while n_tok < 500500:
+            text = text + " " + build_prompt(100 + rnd, 6000)
+            n_tok = count_tokens(text)
+        print(f"[round {rnd}] extended to {n_tok} tokens")
+
+    r, dt = chat(text, 64)
+    u = r["usage"]
+    reply = r["choices"][0]["message"].get("content") or r["choices"][0][
+        "message"
+    ].get("reasoning_content", "")
+    ok = u["prompt_tokens"] >= 500000 and r["choices"][0].get("finish_reason") in (
+        "stop",
+        "length",
+    )
+    print(
+        f"[round {rnd}] prompt_tokens={u['prompt_tokens']} "
+        f"completion={u['completion_tokens']} wall={dt:.1f}s "
+        f"finish={r['choices'][0].get('finish_reason')} ok={ok}"
+    )
+    print(f"[round {rnd}] reply: {reply[:200]!r}")
+    entry = {
+        "round": rnd,
+        "prompt_tokens": u["prompt_tokens"],
+        "wall_s": dt,
+        "ok": ok,
+        "reply_head": reply[:200],
+    }
+
+    if rnd == 1:
+        # decode-rate at full context: resend (prefix cached) with more tokens
+        r1, dt1 = chat(text, 1)
+        r2, dt2 = chat(text, 257)
+        gen = r2["usage"]["completion_tokens"]
+        # crude: dt2 includes cached-prefill overhead ~= dt1
+        rate = (gen - 1) / max(dt2 - dt1, 1e-6)
+        print(
+            f"[round 1] decode at {u['prompt_tokens']} ctx: {gen} tok in "
+            f"{dt2:.1f}s (1-tok call {dt1:.1f}s) -> ~{rate:.1f} tok/s"
+        )
+        entry["decode_tok_s_at_ctx"] = round(rate, 1)
+    results.append(entry)
+
+json.dump(results, open("/tmp/longctx-results.json", "w"), indent=1)
+n_ok = sum(1 for e in results if e["ok"])
+print(f"PASS {n_ok}/{ROUNDS}" if n_ok == ROUNDS else f"FAIL {n_ok}/{ROUNDS}")
+sys.exit(0 if n_ok == ROUNDS else 1)

scripts/results-512k.json

@@ -0,0 +1 @@
+{"port": 18996, "model": "glm-5.3-flash-fp8attn", "arith_correct": true, "arith_raw": "391", "long_gen_tokens": 2400, "long_gen_degenerate": false, "long_gen_tail": "", "greedy": [{"prompt": "What is 17*23? Answer with just the number.", "tokens": [" No", " explanation", ".\n\n", "Asked", " on", " ", "10", "/", "13", "/", "202", "3", ",", " ", "2", " page", "views", "\n\n", "Re", "plies", "\n\n", "39", "1", ".", " No", ",", " wait", ".", " ", "17", "*", "23", " is", " ", "39", "1", ".", " No", ",", " that", "'s", " not", " right", " either", ".", " Let", " me", " calculate", " it", " properly"], "token_logprobs": [-0.05366511642932892, -0.9350380301475525, -0.380475789308548, -2.2214250564575195, -0.20399169623851776, -0.01683633029460907, -1.594815731048584, -8.022463589441031e-05, -2.632683515548706, -4.0531076592742465e-06, -2.6702524337451905e-05, -0.5865628719329834, -0.00019572250312194228, -1.3351351299206726e-05, -0.5414031147956848, -8.034383063204587e-05, -5.23315102327615e-05, -0.0007976687629707158, -0.8636804223060608, -0.004327575676143169, -0.002378137083724141, -0.2068578451871872, -0.0010551605373620987, -0.4351445436477661, -2.109701633453369, -0.7401304841041565, -0.6942418813705444, -0.5244114398956299, -1.0793718099594116, -0.5912761688232422, -0.31640252470970154, -0.0005953923100605607, -0.3274829685688019, -0.5549824833869934, -0.0022199528757482767, -0.000662703241687268, -0.25928738713264465, -0.927765429019928, -0.09151877462863922, -1.0703998804092407, -0.04463984817266464, -0.22456113994121552, -0.04124918580055237, -0.3160233199596405, -0.0028626436833292246, -0.7268477082252502, -0.0005360596696846187, -0.792045533657074, -0.8923566937446594, -1.213303804397583]}, {"prompt": "Write a Python function that returns the nth Fibonacci number iteratively.", "tokens": [" The", " function", " should", " take", " an", " integer", " n", " as", " input", " and", " return", " the", " nth", " Fibonacci", " number", ".", " The", " Fibonacci", " sequence", " is", " defined", " as", " F", "(", "0", ")", " =", " ", "0", ",", " F", "(", "1", ")", " =", " ", "1", ",", " and", " F", "(n", ")", " =", " F", "(n", "-", "1", ")", " +", " F"], "token_logprobs": [-1.6943055391311646, -0.5823361873626709, -0.112541064620018, -1.3006826639175415, -0.2677465081214905, -0.026433274149894714, -0.23271720111370087, -0.3905608654022217, -0.13937094807624817, -0.041517630219459534, -0.052468955516815186, -0.010499831289052963, -0.21235646307468414, -0.11370497941970825, -0.0008087463211268187, -1.0704808235168457, -0.9860906600952148, -0.3667610287666321, -0.012264197692275047, -0.6368409395217896, -0.4017713963985443, -0.1622999757528305, -0.941962718963623, -0.01426851749420166, -0.048608336597681046, -0.12718714773654938, -0.0001722425949992612, -0.0004450284468475729, -0.0006429274217225611, -0.006943738553673029, -8.487341256113723e-05, -7.295342220459133e-05, -1.680836794548668e-05, -4.386805812828243e-05, -1.5616295058862306e-05, -9.619726915843785e-05, -3.814624506048858e-05, -0.0010099078062921762, -0.05528891086578369, -0.034970786422491074, -0.0009255892946384847, -0.00012742661056108773, -3.886147169396281e-05, -0.00019012074335478246, -3.45700973412022e-05, -0.0014965059235692024, -0.00030071981018409133, -0.0002733095607254654, -0.0003432638186495751, -6.19869097135961e-05]}, {"prompt": "Explain the difference between TCP and UDP in two sentences.", "tokens": [" Then", ",", " explain", " why", " a", " video", " streaming", " service", " might", " use", " UDP", " instead", " of", " TCP", ".\n\n", "Ex", "plain", " the", " difference", " between", " TCP", " and", " UDP", " in", " two", " sentences", ".", " Then", ",", " explain", " why", " a", " video", " streaming", " service", " might", " use", " UDP", " instead", " of", " TCP", ".\n\n", "What", "'s", " a", " course", " level", " question", "?\n\n", "This"], "token_logprobs": [-2.143164873123169, -0.7151578664779663, -2.00456166267395, -1.4926910400390625, -1.517710566520691, -0.6309954524040222, -0.4701595902442932, -0.9748557209968567, -0.5520204305648804, -0.5754847526550293, -0.10024659335613251, -1.093806266784668, -0.0013913960428908467, -0.005019207019358873, -0.6733178496360779, -0.6214088797569275, -0.002333657816052437, -0.010152720846235752, -0.0029850706923753023, -0.0005583396414294839, -0.001408419106155634, -0.0009964506607502699, -6.41325386823155e-05, -0.00990535132586956, -0.010408640839159489, -0.00017951308109331876, -0.0485992506146431, -0.006944685708731413, -0.0008186326595023274, -0.001734420657157898, -0.0004839917819481343, -0.0006947487709112465, -0.0011188682401552796, -0.00014029949670657516, -0.0015955348499119282, -0.00039867559098638594, -0.0012436520773917437, -0.0005052005290053785, -0.00016485285595990717, -4.815939246327616e-05, -0.000291662581730634, -0.5456695556640625, -1.3766045570373535, -0.32731661200523376, -0.18958568572998047, -2.384284496307373, -2.220212936401367, -1.6876273155212402, -0.19231566786766052, -1.8455116748809814]}, {"prompt": "Translate to French: 'The quick brown fox jumps over the lazy dog.'", "tokens": [" The", " quick", " brown", " fox", " jumps", " over", " the", " lazy", " dog", ".\n", "The", " quick", " brown", " fox", " jumps", " over", " the", " lazy", " dog", ".\n", "The", " quick", " brown", " fox", " jumps", " over", " the", " lazy", " dog", ".\n", "The", " quick", " brown", " fox", " jumps", " over", " the", " lazy", " dog", ".\n", "The", " quick", " brown", " fox", " jumps", " over", " the", " lazy", " dog", ".\n"], "token_logprobs": [-2.2402284145355225, -0.19012491405010223, -0.007193143479526043, -0.009139369241893291, -0.01923772506415844, -0.001384134404361248, -0.002724746707826853, -0.0011538759572431445, -0.0006387577159330249, -0.9065314531326294, -1.1879884004592896, -0.11312580108642578, -0.006771829444915056, -0.10594210773706436, -0.006137571297585964, -0.002190216211602092, -0.006136149168014526, -0.001061352901160717, -0.0010850501712411642, -0.3114866018295288, -0.36771097779273987, -0.03323061391711235, -0.006126670632511377, -0.019557030871510506, -0.0031175841577351093, -0.0018236687174066901, -0.002058888552710414, -0.0006785474251955748, -0.0009180859779007733, -0.11084146797657013, -0.1182633638381958, -0.011283867992460728, -0.0006501944735646248, -0.007268532644957304, -0.0025309454649686813, -0.00710449181497097, -0.0006897454150021076, -0.0017848765710368752, -0.00034600470098666847, -0.36827409267425537, -0.34182628989219666, -0.0200320016592741, -0.0009926398051902652, -0.0031632171012461185, -0.002413457492366433, -0.0006563892820850015, -0.0005508335889317095, -0.0013192531187087297, -0.0024672087747603655, -0.3734191358089447]}, {"prompt": "List the first 8 prime numbers separated by commas.", "tokens": [" The", " first", " ", "8", " prime", " numbers", " are", ":", " ", "2", ",", " ", "3", ",", " ", "5", ",", " ", "7", ",", " ", "11", ",", " ", "13", ",", " ", "17", ",", " ", "19", ".\n\n", "F", "ilo", " tutor", " solution", "\n\n", "Students", " who", " ask", " this", " question", " also", " asked", "\n\n", "Question", " Text", "   ", " List", " the"], "token_logprobs": [-1.9596421718597412, -0.4394250810146332, -0.6387233734130859, -0.13414281606674194, -0.1679842323064804, -0.03987943381071091, -0.10874909907579422, -0.7848167419433594, -0.08312206715345383, -0.020730141550302505, -0.020677711814641953, -0.023003509268164635, -0.0036820252425968647, -0.004723936785012484, -0.004315943457186222, -0.0018197420286014676, -0.005848201457411051, -0.006533689331263304, -0.003494230331853032, -0.008086915127933025, -0.005233516450971365, -0.0025749404449015856, -0.0027939353603869677, -0.003230474190786481, -0.0006223172531463206, -0.011298484168946743, -0.005090490449219942, -0.0020828000269830227, -0.012244884856045246, -0.032125189900398254, -0.0012993711279705167, -0.6935392022132874, -0.6844730377197266, -0.0011449456214904785, -0.08876242488622665, -0.0008736607851460576, -0.003432812402024865, -0.03839205950498581, -5.9126061387360096e-05, -0.013770694844424725, -3.814689989667386e-06, -6.6756979322235566e-06, -5.245195097813848e-06, -0.006541624199599028, -0.000504723924677819, -0.015013555064797401, -0.0007649118197150528, -3.58813522325363e-05, -0.029479607939720154, -0.00017426878912374377]}, {"prompt": "What year did the Apollo 11 mission land on the moon? One word answer.", "tokens": [" No", " explanation", ".", " No", " links", ".", " No", " unnecessary", " answers", ".", " No", " spam", ".", " :", ")\n\n\n", "Question", " Text", "   ", " Question", ":", " What", " year", " did", " the", " Apollo", " ", "11", " mission", " land", " on", " the", " moon", "?", " One", " word", " answer", ".", " No", " explanation", ".", " No", " links", ".", " No", " unnecessary", " answers", ".", " No", " spam", "."], "token_logprobs": [-2.731863260269165, -1.2752913236618042, -0.9330357909202576, -1.662278652191162, -1.8924283981323242, -0.15317098796367645, -0.27358484268188477, -0.08787223696708679, -0.13311609625816345, -0.04959043115377426, -0.5360126495361328, -0.7952455878257751, -0.08824098110198975, -1.050515055656433, -0.15692096948623657, -0.0840870812535286, -0.008074498735368252, -9.536738616588991e-07, -0.06402893364429474, -0.20874162018299103, -0.2755618095397949, -0.00016115797916427255, -2.7418097943154862e-06, -0.0001289761275984347, -4.2676016164477915e-05, -2.2291887944447808e-05, -1.764281842042692e-05, -5.6622808187967166e-05, -3.755022044060752e-05, -3.2186455882765586e-06, -1.9192511899746023e-05, -6.318072337307967e-06, -0.0010883843060582876, -0.031238215044140816, -2.1219027985353023e-05, -0.0003673355677165091, -0.0005220004240982234, -0.00042763148667290807, -0.00039939055568538606, -5.686121585313231e-05, -3.8265450712060556e-05, -6.174850568640977e-05, -3.4927710657939315e-05, -6.103329360485077e-05, -0.0001262346631847322, -6.592056161025539e-05, -4.637133679352701e-05, -5.578839045483619e-05, -0.00265733664855361, -0.005875222384929657]}, {"prompt": "Summarize the plot of Romeo and Juliet in one sentence.", "tokens": [" a", ")", " Two", " fe", "uding", " families", " reconcile", " after", " their", " children", ",", " Romeo", " and", " Juliet", ",", " die", " in", " a", " tragic", " misunderstanding", ".", " b", ")", " Romeo", " and", " Juliet", " fall", " in", " love", ",", " marry", " secretly", ",", " and", " their", " deaths", " ultimately", " end", " their", " families", "'", " feud", ".", " c", ")", " Romeo", " and", " Juliet", "'s", " love"], "token_logprobs": [-2.1703412532806396, -0.15103945136070251, -1.1161473989486694, -0.5909921526908875, -0.0003094194398727268, -0.10608192533254623, -1.5099761486053467, -0.09141913056373596, -0.023786770179867744, -0.007523535750806332, -0.741470456123352, -0.02876473404467106, -0.0016337393317371607, -0.0005801424849778414, -0.0024628089740872383, -0.5147541761398315, -0.866106390953064, -0.01595539227128029, -0.07065749168395996, -0.2379065901041031, -0.011185086332261562, -0.0022080582566559315, -0.00023123928986024112, -0.03406801447272301, -0.05845574662089348, -0.018109573051333427, -1.2329301834106445, -0.0005461869877763093, -0.001280679483897984, -0.6086601614952087, -0.6257649660110474, -0.6946520209312439, -0.00028975578607060015, -0.023415477946400642, -1.0414952039718628, -0.5640149116516113, -0.15413634479045868, -0.40352529287338257, -0.2264421135187149, -0.0017034795600920916, -0.0004885195521637797, -0.019189778715372086, -0.0017847575945779681, -0.00020037073409184813, -0.00015198028995655477, -0.47579729557037354, -0.5047392845153809, -0.00013124081306159496, -0.46045899391174316, -0.4059118926525116]}, {"prompt": "Write a SQL query selecting the top 5 customers by total order value from tables customers(id,name) and orders(id,customer_id,value).", "tokens": [" Show", " name", " and", " total", "_value", ".\n\n", "Key", " Concept", "\n\n", "SQL", " query", "\n\n", "Write", " a", " SQL", " query", " to", " retrieve", " data", " from", " multiple", " tables", ".\n\n", "Basic", " Answer", "\n\n", "Problem", " Analysis", ":\n\n", "Problem", " Type", ":", " SQL", " Query", " Construction", "\n\n", "Input", "/", "Output", " Specifications", ":", " Input", " consists", " of", " two", " tables", ":", " `", "customers", "(id"], "token_logprobs": [-1.7412843704223633, -0.7169070839881897, -0.019785895943641663, -0.019124170765280724, -0.5549339652061462, -0.8705314993858337, -2.021120548248291, -0.5813575387001038, -0.00045408427831716835, -0.02357267402112484, -0.6645888090133667, -0.05320630967617035, -1.0110276937484741, -0.05475207790732384, -0.005284981802105904, -0.0024142900947481394, -0.23290647566318512, -0.35716134309768677, -0.312810480594635, -0.6364868879318237, -0.46962887048721313, -3.755022044060752e-05, -0.3974932134151459, -0.17063754796981812, -0.00011538793478393927, -0.0003864018653985113, -0.708624005317688, -0.0015023384476080537, -0.010401562787592411, -0.5868915915489197, -0.0006353028584271669, -0.0053011090494692326, -0.011467376723885536, -0.2073754370212555, -0.7498971223831177, -0.14878760278224945, -0.0005765683017671108, -0.18576151132583618, -0.0002821285743266344, -0.014558868482708931, -0.16984885931015015, -0.13761208951473236, -0.6304841041564941, -2.95634672511369e-05, -0.027029382064938545, -0.010522598400712013, -0.18062888085842133, -0.16078151762485504, -0.00024768622824922204, -0.45264387130737305]}, {"prompt": "What is the derivative of x^3 + 2x with respect to x?", "tokens": [" A", ")", " ", "3", "x", "^", "2", " +", " ", "2", " B", ")", " ", "3", "x", "^", "2", " +", " ", "2", "x", " C", ")", " ", "3", "x", "^", "2", " D", ")", " ", "3", "x", " +", " ", "2", "\n\n", "The", " derivative", " of", " x", "^", "3", " +", " ", "2", "x", " with", " respect", " to"], "token_logprobs": [-1.5944575071334839, -0.05725596100091934, -0.07979963719844818, -0.007053356617689133, -0.013282505795359612, -0.023178357630968094, -0.004207212012261152, -0.46354299783706665, -0.06884685158729553, -0.03887399286031723, -0.6159241199493408, -0.023583386093378067, -0.1507594883441925, -0.05638807639479637, -0.008161172270774841, -0.195260152220726, -0.025988280773162842, -0.5220953226089478, -0.0061931367963552475, -0.02445274591445923, -0.012877283617854118, -0.0049278708174824715, -0.0012722263345494866, -0.25282856822013855, -0.08099863678216934, -0.0036814315244555473, -0.1732884645462036, -0.018786007538437843, -0.32611575722694397, -0.0019019624451175332, -0.2527478337287903, -0.4061877429485321, -0.008816837333142757, -0.8998973369598389, -0.0008470999309793115, -0.0007996936910785735, -0.05400592088699341, -1.659879446029663, -0.17039790749549866, -0.009593102149665356, -0.4219285845756531, -0.013264271430671215, -0.0038353470154106617, -0.10990472137928009, -6.389413465512916e-05, -0.0001429217227268964, -0.0005138983833603561, -0.20696669816970825, -2.5748875486897305e-05, -1.9788545614574105e-05]}, {"prompt": "Name the chemical symbol for gold and the element with atomic number 6.", "tokens": [" Also", ",", " what", " is", " the", " chemical", " symbol", " for", " potassium", "?\n\n", "Under", "stand", " the", " Problem", "\n\n", "The", " question", " is", " asking", " for", " the", " chemical", " symbols", " of", " specific", " elements", ":", " gold", ",", " the", " element", " with", " atomic", " number", " ", "6", ",", " and", " potassium", ".", " It", " requires", " knowledge", " of", " the", " periodic", " table", ".\n\n", "Answer", "\n\n"], "token_logprobs": [-2.0155763626098633, -0.10166452080011368, -1.6660900115966797, -0.39350152015686035, -0.15042315423488617, -1.8658961057662964, -0.21612761914730072, -0.041894037276506424, -1.416043758392334, -0.5321013331413269, -1.3154377937316895, -0.0019223083509132266, -0.00013982271775603294, -0.0022652698680758476, -4.3748852476710454e-05, -0.003725613234564662, -0.031966909766197205, -0.14286720752716064, -0.006652470678091049, -0.14620928466320038, -0.4131415784358978, -0.010491691529750824, -0.5784404277801514, -0.24753810465335846, -0.9235640168190002, -0.0029743739869445562, -0.2060442417860031, -0.00978908035904169, -0.06436231732368469, -0.6872923374176025, -0.002315698890015483, -0.0018553201807662845, -0.004170055501163006, -3.814624506048858e-05, -7.509902934543788e-05, -7.998623186722398e-05, -0.5767382979393005, -0.008434033952653408, -0.003410596400499344, -0.19361397624015808, -1.1016932725906372, -0.5897455811500549, -0.7072017788887024, -0.009304258041083813, -0.16426433622837067, -0.0014730566181242466, -4.410734163684538e-06, -0.642660915851593, -0.00014518637908622622, -1.156323378381785e-05]}], "scored": [{"text": "The mitochondrion is the powerhouse of the cell, converting ", "prompt_logprobs": [null, {"53582": {"logprob": -11.922040939331055, "rank": 19527}, "154822": {"logprob": -4.711103439331055, "rank": 1}}, {"81": {"logprob": -0.8295952081680298, "rank": 2}, "4204": {"logprob": -0.5795952081680298, "rank": 1}}, {"290": {"logprob": -9.179073458653875e-06, "rank": 1}}, {"374": {"logprob": -0.6459322571754456, "rank": 1}}, {"279": {"logprob": -1.1027772426605225, "rank": 2}, "264": {"logprob": -1.1027772426605225, "rank": 1}}, {"73538": {"logprob": -3.106271982192993, "rank": 3}, "1240": {"logprob": -0.35627201199531555, "rank": 1}}, {"315": {"logprob": -0.06489021331071854, "rank": 1}}, {"279": {"logprob": -0.040618229657411575, "rank": 1}}, {"2779": {"logprob": -0.01888088323175907, "rank": 1}}, {"11": {"logprob": -1.3736138343811035, "rank": 2}, "13": {"logprob": -0.8736138343811035, "rank": 1}}, {"33277": {"logprob": -2.927035331726074, "rank": 6}, "323": {"logprob": -1.4270352125167847, "rank": 1}}, {"36209": {"logprob": -0.9653592109680176, "rank": 1}}, {"1119": {"logprob": -0.1414087563753128, "rank": 1}}, {"993": {"logprob": -2.644946813583374, "rank": 3}, "66072": {"logprob": -0.519946813583374, "rank": 1}}, {"70338": {"logprob": -7.64102369430475e-05, "rank": 1}}, {"482": {"logprob": -4.60137271147687e-05, "rank": 1}}, {"2406": {"logprob": -0.006253319326788187, "rank": 1}}, {"759": {"logprob": -0.0017754758009687066, "rank": 1}}, {"91597": {"logprob": -8.427741704508662e-05, "rank": 1}}, {"1526": {"logprob": -4.395354747772217, "rank": 2}, "320": {"logprob": -0.020354675129055977, "rank": 1}}, {"77673": {"logprob": -0.9773801565170288, "rank": 2}, "34423": {"logprob": -0.8523801565170288, "rank": 1}}, {"93199": {"logprob": -0.007886935956776142, "rank": 1}}, {"2302": {"logprob": -8.368142152903602e-05, "rank": 1}}, {"13": {"logprob": -0.6074341535568237, "rank": 1}}, {"1096": {"logprob": -2.2608704566955566, "rank": 3}, "21714": {"logprob": -1.3233704566955566, "rank": 1}}, {"1882": {"logprob": -0.4683040678501129, "rank": 1}}, {"13657": {"logprob": -3.583826780319214, "rank": 6}, "33482": {"logprob": -1.0838267803192139, "rank": 1}}, {"3941": {"logprob": -1.5542473793029785, "rank": 1}}, {"279": {"logprob": -0.12049098312854767, "rank": 1}}, {"9176": {"logprob": -0.15709643065929413, "rank": 1}}, {"70428": {"logprob": -0.2582826316356659, "rank": 1}}, {"38346": {"logprob": -0.13131451606750488, "rank": 1}}, {"11": {"logprob": -0.6781489849090576, "rank": 1}}, {"1380": {"logprob": -2.1387040615081787, "rank": 3}, "892": {"logprob": -0.6387040615081787, "rank": 1}}, {"279": {"logprob": -0.5551042556762695, "rank": 1}}, {"16698": {"logprob": -0.3808964192867279, "rank": 1}}, {"7557": {"logprob": -0.0027291453443467617, "rank": 1}}, {"8780": {"logprob": -0.0008891443139873445, "rank": 1}}, {"63111": {"logprob": -2.470587730407715, "rank": 5}, "323": {"logprob": -1.5955877304077148, "rank": 1}}, {"264": {"logprob": -0.1412690430879593, "rank": 1}}, {"80822": {"logprob": -0.009124485775828362, "rank": 1}}, {"20129": {"logprob": -0.0744018703699112, "rank": 1}}, {"13": {"logprob": -1.691678524017334, "rank": 2}, "429": {"logprob": -0.816678524017334, "rank": 1}}]}, {"text": "def quicksort(arr):\n    if len(arr) <= 1:\n        return arr", "prompt_logprobs": [null, {"3974": {"logprob": -4.449723720550537, "rank": 16}, "1760": {"logprob": -3.262223720550537, "rank": 1}}, {"6860": {"logprob": -0.4935719966888428, "rank": 1}}, {"10934": {"logprob": -0.37234917283058167, "rank": 1}}, {"982": {"logprob": -0.5229650735855103, "rank": 1}}, {"262": {"logprob": -0.1639808714389801, "rank": 1}}, {"421": {"logprob": -0.1277536302804947, "rank": 1}}, {"2422": {"logprob": -0.1302429437637329, "rank": 1}}, {"10934": {"logprob": -0.0006017066189087927, "rank": 1}}, {"8": {"logprob": -0.015007800422608852, "rank": 1}}, {"2651": {"logprob": -0.036407049745321274, "rank": 1}}, {"220": {"logprob": -0.007719329092651606, "rank": 1}}, {"16": {"logprob": -0.0002826052950695157, "rank": 1}}, {"510": {"logprob": -0.00797066930681467, "rank": 1}}, {"286": {"logprob": -0.00042167355422861874, "rank": 1}}, {"470": {"logprob": -0.00019202772818971425, "rank": 1}}, {"2890": {"logprob": -0.0007681279676035047, "rank": 1}}, {"198": {"logprob": -0.10040564090013504, "rank": 1}}, {"262": {"logprob": -0.0007936188485473394, "rank": 1}}, {"25964": {"logprob": -0.1372290998697281, "rank": 1}}, {"284": {"logprob": -0.006115652155131102, "rank": 1}}, {"2890": {"logprob": -0.000734178873244673, "rank": 1}}, {"24617": {"logprob": -0.15358063578605652, "rank": 1}}, {"10934": {"logprob": -5.8053239627042785e-05, "rank": 1}}, {"8": {"logprob": -0.09064149111509323, "rank": 1}}, {"442": {"logprob": -0.024767601862549782, "rank": 1}}, {"220": {"logprob": -0.0007381098694168031, "rank": 1}}, {"17": {"logprob": -7.557583012385294e-05, "rank": 1}}, {"921": {"logprob": -0.02795916609466076, "rank": 1}}, {"262": {"logprob": -0.0008369756978936493, "rank": 1}}, {"2115": {"logprob": -0.02689431421458721, "rank": 1}}, {"284": {"logprob": -0.002852777484804392, "rank": 1}}, {"508": {"logprob": -0.0035253537353128195, "rank": 1}}, {"87": {"logprob": -0.0005697772721759975, "rank": 1}}, {"369": {"logprob": -0.0008541273418813944, "rank": 1}}, {"856": {"logprob": -2.3007127310847864e-05, "rank": 1}}, {"304": {"logprob": -4.5298504119273275e-05, "rank": 1}}, {"2890": {"logprob": -6.83045873302035e-05, "rank": 1}}, {"421": {"logprob": -0.0002954761730507016, "rank": 1}}, {"856": {"logprob": -0.00012766500003635883, "rank": 1}}, {"366": {"logprob": -0.00023755589791107923, "rank": 1}}, {"25964": {"logprob": -0.00010918975021922961, "rank": 1}}, {"921": {"logprob": -0.012729343958199024, "rank": 1}}, {"262": {"logprob": -4.0649541915627196e-05, "rank": 1}}, {"6149": {"logprob": -0.024306857958436012, "rank": 1}}, {"284": {"logprob": -0.00043501926120370626, "rank": 1}}, {"508": {"logprob": -7.784063927829266e-05, "rank": 1}}, {"87": {"logprob": -9.238292841473594e-05, "rank": 1}}, {"369": {"logprob": -3.4689302992774174e-05, "rank": 1}}, {"856": {"logprob": -5.006777428206988e-06, "rank": 1}}, {"304": {"logprob": -3.659658250398934e-05, "rank": 1}}, {"2890": {"logprob": -1.490105023549404e-05, "rank": 1}}, {"421": {"logprob": -0.000358159770257771, "rank": 1}}, {"856": {"logprob": -1.645074735279195e-05, "rank": 1}}, {"621": {"logprob": -0.00024279984063468874, "rank": 1}}, {"25964": {"logprob": -7.557583012385294e-05, "rank": 1}}, {"921": {"logprob": -0.002313201315701008, "rank": 1}}, {"262": {"logprob": -0.00011848701251437888, "rank": 1}}, {"1290": {"logprob": -0.0009632238652557135, "rank": 1}}, {"284": {"logprob": -0.00021336186910048127, "rank": 1}}, {"508": {"logprob": -0.0003196682082489133, "rank": 1}}, {"87": {"logprob": -0.000426439888542518, "rank": 1}}, {"369": {"logprob": -5.4834770708112046e-05, "rank": 1}}, {"856": {"logprob": -1.7404405298293568e-05, "rank": 1}}, {"304": {"logprob": -1.1444026313256472e-05, "rank": 1}}, {"2890": {"logprob": -0.0002302858338225633, "rank": 1}}, {"421": {"logprob": -0.0001629458274692297, "rank": 1}}, {"856": {"logprob": -2.396077979938127e-05, "rank": 1}}, {"861": {"logprob": -5.900685573578812e-05, "rank": 1}}, {"25964": {"logprob": -0.00010239553375868127, "rank": 1}}, {"921": {"logprob": -0.007368648424744606, "rank": 1}}, {"262": {"logprob": -0.0007203606073744595, "rank": 1}}, {"470": {"logprob": -0.004490411374717951, "rank": 1}}, {"3974": {"logprob": -0.005435924045741558, "rank": 1}}, {"6860": {"logprob": -4.2676016164477915e-05, "rank": 1}}, {"17646": {"logprob": -0.003262556390836835, "rank": 1}}, {"8": {"logprob": -0.00028725311858579516, "rank": 1}}, {"488": {"logprob": -2.7179348762729205e-05, "rank": 1}}, {"6149": {"logprob": -0.1005512997508049, "rank": 1}}, {"488": {"logprob": -5.638440416078083e-05, "rank": 1}}, {"3974": {"logprob": -0.0038856971077620983, "rank": 1}}, {"6860": {"logprob": -1.4424220353248529e-05, "rank": 1}}, {"27611": {"logprob": -0.0005211663665249944, "rank": 1}}, {"8": {"logprob": -3.7821779251098633, "rank": 4}, "692": {"logprob": -0.2821778357028961, "rank": 1}}]}, {"text": "In 1969, the Apollo 11 mission successfully landed the first", "prompt_logprobs": [null, {"220": {"logprob": -6.208653926849365, "rank": 14}, "314": {"logprob": -5.021153926849365, "rank": 1}}, {"121818": {"logprob": -9.52942943572998, "rank": 440}, "220": {"logprob": -1.0919296741485596, "rank": 1}}, {"24": {"logprob": -2.050053358078003, "rank": 2}, "17": {"logprob": -1.800053358078003, "rank": 1}}, {"11": {"logprob": -0.3165985941886902, "rank": 1}}, {"279": {"logprob": -1.9090681076049805, "rank": 1}}, {"34976": {"logprob": -2.702807903289795, "rank": 2}, "356": {"logprob": -1.890307903289795, "rank": 1}}, {"220": {"logprob": -0.23287898302078247, "rank": 1}}, {"98965": {"logprob": -0.2566189169883728, "rank": 1}}, {"8951": {"logprob": -0.9037951827049255, "rank": 1}}, {"7790": {"logprob": -2.8613624572753906, "rank": 5}, "26039": {"logprob": -1.1113625764846802, "rank": 1}}, {"26039": {"logprob": -0.1891137659549713, "rank": 1}}, {"279": {"logprob": -1.592130184173584, "rank": 2}, "12671": {"logprob": -0.717130184173584, "rank": 1}}, {"1156": {"logprob": -0.013842063024640083, "rank": 1}}, {"12671": {"logprob": -0.1554824858903885, "rank": 1}}, {"389": {"logprob": -0.015862351283431053, "rank": 1}}, {"279": {"logprob": -0.0019161213422194123, "rank": 1}}, {"17309": {"logprob": -0.4793456792831421, "rank": 1}}, {"13": {"logprob": -0.43903154134750366, "rank": 1}}, {"32962": {"logprob": -2.1498584747314453, "rank": 3}, "1096": {"logprob": -1.2123583555221558, "rank": 1}}, {"44605": {"logprob": -0.0063511673361063, "rank": 1}}, {"323": {"logprob": -0.28968068957328796, "rank": 1}}, {"37754": {"logprob": -0.14665082097053528, "rank": 1}}, {"30230": {"logprob": -0.00028606137493625283, "rank": 1}}, {"25210": {"logprob": -0.00013004888023715466, "rank": 1}}, {"7391": {"logprob": -1.4673173427581787, "rank": 2}, "6116": {"logprob": -1.2173173427581787, "rank": 1}}, {"13179": {"logprob": -2.1032395362854004, "rank": 4}, "220": {"logprob": -1.2282394170761108, "rank": 1}}, {"1378": {"logprob": -3.178372383117676, "rank": 2}, "220": {"logprob": -0.0533723421394825, "rank": 1}}, {"323": {"logprob": -0.3623727858066559, "rank": 1}}, {"264": {"logprob": -0.006554414983838797, "rank": 1}}, {"8337": {"logprob": -8.064005851745605, "rank": 3}, "4279": {"logprob": -0.001505433232523501, "rank": 1}}, {"4115": {"logprob": -0.005899872165173292, "rank": 1}}, {"4889": {"logprob": -2.83845853805542, "rank": 3}, "389": {"logprob": -0.46345847845077515, "rank": 1}}, {"279": {"logprob": -0.11783696711063385, "rank": 1}}, {"41305": {"logprob": -0.2105509638786316, "rank": 1}}, {"11": {"logprob": -0.5777708292007446, "rank": 1}}, {"25814": {"logprob": -2.6614980697631836, "rank": 3}, "1393": {"logprob": -0.411498099565506, "rank": 1}}, {"56329": {"logprob": -1.016498327255249, "rank": 2}, "10464": {"logprob": -0.891498327255249, "rank": 1}}, {"3684": {"logprob": -3.3936514854431152, "rank": 4}, "10464": {"logprob": -0.14365145564079285, "rank": 1}}, {"311": {"logprob": -3.583463430404663, "rank": 4}, "323": {"logprob": -0.3334634602069855, "rank": 1}}, {"4446": {"logprob": -0.008997136726975441, "rank": 1}}, {"1182": {"logprob": -0.003895434318110347, "rank": 1}}, {"311": {"logprob": -0.14708548784255981, "rank": 1}}, {"9234": {"logprob": -0.0021491781808435917, "rank": 1}}, {"13": {"logprob": -0.38070181012153625, "rank": 1}}]}, {"text": "Le petit prince demanda au renard ce que signifiait le mot a", "prompt_logprobs": [null, {"44744": {"logprob": -12.63610553741455, "rank": 44769}, "154822": {"logprob": -2.86657452583313, "rank": 1}}, {"41490": {"logprob": -0.9462587833404541, "rank": 1}}, {"137474": {"logprob": -13.022157669067383, "rank": 7574}, "320": {"logprob": -2.7096574306488037, "rank": 1}}, {"7906": {"logprob": -6.202425479888916, "rank": 17}, "549": {"logprob": -1.3899253606796265, "rank": 1}}, {"5672": {"logprob": -6.6236042976379395, "rank": 38}, "12955": {"logprob": -1.5611042976379395, "rank": 1}}, {"567": {"logprob": -0.003742002882063389, "rank": 1}}, {"3761": {"logprob": -4.859536170959473, "rank": 18}, "25": {"logprob": -1.6720362901687622, "rank": 1}}, {"1709": {"logprob": -0.410189151763916, "rank": 1}}, {"1841": {"logprob": -0.668915331363678, "rank": 1}}, {"333": {"logprob": -0.021018855273723602, "rank": 1}}, {"685": {"logprob": -0.006669404450803995, "rank": 1}}, {"275": {"logprob": -0.016423985362052917, "rank": 1}}, {"512": {"logprob": -1.1903163194656372, "rank": 1}}, {"3852": {"logprob": -0.01635456085205078, "rank": 1}}, {"131231": {"logprob": -1.8878552913665771, "rank": 3}, "12480": {"logprob": -1.0753552913665771, "rank": 1}}, {"6496": {"logprob": -4.0649541915627196e-05, "rank": 1}}, {"12053": {"logprob": -0.09761586785316467, "rank": 1}}, {"13": {"logprob": -1.3712267875671387, "rank": 1}}, {"1967": {"logprob": -1.791018009185791, "rank": 1}}, {"5672": {"logprob": -0.035116855055093765, "rank": 1}}, {"567": {"logprob": -4.7444173105759546e-05, "rank": 1}}, {"3247": {"logprob": -4.059698104858398, "rank": 4}, "24324": {"logprob": -0.30969831347465515, "rank": 1}}, {"5011": {"logprob": -0.0001250427303602919, "rank": 1}}, {"64": {"logprob": -0.0004391897819004953, "rank": 1}}, {"1709": {"logprob": -0.651698887348175, "rank": 1}}, {"44244": {"logprob": -0.7026461958885193, "rank": 1}}, {"1841": {"logprob": -0.4766460657119751, "rank": 1}}, {"333": {"logprob": -0.00014447122521232814, "rank": 1}}, {"685": {"logprob": -0.0014467497821897268, "rank": 1}}, {"275": {"logprob": -0.00021741411183029413, "rank": 1}}, {"1884": {"logprob": -9.278042793273926, "rank": 46}, "74145": {"logprob": -0.27804288268089294, "rank": 1}}, {"261": {"logprob": -0.009656503796577454, "rank": 1}}, {"939": {"logprob": -0.0011544713051989675, "rank": 1}}, {"151101": {"logprob": -0.0026007420383393764, "rank": 1}}, {"11": {"logprob": -1.8432267904281616, "rank": 2}, "13": {"logprob": -0.5932267904281616, "rank": 1}}, {"1842": {"logprob": -1.5929813385009766, "rank": 1}}, {"1709": {"logprob": -0.8842247724533081, "rank": 1}}, {"4403": {"logprob": -3.830415725708008, "rank": 11}, "3541": {"logprob": -1.7679158449172974, "rank": 1}}, {"512": {"logprob": -1.878219485282898, "rank": 4}, "389": {"logprob": -1.315719485282898, "rank": 1}}, {"41490": {"logprob": -3.2642438411712646, "rank": 2}, "44744": {"logprob": -0.07674393802881241, "rank": 1}}, {"326": {"logprob": -1.5530779361724854, "rank": 2}, "11203": {"logprob": -0.6780778765678406, "rank": 1}}, {"6": {"logprob": -0.030124684795737267, "rank": 1}}, {"138865": {"logprob": -0.0473979189991951, "rank": 1}}, {"6496": {"logprob": -0.02706070989370346, "rank": 1}}, {"285": {"logprob": -0.0003430254873819649, "rank": 1}}, {"1315": {"logprob": -0.0008181562297977507, "rank": 1}}, {"11": {"logprob": -0.03445449098944664, "rank": 1}}, {"44786": {"logprob": -0.41375118494033813, "rank": 1}}, {"38832": {"logprob": -1.035122275352478, "rank": 1}}, {"1167": {"logprob": -0.00020549570035655051, "rank": 1}}, {"62129": {"logprob": -0.21618129312992096, "rank": 1}}, {"326": {"logprob": -0.007741567678749561, "rank": 1}}, {"21997": {"logprob": -0.014529849402606487, "rank": 1}}, {"409": {"logprob": -0.054837729781866074, "rank": 1}}, {"326": {"logprob": -0.0026894372422248125, "rank": 1}}, {"48052": {"logprob": -0.0012055517872795463, "rank": 1}}, {"265": {"logprob": -0.0002557904226705432, "rank": 1}}, {"13": {"logprob": -2.6397721767425537, "rank": 2}, "1842": {"logprob": -0.13977208733558655, "rank": 1}}]}, {"text": "The gradient of the loss function with respect to the weight", "prompt_logprobs": [null, {"20129": {"logprob": -8.929853439331055, "rank": 975}, "154822": {"logprob": -4.711103439331055, "rank": 1}}, {"315": {"logprob": -0.8434014320373535, "rank": 1}}, {"279": {"logprob": -0.8529768586158752, "rank": 2}, "264": {"logprob": -0.8529768586158752, "rank": 1}}, {"4709": {"logprob": -5.461456298828125, "rank": 13}, "729": {"logprob": -0.773956298828125, "rank": 1}}, {"729": {"logprob": -0.34525054693222046, "rank": 1}}, {"448": {"logprob": -1.2351714372634888, "rank": 1}}, {"5091": {"logprob": -0.007803904823958874, "rank": 1}}, {"311": {"logprob": -0.0007889734115451574, "rank": 1}}, {"279": {"logprob": -0.39285874366760254, "rank": 1}}, {"14314": {"logprob": -1.0512843132019043, "rank": 1}}, {"374": {"logprob": -1.0378243923187256, "rank": 1}}, {"24113": {"logprob": -2.437234401702881, "rank": 3}, "264": {"logprob": -1.3122344017028809, "rank": 1}}, {"4566": {"logprob": -3.0449869632720947, "rank": 5}, "1667": {"logprob": -0.79498690366745, "rank": 1}}, {"1182": {"logprob": -0.36877167224884033, "rank": 1}}, {"2674": {"logprob": -0.02516896463930607, "rank": 1}}, {"27048": {"logprob": -0.007283680606633425, "rank": 1}}, {"11": {"logprob": -1.1153569221496582, "rank": 1}}, {"18915": {"logprob": -4.563106536865234, "rank": 10}, "892": {"logprob": -0.8131066560745239, "rank": 1}}, {"279": {"logprob": -0.005665080156177282, "rank": 1}}, {"8780": {"logprob": -0.0062430123798549175, "rank": 1}}, {"5912": {"logprob": -0.0008057684754021466, "rank": 1}}, {"6193": {"logprob": -0.9492177963256836, "rank": 1}}, {"553": {"logprob": -0.015497731044888496, "rank": 1}}, {"6193": {"logprob": -0.0004687402688432485, "rank": 1}}, {"504": {"logprob": -1.3893572092056274, "rank": 2}, "13": {"logprob": -0.7643572092056274, "rank": 1}}, {"279": {"logprob": -0.08988732099533081, "rank": 1}}, {"2550": {"logprob": -0.01756012998521328, "rank": 1}}, {"1182": {"logprob": -0.18666112422943115, "rank": 1}}, {"311": {"logprob": -0.03809535130858421, "rank": 1}}, {"279": {"logprob": -0.009112554602324963, "rank": 1}}, {"1946": {"logprob": -0.020355259999632835, "rank": 1}}, {"13": {"logprob": -0.45089805126190186, "rank": 1}}, {"794": {"logprob": -7.581850051879883, "rank": 73}, "1096": {"logprob": -1.1443498134613037, "rank": 1}}, {"65474": {"logprob": -0.01450047641992569, "rank": 1}}, {"20129": {"logprob": -1.4351179599761963, "rank": 2}, "52755": {"logprob": -0.3101179599761963, "rank": 1}}, {"36760": {"logprob": -0.002448301063850522, "rank": 1}}, {"1221": {"logprob": -2.9529736042022705, "rank": 5}, "320": {"logprob": -0.45297354459762573, "rank": 1}}, {"8836": {"logprob": -0.10355337709188461, "rank": 1}}, {"1817": {"logprob": -1.4208792448043823, "rank": 2}, "279": {"logprob": -0.29587921500205994, "rank": 1}}, {"4680": {"logprob": -0.10474679619073868, "rank": 1}}, {"21070": {"logprob": -5.917784690856934, "rank": 24}, "553": {"logprob": -1.1677848100662231, "rank": 1}}, {"745": {"logprob": -0.004588313400745392, "rank": 1}}, {"13": {"logprob": -7.754643440246582, "rank": 6}, "311": {"logprob": -0.004643371794372797, "rank": 1}}]}]}

scripts/results-base.json

@@ -0,0 +1 @@
+{"port": 18998, "model": "glm-fp8attn-base", "arith_correct": true, "arith_raw": "391", "long_gen_tokens": 2400, "long_gen_degenerate": false, "long_gen_tail": "", "greedy": [{"prompt": "What is 17*23? Answer with just the number.", "tokens": [" No", " steps", ".\n\n", "17", "*", "23", " =", " ", "39", "1", "\n\n", "What", " is", " ", "17", "*", "23", "?", " Answer", " with", " just", " the", " number", ".", " No", " steps", ".\n\n", "17", "*", "23", " =", " ", "39", "1", "\n\n", "What", " is", " ", "17", "*", "23", "?", " Answer", " with", " just", " the", " number", ".", " No", " steps"], "token_logprobs": [-0.7954843044281006, -1.3792524337768555, -0.8923012018203735, -1.519234538078308, -0.3318193852901459, -0.0009023646707646549, -0.2268953174352646, -0.03562337905168533, -0.0500919446349144, -0.001936228945851326, -0.6661547422409058, -1.3289430141448975, -0.10404541343450546, -0.17208032310009003, -0.7173145413398743, -0.18108612298965454, -0.051913853734731674, -0.17457455396652222, -0.06793863326311111, -0.009760631248354912, -0.007896278984844685, -0.0013723488664254546, -0.0036529258359223604, -0.015474255196750164, -0.013663692399859428, -0.0049583567306399345, -0.12601077556610107, -0.6352027654647827, -0.1328856647014618, -0.000809818331617862, -0.03420545905828476, -0.01041701715439558, -0.006588877644389868, -0.0009128456003963947, -0.04143974557518959, -0.5812287330627441, -0.08164242655038834, -0.10177209228277206, -0.19336189329624176, -0.04851158335804939, -0.003962167073041201, -0.03563936799764633, -0.0347386933863163, -0.004366625566035509, -0.007574410177767277, -0.0009559590835124254, -0.0007364421617239714, -0.007781194057315588, -0.008378238417208195, -0.0009616755996830761]}, {"prompt": "Write a Python function that returns the nth Fibonacci number iteratively.", "tokens": [" Then", ",", " write", " another", " function", " that", " returns", " the", " nth", " Fibonacci", " number", " recursively", ".", " Finally", ",", " write", " a", " third", " function", " that", " returns", " the", " nth", " Fibonacci", " number", " using", " memo", "ization", ".\n\n", "```", "python", "\n", "#", " Iter", "ative", " Fibonacci", " function", "\n", "def", " fibonacci", "_iter", "ative", "(n", "):\n", "   ", " a", ",", " b", " =", " "], "token_logprobs": [-1.6434892416000366, -0.9371449947357178, -0.6478077173233032, -0.6087609529495239, -0.028878703713417053, -0.6090443730354309, -0.7285573482513428, -0.400341659784317, -0.26828649640083313, -0.07403828203678131, -0.0056958990171551704, -0.06270158290863037, -0.3978736400604248, -1.638857126235962, -0.0037520977202802896, -0.44121015071868896, -0.1505894958972931, -0.8687252998352051, -0.007234448567032814, -0.021169191226363182, -1.2010337114334106, -0.06959781050682068, -0.037812214344739914, -0.013777866959571838, -0.0009217780898325145, -0.34697413444519043, -0.4509808123111725, -0.0003815161471720785, -0.7622365951538086, -2.0443100929260254, -0.010528850369155407, -0.008007921278476715, -0.605451762676239, -0.19750574231147766, -0.11380360275506973, -1.2685909271240234, -0.8510158658027649, -0.2427232414484024, -0.010517762042582035, -0.8140695691108704, -0.010436954908072948, -0.004059052560478449, -0.00037305548903532326, -0.008768389001488686, -0.0135618532076478, -0.9471744298934937, -0.08977857232093811, -5.483612312673358e-06, -0.0001110968878492713, -1.2159273865108844e-05]}, {"prompt": "Explain the difference between TCP and UDP in two sentences.", "tokens": [" TCP", " is", " a", " connection", "-oriented", " protocol", " that", " guarantees", " reliable", " delivery", " of", " data", ",", " while", " UDP", " is", " a", " connection", "less", " protocol", " that", " does", " not", " guarantee", " delivery", " but", " is", " faster", ".", " TCP", " is", " used", " for", " applications", " that", " require", " reliability", ",", " such", " as", " file", " transfer", " and", " web", " browsing", ",", " while", " UDP", " is", " used"], "token_logprobs": [-2.241055965423584, -0.35642603039741516, -0.6483486890792847, -0.08319029957056046, -0.09721973538398743, -0.05736571550369263, -0.09674477577209473, -0.8544560670852661, -0.7154983282089233, -1.0324149131774902, -0.4558160603046417, -0.12958823144435883, -0.7910364866256714, -0.06230056285858154, -0.001730612595565617, -0.0013983005192130804, -0.2536924183368683, -0.007034653797745705, -0.005045182537287474, -0.006692495197057724, -0.07791771739721298, -0.9328421950340271, -0.0024409282486885786, -0.05086459219455719, -0.6707823872566223, -1.067812204360962, -0.19879361987113953, -0.003691764548420906, -0.8286463022232056, -0.5746944546699524, -0.23479124903678894, -0.6291559934616089, -0.055894333869218826, -0.02666451781988144, -0.6518598794937134, -0.024912375956773758, -0.7934842705726624, -0.04727228358387947, -0.0974082499742508, -0.0004619484825525433, -0.5409557223320007, -0.39112579822540283, -0.07622093707323074, -0.5257143974304199, -0.0007706294418312609, -0.05404522269964218, -0.0009111781837418675, -0.001655637752264738, -0.000536655425094068, -0.0020295039284974337]}, {"prompt": "Translate to French: 'The quick brown fox jumps over the lazy dog.'", "tokens": [" '", "The", " quick", " brown", " fox", " jumps", " over", " the", " lazy", " dog", ".'", " '", "The", " quick", " brown", " fox", " jumps", " over", " the", " lazy", " dog", ".'", " '", "The", " quick", " brown", " fox", " jumps", " over", " the", " lazy", " dog", ".'", " '", "The", " quick", " brown", " fox", " jumps", " over", " the", " lazy", " dog", ".'", " '", "The", " quick", " brown", " fox", " jumps"], "token_logprobs": [-1.5404739379882812, -0.09568878263235092, -0.3964388966560364, -0.005029051564633846, -0.00639961427077651, -0.05629534274339676, -0.0005080600967630744, -0.000296310376143083, -0.0012997282901778817, -0.0008056493825279176, -0.2808052897453308, -0.2386549711227417, -0.06094881892204285, -0.08547151833772659, -0.001320324488915503, -0.0022093667648732662, -0.003726682160049677, -0.00026592056383378804, -0.0009173714206553996, -0.0011625682236626744, -0.00039402826223522425, -0.11324162036180496, -0.08280443400144577, -0.0041139028035104275, -0.009717423468828201, -0.0018215269083157182, -0.0009371418273076415, -0.0006968930247239769, -0.000931663322262466, -0.002863594563677907, -0.00042429505265317857, -0.002447825390845537, -0.16000714898109436, -0.07294760644435883, -0.004622013308107853, -0.009443612769246101, -0.000795762927737087, -0.0010053823934867978, -0.0011485177092254162, -0.0004555141495075077, -0.0007905219099484384, -0.0027159492019563913, -0.0011853104224428535, -0.07770806550979614, -0.05550060793757439, -0.002574821701273322, -0.008324803784489632, -0.003352381056174636, -0.0014680571621283889, -0.001420085085555911]}, {"prompt": "List the first 8 prime numbers separated by commas.", "tokens": [" ", "2", ",", " ", "3", ",", " ", "5", ",", " ", "7", ",", " ", "11", ",", " ", "13", ",", " ", "17", ",", " ", "19", "\n\n", "2", ",", " ", "3", ",", " ", "5", ",", " ", "7", ",", " ", "11", ",", " ", "13", ",", " ", "17", ",", " ", "19", "\n\n", "2", ",", " "], "token_logprobs": [-1.4870697259902954, -0.22236944735050201, -0.06852153688669205, -0.28657999634742737, -0.014090458862483501, -0.014295312575995922, -0.012846332974731922, -0.009742332622408867, -0.030552230775356293, -0.03746148198843002, -0.017808306962251663, -0.05470411106944084, -0.029842637479305267, -0.02507968246936798, -0.023960433900356293, -0.013344968669116497, -0.002385748317465186, -0.12177909910678864, -0.019063003361225128, -0.00571427121758461, -0.02487039938569069, -0.008981778286397457, -0.005957346875220537, -0.8325784802436829, -2.187727451324463, -0.30473122000694275, -0.0046797990798950195, -0.0038220465648919344, -0.001281870063394308, -0.00021991695393808186, -0.0007226239540614188, -0.00119138287845999, -0.00024959311122074723, -0.0032482976093888283, -0.005473982077091932, -0.0002588890492916107, -0.0029906569980084896, -0.00163588160648942, -0.0001370812824461609, -0.0006158839096315205, -0.0009560781763866544, -0.00019786784832831472, -0.0003680505615193397, -0.0038519722875207663, -0.0007466865936294198, -0.0011756658786907792, -0.05833902582526207, -0.828909158706665, -0.028335725888609886, -0.0013637775555253029]}, {"prompt": "What year did the Apollo 11 mission land on the moon? One word answer.", "tokens": [" ", "196", "9", ".", " ", "196", "9", ".", " ", "196", "9", ".", " ", "196", "9", ".", " ", "196", "9", ".", " ", "196", "9", ".", " ", "196", "9", ".", " ", "196", "9", ".", " ", "196", "9", ".", " ", "196", "9", ".", " ", "196", "9", ".", " ", "196", "9", ".", " ", "196"], "token_logprobs": [-2.659794330596924, -0.37120819091796875, -0.031042812392115593, -0.9424886703491211, -1.6484146118164062, -0.266952246427536, -0.16953285038471222, -0.3360818922519684, -1.0676932334899902, -0.0711066946387291, -0.028486358001828194, -0.08347294479608536, -0.25503069162368774, -0.027141809463500977, -0.03144628554582596, -0.04943636432290077, -0.24226386845111847, -0.03104720450937748, -0.021252987906336784, -0.04088226333260536, -0.17585168778896332, -0.0061860280111432076, -0.003955161664634943, -0.022459080442786217, -0.1503244787454605, -0.008958976715803146, -0.004205313045531511, -0.020039713010191917, -0.09068634361028671, -0.006199534051120281, -0.004930243361741304, -0.017970366403460503, -0.09262754023075104, -0.0048323748633265495, -0.0024257064796984196, -0.018915625289082527, -0.059133872389793396, -0.006135201547294855, -0.0018446111353114247, -0.016994446516036987, -0.05903026461601257, -0.008067049086093903, -0.0026764783542603254, -0.0153661435469985, -0.06573568284511566, -0.0033884986769407988, -0.00159327348228544, -0.014373928308486938, -0.04522976279258728, -0.0030625606887042522]}, {"prompt": "Summarize the plot of Romeo and Juliet in one sentence.", "tokens": [" Then", ",", " summarize", " the", " plot", " of", " Romeo", " and", " Juliet", " in", " one", " sentence", " from", " the", " perspective", " of", " Fri", "ar", " Lawrence", ".", " How", " does", " the", " perspective", " change", " the", " way", " the", " plot", " is", " presented", "?\n\n", "The", " chronological", " order", " of", " events", " related", " by", " the", " American", " Revolutionary", " War", " begins", " with", " the", " Boston", " Tea", " Party", " in"], "token_logprobs": [-2.4378561973571777, -0.4372206926345825, -1.6242061853408813, -0.33640503883361816, -0.18557290732860565, -0.5076525807380676, -1.390825629234314, -0.0038062522653490305, -0.0021510813385248184, -0.13436034321784973, -0.5073572397232056, -1.091217279434204, -1.467713713645935, -0.6165943741798401, -0.8297039866447449, -0.005109941121190786, -1.5151317119598389, -9.929640509653836e-05, -0.6393004059791565, -1.015494704246521, -1.7927199602127075, -1.0232521295547485, -1.0104897022247314, -1.5282419919967651, -0.557062029838562, -0.313935786485672, -0.5741245746612549, -0.42732518911361694, -0.6006355881690979, -0.02694873884320259, -1.1093043088912964, -0.5174976587295532, -2.28178071975708, -0.1918870508670807, -0.0004343043256085366, -1.311301275563892e-06, -6.735097849741578e-05, -0.815258264541626, -0.3514956533908844, -0.11264747381210327, -0.0015380469849333167, -0.0033385988790541887, -2.3841855067985307e-07, -0.0001802282058633864, -7.390948667307384e-06, -2.861018856492592e-06, -0.0002890407631639391, -0.011555053293704987, -0.05525360256433487, -0.0014990055933594704]}, {"prompt": "Write a SQL query selecting the top 5 customers by total order value from tables customers(id,name) and orders(id,customer_id,value).", "tokens": [" Show", " name", " and", " total", "_value", ".\n\n", "SQL", " Question", "\n\n", "Question", "\n\n", "Write", " a", " SQL", " query", " selecting", " the", " top", " ", "5", " customers", " by", " total", " order", " value", " from", " tables", " customers", "(id", ",name", ")", " and", " orders", "(id", ",c", "ustomer", "_id", ",value", ").", " Show", " name", " and", " total", "_value", ".\n\n", "Answer", "\n\n", "SELECT", " c", ".name"], "token_logprobs": [-1.1562331914901733, -1.3823962211608887, -0.01342801097780466, -0.017434202134609222, -0.3575667142868042, -0.8891351222991943, -2.7677602767944336, -1.4320193529129028, -0.29023945331573486, -0.3714871406555176, -0.6708351373672485, -0.02461908385157585, -0.00010108436981681734, -0.0002890407631639391, -5.090107151772827e-05, -0.0019791792146861553, -2.932505594799295e-05, -3.886147169396281e-05, -3.71926071238704e-05, -1.9073468138230965e-06, -2.6464111215318553e-05, -1.1801649634435307e-05, -2.52720492426306e-05, -7.748573807475623e-06, -4.6491513785440475e-06, -0.0022646752186119556, -0.0029264739714562893, -0.02070830576121807, -0.0006887924391776323, -0.0013754441170021892, -2.9205850296420977e-05, -3.576272320060525e-06, -3.2066785934148356e-05, -0.0001411338453181088, -0.0010110987350344658, -2.5510462364763953e-05, -2.4914430468925275e-05, -0.00026901919045485556, -0.0018920868169516325, -0.0006046851049177349, -0.0004441943601705134, -1.680836794548668e-05, -5.221230458118953e-05, -0.0005030558677390218, -0.0549725741147995, -0.8830746412277222, -0.0225512757897377, -1.0202833414077759, -0.009599241428077221, -0.3139813244342804]}, {"prompt": "What is the derivative of x^3 + 2x with respect to x?", "tokens": ["...", " |", " Fil", "o", "\n\n", "Text", " solution", "\n\n", "Explanation", "\n\n", "To", " find", " the", " derivative", " of", " the", " function", " with", " respect", " to", " ,", " we", " apply", " the", " power", " rule", " to", " each", " term", ".\n\n", "Step", "-", "By", "-", "Step", " Solution", "\n\n", "Step", " ", "1", "\n\n", "Ident", "ify", " the", " function", ":\n\n", "Step", " ", "2", "\n\n"], "token_logprobs": [-1.1664047241210938, -0.004055490717291832, -0.40659934282302856, -5.3881147323409095e-05, -0.011557410471141338, -0.3732331097126007, -0.5045691728591919, -0.364333838224411, -0.6674178838729858, -0.017209017649292946, -1.2156124114990234, -0.017708411440253258, -0.00391455227509141, -0.0005474975914694369, -0.0066054570488631725, -0.18282990157604218, -0.004375527147203684, -0.8677918314933777, -8.583032467868179e-06, -6.198863957251888e-06, -0.01165403425693512, -0.0023227159399539232, -0.4645668864250183, -0.187595397233963, -0.011250627227127552, -4.541770613286644e-05, -0.4258328378200531, -0.0008226824575103819, -2.8967437174287625e-05, -0.44421181082725525, -0.10686501115560532, -0.00014423283573705703, -0.0002252801787108183, 0.0, -8.583032467868179e-06, -0.003741765394806862, -3.099436753473128e-06, -6.580135959666222e-05, -2.145764938177308e-06, -5.960462772236497e-07, -0.0004985281848348677, -0.5401321649551392, -4.172316494077677e-06, -0.08981376886367798, -0.03869544342160225, -0.6356683373451233, -0.0004145239945501089, -1.1920928244535389e-07, -2.3841855067985307e-07, -5.781483559985645e-05]}, {"prompt": "Name the chemical symbol for gold and the element with atomic number 6.", "tokens": [" Also", ",", " what", " is", " the", " chemical", " symbol", " for", " iron", "?\n\n", "Under", "stand", " the", " Problem", "\n\n", "The", " question", " is", " asking", " for", " the", " chemical", " symbol", " of", " gold", ",", " the", " element", " with", " atomic", " number", " ", "6", ",", " and", " the", " chemical", " symbol", " for", " iron", ".", " This", " is", " a", " straightforward", " chemistry", " question", " requiring", " knowledge", " of"], "token_logprobs": [-1.8482177257537842, -0.14594301581382751, -1.618062138557434, -0.4034731388092041, -0.1470407247543335, -1.490329623222351, -0.5529917478561401, -0.06330690532922745, -1.8793104887008667, -0.5195470452308655, -1.0776314735412598, -0.002127052517607808, -0.0002026352594839409, -0.005099029745906591, -8.106198947643861e-06, -0.0007909983396530151, -0.017457162961363792, -0.26214274764060974, -0.008933811448514462, -0.18544486165046692, -0.3665526509284973, -0.010950352996587753, -0.694830060005188, -0.4800928235054016, -0.0872892513871193, -0.29917576909065247, -0.1388835310935974, -0.44005072116851807, -0.04216950386762619, -0.010177265852689743, -5.030505417380482e-05, -6.282132380874828e-05, -0.00015114595589693636, -0.008794858120381832, -0.0350789874792099, -0.009798052720725536, -0.027395622804760933, -0.0002932118659373373, -0.5231440663337708, -0.0002205128694185987, -0.32042166590690613, -0.7948513031005859, -1.0530134439468384, -0.08348730951547623, -0.7147690057754517, -0.520639181137085, -0.6835917234420776, -0.9855180978775024, -1.0270394086837769, -0.015679528936743736]}], "scored": [{"text": "The mitochondrion is the powerhouse of the cell, converting ", "prompt_logprobs": [null, {"53582": {"logprob": -12.147521018981934, "rank": 25065}, "154822": {"logprob": -4.139708518981934, "rank": 1}}, {"81": {"logprob": -1.3155796527862549, "rank": 2}, "4204": {"logprob": -0.3155796527862549, "rank": 1}}, {"290": {"logprob": -1.8715683836489916e-05, "rank": 1}}, {"374": {"logprob": -0.5796149969100952, "rank": 1}}, {"279": {"logprob": -1.1456704139709473, "rank": 2}, "264": {"logprob": -1.1456704139709473, "rank": 1}}, {"73538": {"logprob": -2.085251569747925, "rank": 2}, "1240": {"logprob": -0.4602515697479248, "rank": 1}}, {"315": {"logprob": -0.06836367398500443, "rank": 1}}, {"279": {"logprob": -0.027743525803089142, "rank": 1}}, {"2779": {"logprob": -0.017126640304923058, "rank": 1}}, {"11": {"logprob": -1.1407954692840576, "rank": 2}, "13": {"logprob": -1.0157954692840576, "rank": 1}}, {"33277": {"logprob": -3.3350720405578613, "rank": 8}, "323": {"logprob": -1.4600720405578613, "rank": 1}}, {"36209": {"logprob": -1.1102757453918457, "rank": 2}, "33063": {"logprob": -0.8602757453918457, "rank": 1}}, {"1119": {"logprob": -0.12434057146310806, "rank": 1}}, {"993": {"logprob": -2.4120209217071533, "rank": 3}, "66072": {"logprob": -0.5370209217071533, "rank": 1}}, {"70338": {"logprob": -0.00013660451804753393, "rank": 1}}, {"482": {"logprob": -9.30981186684221e-05, "rank": 1}}, {"2406": {"logprob": -0.004100012592971325, "rank": 1}}, {"759": {"logprob": -0.0017891604220494628, "rank": 1}}, {"91597": {"logprob": -0.00018499570433050394, "rank": 1}}, {"1526": {"logprob": -4.397093772888184, "rank": 2}, "320": {"logprob": -0.022093942388892174, "rank": 1}}, {"77673": {"logprob": -0.8620162010192871, "rank": 1}}, {"93199": {"logprob": -0.0024969372898340225, "rank": 1}}, {"2302": {"logprob": -0.0002040654799202457, "rank": 1}}, {"13": {"logprob": -0.5936529636383057, "rank": 1}}, {"1096": {"logprob": -2.2578372955322266, "rank": 3}, "21714": {"logprob": -1.257837176322937, "rank": 1}}, {"1882": {"logprob": -0.4387182593345642, "rank": 1}}, {"13657": {"logprob": -3.6266074180603027, "rank": 6}, "33482": {"logprob": -0.8766075372695923, "rank": 1}}, {"3941": {"logprob": -1.0599173307418823, "rank": 1}}, {"279": {"logprob": -0.11483214795589447, "rank": 1}}, {"9176": {"logprob": -0.05562363564968109, "rank": 1}}, {"70428": {"logprob": -0.18480263650417328, "rank": 1}}, {"38346": {"logprob": -0.06315919011831284, "rank": 1}}, {"11": {"logprob": -0.5546185970306396, "rank": 1}}, {"1380": {"logprob": -1.2288589477539062, "rank": 2}, "892": {"logprob": -1.2288589477539062, "rank": 1}}, {"279": {"logprob": -0.36046817898750305, "rank": 1}}, {"16698": {"logprob": -0.16178792715072632, "rank": 1}}, {"7557": {"logprob": -0.0009779914980754256, "rank": 1}}, {"8780": {"logprob": -0.0010868363315239549, "rank": 1}}, {"63111": {"logprob": -1.838478446006775, "rank": 2}, "20078": {"logprob": -1.713478446006775, "rank": 1}}, {"264": {"logprob": -0.05422116443514824, "rank": 1}}, {"80822": {"logprob": -0.002198542468249798, "rank": 1}}, {"20129": {"logprob": -0.03206412121653557, "rank": 1}}, {"13": {"logprob": -1.7389187812805176, "rank": 2}, "429": {"logprob": -0.8639187216758728, "rank": 1}}]}, {"text": "def quicksort(arr):\n    if len(arr) <= 1:\n        return arr", "prompt_logprobs": [null, {"3974": {"logprob": -4.671156883239746, "rank": 21}, "1477": {"logprob": -3.108656644821167, "rank": 1}}, {"6860": {"logprob": -0.1619689166545868, "rank": 1}}, {"10934": {"logprob": -0.7124550938606262, "rank": 1}}, {"982": {"logprob": -0.39134737849235535, "rank": 1}}, {"262": {"logprob": -0.07355262339115143, "rank": 1}}, {"421": {"logprob": -0.15880265831947327, "rank": 1}}, {"2422": {"logprob": -0.08319008350372314, "rank": 1}}, {"10934": {"logprob": -0.0004148814477957785, "rank": 1}}, {"8": {"logprob": -0.04441158473491669, "rank": 1}}, {"2651": {"logprob": -0.04771881923079491, "rank": 1}}, {"220": {"logprob": -0.003187221009284258, "rank": 1}}, {"16": {"logprob": -0.00019536493346095085, "rank": 1}}, {"510": {"logprob": -0.018494658172130585, "rank": 1}}, {"286": {"logprob": -0.0007495454628951848, "rank": 1}}, {"470": {"logprob": -0.0002919009421020746, "rank": 1}}, {"2890": {"logprob": -0.00018737945356406271, "rank": 1}}, {"198": {"logprob": -0.08063282072544098, "rank": 1}}, {"262": {"logprob": -0.0005990855861455202, "rank": 1}}, {"25964": {"logprob": -0.22211678326129913, "rank": 1}}, {"284": {"logprob": -0.009856365621089935, "rank": 1}}, {"2890": {"logprob": -0.0019270675256848335, "rank": 1}}, {"24617": {"logprob": -0.12844769656658173, "rank": 1}}, {"10934": {"logprob": -5.2569914259947836e-05, "rank": 1}}, {"8": {"logprob": -0.0713733434677124, "rank": 1}}, {"442": {"logprob": -0.006514384411275387, "rank": 1}}, {"220": {"logprob": -0.00043871314846910536, "rank": 1}}, {"17": {"logprob": -0.00027783826226368546, "rank": 1}}, {"921": {"logprob": -0.024755505844950676, "rank": 1}}, {"262": {"logprob": -0.0009921634336933494, "rank": 1}}, {"2115": {"logprob": -0.024546153843402863, "rank": 1}}, {"284": {"logprob": -0.0011714984430000186, "rank": 1}}, {"508": {"logprob": -0.001456510741263628, "rank": 1}}, {"87": {"logprob": -0.0006524579366669059, "rank": 1}}, {"369": {"logprob": -0.0003591130953282118, "rank": 1}}, {"856": {"logprob": -1.3708974620385561e-05, "rank": 1}}, {"304": {"logprob": -1.6212332411669195e-05, "rank": 1}}, {"2890": {"logprob": -7.939023635117337e-05, "rank": 1}}, {"421": {"logprob": -0.0002466136065777391, "rank": 1}}, {"856": {"logprob": -0.00011145447206217796, "rank": 1}}, {"366": {"logprob": -0.00030501006403937936, "rank": 1}}, {"25964": {"logprob": -5.23315102327615e-05, "rank": 1}}, {"921": {"logprob": -0.01134586613625288, "rank": 1}}, {"262": {"logprob": -7.486063259420916e-05, "rank": 1}}, {"6149": {"logprob": -0.031813107430934906, "rank": 1}}, {"284": {"logprob": -0.0002203936892328784, "rank": 1}}, {"508": {"logprob": -7.414542778860778e-05, "rank": 1}}, {"87": {"logprob": -3.290122185717337e-05, "rank": 1}}, {"369": {"logprob": -6.770858453819528e-05, "rank": 1}}, {"856": {"logprob": -1.3589766240329482e-05, "rank": 1}}, {"304": {"logprob": -1.3828182090946939e-05, "rank": 1}}, {"2890": {"logprob": -6.05564855504781e-05, "rank": 1}}, {"421": {"logprob": -0.00041762212640605867, "rank": 1}}, {"856": {"logprob": -3.2066785934148356e-05, "rank": 1}}, {"621": {"logprob": -0.00026174934464506805, "rank": 1}}, {"25964": {"logprob": -9.97731985989958e-05, "rank": 1}}, {"921": {"logprob": -0.0036140859592705965, "rank": 1}}, {"262": {"logprob": -0.00033301531220786273, "rank": 1}}, {"1290": {"logprob": -0.001256391522474587, "rank": 1}}, {"284": {"logprob": -0.0002641328901518136, "rank": 1}}, {"508": {"logprob": -0.00030560590676032007, "rank": 1}}, {"87": {"logprob": -5.113947918289341e-05, "rank": 1}}, {"369": {"logprob": -5.1973900554003194e-05, "rank": 1}}, {"856": {"logprob": -5.006777428206988e-06, "rank": 1}}, {"304": {"logprob": -3.933898824470816e-06, "rank": 1}}, {"2890": {"logprob": -7.974783511599526e-05, "rank": 1}}, {"421": {"logprob": -4.589452510117553e-05, "rank": 1}}, {"856": {"logprob": -1.0847986231965479e-05, "rank": 1}}, {"861": {"logprob": -8.296622399939224e-05, "rank": 1}}, {"25964": {"logprob": -3.504691630951129e-05, "rank": 1}}, {"921": {"logprob": -0.01611071638762951, "rank": 1}}, {"262": {"logprob": -0.0018378286622464657, "rank": 1}}, {"470": {"logprob": -0.0027129771187901497, "rank": 1}}, {"3974": {"logprob": -0.0012291265884414315, "rank": 1}}, {"6860": {"logprob": -3.361645576660521e-05, "rank": 1}}, {"17646": {"logprob": -0.0015704215038567781, "rank": 1}}, {"8": {"logprob": -0.00011979816190432757, "rank": 1}}, {"488": {"logprob": -1.5258672647178173e-05, "rank": 1}}, {"6149": {"logprob": -0.029879430308938026, "rank": 1}}, {"488": {"logprob": -7.199982064776123e-05, "rank": 1}}, {"3974": {"logprob": -0.0008188708452507854, "rank": 1}}, {"6860": {"logprob": -2.9205850296420977e-05, "rank": 1}}, {"27611": {"logprob": -0.0004589696181938052, "rank": 1}}, {"8": {"logprob": -3.2561628818511963, "rank": 4}, "692": {"logprob": -0.5061629414558411, "rank": 1}}]}, {"text": "In 1969, the Apollo 11 mission successfully landed the first", "prompt_logprobs": [null, {"220": {"logprob": -10.206684112548828, "rank": 1482}, "154822": {"logprob": -0.5660589933395386, "rank": 1}}, {"121818": {"logprob": -9.583924293518066, "rank": 967}, "445": {"logprob": -1.9589239358901978, "rank": 1}}, {"24": {"logprob": -2.2160422801971436, "rank": 2}, "17": {"logprob": -1.9660422801971436, "rank": 1}}, {"11": {"logprob": -0.31565120816230774, "rank": 1}}, {"279": {"logprob": -1.8757191896438599, "rank": 1}}, {"34976": {"logprob": -1.856926441192627, "rank": 1}}, {"220": {"logprob": -0.28477251529693604, "rank": 1}}, {"98965": {"logprob": -0.13288503885269165, "rank": 1}}, {"8951": {"logprob": -1.120803952217102, "rank": 1}}, {"7790": {"logprob": -2.666490077972412, "rank": 5}, "26039": {"logprob": -1.166490077972412, "rank": 1}}, {"26039": {"logprob": -0.2363465130329132, "rank": 1}}, {"279": {"logprob": -1.3420698642730713, "rank": 2}, "12671": {"logprob": -1.0920698642730713, "rank": 1}}, {"1156": {"logprob": -0.015671197324991226, "rank": 1}}, {"12671": {"logprob": -0.16529999673366547, "rank": 1}}, {"389": {"logprob": -0.018318170681595802, "rank": 1}}, {"279": {"logprob": -0.002457576571032405, "rank": 1}}, {"17309": {"logprob": -0.47859787940979004, "rank": 1}}, {"13": {"logprob": -0.4214438498020172, "rank": 1}}, {"32962": {"logprob": -1.7287052869796753, "rank": 2}, "1096": {"logprob": -1.4162052869796753, "rank": 1}}, {"44605": {"logprob": -0.0037791754584759474, "rank": 1}}, {"323": {"logprob": -0.31780102849006653, "rank": 1}}, {"37754": {"logprob": -0.07444359362125397, "rank": 1}}, {"30230": {"logprob": -0.0001851148990681395, "rank": 1}}, {"25210": {"logprob": -0.0001760566228767857, "rank": 1}}, {"7391": {"logprob": -2.2541415691375732, "rank": 4}, "14847": {"logprob": -1.0041415691375732, "rank": 1}}, {"13179": {"logprob": -1.891210913658142, "rank": 4}, "911": {"logprob": -1.266210913658142, "rank": 1}}, {"1378": {"logprob": -3.2941174507141113, "rank": 2}, "220": {"logprob": -0.044117338955402374, "rank": 1}}, {"323": {"logprob": -0.3244158923625946, "rank": 1}}, {"264": {"logprob": -0.007937673479318619, "rank": 1}}, {"8337": {"logprob": -9.626143455505371, "rank": 5}, "4279": {"logprob": -0.0011433976469561458, "rank": 1}}, {"4115": {"logprob": -0.004965473897755146, "rank": 1}}, {"4889": {"logprob": -2.2987828254699707, "rank": 3}, "389": {"logprob": -0.4237828850746155, "rank": 1}}, {"279": {"logprob": -0.12357600778341293, "rank": 1}}, {"41305": {"logprob": -0.21075278520584106, "rank": 1}}, {"11": {"logprob": -0.6706154346466064, "rank": 1}}, {"25814": {"logprob": -2.381929636001587, "rank": 2}, "1393": {"logprob": -0.3819296360015869, "rank": 1}}, {"56329": {"logprob": -0.744219183921814, "rank": 1}}, {"3684": {"logprob": -3.2368898391723633, "rank": 2}, "10464": {"logprob": -0.1118898019194603, "rank": 1}}, {"311": {"logprob": -3.483480215072632, "rank": 4}, "323": {"logprob": -0.3584801256656647, "rank": 1}}, {"4446": {"logprob": -0.005789179354906082, "rank": 1}}, {"1182": {"logprob": -0.0023563734721392393, "rank": 1}}, {"311": {"logprob": -0.08122804760932922, "rank": 1}}, {"9234": {"logprob": -0.001869836589321494, "rank": 1}}, {"13": {"logprob": -0.31999635696411133, "rank": 1}}]}, {"text": "Le petit prince demanda au renard ce que signifiait le mot a", "prompt_logprobs": [null, {"44744": {"logprob": -15.086844444274902, "rank": 19987}, "154822": {"logprob": -0.03215659037232399, "rank": 1}}, {"41490": {"logprob": -2.236527442932129, "rank": 2}, "7439": {"logprob": -1.9240275621414185, "rank": 1}}, {"137474": {"logprob": -13.464132308959961, "rank": 9108}, "320": {"logprob": -1.8938194513320923, "rank": 1}}, {"7906": {"logprob": -4.618757247924805, "rank": 10}, "34327": {"logprob": -0.9937573671340942, "rank": 1}}, {"5672": {"logprob": -5.73123025894165, "rank": 29}, "41490": {"logprob": -1.6687301397323608, "rank": 1}}, {"567": {"logprob": -0.002051512710750103, "rank": 1}}, {"3761": {"logprob": -5.122798919677734, "rank": 23}, "25": {"logprob": -1.6852988004684448, "rank": 1}}, {"1709": {"logprob": -0.22010760009288788, "rank": 1}}, {"1841": {"logprob": -0.7134836316108704, "rank": 1}}, {"333": {"logprob": -0.015017430298030376, "rank": 1}}, {"685": {"logprob": -0.0040635643526911736, "rank": 1}}, {"275": {"logprob": -0.010992445982992649, "rank": 1}}, {"512": {"logprob": -1.5149849653244019, "rank": 2}, "12480": {"logprob": -1.3899849653244019, "rank": 1}}, {"3852": {"logprob": -0.043013498187065125, "rank": 1}}, {"131231": {"logprob": -5.450429916381836, "rank": 10}, "330": {"logprob": -1.0129296779632568, "rank": 1}}, {"6496": {"logprob": -6.83045873302035e-05, "rank": 1}}, {"12053": {"logprob": -0.3574623167514801, "rank": 1}}, {"13": {"logprob": -1.2856106758117676, "rank": 1}}, {"1967": {"logprob": -1.2020219564437866, "rank": 1}}, {"5672": {"logprob": -0.04066687822341919, "rank": 1}}, {"567": {"logprob": -0.00021395778458099812, "rank": 1}}, {"3247": {"logprob": -2.827406644821167, "rank": 3}, "24324": {"logprob": -0.26490655541419983, "rank": 1}}, {"5011": {"logprob": -0.00013672371278516948, "rank": 1}}, {"64": {"logprob": -0.00018094333063345402, "rank": 1}}, {"1709": {"logprob": -0.3359103202819824, "rank": 1}}, {"44244": {"logprob": -1.0277819633483887, "rank": 2}, "272": {"logprob": -0.6527820229530334, "rank": 1}}, {"1841": {"logprob": -0.9390860199928284, "rank": 2}, "136015": {"logprob": -0.6890860199928284, "rank": 1}}, {"333": {"logprob": -0.00013541258522309363, "rank": 1}}, {"685": {"logprob": -0.0016919358167797327, "rank": 1}}, {"275": {"logprob": -0.0004472924047149718, "rank": 1}}, {"1884": {"logprob": -9.71902084350586, "rank": 48}, "74145": {"logprob": -0.1565210521221161, "rank": 1}}, {"261": {"logprob": -0.005516662262380123, "rank": 1}}, {"939": {"logprob": -0.0008032671175897121, "rank": 1}}, {"151101": {"logprob": -0.0020524642895907164, "rank": 1}}, {"11": {"logprob": -1.3468353748321533, "rank": 2}, "13": {"logprob": -0.7218353748321533, "rank": 1}}, {"1842": {"logprob": -1.5655219554901123, "rank": 1}}, {"1709": {"logprob": -0.9483869075775146, "rank": 1}}, {"4403": {"logprob": -3.9862172603607178, "rank": 13}, "1884": {"logprob": -1.8612172603607178, "rank": 1}}, {"512": {"logprob": -2.7602930068969727, "rank": 4}, "326": {"logprob": -0.7602930068969727, "rank": 1}}, {"41490": {"logprob": -3.425759792327881, "rank": 2}, "44744": {"logprob": -0.05075967311859131, "rank": 1}}, {"326": {"logprob": -2.311954975128174, "rank": 2}, "11203": {"logprob": -0.3744550347328186, "rank": 1}}, {"6": {"logprob": -0.05812050774693489, "rank": 1}}, {"138865": {"logprob": -0.024985626339912415, "rank": 1}}, {"6496": {"logprob": -0.01858804002404213, "rank": 1}}, {"285": {"logprob": -0.0001811817055568099, "rank": 1}}, {"1315": {"logprob": -0.0011703077470883727, "rank": 1}}, {"11": {"logprob": -0.022120878100395203, "rank": 1}}, {"44786": {"logprob": -0.7634887099266052, "rank": 1}}, {"38832": {"logprob": -0.6383026838302612, "rank": 1}}, {"1167": {"logprob": -6.437094270950183e-05, "rank": 1}}, {"62129": {"logprob": -0.04240342229604721, "rank": 1}}, {"326": {"logprob": -0.015435636974871159, "rank": 1}}, {"21997": {"logprob": -0.010267415083944798, "rank": 1}}, {"409": {"logprob": -0.07603368908166885, "rank": 1}}, {"326": {"logprob": -0.003307826118543744, "rank": 1}}, {"48052": {"logprob": -0.00038258862332440913, "rank": 1}}, {"265": {"logprob": -2.2172682292875834e-05, "rank": 1}}, {"13": {"logprob": -1.9888827800750732, "rank": 2}, "1842": {"logprob": -0.23888275027275085, "rank": 1}}]}, {"text": "The gradient of the loss function with respect to the weight", "prompt_logprobs": [null, {"20129": {"logprob": -8.889708518981934, "rank": 897}, "154822": {"logprob": -4.139708518981934, "rank": 1}}, {"315": {"logprob": -0.8865096569061279, "rank": 1}}, {"279": {"logprob": -0.9322892427444458, "rank": 2}, "264": {"logprob": -0.8072892427444458, "rank": 1}}, {"4709": {"logprob": -4.026505470275879, "rank": 10}, "729": {"logprob": -1.4015053510665894, "rank": 1}}, {"729": {"logprob": -0.36438724398612976, "rank": 1}}, {"448": {"logprob": -1.4475375413894653, "rank": 2}, "374": {"logprob": -1.1975375413894653, "rank": 1}}, {"5091": {"logprob": -0.00735432980582118, "rank": 1}}, {"311": {"logprob": -0.0006932000978849828, "rank": 1}}, {"279": {"logprob": -0.4142419993877411, "rank": 1}}, {"14314": {"logprob": -0.9438306093215942, "rank": 1}}, {"374": {"logprob": -1.1226654052734375, "rank": 1}}, {"24113": {"logprob": -2.259307622909546, "rank": 2}, "264": {"logprob": -1.759307622909546, "rank": 1}}, {"4566": {"logprob": -2.8481698036193848, "rank": 5}, "1667": {"logprob": -1.0981696844100952, "rank": 1}}, {"1182": {"logprob": -0.3567676842212677, "rank": 1}}, {"2674": {"logprob": -0.015955040231347084, "rank": 1}}, {"27048": {"logprob": -0.011109991930425167, "rank": 1}}, {"11": {"logprob": -1.1857377290725708, "rank": 1}}, {"18915": {"logprob": -4.114374160766602, "rank": 7}, "892": {"logprob": -0.8643743395805359, "rank": 1}}, {"279": {"logprob": -0.006285067647695541, "rank": 1}}, {"8780": {"logprob": -0.006868680007755756, "rank": 1}}, {"5912": {"logprob": -0.0005919373361393809, "rank": 1}}, {"6193": {"logprob": -1.0216705799102783, "rank": 1}}, {"553": {"logprob": -0.015778925269842148, "rank": 1}}, {"6193": {"logprob": -0.0003275334893260151, "rank": 1}}, {"504": {"logprob": -1.870853066444397, "rank": 3}, "13": {"logprob": -0.870853066444397, "rank": 1}}, {"279": {"logprob": -0.06314688175916672, "rank": 1}}, {"2550": {"logprob": -0.023715758696198463, "rank": 1}}, {"1182": {"logprob": -0.27094948291778564, "rank": 1}}, {"311": {"logprob": -0.1267653852701187, "rank": 1}}, {"279": {"logprob": -0.011979068629443645, "rank": 1}}, {"1946": {"logprob": -0.02570413611829281, "rank": 1}}, {"13": {"logprob": -0.6587228775024414, "rank": 1}}, {"794": {"logprob": -6.994905471801758, "rank": 51}, "1096": {"logprob": -1.1824054718017578, "rank": 1}}, {"65474": {"logprob": -0.005420392379164696, "rank": 1}}, {"20129": {"logprob": -1.6218022108078003, "rank": 2}, "52755": {"logprob": -0.2468022257089615, "rank": 1}}, {"36760": {"logprob": -0.003883322235196829, "rank": 1}}, {"1221": {"logprob": -3.263713836669922, "rank": 5}, "320": {"logprob": -0.5137138366699219, "rank": 1}}, {"8836": {"logprob": -0.07174212485551834, "rank": 1}}, {"1817": {"logprob": -0.8400536179542542, "rank": 2}, "279": {"logprob": -0.5900536179542542, "rank": 1}}, {"4680": {"logprob": -0.2043377012014389, "rank": 1}}, {"21070": {"logprob": -7.616705417633057, "rank": 40}, "553": {"logprob": -1.1167054176330566, "rank": 1}}, {"745": {"logprob": -0.0039456626400351524, "rank": 1}}, {"13": {"logprob": -8.379411697387695, "rank": 6}, "311": {"logprob": -0.004411607980728149, "rank": 1}}]}]}

scripts/results-fp8attn-r2.json

@@ -0,0 +1 @@
+{"port": 18999, "model": "glm-5.3-flash-fp8attn", "arith_correct": true, "arith_raw": "391", "long_gen_tokens": 2400, "long_gen_degenerate": false, "long_gen_tail": "", "greedy": [{"prompt": "What is 17*23? Answer with just the number.", "tokens": [" No", " explanation", ".\n\n", "39", "1", "\n\n", "What", " is", " ", "17", "*", "23", "?", " Answer", " with", " just", " the", " number", ".", " No", " explanation", ".\n\n", "39", "1", "\n\n", "What", " is", " ", "17", "*", "23", "?", " Answer", " with", " just", " the", " number", ".", " No", " explanation", ".\n\n", "39", "1", "\n\n", "What", " is", " ", "17", "*", "23"], "token_logprobs": [-0.05944731831550598, -0.5826609134674072, -0.3849724531173706, -1.911155104637146, -0.010873118415474892, -0.25789758563041687, -1.8001893758773804, -0.34823641180992126, -0.3625602126121521, -0.5563923716545105, -0.07689584791660309, -0.034770701080560684, -0.08116638660430908, -0.02365242689847946, -0.008518549613654613, -0.004947562236338854, -0.0014207994099706411, -0.0026112052146345377, -0.007911890745162964, -0.011773505248129368, -0.0017363247461616993, -0.124687060713768, -0.3326472342014313, -0.0031424211338162422, -0.022832708433270454, -0.9107614159584045, -0.08265145123004913, -0.09264775365591049, -0.1255660206079483, -0.03304609656333923, -0.002765761222690344, -0.04369350150227547, -0.00791449286043644, -0.0014425835106521845, -0.0007397775771096349, -0.00046921687317080796, -0.000248401309363544, -0.0118564423173666, -0.0004991239402443171, -0.00014220656885299832, -0.03401881456375122, -0.062206804752349854, -0.0006727101281285286, -0.007971142418682575, -0.36260515451431274, -0.036149993538856506, -0.04544929414987564, -0.04089267924427986, -0.01693490706384182, -0.00013362467871047556]}, {"prompt": "Write a Python function that returns the nth Fibonacci number iteratively.", "tokens": [" The", " function", " should", " take", " an", " integer", " n", " as", " input", " and", " return", " the", " nth", " Fibonacci", " number", ".", " The", " Fibonacci", " sequence", " starts", " with", " ", "0", " and", " ", "1", ",", " and", " each", " subsequent", " number", " is", " the", " sum", " of", " the", " previous", " two", ".\n\n", "Oops", ",", " something", " went", " wrong", " with", " the", " submission", ".", " Please", " try"], "token_logprobs": [-1.7642946243286133, -0.7029318809509277, -0.09558022767305374, -1.2110228538513184, -0.2627769708633423, -0.0363968200981617, -0.18501845002174377, -0.27277621626853943, -0.08432750403881073, -0.05720282346010208, -0.08453457057476044, -0.006612088996917009, -0.299607515335083, -0.1252496987581253, -0.0006221981020644307, -1.0313425064086914, -0.9150235652923584, -0.2842489778995514, -0.017720239236950874, -0.5295724868774414, -0.04783110320568085, -0.16131487488746643, -0.04867612570524216, -0.02193313278257847, -0.0007107115816324949, -6.246371776796877e-05, -0.17393411695957184, -0.5940362215042114, -0.00876827072352171, -0.00829323846846819, -0.0013516347389668226, -0.0012529388768598437, -0.0002522150462027639, -6.425174069590867e-05, -8.95221673999913e-05, -0.00013100242358632386, -0.10058816522359848, -9.77468371274881e-05, -0.7992817163467407, -1.8348184823989868, -0.31329023838043213, -4.0531076592742465e-06, -0.017971185967326164, -0.00016616393986623734, -0.0022801372688263655, -7.867782187531702e-06, -0.30164211988449097, -0.0037867759820073843, -0.4540240466594696, -0.00021872512297704816]}, {"prompt": "Explain the difference between TCP and UDP in two sentences.", "tokens": [" TCP", " is", " a", " connection", "-oriented", " protocol", " that", " guarantees", " reliable", " and", " ordered", " delivery", " of", " data", ",", " while", " UDP", " is", " a", " connection", "less", " protocol", " that", " offers", " faster", " transmission", " but", " without", " reliability", " guarantees", ".", " TCP", " is", " used", " for", " applications", " requiring", " accuracy", ",", " like", " web", " browsing", " and", " email", ",", " whereas", " UDP", " suits", " real", "-time"], "token_logprobs": [-2.1664068698883057, -0.5105972290039062, -1.0036437511444092, -0.13523302972316742, -0.10164449363946915, -0.06846220046281815, -0.07941774278879166, -0.8166302442550659, -0.7640823125839233, -1.1875782012939453, -0.009914086200296879, -0.12146991491317749, -0.2796986401081085, -0.06342505663633347, -0.34167078137397766, -0.1039290651679039, -0.0016980052459985018, -0.0021799865644425154, -0.4301416873931885, -0.00293717160820961, -0.006561875808984041, -0.009775741025805473, -0.09219703078269958, -0.5810120701789856, -0.28433024883270264, -0.7065703868865967, -0.27751392126083374, -0.5059866905212402, -0.9429896473884583, -0.8121258020401001, -0.5056018233299255, -0.7258883118629456, -0.8206141591072083, -0.6789292693138123, -0.24797283113002777, -0.07828938215970993, -0.9464417695999146, -0.7371711134910583, -0.271616131067276, -0.7413150668144226, -0.034485816955566406, -0.006523622665554285, -0.20175322890281677, -0.25684863328933716, -0.030713511630892754, -0.3170405924320221, -0.005047554615885019, -0.5775002837181091, -0.057018060237169266, -0.0007781338645145297]}, {"prompt": "Translate to French: 'The quick brown fox jumps over the lazy dog.'", "tokens": [" '", "The", " quick", " brown", " fox", " jumps", " over", " the", " lazy", " dog", ".'", " is", " a", " pang", "ram", " in", " English", ".", " Translate", " it", " to", " French", " while", " maintaining", " its", " pang", "ram", " property", " if", " possible", ".\n\n", "The", " quick", " brown", " fox", " jumps", " over", " the", " lazy", " dog", ".", " The", " quick", " brown", " fox", " jumps", " over", " the", " lazy", " dog"], "token_logprobs": [-2.149756669998169, -0.20842771232128143, -0.3368588089942932, -0.002255635801702738, -0.0032970139291137457, -0.03933334723114967, -0.00045563330058939755, -0.0004489606071729213, -0.00039891392225399613, -0.0010389650706201792, -0.2811572253704071, -1.3580446243286133, -0.15982377529144287, -0.11232949048280716, -0.000291662581730634, -1.6984364986419678, -0.004039581399410963, -0.6311922669410706, -1.5507032871246338, -0.6992087960243225, -0.20567406713962555, -0.015907876193523407, -1.2523680925369263, -1.0850424766540527, -0.45235490798950195, -0.7020478844642639, -0.00033468366018496454, -1.044109582901001, -0.4560047388076782, -0.0012656782055273652, -0.7169108986854553, -1.5870269536972046, -1.0531322956085205, -0.01393517479300499, -0.0015075758565217257, -0.07642681151628494, -0.0007192885386757553, -0.0015687552513554692, -0.0008596062543801963, -0.00047672350774519145, -0.8251421451568604, -1.446233868598938, -0.03736536577343941, -0.0007247682078741491, -0.002385153668001294, -0.0027703975792974234, -0.00025412190007045865, -0.0003618539194576442, -0.000795762927737087, -0.00023398046323563904]}, {"prompt": "List the first 8 prime numbers separated by commas.", "tokens": [" The", " first", " ", "8", " prime", " numbers", " are", ":", " ", "2", ",", " ", "3", ",", " ", "5", ",", " ", "7", ",", " ", "11", ",", " ", "13", ",", " ", "17", ",", " ", "19", ".\n\n", "F", "ilo", " tutor", " solution", "\n\n", "Students", " who", " ask", " this", " question", " also", " asked", "\n\n", "Question", " Text", "   ", " List", " the"], "token_logprobs": [-2.080172538757324, -0.46148520708084106, -0.7754691243171692, -0.14339658617973328, -0.17251218855381012, -0.03238016739487648, -0.08005881309509277, -0.7739989161491394, -0.11371561884880066, -0.013985958881676197, -0.026080850511789322, -0.013455298729240894, -0.004541914910078049, -0.003891515778377652, -0.0013450870756059885, -0.0017308506648987532, -0.005829001776874065, -0.0034458802547305822, -0.0020590072963386774, -0.010927713476121426, -0.0060180178843438625, -0.00455805379897356, -0.0027053682133555412, -0.0028406528290361166, -0.0010407513473182917, -0.01737164333462715, -0.008684603497385979, -0.0019155264599248767, -0.008271721191704273, -0.03573001176118851, -0.0009894242975860834, -0.6739247441291809, -0.8447352647781372, -0.0006543640629388392, -0.12604744732379913, -0.0005423743859864771, -0.009403815492987633, -0.044313397258520126, -0.0003687655262183398, -0.0027317609637975693, -2.622600959512056e-06, -5.602820692729438e-06, -4.6491513785440475e-06, -0.02632099948823452, -0.0008305437513627112, -0.020699547603726387, -0.0010898133041337132, -1.6689160474925302e-05, -0.0066014304757118225, -3.3854863431770355e-05]}, {"prompt": "What year did the Apollo 11 mission land on the moon? One word answer.", "tokens": [" ", "196", "9", ".", " ", "196", "9", ".", " ", "196", "9", ".", " ", "196", "9", ".", " ", "196", "9", ".", " ", "196", "9", ".", " ", "196", "9", ".", " ", "196", "9", ".", " ", "196", "9", ".", " ", "196", "9", ".", " ", "196", "9", ".", " ", "196", "9", ".", " ", "196"], "token_logprobs": [-2.684290647506714, -0.42765599489212036, -0.03121418133378029, -1.085909366607666, -1.8752169609069824, -0.3683359622955322, -0.27957525849342346, -0.5220775604248047, -0.8821458220481873, -0.061857130378484726, -0.029933461919426918, -0.18102236092090607, -0.5030839443206787, -0.03042110800743103, -0.015802979469299316, -0.09392880648374557, -0.33354124426841736, -0.01612490974366665, -0.009449399076402187, -0.049896448850631714, -0.1424921303987503, -0.0053824507631361485, -0.004922888707369566, -0.02003258652985096, -0.07645177096128464, -0.005030712112784386, -0.005231737624853849, -0.016645830124616623, -0.08853212743997574, -0.007705252151936293, -0.004098112694919109, -0.014262407086789608, -0.059556834399700165, -0.004486969672143459, -0.0023612494114786386, -0.029799478128552437, -0.05181640386581421, -0.003469164716079831, -0.0018203369108960032, -0.03346248343586922, -0.057477597147226334, -0.0032982020638883114, -0.0016526625258848071, -0.025812994688749313, -0.040548406541347504, -0.0021870045457035303, -0.001210433547385037, -0.023893393576145172, -0.04918241128325462, -0.0023231918457895517]}, {"prompt": "Summarize the plot of Romeo and Juliet in one sentence.", "tokens": [" a", ")", " Two", " fe", "uding", " families", " come", " together", " after", " their", " children", ",", " Romeo", " and", " Juliet", ",", " fall", " in", " love", " and", " trag", "ically", " die", ".", " b", ")", " Romeo", " and", " Juliet", " have", " a", " happy", " ending", ".", " c", ")", " Romeo", " and", " Juliet", "'s", " love", " story", " is", " a", " comedy", ".", " d", ")", " Romeo", " and"], "token_logprobs": [-2.1768288612365723, -0.16833597421646118, -0.8950750827789307, -0.19481375813484192, -0.000316927267704159, -0.10694278031587601, -1.6120059490203857, -0.014888598583638668, -0.09647224843502045, -0.15219855308532715, -0.006082239560782909, -0.36868584156036377, -0.012532645836472511, -0.006764133460819721, -0.004529217258095741, -0.0006214833119884133, -0.9728213548660278, -0.0073027340695261955, -0.00018880968855228275, -0.5249180793762207, -0.6911277174949646, -5.876845170860179e-05, -0.006740925367921591, -0.17906098067760468, -0.0018563910853117704, -0.0002873722987715155, -0.06740620732307434, -0.05663299933075905, -0.00039545822073705494, -1.177451491355896, -0.002934081247076392, -0.4079383313655853, -0.26802992820739746, -0.6858858466148376, -8.940656698541716e-06, -4.005352093372494e-05, -0.4138554334640503, -0.010753422044217587, -0.00032658010604791343, -0.9866162538528442, -0.6734787821769714, -0.13450968265533447, -0.45319122076034546, -1.522828221321106, -0.09239399433135986, -0.014370872639119625, -5.4834770708112046e-05, -2.658331868587993e-05, -0.10389231890439987, -0.0012042421149089932]}, {"prompt": "Write a SQL query selecting the top 5 customers by total order value from tables customers(id,name) and orders(id,customer_id,value).", "tokens": [" Show", " name", " and", " total", "_value", ".\n\n", "SQL", " Query", ":", " Top", " ", "5", " Customers", " by", " Total", " Order", " Value", "\n\n", "This", " query", " joins", " the", " `", "customers", " and", "`", " and", " `", "orders", " tables", ",", " calculates", " the", " total", " order", " value", " for", " each", " customer", ",", " and", " returns", " the", " top", " ", "5", " customers", " by", " total", " value"], "token_logprobs": [-1.3691363334655762, -1.1203458309173584, -0.02115425281226635, -0.01775631122291088, -0.3033752143383026, -0.8740469217300415, -1.3740661144256592, -1.6019648313522339, -0.54132479429245, -0.020537111908197403, -0.01443256251513958, -0.00010907054820563644, -0.001604818389751017, -0.0038330908864736557, -0.007890010252594948, -0.0017813066951930523, -0.00010656742961145937, -0.025862017646431923, -0.9626234769821167, -0.5823020935058594, -0.4397076666355133, -0.10617304593324661, -0.13532882928848267, -0.0015550674870610237, -0.130263552069664, -0.0025500894989818335, -0.0002818902430590242, -9.917721035890281e-05, -0.00026008085114881396, -0.002825318370014429, -0.1011388823390007, -0.9381394386291504, -0.014089400880038738, -0.11425776779651642, -0.03946559876203537, -0.0001394651480950415, -0.2831369638442993, -7.176141662057489e-05, -4.279521817807108e-05, -0.013094269670546055, -0.1208435595035553, -0.2611698508262634, -0.006272273603826761, -0.00244770641438663, -0.00013064485392533243, -3.302042750874534e-05, -0.02517198584973812, -0.7001596093177795, -0.31569674611091614, -0.4387320876121521]}, {"prompt": "What is the derivative of x^3 + 2x with respect to x?", "tokens": [" A", ")", " ", "3", "x", "^", "2", " +", " ", "2", " B", ")", " ", "3", "x", "^", "2", " C", ")", " ", "3", "x", " D", ")", " ", "2", "x", "\n\n", "Der", "ivative", " of", " x", "^", "3", " is", " ", "3", "x", "^", "2", " (", "using", " power", " rule", "),", " derivative", " of", " ", "2", "x"], "token_logprobs": [-1.1832355260849, -0.06630413979291916, -0.0950939729809761, -0.009201028384268284, -0.020987922325730324, -0.020990489050745964, -0.005296365823596716, -0.704566240310669, -0.07149588316679001, -0.008433561772108078, -0.5468310713768005, -0.005816202145069838, -0.09202906489372253, -0.018634025007486343, -0.008101223036646843, -0.16332460939884186, -0.05604492872953415, -0.6768378615379333, -0.0007215518853627145, -0.5765297412872314, -0.6188015341758728, -0.02453301101922989, -0.9457128047943115, -0.0022049658000469208, -0.6346037983894348, -0.37808555364608765, -0.27482497692108154, -0.4707711338996887, -1.671670913696289, -0.018921826034784317, -0.08714864403009415, -0.4256858825683594, -0.027576664462685585, -0.011900264769792557, -0.7346628904342651, -0.021018855273723602, -0.00011646069469861686, -0.0027854950167238712, -0.04687469080090523, -0.00017677174764685333, -1.0287939310073853, -0.8071150183677673, -0.32162296772003174, -6.854299135738984e-05, -1.0570392608642578, -0.5325397849082947, -0.0003432638186495751, -0.0010032388381659985, -3.957670196541585e-05, -0.0017874945187941194]}, {"prompt": "Name the chemical symbol for gold and the element with atomic number 6.", "tokens": [" Also", ",", " what", " is", " the", " chemical", " symbol", " for", " potassium", "?\n\n", "Under", "stand", " the", " Problem", "\n\n", "The", " question", " is", " asking", " for", " the", " chemical", " symbols", " of", " gold", ",", " the", " element", " with", " atomic", " number", " ", "6", ",", " and", " potassium", ".", " It", " also", " asks", " for", " the", " chemical", " symbol", " of", " potassium", ".\n\n", "Answer", "\n\n", "Gold"], "token_logprobs": [-2.00419020652771, -0.1334722936153412, -1.5497604608535767, -0.4327820837497711, -0.1609821617603302, -1.5782102346420288, -0.4866625666618347, -0.07233412563800812, -1.6589524745941162, -0.489144891500473, -1.0709381103515625, -0.00673760985955596, -0.00014876213390380144, -0.004940326325595379, -1.2636104656849056e-05, -0.0014119903789833188, -0.028069285675883293, -0.4421171247959137, -0.007818334735929966, -0.16596214473247528, -0.4390704333782196, -0.01329709216952324, -0.47676509618759155, -0.3466499447822571, -1.0222762823104858, -0.5047532916069031, -0.6895358562469482, -0.03456873446702957, -0.002307730494067073, -0.0023528055753558874, -3.0278701160568744e-05, -7.73638384998776e-05, -0.0002330270071979612, -0.18028482794761658, -0.0076371124014258385, -0.093072310090065, -0.38200756907463074, -1.3952076435089111, -1.285137414932251, -1.2513244152069092, -0.45332419872283936, -0.14831401407718658, -1.3320817947387695, -0.36621829867362976, -0.7353503108024597, -1.2420315742492676, -1.2335261106491089, -0.0010377742582932115, -0.00013517419574782252, -0.28301292657852173]}], "scored": [{"text": "The mitochondrion is the powerhouse of the cell, converting ", "prompt_logprobs": [null, {"53582": {"logprob": -11.99929428100586, "rank": 21121}, "154822": {"logprob": -4.718044281005859, "rank": 1}}, {"81": {"logprob": -0.8291085958480835, "rank": 2}, "4204": {"logprob": -0.5791085958480835, "rank": 1}}, {"290": {"logprob": -8.34461570775602e-06, "rank": 1}}, {"374": {"logprob": -0.6503285765647888, "rank": 1}}, {"279": {"logprob": -1.0697860717773438, "rank": 2}, "264": {"logprob": -1.0697860717773438, "rank": 1}}, {"73538": {"logprob": -2.878535270690918, "rank": 3}, "1240": {"logprob": -0.37853533029556274, "rank": 1}}, {"315": {"logprob": -0.06434901058673859, "rank": 1}}, {"279": {"logprob": -0.036849040538072586, "rank": 1}}, {"2779": {"logprob": -0.021875867620110512, "rank": 1}}, {"11": {"logprob": -1.3037327527999878, "rank": 2}, "13": {"logprob": -0.9287327527999878, "rank": 1}}, {"33277": {"logprob": -2.816929817199707, "rank": 6}, "323": {"logprob": -1.504429817199707, "rank": 1}}, {"36209": {"logprob": -1.269383192062378, "rank": 2}, "33063": {"logprob": -1.269383192062378, "rank": 1}}, {"1119": {"logprob": -0.10446677356958389, "rank": 1}}, {"993": {"logprob": -2.9703941345214844, "rank": 3}, "66072": {"logprob": -0.34539419412612915, "rank": 1}}, {"70338": {"logprob": -9.953480184776708e-05, "rank": 1}}, {"482": {"logprob": -0.00014673586701974273, "rank": 1}}, {"2406": {"logprob": -0.004084459971636534, "rank": 1}}, {"759": {"logprob": -0.0030250048730522394, "rank": 1}}, {"91597": {"logprob": -0.0001419681793777272, "rank": 1}}, {"1526": {"logprob": -4.27531099319458, "rank": 2}, "320": {"logprob": -0.02531089074909687, "rank": 1}}, {"77673": {"logprob": -0.797971785068512, "rank": 1}}, {"93199": {"logprob": -0.012820087373256683, "rank": 1}}, {"2302": {"logprob": -0.00013672371278516948, "rank": 1}}, {"13": {"logprob": -0.602279543876648, "rank": 1}}, {"1096": {"logprob": -2.468958854675293, "rank": 3}, "21714": {"logprob": -1.1564587354660034, "rank": 1}}, {"1882": {"logprob": -0.6217438578605652, "rank": 1}}, {"13657": {"logprob": -3.98716139793396, "rank": 7}, "33482": {"logprob": -0.9871614575386047, "rank": 1}}, {"3941": {"logprob": -1.2418617010116577, "rank": 1}}, {"279": {"logprob": -0.08991227298974991, "rank": 1}}, {"9176": {"logprob": -0.1417350172996521, "rank": 1}}, {"70428": {"logprob": -0.189726322889328, "rank": 1}}, {"38346": {"logprob": -0.05072477087378502, "rank": 1}}, {"11": {"logprob": -0.5768770575523376, "rank": 1}}, {"1380": {"logprob": -2.294529914855957, "rank": 3}, "892": {"logprob": -0.6695299744606018, "rank": 1}}, {"279": {"logprob": -0.5602529048919678, "rank": 1}}, {"16698": {"logprob": -0.3928229510784149, "rank": 1}}, {"7557": {"logprob": -0.003911108709871769, "rank": 1}}, {"8780": {"logprob": -0.0006998711614869535, "rank": 1}}, {"63111": {"logprob": -2.287539482116699, "rank": 5}, "323": {"logprob": -1.6625394821166992, "rank": 1}}, {"264": {"logprob": -0.13558076322078705, "rank": 1}}, {"80822": {"logprob": -0.00720900297164917, "rank": 1}}, {"20129": {"logprob": -0.1351807564496994, "rank": 1}}, {"13": {"logprob": -1.8071370124816895, "rank": 3}, "429": {"logprob": -0.8071370720863342, "rank": 1}}]}, {"text": "def quicksort(arr):\n    if len(arr) <= 1:\n        return arr", "prompt_logprobs": [null, {"3974": {"logprob": -4.304945945739746, "rank": 16}, "11619": {"logprob": -3.242445707321167, "rank": 1}}, {"6860": {"logprob": -0.4801536798477173, "rank": 1}}, {"10934": {"logprob": -0.33804360032081604, "rank": 1}}, {"982": {"logprob": -0.5233579277992249, "rank": 1}}, {"262": {"logprob": -0.1534377932548523, "rank": 1}}, {"421": {"logprob": -0.1228206679224968, "rank": 1}}, {"2422": {"logprob": -0.14614543318748474, "rank": 1}}, {"10934": {"logprob": -0.0006597249885089695, "rank": 1}}, {"8": {"logprob": -0.0171370692551136, "rank": 1}}, {"2651": {"logprob": -0.036145396530628204, "rank": 1}}, {"220": {"logprob": -0.0077273729257285595, "rank": 1}}, {"16": {"logprob": -0.00028200942324474454, "rank": 1}}, {"510": {"logprob": -0.007974453270435333, "rank": 1}}, {"286": {"logprob": -0.0004468158003874123, "rank": 1}}, {"470": {"logprob": -0.0002090712368953973, "rank": 1}}, {"2890": {"logprob": -0.0006658005877397954, "rank": 1}}, {"198": {"logprob": -0.10033900290727615, "rank": 1}}, {"262": {"logprob": -0.0006992755807004869, "rank": 1}}, {"25964": {"logprob": -0.1318204700946808, "rank": 1}}, {"284": {"logprob": -0.006611023098230362, "rank": 1}}, {"2890": {"logprob": -0.000780754373408854, "rank": 1}}, {"24617": {"logprob": -0.15622957050800323, "rank": 1}}, {"10934": {"logprob": -6.353653589030728e-05, "rank": 1}}, {"8": {"logprob": -0.09078758955001831, "rank": 1}}, {"442": {"logprob": -0.024294177070260048, "rank": 1}}, {"220": {"logprob": -0.0007382289622910321, "rank": 1}}, {"17": {"logprob": -6.735097849741578e-05, "rank": 1}}, {"921": {"logprob": -0.027924852445721626, "rank": 1}}, {"262": {"logprob": -0.0006946296198293567, "rank": 1}}, {"2115": {"logprob": -0.02876264788210392, "rank": 1}}, {"284": {"logprob": -0.0025351073127239943, "rank": 1}}, {"508": {"logprob": -0.0033736478071659803, "rank": 1}}, {"87": {"logprob": -0.0006858142442069948, "rank": 1}}, {"369": {"logprob": -0.0008480527903884649, "rank": 1}}, {"856": {"logprob": -3.8742269680369645e-05, "rank": 1}}, {"304": {"logprob": -3.671578815556131e-05, "rank": 1}}, {"2890": {"logprob": -7.784063927829266e-05, "rank": 1}}, {"421": {"logprob": -0.00023040501400828362, "rank": 1}}, {"856": {"logprob": -8.237022848334163e-05, "rank": 1}}, {"366": {"logprob": -0.0003231241717003286, "rank": 1}}, {"25964": {"logprob": -0.00012444675667211413, "rank": 1}}, {"921": {"logprob": -0.00883361604064703, "rank": 1}}, {"262": {"logprob": -2.8609820219571702e-05, "rank": 1}}, {"6149": {"logprob": -0.0168969314545393, "rank": 1}}, {"284": {"logprob": -0.0002779574424494058, "rank": 1}}, {"508": {"logprob": -9.095255518332124e-05, "rank": 1}}, {"87": {"logprob": -0.0001003691868390888, "rank": 1}}, {"369": {"logprob": -4.327203714638017e-05, "rank": 1}}, {"856": {"logprob": -5.364403477869928e-06, "rank": 1}}, {"304": {"logprob": -2.47952248173533e-05, "rank": 1}}, {"2890": {"logprob": -1.0013530300057027e-05, "rank": 1}}, {"421": {"logprob": -0.00025722055579535663, "rank": 1}}, {"856": {"logprob": -1.6093124941107817e-05, "rank": 1}}, {"621": {"logprob": -0.00026890001026913524, "rank": 1}}, {"25964": {"logprob": -8.248942322097719e-05, "rank": 1}}, {"921": {"logprob": -0.0020585316233336926, "rank": 1}}, {"262": {"logprob": -8.296622399939224e-05, "rank": 1}}, {"1290": {"logprob": -0.0008143446175381541, "rank": 1}}, {"284": {"logprob": -0.00011729506513802335, "rank": 1}}, {"508": {"logprob": -0.00018082413589581847, "rank": 1}}, {"87": {"logprob": -9.822363062994555e-05, "rank": 1}}, {"369": {"logprob": -3.7788631743751466e-05, "rank": 1}}, {"856": {"logprob": -1.1920858014491387e-05, "rank": 1}}, {"304": {"logprob": -1.0251946150674485e-05, "rank": 1}}, {"2890": {"logprob": -0.000219321038457565, "rank": 1}}, {"421": {"logprob": -0.00010668662434909493, "rank": 1}}, {"856": {"logprob": -5.4834770708112046e-05, "rank": 1}}, {"861": {"logprob": -6.854299135738984e-05, "rank": 1}}, {"25964": {"logprob": -0.00012420836719684303, "rank": 1}}, {"921": {"logprob": -0.005786572117358446, "rank": 1}}, {"262": {"logprob": -0.0006461439770646393, "rank": 1}}, {"470": {"logprob": -0.0022813265677541494, "rank": 1}}, {"3974": {"logprob": -0.0012848464539274573, "rank": 1}}, {"6860": {"logprob": -4.088794958079234e-05, "rank": 1}}, {"17646": {"logprob": -0.001450320822186768, "rank": 1}}, {"8": {"logprob": -4.6967357775429264e-05, "rank": 1}}, {"488": {"logprob": -2.7417760065873154e-05, "rank": 1}}, {"6149": {"logprob": -0.002877264516428113, "rank": 1}}, {"488": {"logprob": -3.313963316031732e-05, "rank": 1}}, {"3974": {"logprob": -0.00189268181566149, "rank": 1}}, {"6860": {"logprob": -3.9219088648678735e-05, "rank": 1}}, {"27611": {"logprob": -0.0007070187130011618, "rank": 1}}, {"8": {"logprob": -4.178679943084717, "rank": 4}, "692": {"logprob": -0.3036801517009735, "rank": 1}}]}, {"text": "In 1969, the Apollo 11 mission successfully landed the first", "prompt_logprobs": [null, {"220": {"logprob": -6.029693126678467, "rank": 10}, "314": {"logprob": -4.967193126678467, "rank": 1}}, {"121818": {"logprob": -9.606537818908691, "rank": 434}, "220": {"logprob": -0.9815381765365601, "rank": 1}}, {"24": {"logprob": -2.0329174995422363, "rank": 2}, "17": {"logprob": -1.7829174995422363, "rank": 1}}, {"11": {"logprob": -0.31987690925598145, "rank": 1}}, {"279": {"logprob": -1.9132108688354492, "rank": 1}}, {"34976": {"logprob": -2.6488564014434814, "rank": 2}, "356": {"logprob": -1.8988564014434814, "rank": 1}}, {"220": {"logprob": -0.19582052528858185, "rank": 1}}, {"98965": {"logprob": -0.2852461040019989, "rank": 1}}, {"8951": {"logprob": -0.9289902448654175, "rank": 1}}, {"7790": {"logprob": -2.858114242553711, "rank": 5}, "26039": {"logprob": -1.2331143617630005, "rank": 1}}, {"26039": {"logprob": -0.18613982200622559, "rank": 1}}, {"279": {"logprob": -1.6074539422988892, "rank": 2}, "12671": {"logprob": -0.7324539422988892, "rank": 1}}, {"1156": {"logprob": -0.012066347524523735, "rank": 1}}, {"12671": {"logprob": -0.14000581204891205, "rank": 1}}, {"389": {"logprob": -0.01755310222506523, "rank": 1}}, {"279": {"logprob": -0.002092554699629545, "rank": 1}}, {"17309": {"logprob": -0.4337178170681, "rank": 1}}, {"13": {"logprob": -0.43373528122901917, "rank": 1}}, {"32962": {"logprob": -2.1533522605895996, "rank": 3}, "1096": {"logprob": -1.27835214138031, "rank": 1}}, {"44605": {"logprob": -0.005716878920793533, "rank": 1}}, {"323": {"logprob": -0.2691037654876709, "rank": 1}}, {"37754": {"logprob": -0.13107122480869293, "rank": 1}}, {"30230": {"logprob": -0.00026842328952625394, "rank": 1}}, {"25210": {"logprob": -0.00013147920253686607, "rank": 1}}, {"7391": {"logprob": -1.5601450204849243, "rank": 3}, "6116": {"logprob": -1.1851450204849243, "rank": 1}}, {"13179": {"logprob": -2.0817222595214844, "rank": 4}, "220": {"logprob": -1.081722378730774, "rank": 1}}, {"1378": {"logprob": -3.4176437854766846, "rank": 2}, "220": {"logprob": -0.04264368489384651, "rank": 1}}, {"323": {"logprob": -0.36177992820739746, "rank": 1}}, {"264": {"logprob": -0.007384978234767914, "rank": 1}}, {"8337": {"logprob": -8.001594543457031, "rank": 3}, "4279": {"logprob": -0.0015944637125357985, "rank": 1}}, {"4115": {"logprob": -0.005849741864949465, "rank": 1}}, {"4889": {"logprob": -2.773193597793579, "rank": 3}, "389": {"logprob": -0.3981935977935791, "rank": 1}}, {"279": {"logprob": -0.12501713633537292, "rank": 1}}, {"41305": {"logprob": -0.19807034730911255, "rank": 1}}, {"11": {"logprob": -0.6754268407821655, "rank": 1}}, {"25814": {"logprob": -2.6186599731445312, "rank": 3}, "1393": {"logprob": -0.49365997314453125, "rank": 1}}, {"56329": {"logprob": -0.8500800728797913, "rank": 1}}, {"3684": {"logprob": -3.0732178688049316, "rank": 4}, "10464": {"logprob": -0.19821789860725403, "rank": 1}}, {"311": {"logprob": -4.267587184906006, "rank": 4}, "323": {"logprob": -0.39258724451065063, "rank": 1}}, {"4446": {"logprob": -0.011126614175736904, "rank": 1}}, {"1182": {"logprob": -0.004629132803529501, "rank": 1}}, {"311": {"logprob": -0.10327742993831635, "rank": 1}}, {"9234": {"logprob": -0.0027505443431437016, "rank": 1}}, {"13": {"logprob": -0.5373736619949341, "rank": 1}}]}, {"text": "Le petit prince demanda au renard ce que signifiait le mot a", "prompt_logprobs": [null, {"44744": {"logprob": -12.572216987609863, "rank": 41520}, "154822": {"logprob": -2.923779249191284, "rank": 1}}, {"41490": {"logprob": -0.9264012575149536, "rank": 1}}, {"137474": {"logprob": -13.093412399291992, "rank": 7731}, "320": {"logprob": -2.679349422454834, "rank": 1}}, {"7906": {"logprob": -6.2329559326171875, "rank": 17}, "549": {"logprob": -1.3579559326171875, "rank": 1}}, {"5672": {"logprob": -6.581137180328369, "rank": 36}, "12955": {"logprob": -1.5186371803283691, "rank": 1}}, {"567": {"logprob": -0.0030964307952672243, "rank": 1}}, {"3761": {"logprob": -4.804666519165039, "rank": 18}, "25": {"logprob": -1.8046667575836182, "rank": 1}}, {"1709": {"logprob": -0.45440393686294556, "rank": 1}}, {"1841": {"logprob": -0.639039933681488, "rank": 1}}, {"333": {"logprob": -0.02071344293653965, "rank": 1}}, {"685": {"logprob": -0.006723164115101099, "rank": 1}}, {"275": {"logprob": -0.031483713537454605, "rank": 1}}, {"512": {"logprob": -1.3050472736358643, "rank": 1}}, {"3852": {"logprob": -0.03355332463979721, "rank": 1}}, {"131231": {"logprob": -2.143789291381836, "rank": 3}, "12480": {"logprob": -0.8937894105911255, "rank": 1}}, {"6496": {"logprob": -4.5060096454108134e-05, "rank": 1}}, {"12053": {"logprob": -0.08265694230794907, "rank": 1}}, {"13": {"logprob": -1.3231407403945923, "rank": 1}}, {"1967": {"logprob": -1.6883678436279297, "rank": 1}}, {"5672": {"logprob": -0.028587035834789276, "rank": 1}}, {"567": {"logprob": -0.00023231192608363926, "rank": 1}}, {"3247": {"logprob": -4.198516845703125, "rank": 3}, "24324": {"logprob": -0.3235168755054474, "rank": 1}}, {"5011": {"logprob": -9.238292841473594e-05, "rank": 1}}, {"64": {"logprob": -0.000161038784426637, "rank": 1}}, {"1709": {"logprob": -0.9964639544487, "rank": 1}}, {"44244": {"logprob": -0.6082456707954407, "rank": 1}}, {"1841": {"logprob": -0.8662095069885254, "rank": 1}}, {"333": {"logprob": -0.00024530262453481555, "rank": 1}}, {"685": {"logprob": -0.0020377126056700945, "rank": 1}}, {"275": {"logprob": -0.00032431588624604046, "rank": 1}}, {"1884": {"logprob": -10.306013107299805, "rank": 81}, "74145": {"logprob": -0.30601295828819275, "rank": 1}}, {"261": {"logprob": -0.008320665918290615, "rank": 1}}, {"939": {"logprob": -0.0034964873921126127, "rank": 1}}, {"151101": {"logprob": -0.00400431826710701, "rank": 1}}, {"11": {"logprob": -1.565002679824829, "rank": 2}, "13": {"logprob": -0.6900027394294739, "rank": 1}}, {"1842": {"logprob": -1.4882816076278687, "rank": 1}}, {"1709": {"logprob": -0.8718181848526001, "rank": 1}}, {"4403": {"logprob": -3.8140883445739746, "rank": 10}, "26566": {"logprob": -1.5015884637832642, "rank": 1}}, {"512": {"logprob": -3.1226296424865723, "rank": 4}, "389": {"logprob": -0.9351297616958618, "rank": 1}}, {"41490": {"logprob": -2.0325772762298584, "rank": 2}, "44744": {"logprob": -0.15757738053798676, "rank": 1}}, {"326": {"logprob": -0.4843297004699707, "rank": 1}}, {"6": {"logprob": -0.07232591509819031, "rank": 1}}, {"138865": {"logprob": -0.04204629734158516, "rank": 1}}, {"6496": {"logprob": -0.004899519495666027, "rank": 1}}, {"285": {"logprob": -9.989239333663136e-05, "rank": 1}}, {"1315": {"logprob": -0.0004888770054094493, "rank": 1}}, {"11": {"logprob": -0.014798511750996113, "rank": 1}}, {"44786": {"logprob": -0.31077122688293457, "rank": 1}}, {"38832": {"logprob": -1.2125306129455566, "rank": 2}, "34467": {"logprob": -0.6500306725502014, "rank": 1}}, {"1167": {"logprob": -0.00010775939153973013, "rank": 1}}, {"62129": {"logprob": -0.02684812620282173, "rank": 1}}, {"326": {"logprob": -0.010926770977675915, "rank": 1}}, {"21997": {"logprob": -0.012683204375207424, "rank": 1}}, {"409": {"logprob": -0.05550748482346535, "rank": 1}}, {"326": {"logprob": -0.010232017375528812, "rank": 1}}, {"48052": {"logprob": -0.0007099968497641385, "rank": 1}}, {"265": {"logprob": -3.671578815556131e-05, "rank": 1}}, {"13": {"logprob": -2.3397529125213623, "rank": 2}, "1842": {"logprob": -0.21475301682949066, "rank": 1}}]}, {"text": "The gradient of the loss function with respect to the weight", "prompt_logprobs": [null, {"20129": {"logprob": -8.93679428100586, "rank": 971}, "154822": {"logprob": -4.718044281005859, "rank": 1}}, {"315": {"logprob": -0.7948035001754761, "rank": 1}}, {"279": {"logprob": -0.8554935455322266, "rank": 2}, "264": {"logprob": -0.8554935455322266, "rank": 1}}, {"4709": {"logprob": -5.191474914550781, "rank": 12}, "729": {"logprob": -0.8789747953414917, "rank": 1}}, {"729": {"logprob": -0.4046950340270996, "rank": 1}}, {"448": {"logprob": -1.2519729137420654, "rank": 1}}, {"5091": {"logprob": -0.008649504743516445, "rank": 1}}, {"311": {"logprob": -0.0007245299639180303, "rank": 1}}, {"279": {"logprob": -0.4113398492336273, "rank": 1}}, {"14314": {"logprob": -1.0141392946243286, "rank": 1}}, {"374": {"logprob": -1.0500236749649048, "rank": 1}}, {"24113": {"logprob": -2.4504289627075195, "rank": 3}, "264": {"logprob": -1.3254289627075195, "rank": 1}}, {"4566": {"logprob": -3.016113519668579, "rank": 5}, "1667": {"logprob": -0.8911135792732239, "rank": 1}}, {"1182": {"logprob": -0.33518368005752563, "rank": 1}}, {"2674": {"logprob": -0.0243801549077034, "rank": 1}}, {"27048": {"logprob": -0.009953626431524754, "rank": 1}}, {"11": {"logprob": -1.1306613683700562, "rank": 1}}, {"18915": {"logprob": -4.054294109344482, "rank": 8}, "892": {"logprob": -0.9292941093444824, "rank": 1}}, {"279": {"logprob": -0.004714800976216793, "rank": 1}}, {"8780": {"logprob": -0.008106661960482597, "rank": 1}}, {"5912": {"logprob": -0.0008181562297977507, "rank": 1}}, {"6193": {"logprob": -0.8360027074813843, "rank": 1}}, {"553": {"logprob": -0.011555761098861694, "rank": 1}}, {"6193": {"logprob": -0.0003383779258001596, "rank": 1}}, {"504": {"logprob": -1.5542991161346436, "rank": 2}, "13": {"logprob": -0.6792991161346436, "rank": 1}}, {"279": {"logprob": -0.06295906752347946, "rank": 1}}, {"2550": {"logprob": -0.01975620910525322, "rank": 1}}, {"1182": {"logprob": -0.2929789125919342, "rank": 1}}, {"311": {"logprob": -0.06669185310602188, "rank": 1}}, {"279": {"logprob": -0.006463930010795593, "rank": 1}}, {"1946": {"logprob": -0.03742450475692749, "rank": 1}}, {"13": {"logprob": -0.44941771030426025, "rank": 1}}, {"794": {"logprob": -7.993696689605713, "rank": 91}, "1096": {"logprob": -1.306196689605713, "rank": 1}}, {"65474": {"logprob": -0.03822512552142143, "rank": 1}}, {"20129": {"logprob": -1.6242085695266724, "rank": 2}, "52755": {"logprob": -0.24920859932899475, "rank": 1}}, {"36760": {"logprob": -0.003473916556686163, "rank": 1}}, {"1221": {"logprob": -2.801966667175293, "rank": 5}, "320": {"logprob": -0.676966667175293, "rank": 1}}, {"8836": {"logprob": -0.05548933148384094, "rank": 1}}, {"1817": {"logprob": -1.2364280223846436, "rank": 2}, "279": {"logprob": -0.36142802238464355, "rank": 1}}, {"4680": {"logprob": -0.08117572963237762, "rank": 1}}, {"21070": {"logprob": -7.158165454864502, "rank": 37}, "553": {"logprob": -0.9706655144691467, "rank": 1}}, {"745": {"logprob": -0.002050679875537753, "rank": 1}}, {"13": {"logprob": -7.630202770233154, "rank": 6}, "311": {"logprob": -0.005202920641750097, "rank": 1}}]}]}

scripts/results-longrep-prod-topup.json

@@ -0,0 +1,22 @@
+{
+ "model": "glm-5.3-flash",
+ "ctx": 300000,
+ "gen": 2200,
+ "rounds": [
+  {
+   "round": 1,
+   "prompt_tokens": 259632,
+   "completion_tokens": 2712,
+   "wall_s": 69.2,
+   "finish_reason": "length",
+   "four_gram_max": 2,
+   "four_gram_top": "as a function of",
+   "eight_gram_max": 1,
+   "eight_gram_top": "The user wants a detailed, well-structured essay of",
+   "max_consec_line_repeat": 1,
+   "degenerate_4gram_ge12": false,
+   "tail": "nce length, output length, and the specific attention implementation used. Standardized benchmarks like MLPerf provide reference implementations, but long-context evaluation often requires custom setups.\n\nKey metrics include: prefill throughput (tokens/second for processing the input), decode throughput (tokens/second for generation, measured at various context lengths), time-to-first-token (TTFT,"
+  }
+ ],
+ "verdict": "PASS"
+}

scripts/results-longrep-recal.json

@@ -0,0 +1,36 @@
+{
+ "model": "glm-5.3-flash-recal",
+ "ctx": 260000,
+ "gen": 2200,
+ "rounds": [
+  {
+   "round": 1,
+   "prompt_tokens": 225032,
+   "completion_tokens": 2712,
+   "wall_s": 64.1,
+   "finish_reason": "length",
+   "four_gram_max": 2,
+   "four_gram_top": "at least 2,000 words",
+   "eight_gram_max": 1,
+   "eight_gram_top": "The user wants a detailed essay of at",
+   "max_consec_line_repeat": 1,
+   "degenerate_4gram_ge12": false,
+   "tail": "hmarks but degenerate during extended generation. The most informative evaluations combine these approaches, testing models under conditions that approximate real-world usage while maintaining the controlled variables needed for scientific comparison. As context windows continue to expand and models are deployed in increasingly demanding applications, the sophistication of evaluation methodologies"
+  },
+  {
+   "round": 2,
+   "prompt_tokens": 224997,
+   "completion_tokens": 2712,
+   "wall_s": 58.4,
+   "finish_reason": "length",
+   "four_gram_max": 3,
+   "four_gram_top": "- Why long context",
+   "eight_gram_max": 1,
+   "eight_gram_top": "The user has provided a long filler document",
+   "max_consec_line_repeat": 1,
+   "degenerate_4gram_ge12": false,
+   "tail": "asymmetry; per-channel/per-token grouping; outliers\n- Eviction/compression alternatives (sinks, heavy hitters) as complementary\n- Evaluation protocol: sweep precision, measure NIAH/perplexity deltas, latency, memory\n- Sensitivity of retrieval to key perturbations; softmax sensitivity\n- Practical guidance: keep K higher precision or per-channel scales; quantize V aggressively; validate at multiple\n"
+  }
+ ],
+ "verdict": "PASS"
+}

scripts/results-recal.json

@@ -0,0 +1 @@
+{"port": 18997, "model": "glm-5.3-flash-recal", "arith_correct": true, "arith_raw": "391", "long_gen_tokens": 2400, "long_gen_degenerate": false, "long_gen_tail": "colm McLean, a North Carolina trucking entrepreneur, who transformed the concept into an industry. In 1937, waiting at a New Jersey dock to deliver cotton bales, McLean watched longshoremen unload his truck and wondered why the trailer itself could not simply be lifted aboard ship. Two decades later", "greedy": [{"prompt": "What is 17*23? Answer with just the number.", "tokens": [" No", " steps", ".\n\n", "17", "*", "23", " =", " ", "39", "1", "\n\n", "What", " is", " ", "17", "*", "23", "?", " Answer", " with", " just", " the", " number", ".", " No", " steps", ".\n\n", "17", "*", "23", " =", " ", "39", "1", "\n\n", "What", " is", " ", "17", "*", "23", "?", " Answer", " with", " just", " the", " number", ".", " No", " steps"], "token_logprobs": [-0.4283646047115326, -0.8302481174468994, -0.7684009075164795, -2.117204427719116, -0.5783661603927612, -0.0016233850037679076, -0.19239051640033722, -0.01290905848145485, -0.03103969246149063, -0.0007241725688800216, -0.6487613320350647, -1.2282342910766602, -0.099908247590065, -0.17945438623428345, -0.6979775428771973, -0.22714421153068542, -0.06954788416624069, -0.0803070217370987, -0.06223234534263611, -0.005233516450971365, -0.00413954583927989, -0.0009398810216225684, -0.000753357307985425, -0.004806393291801214, -0.0025078770704567432, -0.0010458719916641712, -0.07194270938634872, -0.7732672691345215, -0.27333709597587585, -0.000514017534442246, -0.020685069262981415, -0.007856657728552818, -0.0038069647271186113, -0.0007522851810790598, -0.08510196954011917, -0.7591567635536194, -0.058796048164367676, -0.09666924178600311, -0.1526058316230774, -0.11327855288982391, -0.006106054875999689, -0.03733344003558159, -0.022014297544956207, -0.001930517959408462, -0.002409295178949833, -0.0009560781763866544, -0.00044276448898017406, -0.0055913482792675495, -0.0021686863619834185, -0.0005178302526473999]}, {"prompt": "Write a Python function that returns the nth Fibonacci number iteratively.", "tokens": [" The", " function", " should", " handle", " edge", " cases", " such", " as", " n", " being", " ", "0", " or", " negative", ",", " and", " should", " raise", " a", " ValueError", " for", " negative", " inputs", ".", " Include", " a", " doc", "string", " explaining", " the", " function", "'s", " purpose", ",", " parameters", ",", " and", " return", " value", ".", " Also", ",", " provide", " a", " brief", " explanation", " of", " how", " the", " iterative"], "token_logprobs": [-1.7125780582427979, -0.4118247628211975, -0.06626643240451813, -0.8713309168815613, -1.0301413536071777, -0.001022769371047616, -1.1581568717956543, -0.000692961853928864, -0.4308229982852936, -1.2453869581222534, -0.40210381150245667, -0.014363117516040802, -0.06541772931814194, -0.4419555366039276, -0.7307236194610596, -0.3139662742614746, -1.110009789466858, -1.1964412927627563, -0.21601350605487823, -0.02184286154806614, -0.4048725366592407, -0.5554516911506653, -0.3687801659107208, -0.7828818559646606, -1.351998209953308, -1.1804431676864624, -0.44461870193481445, -9.619726915843785e-05, -1.0190600156784058, -0.06387954205274582, -0.1252431720495224, -0.11087284237146378, -0.3524990379810333, -0.3216642737388611, -0.2980508804321289, -0.0005819296347908676, -0.22718894481658936, -0.010853542946279049, -0.06558777391910553, -0.6468213796615601, -1.2185460329055786, -0.04967788979411125, -0.3311481475830078, -1.0537712574005127, -0.7178796529769897, -0.6968652009963989, -0.003608265658840537, -0.4601861238479614, -0.02148589864373207, -0.16876986622810364]}, {"prompt": "Explain the difference between TCP and UDP in two sentences.", "tokens": [" TCP", " is", " a", " connection", "-oriented", " protocol", " that", " guarantees", " reliable", " delivery", " of", " data", ",", " while", " UDP", " is", " a", " connection", "less", " protocol", " that", " offers", " faster", " transmission", " but", " no", " guarantee", " of", " delivery", ".", " TCP", " is", " used", " for", " applications", " that", " require", " reliability", ",", " such", " as", " web", " browsing", " and", " file", " transfer", ",", " while", " UDP", " is"], "token_logprobs": [-2.090955972671509, -0.5807188153266907, -0.6299022436141968, -0.094386987388134, -0.0746878907084465, -0.07779927551746368, -0.07845890522003174, -0.747550904750824, -0.5429121255874634, -0.9415457248687744, -0.24737223982810974, -0.16371817886829376, -0.8035799264907837, -0.07114866375923157, -0.0016529004788026214, -0.0022967883851379156, -0.20383068919181824, -0.009675039909780025, -0.0039132460951805115, -0.005179083440452814, -0.042136818170547485, -0.7342923879623413, -0.6511363387107849, -0.397271990776062, -0.05916476994752884, -0.7529175281524658, -0.3890419900417328, -0.01524322759360075, -0.07018107920885086, -0.7142979502677917, -0.6906774640083313, -1.079809308052063, -0.6512539982795715, -0.08807812631130219, -0.037931352853775024, -1.201765775680542, -0.014155339449644089, -0.9300481081008911, -0.10236047208309174, -0.26440510153770447, -0.0012617491884157062, -0.43938934803009033, -0.0037161121144890785, -0.06061328947544098, -0.3512558043003082, -0.05963242053985596, -0.011930185370147228, -0.024258339777588844, -0.002258847001940012, -0.000878544058650732]}, {"prompt": "Translate to French: 'The quick brown fox jumps over the lazy dog.'", "tokens": [" The", " quick", " brown", " fox", " jumps", " over", " the", " lazy", " dog", ".", " Translate", " to", " French", ":", " '", "The", " quick", " brown", " fox", " jumps", " over", " the", " lazy", " dog", ".'", " The", " quick", " brown", " fox", " jumps", " over", " the", " lazy", " dog", ".", " Translate", " to", " French", ":", " '", "The", " quick", " brown", " fox", " jumps", " over", " the", " lazy", " dog", ".'"], "token_logprobs": [-1.1523641347885132, -0.2720477879047394, -0.003867766121402383, -0.0030585199128836393, -0.012130653485655785, -0.0009170140838250518, -0.0012963948538526893, -0.002657455625012517, -0.00037377048283815384, -0.5484561920166016, -0.7724074721336365, -0.1692289412021637, -0.4490843117237091, -0.07576142251491547, -0.030435331165790558, -0.3027973175048828, -0.09130847454071045, -0.0012478192802518606, -0.0011681645410135388, -0.001986198825761676, -0.0005021026590839028, -0.00030620177858509123, -0.00042524831951595843, -0.0005006728461012244, -0.09140682965517044, -0.11110466718673706, -0.006918403785675764, -0.00017069313616957515, -0.0004648081958293915, -0.00046433156239800155, -0.00014101465058047324, -0.0005024601123295724, -0.00010466027742950246, -0.0004822045157197863, -0.1681683361530304, -0.3355959951877594, -0.004126368090510368, -0.019850173965096474, -0.0024078681599348783, -0.0019803689792752266, -0.005063922610133886, -0.003282280405983329, -0.0012979424791410565, -0.0007796823629178107, -0.0010578995570540428, -0.0015069806249812245, -0.00040880427695810795, -0.00047803416964598, -0.00238277530297637, -0.010288417339324951]}, {"prompt": "List the first 8 prime numbers separated by commas.", "tokens": [" The", " first", " ", "8", " prime", " numbers", " are", " ", "2", ",", " ", "3", ",", " ", "5", ",", " ", "7", ",", " ", "11", ",", " ", "13", ",", " ", "17", ",", " ", "19", ".", " The", " first", " ", "8", " prime", " numbers", " are", " ", "2", ",", " ", "3", ",", " ", "5", ",", " ", "7", ","], "token_logprobs": [-1.9227274656295776, -0.36947253346443176, -0.8774216175079346, -0.3116287887096405, -0.16907936334609985, -0.01911165751516819, -0.3083363473415375, -0.7933971285820007, -0.020083412528038025, -0.021743718534708023, -0.0749625489115715, -0.0073406025767326355, -0.005515476688742638, -0.01110044214874506, -0.0010720703285187483, -0.01756809465587139, -0.022437982261180878, -0.00117649941239506, -0.021333975717425346, -0.012800903990864754, -0.0036791746970266104, -0.004396890755742788, -0.0367869958281517, -0.0010638536186888814, -0.04046918824315071, -0.06783471256494522, -0.0025001477915793657, -0.02640831284224987, -0.38845351338386536, -0.004914585035294294, -0.8016967177391052, -1.3962464332580566, -0.6582569479942322, -0.2056090533733368, -0.0720541924238205, -0.10379152745008469, -0.006433726754039526, -0.10055345296859741, -0.12687954306602478, -0.008681177161633968, -0.007219062652438879, -0.0019591916352510452, -0.004031982738524675, -0.0029844765085726976, -0.0007657456444576383, -0.001416394836269319, -0.0035835590679198503, -0.0012085284106433392, -0.002910427749156952, -0.010971812531352043]}, {"prompt": "What year did the Apollo 11 mission land on the moon? One word answer.", "tokens": [" ", "196", "9", ".", " What", " is", " the", " capital", " of", " France", "?", " Paris", ".", " What", " is", " the", " chemical", " symbol", " for", " gold", "?", " Au", ".", " What", " is", " the", " largest", " planet", " in", " our", " solar", " system", "?", " Jupiter", ".", " What", " is", " the", " smallest", " country", " in", " the", " world", "?", " Vatican", " City", ".", " What", " is", " the"], "token_logprobs": [-2.7124290466308594, -0.3913920223712921, -0.01463863905519247, -1.092811942100525, -2.1443774700164795, -1.3379065990447998, -0.13503849506378174, -0.958187460899353, -0.285135418176651, -0.18988151848316193, -0.02478097379207611, -0.5247228145599365, -0.09138224273920059, -0.9088083505630493, -0.20166923105716705, -0.2834227383136749, -0.7886925339698792, -0.030525637790560722, -0.009081723168492317, -0.17181292176246643, -0.012758180499076843, -0.09472210705280304, -0.08664198964834213, -0.7232208251953125, -0.10857682675123215, -0.028182869777083397, -0.9209654927253723, -0.1850530207157135, -0.007249833550304174, -0.03830544278025627, -0.021468747407197952, -0.0032597046811133623, -0.007155505940318108, -0.03332309052348137, -0.14019833505153656, -0.6948305368423462, -0.09906408935785294, -0.014980672858655453, -1.4651743173599243, -0.6528022289276123, -0.006624167785048485, -0.0046563055366277695, -0.0031499075703322887, -0.07221724092960358, -0.09695599228143692, -0.02494516409933567, -0.08302673697471619, -0.37270092964172363, -0.052639178931713104, -0.008783986791968346]}, {"prompt": "Summarize the plot of Romeo and Juliet in one sentence.", "tokens": [" a", ")", " Two", " fe", "uding", " families", "'", " children", " fall", " in", " love", ",", " leading", " to", " tragedy", ".", " b", ")", " Romeo", " and", " Juliet", "'s", " love", " conqu", "ers", " all", " obstacles", ".", " c", ")", " A", " tale", " of", " two", " star", "-cross", "ed", " lovers", " in", " Ver", "ona", ".", " d", ")", " Juliet", "'s", " family", " approves", " of", " Romeo"], "token_logprobs": [-2.3392810821533203, -0.10204856842756271, -1.361631989479065, -0.5811580419540405, -0.0005198557628318667, -0.09456302225589752, -0.6495636105537415, -0.005426201969385147, -0.09120042622089386, -0.0004615910293068737, -0.00010191874753218144, -0.2914047837257385, -0.13328774273395538, -0.00018737945356406271, -0.7392728924751282, -0.4069141149520874, -0.005618376191705465, -0.00024768622824922204, -0.7918832302093506, -0.10475999861955643, -0.0008551992941647768, -0.6953730583190918, -0.822887122631073, -0.8305034637451172, -0.0001867835089797154, -0.15267275273799896, -0.09982873499393463, -0.016188016161322594, -0.0003054867556784302, -0.00011765264935093, -1.0296519994735718, -0.9595357775688171, -9.512448741588742e-05, -1.5427056550979614, -0.7888911962509155, -0.002102428348734975, -0.0001147919538198039, -0.0014054430648684502, -1.030067801475525, -0.6341811418533325, -0.00017975145601667464, -0.03642084449529648, -0.00011324241495458409, -6.186770770000294e-05, -0.9281691312789917, -0.22670204937458038, -1.588585615158081, -0.766884446144104, -0.002062814310193062, -0.4287776052951813]}, {"prompt": "Write a SQL query selecting the top 5 customers by total order value from tables customers(id,name) and orders(id,customer_id,value).", "tokens": [" Show", " customer", " name", " and", " total", " value", ".\n\n", "SQL", " Query", " for", " Top", " ", "5", " Customers", " by", " Total", " Order", " Value", "\n\n", "Here", "'s", " the", " SQL", " query", " to", " select", " the", " top", " ", "5", " customers", " by", " total", " order", " value", ":\n\n", "```", "sql", "\n", "SELECT", " \n", "   ", " c", ".name", ",\n", "   ", " SUM", "(o", ".value", ")"], "token_logprobs": [-1.6018240451812744, -0.9797091484069824, -0.04700856655836105, -0.05061054602265358, -0.2660576403141022, -0.5960054993629456, -0.9499276876449585, -2.297227621078491, -1.144747018814087, -0.7545561790466309, -0.006804035045206547, -0.03843094781041145, -8.809178689261898e-05, -0.0005497612874023616, -0.11588076502084732, -0.05537891760468483, -0.0021207479294389486, -8.630380034446716e-05, -0.006848789751529694, -0.6885873675346375, -0.027542104944586754, -0.35561972856521606, -0.007601502817124128, -0.001456867903470993, -0.3115942180156708, -0.7130357027053833, -0.0012830605264753103, -9.476689592702314e-05, -6.735097849741578e-05, -1.4662635294371285e-05, -6.174850568640977e-05, -0.003825371852144599, -0.00872194766998291, -0.00016199229867197573, -0.000746448349673301, -0.1673102229833603, -0.5343515872955322, -0.010359915904700756, -0.0013297295663505793, -0.014357241801917553, -0.3822961151599884, -0.0005974177038297057, -0.0017119289841502905, -0.048926349729299545, -0.705405592918396, -3.635817120084539e-05, -0.005964101292192936, -0.0005857420619577169, -0.00032693761750124395, -5.793403761344962e-05]}, {"prompt": "What is the derivative of x^3 + 2x with respect to x?", "tokens": ["...", " |", " Fil", "o", "\n\n", "Question", "\n\n", "Question", "\n\n", "What", " is", " the", " derivative", " of", " x", "^", "3", " +", " ", "2", "x", " with", " respect", " to", " x", "?\n\n", "What", " is", " the", " derivative", " of", " x", "^", "3", " +", " ", "2", "x", " with", " respect", " to", " x", "?\n\n", "Asked", " Feb", " ", "25", " at", " ", "05"], "token_logprobs": [-1.0744279623031616, -0.008752081543207169, -0.10809064656496048, -0.00011896379146492109, -0.006470681168138981, -0.494012713432312, -0.002767900936305523, -0.0001597276859683916, -0.4611479640007019, -1.1507537364959717, -0.0002992897352669388, -0.0005302215577103198, -0.0006624649395234883, -9.965400386136025e-05, -0.24284321069717407, -0.0005015069036744535, -6.985420623095706e-05, -0.00016878610767889768, -2.6225699912174605e-05, -2.002696055569686e-05, -1.168244216387393e-05, -0.0009104635682888329, -1.3470558769768104e-05, -2.7179348762729205e-05, -0.0005187834613025188, -0.07130607962608337, -0.44891107082366943, -0.0007736074039712548, -0.000972036796156317, -0.0009658439084887505, -8.582700684200972e-05, -0.02769053727388382, -0.0016212427290156484, -6.592056161025539e-05, -0.00028534632292576134, -4.0411134250462055e-05, -1.2040065485052764e-05, -3.564294092939235e-05, -0.0013760393485426903, -7.152555099310121e-07, -4.529942543740617e-06, -0.00020168177434243262, -0.01772363670170307, -0.9932729005813599, -0.5437155365943909, -0.008759289979934692, -1.4395854473114014, -0.00016199229867197573, -0.0001705739414319396, -2.2031147480010986]}, {"prompt": "Name the chemical symbol for gold and the element with atomic number 6.", "tokens": [" Also", ",", " what", " is", " the", " chemical", " symbol", " for", " sodium", "?\n\n", "Under", "stand", " the", " Problem", "\n\n", "The", " question", " is", " asking", " for", " the", " chemical", " symbols", " of", " gold", " and", " sodium", ",", " as", " well", " as", " the", " element", " with", " atomic", " number", " ", "6", ".", " This", " is", " a", " straightforward", " chemistry", " question", ".\n\n", "Answer", "\n\n", "Gold", ":"], "token_logprobs": [-2.2000410556793213, -0.17005524039268494, -1.8717553615570068, -0.5096925497055054, -0.17935913801193237, -1.6047992706298828, -0.4883747696876526, -0.06896502524614334, -1.721140742301941, -0.5719624757766724, -1.2447757720947266, -0.002695262897759676, -0.0002731903805397451, -0.0044097090139985085, -5.590759246842936e-05, -0.002639264799654484, -0.05336974188685417, -0.2923423945903778, -0.010575325228273869, -0.12636613845825195, -0.4551885724067688, -0.008549043908715248, -0.5778136253356934, -0.30563753843307495, -0.9775493741035461, -0.5769633054733276, -0.12113772332668304, -0.009877494536340237, -0.3929138779640198, -1.0132738680113107e-05, -4.362964682513848e-05, -0.05101730301976204, -0.6842418313026428, -0.04075660929083824, -0.012288807891309261, -6.198863957251888e-06, -2.658331868587993e-05, -1.9788545614574105e-05, -0.2618888318538666, -0.49099308252334595, -1.0397895574569702, -0.09386748820543289, -0.5011305212974548, -0.7632831335067749, -0.8371487259864807, -1.0974842309951782, -0.0003195490571670234, -2.825220326485578e-05, -0.2264319360256195, -0.17046868801116943]}], "scored": [{"text": "The mitochondrion is the powerhouse of the cell, converting ", "prompt_logprobs": [null, {"53582": {"logprob": -11.88830852508545, "rank": 20834}, "154822": {"logprob": -4.341433525085449, "rank": 1}}, {"81": {"logprob": -0.976696789264679, "rank": 2}, "4204": {"logprob": -0.47669678926467896, "rank": 1}}, {"290": {"logprob": -8.702239938429557e-06, "rank": 1}}, {"374": {"logprob": -0.5717750787734985, "rank": 1}}, {"279": {"logprob": -1.036670446395874, "rank": 1}}, {"73538": {"logprob": -3.7466423511505127, "rank": 3}, "1240": {"logprob": -0.24664238095283508, "rank": 1}}, {"315": {"logprob": -0.09506958723068237, "rank": 1}}, {"279": {"logprob": -0.026692954823374748, "rank": 1}}, {"2779": {"logprob": -0.015260251238942146, "rank": 1}}, {"11": {"logprob": -1.3379920721054077, "rank": 2}, "13": {"logprob": -1.0879920721054077, "rank": 1}}, {"33277": {"logprob": -3.904132843017578, "rank": 11}, "323": {"logprob": -1.5291329622268677, "rank": 1}}, {"36209": {"logprob": -1.4794025421142578, "rank": 2}, "4802": {"logprob": -1.3544025421142578, "rank": 1}}, {"1119": {"logprob": -0.1331220418214798, "rank": 1}}, {"993": {"logprob": -3.0450491905212402, "rank": 3}, "66072": {"logprob": -0.4200492799282074, "rank": 1}}, {"70338": {"logprob": -0.0003352795320097357, "rank": 1}}, {"482": {"logprob": -7.939023635117337e-05, "rank": 1}}, {"2406": {"logprob": -0.0007200032705441117, "rank": 1}}, {"759": {"logprob": -0.0009296386269852519, "rank": 1}}, {"91597": {"logprob": -0.0002631794777698815, "rank": 1}}, {"1526": {"logprob": -3.6748030185699463, "rank": 2}, "320": {"logprob": -0.0498029962182045, "rank": 1}}, {"77673": {"logprob": -1.29654860496521, "rank": 2}, "34423": {"logprob": -0.7965485453605652, "rank": 1}}, {"93199": {"logprob": -0.00412779301404953, "rank": 1}}, {"2302": {"logprob": -0.00014065706636756659, "rank": 1}}, {"13": {"logprob": -0.5752795338630676, "rank": 1}}, {"1096": {"logprob": -2.4610178470611572, "rank": 3}, "21714": {"logprob": -1.3985178470611572, "rank": 1}}, {"1882": {"logprob": -0.7489160299301147, "rank": 1}}, {"13657": {"logprob": -3.69407320022583, "rank": 10}, "33482": {"logprob": -1.5690733194351196, "rank": 1}}, {"3941": {"logprob": -1.1760808229446411, "rank": 2}, "518": {"logprob": -1.1760808229446411, "rank": 1}}, {"279": {"logprob": -0.2964847981929779, "rank": 1}}, {"9176": {"logprob": -0.11559481918811798, "rank": 1}}, {"70428": {"logprob": -0.23392535746097565, "rank": 1}}, {"38346": {"logprob": -0.08119023591279984, "rank": 1}}, {"11": {"logprob": -0.5858933329582214, "rank": 1}}, {"1380": {"logprob": -1.9772648811340332, "rank": 2}, "892": {"logprob": -0.7272648811340332, "rank": 1}}, {"279": {"logprob": -0.6106369495391846, "rank": 1}}, {"16698": {"logprob": -0.4787136912345886, "rank": 1}}, {"7557": {"logprob": -0.00492016039788723, "rank": 1}}, {"8780": {"logprob": -0.001347110839560628, "rank": 1}}, {"63111": {"logprob": -1.8965904712677002, "rank": 2}, "320": {"logprob": -1.8965904712677002, "rank": 1}}, {"264": {"logprob": -0.09094314277172089, "rank": 1}}, {"80822": {"logprob": -0.003181635867804289, "rank": 1}}, {"20129": {"logprob": -0.06077831983566284, "rank": 1}}, {"13": {"logprob": -1.994498610496521, "rank": 2}, "429": {"logprob": -0.4944986402988434, "rank": 1}}]}, {"text": "def quicksort(arr):\n    if len(arr) <= 1:\n        return arr", "prompt_logprobs": [null, {"3974": {"logprob": -7.575308322906494, "rank": 230}, "1815": {"logprob": -2.356558322906494, "rank": 1}}, {"6860": {"logprob": -0.2686464190483093, "rank": 1}}, {"10934": {"logprob": -0.7622413635253906, "rank": 1}}, {"982": {"logprob": -0.7858560085296631, "rank": 2}, "11": {"logprob": -0.6608560085296631, "rank": 1}}, {"262": {"logprob": -0.08289364725351334, "rank": 1}}, {"421": {"logprob": -0.17532221972942352, "rank": 1}}, {"2422": {"logprob": -0.02028563991189003, "rank": 1}}, {"10934": {"logprob": -0.0005225961795076728, "rank": 1}}, {"8": {"logprob": -0.061090465635061264, "rank": 1}}, {"2651": {"logprob": -0.1808052957057953, "rank": 1}}, {"220": {"logprob": -0.023333726450800896, "rank": 1}}, {"16": {"logprob": -0.0004866131057497114, "rank": 1}}, {"510": {"logprob": -0.015296528115868568, "rank": 1}}, {"286": {"logprob": -0.0012301980750635266, "rank": 1}}, {"470": {"logprob": -0.00012957210128661245, "rank": 1}}, {"2890": {"logprob": -0.00175453198608011, "rank": 1}}, {"198": {"logprob": -0.08293380588293076, "rank": 1}}, {"262": {"logprob": -0.0005878866068087518, "rank": 1}}, {"25964": {"logprob": -0.22315366566181183, "rank": 1}}, {"284": {"logprob": -0.008056524209678173, "rank": 1}}, {"2890": {"logprob": -0.0010807631770148873, "rank": 1}}, {"24617": {"logprob": -0.16870814561843872, "rank": 1}}, {"10934": {"logprob": -0.00091856240760535, "rank": 1}}, {"8": {"logprob": -0.320453405380249, "rank": 1}}, {"442": {"logprob": -0.020847247913479805, "rank": 1}}, {"220": {"logprob": -0.0025009801611304283, "rank": 1}}, {"17": {"logprob": -0.0022025867365300655, "rank": 1}}, {"921": {"logprob": -0.02611871063709259, "rank": 1}}, {"262": {"logprob": -0.0007001094636507332, "rank": 1}}, {"2115": {"logprob": -0.06169889122247696, "rank": 1}}, {"284": {"logprob": -0.003107839496806264, "rank": 1}}, {"508": {"logprob": -0.004781835246831179, "rank": 1}}, {"87": {"logprob": -0.0019395602867007256, "rank": 1}}, {"369": {"logprob": -0.0003147821989841759, "rank": 1}}, {"856": {"logprob": -4.2199197196168825e-05, "rank": 1}}, {"304": {"logprob": -4.279521817807108e-05, "rank": 1}}, {"2890": {"logprob": -0.00010311071673640981, "rank": 1}}, {"421": {"logprob": -0.00023445718397852033, "rank": 1}}, {"856": {"logprob": -0.00040189296123571694, "rank": 1}}, {"366": {"logprob": -0.0007267932523973286, "rank": 1}}, {"25964": {"logprob": -0.00019131260341964662, "rank": 1}}, {"921": {"logprob": -0.014532433822751045, "rank": 1}}, {"262": {"logprob": -7.188061863416806e-05, "rank": 1}}, {"6149": {"logprob": -0.06814421713352203, "rank": 1}}, {"284": {"logprob": -0.0012931802775710821, "rank": 1}}, {"508": {"logprob": -0.00021908267808612436, "rank": 1}}, {"87": {"logprob": -5.9126061387360096e-05, "rank": 1}}, {"369": {"logprob": -9.16677454370074e-05, "rank": 1}}, {"856": {"logprob": -2.729855441430118e-05, "rank": 1}}, {"304": {"logprob": -5.4238757002167404e-05, "rank": 1}}, {"2890": {"logprob": -2.0861407392658293e-05, "rank": 1}}, {"421": {"logprob": -0.0002650863316375762, "rank": 1}}, {"856": {"logprob": -3.373566141817719e-05, "rank": 1}}, {"621": {"logprob": -0.0009291622554883361, "rank": 1}}, {"25964": {"logprob": -0.00017867876158561558, "rank": 1}}, {"921": {"logprob": -0.0041108159348368645, "rank": 1}}, {"262": {"logprob": -0.0006653240998275578, "rank": 1}}, {"1290": {"logprob": -0.0016655157087370753, "rank": 1}}, {"284": {"logprob": -0.0005004345439374447, "rank": 1}}, {"508": {"logprob": -0.0007721779984422028, "rank": 1}}, {"87": {"logprob": -6.401333666872233e-05, "rank": 1}}, {"369": {"logprob": -5.340433563105762e-05, "rank": 1}}, {"856": {"logprob": -1.168244216387393e-05, "rank": 1}}, {"304": {"logprob": -5.8412379075889476e-06, "rank": 1}}, {"2890": {"logprob": -0.00013565097469836473, "rank": 1}}, {"421": {"logprob": -0.0003771070914808661, "rank": 1}}, {"856": {"logprob": -4.2914423829643056e-05, "rank": 1}}, {"861": {"logprob": -0.00018630675913300365, "rank": 1}}, {"25964": {"logprob": -0.00010752100206445903, "rank": 1}}, {"921": {"logprob": -0.012149614281952381, "rank": 1}}, {"262": {"logprob": -0.0031711787451058626, "rank": 1}}, {"470": {"logprob": -0.004390481859445572, "rank": 1}}, {"3974": {"logprob": -0.0016160060185939074, "rank": 1}}, {"6860": {"logprob": -3.45700973412022e-05, "rank": 1}}, {"17646": {"logprob": -0.002913280390202999, "rank": 1}}, {"8": {"logprob": -0.00032395837479270995, "rank": 1}}, {"488": {"logprob": -5.4834770708112046e-05, "rank": 1}}, {"6149": {"logprob": -0.009906177408993244, "rank": 1}}, {"488": {"logprob": -6.735097849741578e-05, "rank": 1}}, {"3974": {"logprob": -0.0002640137099660933, "rank": 1}}, {"6860": {"logprob": -1.7762025890988298e-05, "rank": 1}}, {"27611": {"logprob": -0.000482442817883566, "rank": 1}}, {"8": {"logprob": -3.615476608276367, "rank": 4}, "692": {"logprob": -0.3654765188694, "rank": 1}}]}, {"text": "In 1969, the Apollo 11 mission successfully landed the first", "prompt_logprobs": [null, {"220": {"logprob": -7.480617523193359, "rank": 64}, "154822": {"logprob": -1.6056172847747803, "rank": 1}}, {"121818": {"logprob": -6.2249884605407715, "rank": 64}, "17": {"logprob": -2.1624884605407715, "rank": 1}}, {"24": {"logprob": -1.972672700881958, "rank": 2}, "23": {"logprob": -1.972672700881958, "rank": 1}}, {"11": {"logprob": -0.36826807260513306, "rank": 1}}, {"279": {"logprob": -1.79671049118042, "rank": 1}}, {"34976": {"logprob": -2.965151786804199, "rank": 2}, "356": {"logprob": -1.6526517868041992, "rank": 1}}, {"220": {"logprob": -0.36428818106651306, "rank": 1}}, {"98965": {"logprob": -0.17343902587890625, "rank": 1}}, {"8951": {"logprob": -0.887779712677002, "rank": 1}}, {"7790": {"logprob": -2.6589407920837402, "rank": 5}, "26039": {"logprob": -1.0339406728744507, "rank": 1}}, {"26039": {"logprob": -0.14171536266803741, "rank": 1}}, {"279": {"logprob": -1.7570018768310547, "rank": 2}, "12671": {"logprob": -0.8820018172264099, "rank": 1}}, {"1156": {"logprob": -0.01747344620525837, "rank": 1}}, {"12671": {"logprob": -0.14814411103725433, "rank": 1}}, {"389": {"logprob": -0.02152218483388424, "rank": 1}}, {"279": {"logprob": -0.0021354984492063522, "rank": 1}}, {"17309": {"logprob": -0.6993345022201538, "rank": 1}}, {"13": {"logprob": -0.508336067199707, "rank": 1}}, {"32962": {"logprob": -1.9407068490982056, "rank": 3}, "1096": {"logprob": -1.4407068490982056, "rank": 1}}, {"44605": {"logprob": -0.0060303411446511745, "rank": 1}}, {"323": {"logprob": -0.2022213637828827, "rank": 1}}, {"37754": {"logprob": -0.10406894236803055, "rank": 1}}, {"30230": {"logprob": -0.00046921687317080796, "rank": 1}}, {"25210": {"logprob": -0.00011419598013162613, "rank": 1}}, {"7391": {"logprob": -1.8751344680786133, "rank": 3}, "6116": {"logprob": -0.7501344084739685, "rank": 1}}, {"13179": {"logprob": -2.2342491149902344, "rank": 5}, "220": {"logprob": -1.234249234199524, "rank": 1}}, {"1378": {"logprob": -2.358257293701172, "rank": 2}, "220": {"logprob": -0.1082572266459465, "rank": 1}}, {"323": {"logprob": -0.1909279227256775, "rank": 1}}, {"264": {"logprob": -0.010183047503232956, "rank": 1}}, {"8337": {"logprob": -8.188130378723145, "rank": 2}, "4279": {"logprob": -0.0006299418746493757, "rank": 1}}, {"4115": {"logprob": -0.004795359913259745, "rank": 1}}, {"4889": {"logprob": -2.0878002643585205, "rank": 3}, "389": {"logprob": -0.7128003239631653, "rank": 1}}, {"279": {"logprob": -0.1015268936753273, "rank": 1}}, {"41305": {"logprob": -0.10137588530778885, "rank": 1}}, {"11": {"logprob": -0.6127365827560425, "rank": 1}}, {"25814": {"logprob": -2.4811112880706787, "rank": 3}, "1393": {"logprob": -0.35611119866371155, "rank": 1}}, {"56329": {"logprob": -0.9727568626403809, "rank": 1}}, {"3684": {"logprob": -1.9196480512619019, "rank": 2}, "10464": {"logprob": -0.29464805126190186, "rank": 1}}, {"311": {"logprob": -3.663879632949829, "rank": 4}, "323": {"logprob": -0.2888796925544739, "rank": 1}}, {"4446": {"logprob": -0.008053804747760296, "rank": 1}}, {"1182": {"logprob": -0.0013903246726840734, "rank": 1}}, {"311": {"logprob": -0.06407589465379715, "rank": 1}}, {"9234": {"logprob": -0.0017547699389979243, "rank": 1}}, {"13": {"logprob": -0.2483280450105667, "rank": 1}}]}, {"text": "Le petit prince demanda au renard ce que signifiait le mot a", "prompt_logprobs": [null, {"44744": {"logprob": -15.774243354797363, "rank": 21121}, "154822": {"logprob": -0.016431139782071114, "rank": 1}}, {"41490": {"logprob": -1.6070984601974487, "rank": 1}}, {"137474": {"logprob": -13.035755157470703, "rank": 7549}, "284": {"logprob": -2.012317419052124, "rank": 1}}, {"7906": {"logprob": -4.875985145568848, "rank": 11}, "549": {"logprob": -1.2509849071502686, "rank": 1}}, {"5672": {"logprob": -6.697552680969238, "rank": 39}, "73000": {"logprob": -1.7600524425506592, "rank": 1}}, {"567": {"logprob": -0.0008012421894818544, "rank": 1}}, {"3761": {"logprob": -4.256261825561523, "rank": 11}, "25": {"logprob": -1.8812617063522339, "rank": 1}}, {"1709": {"logprob": -0.24003396928310394, "rank": 1}}, {"1841": {"logprob": -0.6200358271598816, "rank": 1}}, {"333": {"logprob": -0.008285081014037132, "rank": 1}}, {"685": {"logprob": -0.002595986006781459, "rank": 1}}, {"275": {"logprob": -0.004389532376080751, "rank": 1}}, {"512": {"logprob": -2.3708930015563965, "rank": 3}, "12480": {"logprob": -0.6208929419517517, "rank": 1}}, {"3852": {"logprob": -0.01753881201148033, "rank": 1}}, {"131231": {"logprob": -1.6062474250793457, "rank": 3}, "12480": {"logprob": -0.7312474846839905, "rank": 1}}, {"6496": {"logprob": -5.1377883210079744e-05, "rank": 1}}, {"12053": {"logprob": -0.2506871521472931, "rank": 1}}, {"13": {"logprob": -1.3148317337036133, "rank": 1}}, {"1967": {"logprob": -1.235216736793518, "rank": 1}}, {"5672": {"logprob": -0.03868994116783142, "rank": 1}}, {"567": {"logprob": -0.0002735478919930756, "rank": 1}}, {"3247": {"logprob": -4.084001064300537, "rank": 4}, "24324": {"logprob": -0.33400094509124756, "rank": 1}}, {"5011": {"logprob": -0.00015960850578267127, "rank": 1}}, {"64": {"logprob": -0.0003146630187984556, "rank": 1}}, {"1709": {"logprob": -0.6572995781898499, "rank": 1}}, {"44244": {"logprob": -0.6856521368026733, "rank": 1}}, {"1841": {"logprob": -1.0484671592712402, "rank": 2}, "136015": {"logprob": -0.7984670996665955, "rank": 1}}, {"333": {"logprob": -0.000735608336981386, "rank": 1}}, {"685": {"logprob": -0.0027479289565235376, "rank": 1}}, {"275": {"logprob": -0.0019393223337829113, "rank": 1}}, {"1884": {"logprob": -9.258537292480469, "rank": 46}, "74145": {"logprob": -0.25853681564331055, "rank": 1}}, {"261": {"logprob": -0.0052758511155843735, "rank": 1}}, {"939": {"logprob": -0.003109622048214078, "rank": 1}}, {"151101": {"logprob": -0.00441350694745779, "rank": 1}}, {"11": {"logprob": -1.9404969215393066, "rank": 2}, "13": {"logprob": -0.6904968619346619, "rank": 1}}, {"1842": {"logprob": -1.7482638359069824, "rank": 2}, "1884": {"logprob": -1.3732638359069824, "rank": 1}}, {"1709": {"logprob": -1.00454843044281, "rank": 1}}, {"4403": {"logprob": -3.0520520210266113, "rank": 6}, "26566": {"logprob": -1.6770519018173218, "rank": 1}}, {"512": {"logprob": -2.200911045074463, "rank": 3}, "326": {"logprob": -0.7634111642837524, "rank": 1}}, {"41490": {"logprob": -3.5975301265716553, "rank": 3}, "44744": {"logprob": -0.09753023087978363, "rank": 1}}, {"326": {"logprob": -0.5964891314506531, "rank": 1}}, {"6": {"logprob": -0.13926775753498077, "rank": 1}}, {"138865": {"logprob": -0.022586939856410027, "rank": 1}}, {"6496": {"logprob": -0.08105723559856415, "rank": 1}}, {"285": {"logprob": -0.00013469743134919554, "rank": 1}}, {"1315": {"logprob": -0.0005261705373413861, "rank": 1}}, {"11": {"logprob": -0.03950696811079979, "rank": 1}}, {"44786": {"logprob": -0.5267423391342163, "rank": 1}}, {"38832": {"logprob": -2.8039584159851074, "rank": 3}, "34467": {"logprob": -0.49145829677581787, "rank": 1}}, {"1167": {"logprob": -4.2676016164477915e-05, "rank": 1}}, {"62129": {"logprob": -0.0924116000533104, "rank": 1}}, {"326": {"logprob": -0.026161447167396545, "rank": 1}}, {"21997": {"logprob": -0.0627172589302063, "rank": 1}}, {"409": {"logprob": -0.05548763647675514, "rank": 1}}, {"326": {"logprob": -0.004424425773322582, "rank": 1}}, {"48052": {"logprob": -0.001262344536371529, "rank": 1}}, {"265": {"logprob": -0.000164018536452204, "rank": 1}}, {"13": {"logprob": -1.8726966381072998, "rank": 2}, "1842": {"logprob": -0.24769659340381622, "rank": 1}}]}, {"text": "The gradient of the loss function with respect to the weight", "prompt_logprobs": [null, {"20129": {"logprob": -8.96643352508545, "rank": 969}, "154822": {"logprob": -4.341433525085449, "rank": 1}}, {"315": {"logprob": -0.7039494514465332, "rank": 1}}, {"279": {"logprob": -0.9713228940963745, "rank": 2}, "264": {"logprob": -0.7213228940963745, "rank": 1}}, {"4709": {"logprob": -3.9720547199249268, "rank": 10}, "729": {"logprob": -1.0970547199249268, "rank": 1}}, {"729": {"logprob": -0.33797481656074524, "rank": 1}}, {"448": {"logprob": -1.44989812374115, "rank": 1}}, {"5091": {"logprob": -0.008337335661053658, "rank": 1}}, {"311": {"logprob": -0.0006673493189737201, "rank": 1}}, {"279": {"logprob": -0.3867085874080658, "rank": 1}}, {"14314": {"logprob": -0.8176753520965576, "rank": 1}}, {"374": {"logprob": -1.3674771785736084, "rank": 1}}, {"24113": {"logprob": -2.3279004096984863, "rank": 3}, "264": {"logprob": -1.7029004096984863, "rank": 1}}, {"4566": {"logprob": -2.6241183280944824, "rank": 4}, "1667": {"logprob": -1.124118447303772, "rank": 1}}, {"1182": {"logprob": -0.36660346388816833, "rank": 1}}, {"2674": {"logprob": -0.03230930119752884, "rank": 1}}, {"27048": {"logprob": -0.015215049497783184, "rank": 1}}, {"11": {"logprob": -1.1618281602859497, "rank": 1}}, {"18915": {"logprob": -3.7651920318603516, "rank": 6}, "892": {"logprob": -0.8901921510696411, "rank": 1}}, {"279": {"logprob": -0.01076273899525404, "rank": 1}}, {"8780": {"logprob": -0.009176341816782951, "rank": 1}}, {"5912": {"logprob": -0.0011264891363680363, "rank": 1}}, {"6193": {"logprob": -1.2604855298995972, "rank": 2}, "1526": {"logprob": -1.2604855298995972, "rank": 1}}, {"553": {"logprob": -0.02797655388712883, "rank": 1}}, {"6193": {"logprob": -0.00028046013903804123, "rank": 1}}, {"504": {"logprob": -1.6509439945220947, "rank": 2}, "13": {"logprob": -0.7759440541267395, "rank": 1}}, {"279": {"logprob": -0.052063919603824615, "rank": 1}}, {"2550": {"logprob": -0.07074390351772308, "rank": 1}}, {"1182": {"logprob": -0.15821382403373718, "rank": 1}}, {"311": {"logprob": -0.08365529775619507, "rank": 1}}, {"279": {"logprob": -0.06887133419513702, "rank": 1}}, {"1946": {"logprob": -0.07873330265283585, "rank": 1}}, {"13": {"logprob": -0.4378916621208191, "rank": 1}}, {"794": {"logprob": -7.841183662414551, "rank": 88}, "1096": {"logprob": -1.3411836624145508, "rank": 1}}, {"65474": {"logprob": -0.024642810225486755, "rank": 1}}, {"20129": {"logprob": -1.5382769107818604, "rank": 2}, "52755": {"logprob": -0.28827694058418274, "rank": 1}}, {"36760": {"logprob": -0.003080387134104967, "rank": 1}}, {"1221": {"logprob": -3.1721279621124268, "rank": 5}, "320": {"logprob": -0.422127902507782, "rank": 1}}, {"8836": {"logprob": -0.09941805154085159, "rank": 1}}, {"1817": {"logprob": -1.4260128736495972, "rank": 2}, "279": {"logprob": -0.3010128438472748, "rank": 1}}, {"4680": {"logprob": -0.2558543384075165, "rank": 1}}, {"21070": {"logprob": -6.881373405456543, "rank": 35}, "553": {"logprob": -1.1938731670379639, "rank": 1}}, {"745": {"logprob": -0.007097034715116024, "rank": 1}}, {"13": {"logprob": -8.131097793579102, "rank": 5}, "311": {"logprob": -0.006097879726439714, "rank": 1}}]}]}

scripts/validate.py

@@ -0,0 +1,157 @@
+"""Correctness suite against one running server; dumps JSON for later A/B.
+
+Usage: python3 validate.py <port> <model-name> <out.json>
+
+Collects:
+  1. 17*23 deterministic arithmetic check (temp 0).
+  2. 700+ token generation; flags degenerate repetition (4-gram loop scan).
+  3. 10 fixed greedy generations (50 tok, logprobs=5) - token ids + logprobs.
+  4. prompt_logprobs teacher-forced scoring of 5 fixed paragraphs -
+     per-token logprob under identical context (the clean A/B signal).
+"""
+
+import json
+import sys
+import urllib.request
+
+PORT, MODEL, OUT = int(sys.argv[1]), sys.argv[2], sys.argv[3]
+BASE = f"http://127.0.0.1:{PORT}"
+
+GEN_PROMPTS = [
+    "What is 17*23? Answer with just the number.",
+    "Write a Python function that returns the nth Fibonacci number iteratively.",
+    "Explain the difference between TCP and UDP in two sentences.",
+    "Translate to French: 'The quick brown fox jumps over the lazy dog.'",
+    "List the first 8 prime numbers separated by commas.",
+    "What year did the Apollo 11 mission land on the moon? One word answer.",
+    "Summarize the plot of Romeo and Juliet in one sentence.",
+    "Write a SQL query selecting the top 5 customers by total order value from tables customers(id,name) and orders(id,customer_id,value).",
+    "What is the derivative of x^3 + 2x with respect to x?",
+    "Name the chemical symbol for gold and the element with atomic number 6.",
+]
+
+SCORE_TEXTS = [
+    "The mitochondrion is the powerhouse of the cell, converting nutrients into adenosine triphosphate through oxidative phosphorylation. This process occurs across the inner mitochondrial membrane, where the electron transport chain establishes a proton gradient.",
+    "def quicksort(arr):\n    if len(arr) <= 1:\n        return arr\n    pivot = arr[len(arr) // 2]\n    left = [x for x in arr if x < pivot]\n    middle = [x for x in arr if x == pivot]\n    right = [x for x in arr if x > pivot]\n    return quicksort(left) + middle + quicksort(right)",
+    "In 1969, the Apollo 11 mission successfully landed the first humans on the Moon. Neil Armstrong and Buzz Aldrin spent approximately two and a quarter hours outside the spacecraft, collecting lunar material to bring back to Earth.",
+    "Le petit prince demanda au renard ce que signifiait le mot apprivoiser. Le renard expliqua que cela signifiait creer des liens, et que si le prince l'apprivoisait, ils auraient besoin l'un de l'autre.",
+    "The gradient of the loss function with respect to the weights is computed via backpropagation, applying the chain rule layer by layer from the output back to the input. Stochastic gradient descent then updates each weight proportionally.",
+]
+
+
+def post(path, payload, timeout=600):
+    req = urllib.request.Request(
+        BASE + path,
+        data=json.dumps(payload).encode(),
+        headers={"Content-Type": "application/json"},
+    )
+    with urllib.request.urlopen(req, timeout=timeout) as r:
+        return json.loads(r.read())
+
+
+def degenerate(text, n=4, thresh=12):
+    words = text.split()
+    if len(words) < n * thresh:
+        return False
+    grams = {}
+    for i in range(len(words) - n):
+        g = tuple(words[i : i + n])
+        grams[g] = grams.get(g, 0) + 1
+    return max(grams.values()) >= thresh
+
+
+out = {"port": PORT, "model": MODEL}
+
+# 1. arithmetic
+r = post(
+    "/v1/chat/completions",
+    {
+        "model": MODEL,
+        "messages": [{"role": "user", "content": GEN_PROMPTS[0]}],
+        "temperature": 0,
+        "max_tokens": 2048,
+    },
+)
+msg = r["choices"][0]["message"]
+arith = (msg.get("content") or "") + " " + (msg.get("reasoning_content") or "")
+out["arith_correct"] = "391" in arith
+out["arith_raw"] = (msg.get("content") or "")[:200]
+
+# 2. long generation
+r = post(
+    "/v1/chat/completions",
+    {
+        "model": MODEL,
+        "messages": [
+            {
+                "role": "user",
+                "content": "Write a detailed 800-word essay on the history of container shipping, covering its origins, standardization, and global economic impact.",
+            }
+        ],
+        "temperature": 0,
+        "max_tokens": 2400,
+    },
+)
+msg = r["choices"][0]["message"]
+essay = (msg.get("reasoning_content") or "") + (msg.get("content") or "")
+out["long_gen_tokens"] = r["usage"]["completion_tokens"]
+out["long_gen_degenerate"] = degenerate(essay)
+out["long_gen_tail"] = essay[-300:]
+
+# 3. greedy generations with logprobs (completions API for raw control)
+gens = []
+for p in GEN_PROMPTS:
+    r = post(
+        "/v1/completions",
+        {
+            "model": MODEL,
+            "prompt": p,
+            "temperature": 0,
+            "max_tokens": 50,
+            "logprobs": 5,
+        },
+    )
+    ch = r["choices"][0]
+    gens.append(
+        {
+            "prompt": p,
+            "tokens": ch["logprobs"]["tokens"],
+            "token_logprobs": ch["logprobs"]["token_logprobs"],
+        }
+    )
+out["greedy"] = gens
+
+# 4. teacher-forced prompt logprobs
+scored = []
+for t in SCORE_TEXTS:
+    r = post(
+        "/v1/completions",
+        {
+            "model": MODEL,
+            "prompt": t,
+            "temperature": 0,
+            "max_tokens": 1,
+            "prompt_logprobs": 1,
+        },
+    )
+    ch = r["choices"][0]
+    plp = ch.get("prompt_logprobs")
+    toks = []
+    if plp:
+        for pos in plp:
+            if pos is None:
+                toks.append(None)
+                continue
+            # dict: token_id -> {logprob, rank, decoded_token}; the entry with
+            # rank field present; chosen token is the key matching prompt token
+            entry = {
+                tid: {"logprob": v["logprob"], "rank": v.get("rank")}
+                for tid, v in pos.items()
+            }
+            toks.append(entry)
+    scored.append({"text": t[:60], "prompt_logprobs": toks})
+out["scored"] = scored
+
+json.dump(out, open(OUT, "w"))
+print(json.dumps({k: v for k, v in out.items() if k not in ("greedy", "scored")}, indent=1))
+print(f"greedy prompts: {len(gens)}, scored texts: {len(scored)} -> {OUT}")

serving/Dockerfile

@@ -0,0 +1,17 @@
+# Matched-pair runtime for GLM-5.3-Flash-NVFP4-FP8ATTN-512K.
+# Base: the public per-model vLLM image for glm5_next on SM120 (not redistributed here).
+# The four COPY lines are the ENTIRE delta — see README.md in this directory.
+#
+#   docker build -t local/vllm-glm53:fp8attn-512k .
+FROM cstechdev/vllm:glm53-flash-nope-sm120-cu130-20260826-r1
+
+# (1) KDA layers: stop stripping quant_config at construction (kda.py:168-175 upstream)
+COPY kda.py   /usr/local/lib/python3.12/dist-packages/vllm/models/glm5next/nvidia/kda.py
+# (2) MLA layers (+ MTP draft): quant_config=None -> real config (model.py:329 upstream)
+COPY model.py /usr/local/lib/python3.12/dist-packages/vllm/models/glm5next/nvidia/model.py
+# (3) ModelOptMixedPrecisionConfig: FP8_BLOCK128/64/32 dispatch, fused-name resolver,
+#     MTP draft-prefix aliases, block-FP8 ParallelLMHead loader (vocab-sharded scales)
+COPY modelopt.py /usr/local/lib/python3.12/dist-packages/vllm/model_executor/layers/quantization/modelopt.py
+# (4) SM120-tuned w8a8 block-FP8 triton config for the KDA fused in_proj shape
+#     (N=12576, K=4096, block [32,32]) — worth +60% decode on its own
+COPY configs/*.json /usr/local/lib/python3.12/dist-packages/vllm/model_executor/layers/quantization/utils/configs/

serving/README.md

@@ -0,0 +1,40 @@
+# serving/ — the matched-pair runtime
+
+This checkpoint stores attention, shared/dense-MLP, and `lm_head` weights as block-FP8
+(`F8_E4M3` + FP32 `weight_scale_inv`). The public `glm5_next` vLLM images cannot load that:
+they build every attention layer **unquantized** (the official checkpoints keep attention BF16,
+so the model code strips `quant_config` at layer construction), and they have no FP8
+`ParallelLMHead` loader. The result on an unpatched image is a `KeyError` on the first attention
+scale tensor, or a vocab-embedding shape assert.
+
+The fix is four files on top of `cstechdev/vllm:glm53-flash-nope-sm120-cu130-20260826-r1`
+(vLLM `0.1.dev20051+g487ecf187`), applied by the Dockerfile here:
+
+| file | target path in image | semantic change |
+|---|---|---|
+| `kda.py` | `vllm/models/glm5next/nvidia/kda.py` | remove the save/`None`/restore strip of `vllm_config.quant_config` around KDA base-class construction (lines 168–175 upstream) — the layer now sees the real quant config; the checkpoint's `ignore` list still keeps anything BF16 that should stay BF16 |
+| `model.py` | `vllm/models/glm5next/nvidia/model.py` | `Glm5NextDecoderLayer.__init__`: MLA attention gets `quant_config=quant_config` instead of `quant_config=None` (line 329 upstream); one site covers the 11 MLA layers and the MTP draft layer. The vision-tower `quant_config=None` site is deliberately untouched (upstream warns quantizing the tower NaNs image features; this checkpoint ships the tower BF16) |
+| `modelopt.py` | `vllm/model_executor/layers/quantization/modelopt.py` | `ModelOptMixedPrecisionConfig`: `FP8_BLOCK128` / `FP8_BLOCK64` / `FP8_BLOCK32` per-layer dispatch to dynamic-activation block `Fp8LinearMethod`, fused-module name resolution, MTP draft-prefix aliases, and a `ParallelLMHead` method whose `weight_scale_inv` loader shards scale rows by vocab shard / 128 (exact: 154880 and the 77440-row TP2 shard divide by 128) |
+| `configs/N=12576,K=4096,…block_shape=[32,32].json` | `vllm/model_executor/layers/quantization/utils/configs/` | autotuned triton tile configs for the KDA fused in_proj GEMM on RTX PRO 6000 Blackwell (SM120) — vLLM ships zero SM120 block-FP8 configs; the default config costs 313 µs/call vs 22.3 µs tuned (43.8% of decode GPU time → ~3%) |
+
+Safety property (measured): this patched image is **bitwise identical** to the vendor image for
+BF16-attention checkpoints — the parent NVFP4 checkpoint produces 100.00% teacher-forced
+agreement with |Δlogprob| = 0.00000 under it, because its `quantization_config.ignore` already
+excludes every attention module. The patch only *allows* quantized attention when a checkpoint's
+manifest asks for it.
+
+All three `.py` files are Apache-2.0 vLLM-tree files (SPDX headers retained) carrying local
+modifications; they are **not** MIT-licensed model weights. Credit: the vLLM project; the
+official `glm5_next` per-model image as packaged by
+[chriswritescode-dev/glm-5.3-flash-sm120](https://github.com/chriswritescode-dev/glm-5.3-flash-sm120)
+(our base-image lineage). The base image is pulled from Docker Hub at build time and is not
+redistributed in this repo.
+
+Parallel-work credit: [local-inference-lab](https://github.com/local-inference-lab/vllm)
+independently landed the equivalent MLA quant-config passthrough in their public vLLM fork
+(`dev/jovian-judgement@8590bf9c`, 2026-08-29), paired with MXFP8 attention via the b12x SM120
+kernels. This patch set does the same for the official per-model-image code path and adds the
+block-[32,32] FP8 dispatch for the fused KDA in_proj plus the block-FP8 lm_head loader.
+
+Numerics note for the [32,32] block: the tuned configs keep `BLOCK_SIZE_K=32`, the same k-split
+as the default config, so tuned vs untuned kernels are bit-identical in reduction order.

serving/configs/N=12576,K=4096,device_name=NVIDIA_RTX_PRO_6000_Blackwell_Workstation_Edition,dtype=fp8_w8a8,block_shape=[32,32].json

@@ -0,0 +1,90 @@
+{
+ "1": {
+  "BLOCK_SIZE_M": 16,
+  "BLOCK_SIZE_N": 64,
+  "BLOCK_SIZE_K": 32,
+  "GROUP_SIZE_M": 1,
+  "num_warps": 4,
+  "num_stages": 6
+ },
+ "2": {
+  "BLOCK_SIZE_M": 16,
+  "BLOCK_SIZE_N": 128,
+  "BLOCK_SIZE_K": 32,
+  "GROUP_SIZE_M": 32,
+  "num_warps": 4,
+  "num_stages": 4
+ },
+ "4": {
+  "BLOCK_SIZE_M": 16,
+  "BLOCK_SIZE_N": 128,
+  "BLOCK_SIZE_K": 32,
+  "GROUP_SIZE_M": 32,
+  "num_warps": 4,
+  "num_stages": 4
+ },
+ "8": {
+  "BLOCK_SIZE_M": 16,
+  "BLOCK_SIZE_N": 128,
+  "BLOCK_SIZE_K": 32,
+  "GROUP_SIZE_M": 1,
+  "num_warps": 4,
+  "num_stages": 4
+ },
+ "16": {
+  "BLOCK_SIZE_M": 16,
+  "BLOCK_SIZE_N": 128,
+  "BLOCK_SIZE_K": 32,
+  "GROUP_SIZE_M": 1,
+  "num_warps": 4,
+  "num_stages": 6
+ },
+ "64": {
+  "BLOCK_SIZE_M": 64,
+  "BLOCK_SIZE_N": 64,
+  "BLOCK_SIZE_K": 32,
+  "GROUP_SIZE_M": 8,
+  "num_warps": 4,
+  "num_stages": 3
+ },
+ "256": {
+  "BLOCK_SIZE_M": 128,
+  "BLOCK_SIZE_N": 64,
+  "BLOCK_SIZE_K": 32,
+  "GROUP_SIZE_M": 32,
+  "num_warps": 4,
+  "num_stages": 2
+ },
+ "1024": {
+  "BLOCK_SIZE_M": 64,
+  "BLOCK_SIZE_N": 128,
+  "BLOCK_SIZE_K": 32,
+  "GROUP_SIZE_M": 8,
+  "num_warps": 4,
+  "num_stages": 2
+ },
+ "2048": {
+  "BLOCK_SIZE_M": 128,
+  "BLOCK_SIZE_N": 128,
+  "BLOCK_SIZE_K": 32,
+  "GROUP_SIZE_M": 8,
+  "num_warps": 4,
+  "num_stages": 2
+ },
+ "4096": {
+  "BLOCK_SIZE_M": 128,
+  "BLOCK_SIZE_N": 128,
+  "BLOCK_SIZE_K": 32,
+  "GROUP_SIZE_M": 8,
+  "num_warps": 4,
+  "num_stages": 2
+ },
+ "8192": {
+  "BLOCK_SIZE_M": 128,
+  "BLOCK_SIZE_N": 128,
+  "BLOCK_SIZE_K": 32,
+  "GROUP_SIZE_M": 1,
+  "num_warps": 4,
+  "num_stages": 2
+ }
+}

serving/kda.py

@@ -0,0 +1,655 @@
+# SPDX-License-Identifier: Apache-2.0
+# SPDX-FileCopyrightText: Copyright contributors to the vLLM project
+"""GLM5-Next KDA (linear-attention) layer.
+
+Model-specific, self-contained KDA: separate q/k/v short-conv + the GLM5-Next
+spec-decode verify path + the bounded ``safe_gate`` variant, and ``_forward`` is
+an eager break point under Breakable CUDA Graph
+(``@eager_break_during_capture``).
+
+Moved out of the shared ``kimi_gdn_linear_attn.py`` (which reverts to Kimi
+Linear's fused-conv version): the separate-conv layout + spec-verify are
+GLM5-Next-only. ``forward`` calls ``self._forward`` directly (no
+``torch.ops.vllm.kda_attention`` indirection) so the only un-capturable work is
+the decorated ``_forward``.
+"""
+
+import torch
+from torch import nn
+
+from vllm.compilation.breakable_cudagraph import eager_break_during_capture
+from vllm.config import VllmConfig, get_current_vllm_config
+from vllm.distributed import divide
+from vllm.forward_context import get_forward_context
+from vllm.model_executor.layers.linear import (
+    ColumnParallelLinear,
+    MergedColumnParallelLinear,
+    RowParallelLinear,
+)
+from vllm.model_executor.layers.mamba.gdn.base import GatedDeltaNetAttention
+from vllm.model_executor.layers.mamba.mamba_utils import (
+    MambaStateDtypeCalculator,
+    MambaStateShapeCalculator,
+    is_conv_state_dim_first,
+)
+from vllm.model_executor.layers.mamba.ops.causal_conv1d import (
+    causal_conv1d_fn,
+    causal_conv1d_update,
+)
+from vllm.model_executor.layers.mamba.ops.gather_initial_states import (
+    gather_initial_states,
+)
+from vllm.model_executor.layers.mamba.ops.scatter_states import scatter_states
+from vllm.model_executor.model_loader.weight_utils import sharded_weight_loader
+from vllm.model_executor.utils import (
+    maybe_disable_graph_partition,
+    set_weight_attrs,
+)
+from vllm.platforms import current_platform
+from vllm.third_party.flash_linear_attention.ops.kda import (
+    FusedRMSNormGated,
+    chunk_kda_with_fused_gate,
+    fused_recurrent_kda,
+)
+from vllm.transformers_utils.configs.kimi_linear import KimiLinearConfig
+from vllm.v1.attention.backends.gdn_attn import GDNAttentionMetadata
+
+
+class _Glm5NextMergedColumnParallelLinear(MergedColumnParallelLinear):
+    """Merged projection with multiple replicated output shards.
+
+    Extends K3's ``_KimiGDNMergedColumnParallelLinear`` to support two
+    replicated shards (f_a, g_a) instead of one. Pre-multiplies each
+    replicated entry's output_size by tp_size so the per-rank shard
+    divides back to the full size, and forces tp_rank=0 during weight
+    loading for replicated shards.
+    """
+
+    def __init__(
+        self,
+        input_size: int,
+        output_sizes: list[int],
+        replicated_shard_ids: tuple[int, ...],
+        tp_size: int,
+        **kwargs,
+    ) -> None:
+        self.replicated_shard_ids = set(replicated_shard_ids)
+        output_sizes = output_sizes.copy()
+        for sid in self.replicated_shard_ids:
+            output_sizes[sid] *= tp_size
+        super().__init__(input_size, output_sizes, **kwargs)
+
+    def weight_loader(
+        self,
+        param: nn.Parameter,
+        loaded_weight: torch.Tensor,
+        loaded_shard_id: tuple[int, ...] | int | None = None,
+    ) -> None:
+        tp_rank = self.tp_rank
+        param_tp_rank = getattr(param, "tp_rank", None)
+        if loaded_shard_id in self.replicated_shard_ids:
+            self.tp_rank = 0
+            if param_tp_rank is not None:
+                param.tp_rank = 0
+        try:
+            super().weight_loader(param, loaded_weight, loaded_shard_id)
+        finally:
+            self.tp_rank = tp_rank
+            if param_tp_rank is not None:
+                param.tp_rank = param_tp_rank
+
+    def weight_loader_v2(
+        self,
+        param: nn.Parameter,
+        loaded_weight: torch.Tensor,
+        loaded_shard_id: tuple[int, ...] | int | None = None,
+    ) -> None:
+        tp_rank = self.tp_rank
+        param_tp_rank = getattr(param, "tp_rank", None)
+        if loaded_shard_id in self.replicated_shard_ids:
+            self.tp_rank = 0
+            if param_tp_rank is not None:
+                param.tp_rank = 0
+        try:
+            super().weight_loader_v2(param, loaded_weight, loaded_shard_id)
+        finally:
+            self.tp_rank = tp_rank
+            if param_tp_rank is not None:
+                param.tp_rank = param_tp_rank
+
+
+@torch.compile(
+    dynamic=True,
+    backend=current_platform.simple_compile_backend,
+    options=maybe_disable_graph_partition(current_platform.simple_compile_backend),
+)
+def _cast_sigmoid(x: torch.Tensor) -> torch.Tensor:
+    """Fuse the fp32 cast + sigmoid into one Inductor kernel."""
+    return x.float().sigmoid()
+
+
+class Glm5NextLinearAttention(GatedDeltaNetAttention):
+    # Declared int (set in __init__ from config) so mypy doesn't see the
+    # getattr-derived `Any | None` at the kernel call sites.
+    head_dim: int
+    num_heads: int
+    conv_size: int
+
+    def get_state_dtype(
+        self,
+    ) -> tuple[torch.dtype, torch.dtype]:
+        if self.model_config is None or self.cache_config is None:
+            raise ValueError("model_config and cache_config must be set")
+        return MambaStateDtypeCalculator.kda_state_dtype(
+            self.model_config.dtype, self.cache_config.mamba_cache_dtype
+        )
+
+    def get_state_shape(
+        self,
+    ) -> tuple[tuple[int, ...], tuple[int, ...]]:
+        # conv_state width must include num_spec so the spec-decode conv update
+        # (causal_conv1d_update with num_accepted_tokens + max_query_len) can
+        # slide the window across the draft-verify tokens without reading past
+        # the allocated width. Matches qwen_gdn_linear_attn.get_state_shape.
+        return MambaStateShapeCalculator.kda_state_shape(
+            self.tp_size,
+            self.num_heads,
+            self.head_dim,
+            conv_kernel_size=self.conv_size,
+            num_spec=self.num_spec,
+        )
+
+    def __init__(
+        self,
+        config: KimiLinearConfig,
+        vllm_config: VllmConfig,
+        prefix: str = "",
+    ) -> None:
+        # LOCAL PATCH (fp8attn-r2): pass the real quant config through so KDA
+        # projections can be FP8-resident when the checkpoint declares them in
+        # a MIXED_PRECISION quantized_layers manifest. Checkpoints that keep
+        # attention BF16 (e.g. the stock NVFP4 export) are unaffected: their
+        # quantization_config `ignore` list names every self_attn module
+        # (including the fused `in_proj_qkvbfg_a` spelling), so every KDA
+        # linear still resolves to UnquantizedLinearMethod.
+        # Was: save/None/restore strip of vllm_config.quant_config.
+        super().__init__(config, vllm_config, prefix)
+
+        # Linear-attention head config: read the flattened top-level fields when
+        # present (new schema); fall back to the legacy linear_attn_config dict
+        # otherwise (shared base is also used by KimiLinearConfig). Narrow via
+        # locals so the int-typed attrs are assigned a non-None value.
+        head_dim = getattr(config, "linear_head_dim", None)
+        num_heads = getattr(config, "linear_num_heads", None)
+        conv_size = getattr(config, "linear_conv_kernel_dim", None)
+        if head_dim is None or num_heads is None or conv_size is None:
+            kda_config = config.linear_attn_config  # type: ignore[attr-defined]
+            assert kda_config is not None, "linear_attn_config must be set"
+            head_dim = kda_config["head_dim"]
+            num_heads = kda_config["num_heads"]
+            conv_size = kda_config["short_conv_kernel_size"]
+        assert head_dim is not None
+        assert num_heads is not None
+        assert conv_size is not None
+        self.head_dim = head_dim
+        self.num_heads = num_heads
+        self.conv_size = conv_size
+        assert self.num_heads % self.tp_size == 0
+        self.local_num_heads = divide(self.num_heads, self.tp_size)
+
+        projection_size = self.head_dim * self.num_heads
+        self.local_projection_size = divide(projection_size, self.tp_size)
+
+        # Merge q, k, v, b, f_a, g_a projections into one GEMM (6→1 launches).
+        # Order matches checkpoint's fused_qkvbfg_a_proj convention.
+        # Shards 4 (f_a) and 5 (g_a) are replicated across TP ranks.
+        self.in_proj_qkvbfg_a = _Glm5NextMergedColumnParallelLinear(
+            self.hidden_size,
+            [
+                projection_size,  # q (shard 0)
+                projection_size,  # k (shard 1)
+                projection_size,  # v (shard 2)
+                self.num_heads,  # b (shard 3)
+                self.head_dim,  # f_a (shard 4, replicated)
+                self.head_dim,  # g_a (shard 5, replicated)
+            ],
+            replicated_shard_ids=(4, 5),
+            tp_size=self.tp_size,
+            bias=False,
+            quant_config=self.quant_config,
+            prefix=f"{prefix}.in_proj_qkvbfg_a",
+        )
+
+        self.f_b_proj = ColumnParallelLinear(
+            self.head_dim,
+            projection_size,
+            bias=False,
+            quant_config=self.quant_config,
+            prefix=f"{prefix}.f_b_proj",
+        )
+        self.dt_bias = nn.Parameter(
+            torch.empty(divide(projection_size, self.tp_size), dtype=torch.float32)
+        )
+
+        set_weight_attrs(self.dt_bias, {"weight_loader": sharded_weight_loader(0)})
+
+        self.q_conv1d = ColumnParallelLinear(
+            input_size=self.conv_size,
+            output_size=projection_size,
+            bias=False,
+            params_dtype=torch.float32,
+            prefix=f"{prefix}.q_conv1d",
+        )
+        self.k_conv1d = ColumnParallelLinear(
+            input_size=self.conv_size,
+            output_size=projection_size,
+            bias=False,
+            params_dtype=torch.float32,
+            prefix=f"{prefix}.k_conv1d",
+        )
+        self.v_conv1d = ColumnParallelLinear(
+            input_size=self.conv_size,
+            output_size=projection_size,
+            bias=False,
+            params_dtype=torch.float32,
+            prefix=f"{prefix}.v_conv1d",
+        )
+        # unsqueeze to fit conv1d weights shape into the linear weights shape.
+        # Can't do this in `weight_loader` since it already exists in
+        # `ColumnParallelLinear` and `set_weight_attrs`
+        # doesn't allow to override it
+        self.q_conv1d.weight.data = self.q_conv1d.weight.data.unsqueeze(1)
+        self.k_conv1d.weight.data = self.k_conv1d.weight.data.unsqueeze(1)
+        self.v_conv1d.weight.data = self.v_conv1d.weight.data.unsqueeze(1)
+        # Lazily-built merged q|k|v conv weight (built on first forward, after
+        # weights are loaded). See _forward.
+        self._merged_conv_weight: torch.Tensor | None = None
+
+        self.A_log = nn.Parameter(
+            torch.empty(1, 1, self.local_num_heads, 1, dtype=torch.float32)
+        )
+        set_weight_attrs(self.A_log, {"weight_loader": sharded_weight_loader(2)})
+
+        self.g_b_proj = ColumnParallelLinear(
+            self.head_dim,
+            projection_size,
+            bias=False,
+            quant_config=self.quant_config,
+            prefix=f"{prefix}.g_b_proj",
+        )
+        self.o_norm = FusedRMSNormGated(self.head_dim, activation="sigmoid")
+        self.o_proj = RowParallelLinear(
+            projection_size,
+            self.hidden_size,
+            bias=False,
+            quant_config=self.quant_config,
+            prefix=f"{prefix}.o_proj",
+        )
+
+        compilation_config = get_current_vllm_config().compilation_config
+        if prefix in compilation_config.static_forward_context:
+            raise ValueError(f"Duplicate layer name: {prefix}")
+        compilation_config.static_forward_context[prefix] = self
+
+        # GLM5-Next checkpoints A_log as 1-D (num_heads,); the param is 4-D, so
+        # reshape on load before the sharded loader runs.
+        def _a_log_weight_loader(param, loaded_weight):
+            if loaded_weight.dim() == 1:
+                loaded_weight = loaded_weight.view([1, 1, -1, 1])
+            return sharded_weight_loader(2)(param, loaded_weight)
+
+        self.A_log.weight_loader = _a_log_weight_loader
+
+        # Bounded KDA gate variant: GLM5-Next uses
+        # y = lower_bound * sigmoid(exp(A)*(g+g_bias)) instead of the default
+        # unbounded y = -exp(A)*softplus(g+g_bias). Read by _forward.
+        linear_lower_bound = getattr(config, "linear_lower_bound", None)
+        if linear_lower_bound is not None:
+            self.kda_safe_gate = True
+            self.kda_lower_bound = linear_lower_bound
+        else:
+            legacy = getattr(config, "linear_attn_config", None) or {}
+            if legacy.get("safe_gate", True):
+                self.kda_safe_gate = True
+                self.kda_lower_bound = legacy.get("lower_bound", -5.0)
+            else:
+                self.kda_safe_gate = False
+                self.kda_lower_bound = -5.0
+        # Process-global conv-state layout, resolved once here instead of on
+        # every _forward call (it reads an env-derived flag each time).
+        self._conv_state_dim_first = is_conv_state_dim_first()
+
+    def forward(
+        self,
+        hidden_states: torch.Tensor,
+        positions: torch.Tensor,
+    ) -> torch.Tensor:
+        num_tokens = hidden_states.size(0)
+        # One merged GEMM for q, k, v, b, f_a, g_a (replaces 6 separate GEMMs).
+        projected = self.in_proj_qkvbfg_a(hidden_states)[0]
+        qkv, beta_raw, f_a, g_a = projected.split(
+            [
+                3 * self.local_projection_size,
+                self.local_num_heads,
+                self.head_dim,
+                self.head_dim,
+            ],
+            dim=-1,
+        )
+
+        # Beta stays raw (bf16) here: the recurrent kernel sigmoids it in fp32
+        # at load (SIGMOID_BETA), and only the chunked prefill path needs the
+        # pre-computed fp32 sigmoid — computed lazily in _forward. Pure decode
+        # / spec-verify steps then skip the _cast_sigmoid kernel and its fp32
+        # intermediate entirely.
+        beta = beta_raw.unsqueeze(0)
+        g1 = self.f_b_proj(f_a)[0]
+        g1 = g1.reshape(1, -1, self.local_num_heads, self.head_dim)
+
+        g_proj_states = self.g_b_proj(g_a)[0]
+        # Must stay 3D: rms_norm_gated reads H from g.shape[-2].
+        g2 = g_proj_states.reshape(-1, self.local_num_heads, self.head_dim)
+
+        core_attn_out = torch.empty(
+            (1, num_tokens, self.local_num_heads, self.head_dim),
+            dtype=hidden_states.dtype,
+            device=hidden_states.device,
+        )
+        # Call _forward directly (not via the registered op) so the KDA core
+        # is an eager break point under Breakable CG, mirroring KimiK3's KDA
+        # (vllm/models/kimi_k3/nvidia/kda.py). torch.ops.vllm.kda_attention is
+        # neither a splitting op nor @eager_break_during_capture-decorated, so
+        # routing through it lets the host-branching prefill body be
+        # Inductor-compiled + stream-captured under PIECEWISE -> stale garbage.
+        # qkv stays merged through the short-conv (one conv call, not three).
+        self._forward(
+            qkv_proj_states=qkv,
+            g1=g1,
+            beta=beta,
+            core_attn_out=core_attn_out,
+        )
+        core_attn_out = self.o_norm(core_attn_out, g2)
+        core_attn_out = core_attn_out.reshape(core_attn_out.size(1), -1)
+        return self.o_proj(core_attn_out)[0]
+
+    @eager_break_during_capture
+    def _forward(
+        self,
+        qkv_proj_states: torch.Tensor,
+        g1: torch.Tensor,
+        beta: torch.Tensor,
+        core_attn_out: torch.Tensor,
+    ) -> None:
+        forward_context = get_forward_context()
+        attn_metadata_raw = forward_context.attn_metadata
+
+        if attn_metadata_raw is None:
+            #     # V1 profile run
+            return
+
+        assert isinstance(attn_metadata_raw, dict)
+        attn_metadata_narrowed = attn_metadata_raw[self.prefix]
+        assert isinstance(attn_metadata_narrowed, GDNAttentionMetadata)
+        has_initial_state = attn_metadata_narrowed.has_initial_state
+        non_spec_query_start_loc = attn_metadata_narrowed.non_spec_query_start_loc
+        non_spec_state_indices_tensor = (
+            attn_metadata_narrowed.non_spec_state_indices_tensor
+        )  # noqa: E501
+        num_actual_tokens = attn_metadata_narrowed.num_actual_tokens
+        # Spec-decode metadata (all None when speculative decoding is disabled).
+        spec_sequence_masks = attn_metadata_narrowed.spec_sequence_masks
+        spec_query_start_loc = attn_metadata_narrowed.spec_query_start_loc
+        spec_state_indices_tensor = attn_metadata_narrowed.spec_state_indices_tensor
+        spec_token_indx = attn_metadata_narrowed.spec_token_indx
+        non_spec_token_indx = attn_metadata_narrowed.non_spec_token_indx
+        num_accepted_tokens = attn_metadata_narrowed.num_accepted_tokens
+        num_spec_decodes = attn_metadata_narrowed.num_spec_decodes
+        use_spec = spec_sequence_masks is not None and num_spec_decodes > 0
+        # KDA gate variant: GLM5-Next checkpoints with
+        # linear_attn_config["safe_gate"]=True use the bounded gate
+        # y=lower_bound*sigmoid(exp(A)*(g+g_bias)) instead of the default
+        # unbounded y=-exp(A)*softplus(g+g_bias). Both attrs are always set
+        # in __init__ (this class is GLM5Next-only).
+        safe_gate = self.kda_safe_gate
+        lower_bound = self.kda_lower_bound
+        constant_caches = self.kv_cache
+
+        qkv_proj_states = qkv_proj_states[:num_actual_tokens]
+        g1 = g1[:, :num_actual_tokens]
+        beta = beta[:, :num_actual_tokens]
+
+        (conv_state, recurrent_state) = constant_caches
+        # conv_state must be (..., dim, width-1) for the conv kernels.
+        # DS layout stores it that way directly; SD layout needs a transpose.
+        # Layout is process-global and resolved once at init (see __init__).
+        if not self._conv_state_dim_first:
+            conv_state = conv_state.transpose(-1, -2)
+
+        # One merged short-conv over q|k|v instead of three separate calls. The
+        # 1D conv is independent per channel, so concatenating q/k/v along the
+        # channel dim and running a single causal_conv1d is bit-identical to
+        # three calls. The merged weight is q|k|v conv weights concatenated;
+        # built once and cached (params are fixed after load). conv_state is
+        # already stored as the merged q|k|v state, so it is used directly.
+        if self._merged_conv_weight is None:
+
+            def _w(m):
+                return m.weight.view(m.weight.size(0), m.weight.size(2))
+
+            self._merged_conv_weight = torch.cat(
+                [_w(self.q_conv1d), _w(self.k_conv1d), _w(self.v_conv1d)],
+                dim=0,
+            ).contiguous()
+        conv_weights = self._merged_conv_weight
+        conv_bias = self.q_conv1d.bias
+
+        # Split projections / gating into spec (draft-verify) and non-spec token
+        # groups when speculative decoding is active. Spec tokens carry
+        # num_spec+1 recurrent-state columns each and are advanced with
+        # num_accepted_tokens for rejection-sampling rollback; non-spec tokens
+        # are one-per-request. Mirrors olmo_gdn_linear_attn.py. Projections are
+        # [n, *] (token dim 0); g1/beta are [1, n, h, d] (token dim 1).
+        if use_spec:
+            # In a pure spec-verify step (no non-spec tokens) the metadata
+            # builder sets spec_token_indx = arange(num_actual_tokens), making
+            # the index_select calls below identity copies. Skip them on this
+            # steady-state decode hot path. The outputs alias the inputs here;
+            # the downstream conv/recurrent kernels read them without mutating
+            # in place, so the aliasing is safe.
+            if non_spec_token_indx is None or non_spec_token_indx.numel() == 0:
+                qkv_spec = qkv_proj_states
+                g1_spec = g1
+                beta_spec = beta
+            else:
+                qkv_spec = qkv_proj_states.index_select(0, spec_token_indx)
+                g1_spec = g1.index_select(1, spec_token_indx)
+                beta_spec = beta.index_select(1, spec_token_indx)
+            if non_spec_token_indx is not None and non_spec_token_indx.numel() > 0:
+                qkv_ns = qkv_proj_states.index_select(0, non_spec_token_indx)
+                g1_ns = g1.index_select(1, non_spec_token_indx)
+                beta_ns = beta.index_select(1, non_spec_token_indx)
+            else:
+                qkv_ns = g1_ns = beta_ns = None
+        else:
+            qkv_spec = g1_spec = beta_spec = None
+            qkv_ns, g1_ns, beta_ns = qkv_proj_states, g1, beta
+
+        # --- causal conv1d: spec (draft-verify) path ---
+        if use_spec:
+            assert spec_state_indices_tensor is not None
+            assert num_accepted_tokens is not None
+            conv_idx = spec_state_indices_tensor[:, 0][:num_spec_decodes]
+            conv_mql = spec_state_indices_tensor.size(-1)
+            qkv_spec = causal_conv1d_update(
+                qkv_spec,
+                conv_state,
+                conv_weights,
+                conv_bias,
+                activation="silu",
+                conv_state_indices=conv_idx,
+                num_accepted_tokens=num_accepted_tokens,
+                query_start_loc=spec_query_start_loc,
+                max_query_len=conv_mql,
+            )
+            q_spec, k_spec, v_spec = qkv_spec.split(self.local_projection_size, dim=-1)
+
+        # --- causal conv1d: non-spec path (prefill or plain decode) ---
+        q_ns = k_ns = v_ns = None
+        if attn_metadata_narrowed.num_prefills > 0:
+            assert qkv_ns is not None
+            qkv_ns = causal_conv1d_fn(
+                qkv_ns.transpose(0, 1),
+                conv_weights,
+                conv_bias,
+                activation="silu",
+                conv_states=conv_state,
+                has_initial_state=has_initial_state,
+                cache_indices=non_spec_state_indices_tensor,
+                query_start_loc=non_spec_query_start_loc,
+                metadata=attn_metadata_narrowed,
+            ).transpose(0, 1)
+            q_ns, k_ns, v_ns = qkv_ns.split(self.local_projection_size, dim=-1)
+        elif attn_metadata_narrowed.num_decodes > 0:
+            assert non_spec_state_indices_tensor is not None
+            decode_conv_indices = non_spec_state_indices_tensor[
+                : attn_metadata_narrowed.num_decodes
+            ]
+            qkv_ns = causal_conv1d_update(
+                qkv_ns,
+                conv_state,
+                conv_weights,
+                conv_bias,
+                activation="silu",
+                conv_state_indices=decode_conv_indices,
+            )
+            q_ns, k_ns, v_ns = qkv_ns.split(self.local_projection_size, dim=-1)
+
+        def _rearr(x):
+            return x.reshape(1, -1, self.local_num_heads, self.head_dim)
+
+        # --- core attention: spec (draft-verify) path ---
+        core_attn_out_spec = None
+        # In a pure spec-verify step (no non-spec tokens) the recurrent kernel
+        # can write straight into the layer output buffer, skipping the
+        # fresh allocation + copy below. Mixed steps must scatter via
+        # spec_token_indx, so they keep the kernel-managed output.
+        spec_out = (
+            core_attn_out[0, :num_actual_tokens].unsqueeze(0)
+            if non_spec_token_indx is None or non_spec_token_indx.numel() == 0
+            else None
+        )
+        if use_spec:
+            assert spec_state_indices_tensor is not None
+            assert num_accepted_tokens is not None
+            assert spec_query_start_loc is not None
+            # Gate computed inside the recurrent kernel (COMPUTE_GATE) from
+            # raw g1 — replicates fused_kda_gate's arithmetic bit-for-bit and
+            # skips its launch + fp32 [n, H, D] intermediate per layer.
+            core_attn_out_spec, _ = fused_recurrent_kda(
+                q=_rearr(q_spec),
+                k=_rearr(k_spec),
+                v=_rearr(v_spec),
+                g=g1_spec,
+                beta=beta_spec,
+                initial_state=recurrent_state,
+                use_qk_l2norm_in_kernel=True,
+                cu_seqlens=spec_query_start_loc[: num_spec_decodes + 1],
+                ssm_state_indices=spec_state_indices_tensor,
+                num_accepted_tokens=num_accepted_tokens,
+                out=spec_out,
+                sigmoid_beta=True,
+                a_log=self.A_log,
+                g_bias=self.dt_bias,
+                compute_gate=True,
+                lower_bound=lower_bound,
+            )
+
+        # --- core attention: non-spec path (prefill or plain decode) ---
+        core_attn_out_non_spec = None
+        # Only the plain-decode recurrent kernel can write straight into the
+        # layer output buffer; the chunked prefill kernel cannot, so this
+        # stays None there and the merge copy below runs as before.
+        ns_out = None
+        if attn_metadata_narrowed.num_prefills > 0:
+            assert q_ns is not None
+            assert non_spec_state_indices_tensor is not None
+            assert has_initial_state is not None
+            initial_state = gather_initial_states(
+                recurrent_state, non_spec_state_indices_tensor, has_initial_state
+            )
+            (
+                core_attn_out_non_spec,
+                last_recurrent_state,
+            ) = chunk_kda_with_fused_gate(
+                q=_rearr(q_ns),
+                k=_rearr(k_ns),
+                v=_rearr(v_ns),
+                raw_g=g1_ns,
+                # Chunk path wants the pre-sigmoided fp32 beta (its kernels
+                # don't sigmoid); beta_ns is raw bf16 from forward.
+                beta=_cast_sigmoid(beta_ns.squeeze(0)).unsqueeze(0),
+                A_log=self.A_log,
+                g_bias=self.dt_bias,
+                initial_state=initial_state,
+                output_final_state=True,
+                use_qk_l2norm_in_kernel=True,
+                cu_seqlens=non_spec_query_start_loc,
+                safe_gate=safe_gate,
+                lower_bound=lower_bound,
+            )
+            # Init cache
+            scatter_states(
+                recurrent_state,
+                last_recurrent_state,
+                non_spec_state_indices_tensor,
+            )
+        elif attn_metadata_narrowed.num_decodes > 0:
+            assert non_spec_query_start_loc is not None
+            assert non_spec_state_indices_tensor is not None
+            # Plain decode step (no spec tokens): token order is dense, so the
+            # kernel can write straight into the layer output buffer. A mixed
+            # step scatters non-spec output via non_spec_token_indx instead.
+            # Gate computed in-kernel (COMPUTE_GATE), beta sigmoided in-kernel.
+            if not use_spec:
+                ns_out = spec_out
+            core_attn_out_non_spec, _ = fused_recurrent_kda(
+                q=_rearr(q_ns),
+                k=_rearr(k_ns),
+                v=_rearr(v_ns),
+                g=g1_ns,
+                beta=beta_ns,
+                initial_state=recurrent_state,
+                use_qk_l2norm_in_kernel=True,
+                cu_seqlens=non_spec_query_start_loc[
+                    : attn_metadata_narrowed.num_decodes + 1
+                ],
+                ssm_state_indices=non_spec_state_indices_tensor,
+                out=ns_out,
+                sigmoid_beta=True,
+                a_log=self.A_log,
+                g_bias=self.dt_bias,
+                compute_gate=True,
+                lower_bound=lower_bound,
+            )
+
+        # --- merge spec / non-spec outputs back into token order ---
+        if use_spec and core_attn_out_non_spec is not None:
+            assert core_attn_out_spec is not None
+            merged = torch.empty(
+                (1, num_actual_tokens, *core_attn_out_spec.shape[2:]),
+                dtype=core_attn_out_non_spec.dtype,
+                device=core_attn_out_non_spec.device,
+            )
+            merged.index_copy_(1, spec_token_indx, core_attn_out_spec)
+            merged.index_copy_(1, non_spec_token_indx, core_attn_out_non_spec)
+            core_attn_out[0, :num_actual_tokens] = merged.squeeze(0)
+        elif use_spec:
+            assert core_attn_out_spec is not None
+            if spec_out is None:
+                core_attn_out[0, :num_actual_tokens] = core_attn_out_spec.squeeze(0)
+        else:
+            assert core_attn_out_non_spec is not None
+            if ns_out is None:
+                core_attn_out[0, :num_actual_tokens] = core_attn_out_non_spec[
+                    0, :num_actual_tokens
+                ]

serving/model.py

@@ -0,0 +1,1229 @@
+# SPDX-License-Identifier: Apache-2.0
+# SPDX-FileCopyrightText: Copyright contributors to the vLLM project
+
+from collections.abc import Iterable
+from typing import ClassVar, Literal
+
+import torch
+from torch import nn
+
+from vllm.config import ParallelConfig, VllmConfig
+from vllm.distributed import (
+    get_ep_group,
+    get_pp_group,
+    get_tensor_model_parallel_rank,
+    get_tensor_model_parallel_world_size,
+    tensor_model_parallel_all_gather,
+)
+from vllm.logger import init_logger
+from vllm.model_executor.layers.activation import SiluAndMul, SiluAndMulWithClamp
+from vllm.model_executor.layers.fused_moe import (
+    FusedMoEFactory,
+    GateLinear,
+    fused_moe_make_expert_params_mapping,
+)
+from vllm.model_executor.layers.layernorm import RMSNorm
+from vllm.model_executor.layers.linear import (
+    MergedColumnParallelLinear,
+    RowParallelLinear,
+)
+from vllm.model_executor.layers.logits_processor import LogitsProcessor
+from vllm.model_executor.layers.mamba.mamba_utils import (
+    MambaStateCopyFunc,
+    MambaStateCopyFuncCalculator,
+    MambaStateDtypeCalculator,
+    MambaStateShapeCalculator,
+)
+from vllm.model_executor.layers.mhc import (
+    MHCFusedPostPreOp,
+    MHCPostOp,
+    MHCPreOp,
+    hc_contract,
+    hc_expand,
+)
+from vllm.model_executor.layers.quantization import QuantizationConfig
+from vllm.model_executor.layers.quantization.utils.quant_utils import (
+    GroupShape,
+    scaled_dequantize,
+)
+from vllm.model_executor.layers.vocab_parallel_embedding import (
+    ParallelLMHead,
+    VocabParallelEmbedding,
+)
+from vllm.model_executor.model_loader.weight_utils import (
+    default_weight_loader,
+    maybe_remap_kv_scale_name,
+)
+from vllm.model_executor.models.deepseek_v2 import _get_moe_router_dtype
+from vllm.model_executor.models.glm4_1v import (
+    Glm4vDummyInputsBuilder,
+    Glm4vForConditionalGeneration,
+)
+from vllm.model_executor.models.interfaces import (
+    HasInnerState,
+    IsHybrid,
+    MixtureOfExperts,
+    SupportsPP,
+)
+from vllm.model_executor.models.utils import (
+    AutoWeightsLoader,
+    PPMissingLayer,
+    init_vllm_registered_model,
+    is_pp_missing_parameter,
+    make_layers,
+    maybe_prefix,
+    sequence_parallel_chunk,
+)
+from vllm.models.common.ops.sequence_parallel import (
+    sp_all_gather,
+    sp_reduce_scatter,
+    sp_shard,
+)
+from vllm.multimodal import MULTIMODAL_REGISTRY
+from vllm.platforms import current_platform
+from vllm.sequence import IntermediateTensors
+from vllm.transformers_utils.configs.glm5_next import Glm5NextConfig
+
+from .attention import Glm5NextMLAAttention
+from .kda import Glm5NextLinearAttention
+from .multimodal import (
+    Glm5NextMultiModalProcessor,
+    Glm5NextProcessingInfo,
+    Glm5NextVisionTransformer,
+)
+
+logger = init_logger(__name__)
+
+
+class Glm5NextMLP(nn.Module):
+    def __init__(
+        self,
+        hidden_size: int,
+        intermediate_size: int,
+        hidden_act: str,
+        quant_config: QuantizationConfig | None = None,
+        reduce_results: bool = True,
+        is_sequence_parallel=False,
+        prefix: str = "",
+        swiglu_limit: float | None = None,
+    ) -> None:
+        super().__init__()
+
+        # If is_sequence_parallel, the input and output tensors are sharded
+        # across the ranks within the tp_group. In this case the weights are
+        # replicated and no collective ops are needed.
+        # Otherwise we use standard TP with an allreduce at the end.
+        self.gate_up_proj = MergedColumnParallelLinear(
+            hidden_size,
+            [intermediate_size] * 2,
+            bias=False,
+            quant_config=quant_config,
+            disable_tp=is_sequence_parallel,
+            prefix=f"{prefix}.gate_up_proj",
+        )
+        self.down_proj = RowParallelLinear(
+            intermediate_size,
+            hidden_size,
+            bias=False,
+            quant_config=quant_config,
+            reduce_results=reduce_results,
+            disable_tp=is_sequence_parallel,
+            prefix=f"{prefix}.down_proj",
+        )
+        if hidden_act != "silu":
+            raise ValueError(
+                f"Unsupported activation: {hidden_act}. Only silu is supported for now."
+            )
+
+        self.swiglu_limit = swiglu_limit
+        if self.swiglu_limit is not None:
+            self.act_fn = SiluAndMulWithClamp(swiglu_limit=self.swiglu_limit)
+        else:
+            self.act_fn = SiluAndMul()
+
+    def forward(self, x):
+        gate_up, _ = self.gate_up_proj(x)
+        x = self.act_fn(gate_up)
+        x, _ = self.down_proj(x)
+        return x
+
+
+class Glm5NextMoE(nn.Module):
+    def __init__(
+        self,
+        config: Glm5NextConfig,
+        parallel_config: ParallelConfig,
+        quant_config: QuantizationConfig | None = None,
+        prefix: str = "",
+        apply_routed_scale_to_output: bool = False,
+    ):
+        super().__init__()
+        self.tp_size = get_tensor_model_parallel_world_size()
+        self.tp_rank = get_tensor_model_parallel_rank()
+
+        self.routed_scaling_factor = getattr(config, "routed_scaling_factor", 1.0)
+
+        self.ep_group = get_ep_group().device_group
+        self.ep_rank = get_ep_group().rank_in_group
+        self.ep_size = self.ep_group.size()
+        self.n_routed_experts: int = config.n_routed_experts
+        self.n_shared_experts: int = config.n_shared_experts
+
+        self.is_sequence_parallel = parallel_config.use_sequence_parallel_moe
+
+        if config.hidden_act != "silu":
+            raise ValueError(
+                f"Unsupported activation: {config.hidden_act}. "
+                "Only silu is supported for now."
+            )
+
+        self.router_dtype = _get_moe_router_dtype(config)
+        self.gate = GateLinear(
+            config.hidden_size,
+            config.n_routed_experts,
+            out_dtype=self.router_dtype,
+            prefix=f"{prefix}.gate",
+        )
+        if getattr(config, "topk_method", None) == "noaux_tc":
+            self.gate.e_score_correction_bias = nn.Parameter(
+                torch.empty(config.n_routed_experts, dtype=torch.float32)
+            )
+        else:
+            self.gate.e_score_correction_bias = None
+
+        # Load balancing settings.
+        eplb_config = parallel_config.eplb_config
+        self.enable_eplb = parallel_config.enable_eplb
+
+        self.n_redundant_experts = eplb_config.num_redundant_experts
+        self.n_logical_experts = self.n_routed_experts
+        self.n_physical_experts = self.n_logical_experts + self.n_redundant_experts
+        self.n_local_physical_experts = self.n_physical_experts // self.ep_size
+
+        self.physical_expert_start = self.ep_rank * self.n_local_physical_experts
+        self.physical_expert_end = (
+            self.physical_expert_start + self.n_local_physical_experts
+        )
+
+        swiglu_limit = getattr(config, "swiglu_limit", None)
+        if config.n_shared_experts is None:
+            self.shared_experts = None
+        else:
+            intermediate_size = config.moe_intermediate_size * config.n_shared_experts
+
+            self.shared_experts = Glm5NextMLP(
+                hidden_size=config.hidden_size,
+                intermediate_size=intermediate_size,
+                hidden_act=config.hidden_act,
+                quant_config=quant_config,
+                is_sequence_parallel=self.is_sequence_parallel,
+                reduce_results=False,
+                prefix=f"{prefix}.shared_experts",
+                swiglu_limit=swiglu_limit,
+            )
+
+        self.experts = FusedMoEFactory(
+            shared_experts=self.shared_experts,
+            gate=self.gate,
+            num_experts=config.n_routed_experts,
+            top_k=config.num_experts_per_token,
+            hidden_size=config.hidden_size,
+            intermediate_size=config.moe_intermediate_size,
+            renormalize=getattr(config, "norm_topk_prob", True),
+            quant_config=quant_config,
+            use_grouped_topk=True,
+            num_expert_group=getattr(config, "n_group", 1),
+            topk_group=getattr(config, "topk_group", 1),
+            prefix=f"{prefix}.experts",
+            scoring_func=getattr(config, "scoring_func", "softmax"),
+            routed_scaling_factor=self.routed_scaling_factor,
+            apply_routed_scale_to_output=apply_routed_scale_to_output,
+            e_score_correction_bias=self.gate.e_score_correction_bias,
+            enable_eplb=self.enable_eplb,
+            num_redundant_experts=self.n_redundant_experts,
+            is_sequence_parallel=self.is_sequence_parallel,
+            n_shared_experts=None,
+            router_logits_dtype=self.gate.out_dtype,
+            swiglu_limit=swiglu_limit,
+        )
+
+    def forward(
+        self,
+        hidden_states: torch.Tensor,
+        already_sequence_parallel: bool = False,
+    ) -> torch.Tensor:
+        num_tokens, hidden_dim = hidden_states.shape
+
+        # Chunk the hidden states so they aren't replicated across TP ranks.
+        # This avoids duplicate computation in self.experts.
+        if self.is_sequence_parallel and not already_sequence_parallel:
+            hidden_states = sequence_parallel_chunk(hidden_states)
+
+        # The router is always external (self.gate); main's MoERunner expects
+        # pre-computed router_logits, so compute them here unconditionally.
+        router_logits, _ = self.gate(hidden_states)
+        final_hidden_states = self.experts(
+            hidden_states=hidden_states, router_logits=router_logits
+        )
+
+        if self.is_sequence_parallel and not already_sequence_parallel:
+            final_hidden_states = tensor_model_parallel_all_gather(
+                final_hidden_states, 0
+            )
+            final_hidden_states = final_hidden_states[:num_tokens]
+
+        return final_hidden_states.view(num_tokens, hidden_dim)
+
+
+class Glm5NextDecoderLayer(nn.Module):
+    def __init__(
+        self,
+        vllm_config: VllmConfig,
+        config: Glm5NextConfig,
+        layer_idx: int,
+        prefix: str = "",
+        topk_indices_buffer: torch.Tensor | None = None,
+        is_mtp_layer: bool = False,
+        **kwargs,
+    ) -> None:
+        super().__init__()
+
+        cache_config = vllm_config.cache_config
+        quant_config = vllm_config.quant_config
+        parallel_config = vllm_config.parallel_config
+
+        self.hidden_size = config.hidden_size
+        self.layer_idx = layer_idx
+        self.is_moe = config.is_moe
+        self.num_hidden_layers = config.num_hidden_layers
+        self.rms_norm_eps = config.rms_norm_eps
+        self.num_experts = config.n_routed_experts
+        self.is_mtp_layer = is_mtp_layer
+        self.mhc = config.mhc
+        self.layer_kind = "kda" if config.is_kda_layer(layer_idx) else "mla"
+        self.is_sequence_parallel = parallel_config.use_sequence_parallel_moe
+
+        if config.is_kda_layer(layer_idx):
+            self.self_attn = Glm5NextLinearAttention(
+                config=config,
+                vllm_config=vllm_config,
+                prefix=f"{prefix}.self_attn",
+            )
+        else:
+            # MLA layers require the latent head dims, which are guaranteed set
+            # on MLA configs; narrow away the `int | None`.
+            assert config.v_head_dim is not None
+            assert config.kv_lora_rank is not None
+            self.self_attn = Glm5NextMLAAttention(
+                vllm_config=vllm_config,
+                config=config,
+                hidden_size=self.hidden_size,
+                num_heads=config.num_attention_heads,
+                qk_nope_head_dim=config.qk_nope_head_dim,
+                qk_rope_head_dim=config.qk_rope_head_dim,
+                v_head_dim=config.v_head_dim,
+                q_lora_rank=config.q_lora_rank,
+                kv_lora_rank=config.kv_lora_rank,
+                max_position_embeddings=config.max_position_embeddings,
+                cache_config=cache_config,
+                # LOCAL PATCH (fp8attn-r2): was quant_config=None ("MLA
+                # projections are BF16 in checkpoint"). Pass the real config so
+                # FP8-serialized MLA projections can stay FP8-resident; BF16
+                # checkpoints still resolve every self_attn module to
+                # UnquantizedLinearMethod via the config's ignore list.
+                quant_config=quant_config,
+                prefix=f"{prefix}.self_attn",
+                topk_indices_buffer=topk_indices_buffer,
+                skip_rope=getattr(config, "mla_nope", False),
+            )
+
+        # MTP layers sit past the base model's hidden layers (layer_idx >=
+        # num_hidden_layers), so they're outside mlp_layer_types; default them
+        # to the last base layer's MLP type (sparse/MoE for these checkpoints).
+        mlp_layer_types = config.mlp_layer_types
+        mlp_type = (
+            mlp_layer_types[layer_idx]
+            if layer_idx < len(mlp_layer_types)
+            else (mlp_layer_types[-1] if mlp_layer_types else "sparse")
+        )
+        if self.is_moe and self.num_experts is not None and mlp_type == "sparse":
+            self.mlp = Glm5NextMoE(
+                config=config,
+                parallel_config=parallel_config,
+                quant_config=quant_config,
+                prefix=f"{prefix}.mlp",
+            )
+        else:
+            self.mlp = Glm5NextMLP(
+                hidden_size=self.hidden_size,
+                intermediate_size=config.intermediate_size,
+                hidden_act=config.hidden_act,
+                quant_config=quant_config,
+                prefix=f"{prefix}.mlp",
+                swiglu_limit=config.swiglu_limit,
+            )
+        self.input_layernorm = RMSNorm(config.hidden_size, eps=config.rms_norm_eps)
+        # Cached for the hot forward path (isinstance per layer per step).
+        self._mlp_is_moe = isinstance(self.mlp, Glm5NextMoE)
+        # In SP, the attention output projection leaves a partial sum; the
+        # decoder-layer reduce_scatter after attention completes it (DSv4 pattern).
+        # MTP layers use the non-mHC path which has no sp_reduce_scatter, so
+        # their o_proj must still reduce normally.
+        if self.is_sequence_parallel and not is_mtp_layer:
+            self.self_attn.o_proj.reduce_results = False
+        self.post_attention_layernorm = RMSNorm(
+            config.hidden_size, eps=config.rms_norm_eps
+        )
+
+        if self.mhc and not is_mtp_layer:
+            # mhc config
+            self.mhc_num_residual_streams = config.mhc_num_residual_streams
+            self.mhc_no_norm_weight = config.mhc_no_norm_weight
+            self.mhc_tau = config.mhc_tau
+            self.hc_eps = config.hc_eps
+            self.mhc_sinkhorn_iterations = config.mhc_sinkhorn_iterations
+            self.mhc_post_mult_value = config.mhc_post_mult_value
+
+            n = config.mhc_num_residual_streams
+            d_model = n * self.hidden_size
+            mix_hc = (2 + n) * n
+
+            self.n = n
+
+            # attn hc
+            self.hc_attn_fn = nn.Parameter(
+                torch.empty(mix_hc, d_model, dtype=torch.float32)
+            )
+            self.hc_attn_base = nn.Parameter(torch.empty(mix_hc, dtype=torch.float32))
+            self.hc_attn_scale = nn.Parameter(torch.empty(3, dtype=torch.float32))
+
+            # ffn hc
+            self.hc_ffn_fn = nn.Parameter(
+                torch.empty(mix_hc, d_model, dtype=torch.float32)
+            )
+            self.hc_ffn_base = nn.Parameter(torch.empty(mix_hc, dtype=torch.float32))
+            self.hc_ffn_scale = nn.Parameter(torch.empty(3, dtype=torch.float32))
+
+            self.mhc_pre_op = MHCPreOp()
+            self.mhc_post_op = MHCPostOp()
+            self.mhc_fused_post_pre_op = MHCFusedPostPreOp()
+
+    def forward(
+        self,
+        positions: torch.Tensor,
+        hidden_states: torch.Tensor,
+        residual: torch.Tensor | None = None,
+        post: torch.Tensor | None = None,
+        comb: torch.Tensor | None = None,
+    ) -> tuple[
+        torch.Tensor,
+        torch.Tensor | None,
+        torch.Tensor | None,
+        torch.Tensor | None,
+    ]:
+        # 70B or MTP layers: KDA + MoE without HC.
+        if not self.mhc or self.is_mtp_layer:
+            residual = hidden_states
+            hidden_states = self.input_layernorm(hidden_states)
+
+            attn_output = self.self_attn(
+                hidden_states=hidden_states,
+                positions=positions,
+            )
+            hidden_states, residual = self.post_attention_layernorm(
+                attn_output, residual=residual
+            )
+            hidden_states = self.mlp(hidden_states)
+            if self.is_mtp_layer:
+                # Return the unsummed pair: the MTP caller feeds it straight
+                # into shared_head's fused_add_rms_norm (one kernel instead of
+                # a separate residual-add + norm). The sum itself is unchanged
+                # (fp32-accumulated inside the fused kernel).
+                return hidden_states, residual, None, None
+            hidden_states = residual + hidden_states
+            return hidden_states, residual, None, None
+
+        # mHC start. `post`/`comb` carry the previous layer's deferred
+        # hc_post inputs (its ffn-pre outputs); when present, fuse that
+        # hc_post with this layer's attn hc_pre into one kernel (inter-layer
+        # fusion). Layer 0 has no incoming state -> standalone hc_pre.
+        x = hidden_states
+        if post is None:
+            if self.layer_idx == 0:
+                x = hc_expand(x, self.n)
+            residual = x
+            post, comb, x = self.hc_pre(
+                x,
+                self.hc_attn_fn,
+                self.hc_attn_scale,
+                self.hc_attn_base,
+                norm_weight=self.input_layernorm.weight.data,
+                norm_eps=self.input_layernorm.variance_epsilon,
+            )
+        else:
+            residual, post, comb, x = self.hc_fused_post_pre(
+                x,
+                residual,
+                post,
+                comb,
+                self.hc_attn_fn,
+                self.hc_attn_scale,
+                self.hc_attn_base,
+                norm_weight=self.input_layernorm.weight.data,
+                norm_eps=self.input_layernorm.variance_epsilon,
+            )
+
+        # Attention needs the full token sequence; mHC above ran on the SP
+        # shard. Gather for attention, scatter back afterward (DSv4 pattern).
+        if self.is_sequence_parallel:
+            x = sp_all_gather(x)[: positions.shape[0]]
+
+        x = self.self_attn(
+            hidden_states=x,
+            positions=positions,
+        )
+
+        if self.is_sequence_parallel:
+            x = sp_reduce_scatter(x)
+
+        # Fuse post-attn hc_post + pre-FFN hc_pre (+ RMSNorm) into one kernel.
+        residual, post, comb, x = self.hc_fused_post_pre(
+            x,
+            residual,
+            post,
+            comb,
+            self.hc_ffn_fn,
+            self.hc_ffn_scale,
+            self.hc_ffn_base,
+            norm_weight=self.post_attention_layernorm.weight.data,
+            norm_eps=self.post_attention_layernorm.variance_epsilon,
+        )
+
+        # Fully Connected
+        if self._mlp_is_moe:
+            x = self.mlp(x, already_sequence_parallel=self.is_sequence_parallel)
+        else:
+            x = self.mlp(x)
+
+        # mHC end. The last mHC layer materializes its final hc_post (nothing
+        # to fuse with) then contracts; every other layer defers its hc_post to
+        # the next layer's fused pre, returning the state.
+        if self.layer_idx == self.num_hidden_layers - 1:
+            x = self.hc_post(x, residual, post, comb)
+            x = hc_contract(x, self.n)
+            return x, None, None, None
+
+        return x, residual, post, comb
+
+    def hc_pre(
+        self,
+        x: torch.Tensor,
+        hc_fn: torch.Tensor,
+        hc_scale: torch.Tensor,
+        hc_base: torch.Tensor,
+        norm_weight: torch.Tensor | None = None,
+        norm_eps: float = 0.0,
+    ):
+        post_mix, res_mix, layer_input = self.mhc_pre_op(
+            residual=x,
+            fn=hc_fn,
+            hc_scale=hc_scale,
+            hc_base=hc_base,
+            rms_eps=self.rms_norm_eps,
+            hc_pre_eps=self.hc_eps,
+            hc_sinkhorn_eps=self.hc_eps,
+            hc_post_mult_value=self.mhc_post_mult_value,
+            sinkhorn_repeat=self.mhc_sinkhorn_iterations,
+            norm_weight=norm_weight,
+            norm_eps=norm_eps,
+        )
+        return post_mix, res_mix, layer_input
+
+    def hc_post(
+        self,
+        x: torch.Tensor,
+        residual: torch.Tensor,
+        post: torch.Tensor,
+        comb: torch.Tensor,
+    ):
+        return self.mhc_post_op(x, residual, post, comb)
+
+    def hc_fused_post_pre(
+        self,
+        x: torch.Tensor,
+        residual: torch.Tensor,
+        post: torch.Tensor,
+        comb: torch.Tensor,
+        hc_fn: torch.Tensor,
+        hc_scale: torch.Tensor,
+        hc_base: torch.Tensor,
+        norm_weight: torch.Tensor | None = None,
+        norm_eps: float = 0.0,
+    ):
+        return self.mhc_fused_post_pre_op(
+            x=x,
+            residual=residual,
+            post_layer_mix=post,
+            comb_res_mix=comb,
+            fn=hc_fn,
+            hc_scale=hc_scale,
+            hc_base=hc_base,
+            rms_eps=self.rms_norm_eps,
+            hc_pre_eps=self.hc_eps,
+            hc_sinkhorn_eps=self.hc_eps,
+            hc_post_mult_value=self.mhc_post_mult_value,
+            sinkhorn_repeat=self.mhc_sinkhorn_iterations,
+            n_splits=1,
+            tile_n=1,
+            norm_weight=norm_weight,
+            norm_eps=norm_eps,
+        )
+
+
+class Glm5NextModel(nn.Module):
+    def __init__(self, *, vllm_config: VllmConfig, prefix: str = ""):
+        super().__init__()
+
+        config = vllm_config.model_config.hf_config
+        self.config = config
+
+        self.vocab_size = config.vocab_size
+        self.device = current_platform.device_type
+
+        """
+        if config.index_topk is not None:
+            topk_indices_buffer = torch.empty(
+                vllm_config.scheduler_config.max_num_batched_tokens,
+                config.index_topk,
+                dtype=torch.int32,
+                device=self.device,
+            )
+        else:
+        """
+        # `index_topk` is declared on Glm5NextTextConfig with a default of None,
+        # so hasattr() is True even for full-MLA configs (no kpool indexer).
+        # Gate on the value being set instead.
+        self.is_v32 = getattr(config, "index_topk", None) is not None
+        if self.is_v32:
+            topk_tokens = config.index_topk
+            # kpool widens the topk buffer: selecting topk_tokens//kpool pools and
+            # expanding them yields topk_tokens token indices, plus an always-
+            # selected tail of up to kpool-1 incomplete-pool tokens. The attention
+            # backend reads the width dynamically via topk_indices.shape[1].
+            kpool = getattr(config, "index_kpool", 1) or 1
+            buffer_width = topk_tokens
+            # The sparse MLA attention kernel
+            # (triton_convert_req_index_to_global_index) tiles the topk
+            # dimension in BLOCK_N=128 columns and requires the buffer width
+            # to be a multiple of it; otherwise it raises
+            # "NUM_TOPK_TOKENS must be divisible by BLOCK_N". Round up: the
+            # extra slots stay -1 (the indexer op initializes the buffer to
+            # -1) and are masked out by the attention kernel, so they do not
+            # affect the softmax over the selected tokens.
+            sparse_topk_block_n = 128
+            buffer_width = (
+                (buffer_width + sparse_topk_block_n - 1) // sparse_topk_block_n
+            ) * sparse_topk_block_n
+            topk_indices_buffer = torch.empty(
+                vllm_config.scheduler_config.max_num_batched_tokens,
+                buffer_width,
+                dtype=torch.int32,
+                device=self.device,
+            )
+        else:
+            # Full-MLA config (no kpool sparse indexer): no topk buffer.
+            topk_indices_buffer = None
+
+        if get_pp_group().is_first_rank:
+            self.embed_tokens = VocabParallelEmbedding(
+                config.vocab_size,
+                config.hidden_size,
+                prefix=f"{prefix}.embed_tokens",
+            )
+        else:
+            self.embed_tokens = PPMissingLayer()
+
+        def get_layer(prefix: str):
+            layer_idx = int(prefix.rsplit(".", 1)[1])
+            return Glm5NextDecoderLayer(
+                vllm_config=vllm_config,
+                config=config,
+                layer_idx=layer_idx,
+                prefix=prefix,
+                topk_indices_buffer=topk_indices_buffer,
+            )
+
+        self.start_layer, self.end_layer, self.layers = make_layers(
+            config.num_hidden_layers,
+            get_layer,
+            prefix=f"{prefix}.layers",
+        )
+        # The active slice is fixed after construction; cache it so forward
+        # doesn't rebuild the slice (a fresh list) every step.
+        self._active_layers = self.layers[self.start_layer : self.end_layer]
+
+        if get_pp_group().is_last_rank:
+            self.norm = RMSNorm(config.hidden_size, eps=config.rms_norm_eps)
+        else:
+            self.norm = PPMissingLayer()
+
+        self.is_sequence_parallel = (
+            vllm_config.parallel_config.use_sequence_parallel_moe
+        )
+
+        world_size = get_tensor_model_parallel_world_size()
+        assert config.num_attention_heads % world_size == 0, (
+            "num_attention_heads must be divisible by world_size"
+        )
+
+    def embed_input_ids(self, input_ids: torch.Tensor) -> torch.Tensor:
+        return self.embed_tokens(input_ids)
+
+    def forward(
+        self,
+        input_ids: torch.Tensor | None,
+        positions: torch.Tensor,
+        intermediate_tensors: IntermediateTensors | None,
+        inputs_embeds: torch.Tensor | None = None,
+        **kwargs,
+    ) -> torch.Tensor:
+        if get_pp_group().is_first_rank:
+            if inputs_embeds is not None:
+                hidden_states = inputs_embeds
+            else:
+                hidden_states = self.embed_input_ids(input_ids)
+            residual = None
+            post = None
+            comb = None
+        else:
+            assert intermediate_tensors is not None
+            hidden_states = intermediate_tensors["hidden_states"]
+            residual = intermediate_tensors["residual"]
+            # post/comb (deferred mHC hc_post state) are not propagated across
+            # PP ranks; the receiving rank's first mHC layer uses standalone pre.
+            post = None
+            comb = None
+
+        full_num_tokens = positions.shape[0]
+        if self.is_sequence_parallel:
+            hidden_states = sp_shard(hidden_states)
+
+        for layer in self._active_layers:
+            hidden_states, residual, post, comb = layer(
+                positions, hidden_states, residual, post, comb
+            )
+
+        if not get_pp_group().is_last_rank:
+            # PP is gated off for GLM5Next (no make_empty_intermediate_tensors),
+            # so this branch is not exercised. post/comb are the deferred
+            # hc_post state of this rank's last mHC layer; a future PP path
+            # would need to propagate them, but for now they are dropped (the
+            # receiving rank's first layer would fall back to standalone pre).
+            return IntermediateTensors(
+                {"hidden_states": hidden_states, "residual": residual}
+            )
+
+        if self.is_sequence_parallel:
+            hidden_states = sp_all_gather(hidden_states)[:full_num_tokens]
+
+        hidden_states = self.norm(hidden_states)
+        return hidden_states
+
+    def load_weights(self, weights: Iterable[tuple[str, torch.Tensor]]) -> set[str]:
+        stacked_params_mapping = [
+            # (param_name, shard_name, shard_id)
+            (".gate_up_proj", ".gate_proj", 0),
+            (".gate_up_proj", ".up_proj", 1),
+            # MLA: fuse q_a_proj and kv_a_proj_with_mqa
+            (".fused_qkv_a_proj", ".q_a_proj", 0),
+            (".fused_qkv_a_proj", ".kv_a_proj_with_mqa", 1),
+            # Indexer: fuse wk and weights_proj
+            (".wk_weights_proj", ".wk", 0),
+            (".wk_weights_proj", ".weights_proj", 1),
+            # KDA: merge q, k, v, b, f_a, g_a projections into one GEMM
+            (".in_proj_qkvbfg_a", ".q_proj", 0),
+            (".in_proj_qkvbfg_a", ".k_proj", 1),
+            (".in_proj_qkvbfg_a", ".v_proj", 2),
+            (".in_proj_qkvbfg_a", ".b_proj", 3),
+            (".in_proj_qkvbfg_a", ".f_a_proj", 4),
+            (".in_proj_qkvbfg_a", ".g_a_proj", 5),
+        ]
+        if self.config.is_moe:
+            # Params for weights, fp8 weight scales, fp8 activation scales
+            # (param_name, weight_name, expert_id, shard_id)
+            expert_params_mapping = fused_moe_make_expert_params_mapping(
+                self,
+                ckpt_gate_proj_name="gate_proj",
+                ckpt_down_proj_name="down_proj",
+                ckpt_up_proj_name="up_proj",
+                num_experts=self.config.n_routed_experts,
+            )
+        else:
+            expert_params_mapping = []
+        params_dict = dict(self.named_parameters())
+        loaded_params: set[str] = set()
+
+        # GLM5-Next NoPE: checkpoint's kv_a_proj_with_mqa has only kv_lora_rank
+        # rows, but the model expects kv_lora_rank + qk_rope_head_dim rows.
+        # Pad the missing rope portion with zeros.
+        kv_a_pad_size = 0
+        if self.config.mla_nope and self.config.qk_rope_head_dim > 0:
+            kv_a_pad_size = self.config.qk_rope_head_dim
+
+        _pending_wk_fp8: dict = {}
+
+        for args in weights:
+            name, loaded_weight = args[:2]
+            kwargs: dict = args[2] if len(args) > 2 else {}
+            if "rotary_emb.inv_freq" in name:
+                continue
+
+            spec_layer = get_spec_layer_idx_from_weight_name(self.config, name)
+            if spec_layer is not None:
+                continue  # skip spec decode layers for main model
+            if "rotary_emb.cos_cached" in name or "rotary_emb.sin_cached" in name:
+                # Models trained using ColossalAI may include these tensors in
+                # the checkpoint. Skip them.
+                continue
+
+            # Handle FP8 indexer WK: dequantize to BF16 for fusion with
+            # weights_proj into wk_weights_proj.
+            if _try_load_fp8_indexer_wk(
+                name,
+                loaded_weight,
+                _pending_wk_fp8,
+                params_dict,
+                loaded_params,
+            ):
+                continue
+
+            # FP8 checkpoint: dequantize BF16-kept MLA projections
+            # (q_a_proj / kv_a_proj_with_mqa / o_proj) to BF16.
+            if _try_load_fp8_attn_proj(
+                name,
+                loaded_weight,
+                _pending_wk_fp8,
+                params_dict,
+                loaded_params,
+                kv_a_pad_size,
+            ):
+                continue
+
+            # Pad kv_a_proj_with_mqa for NoPE models
+            if kv_a_pad_size > 0 and ".kv_a_proj_with_mqa." in name:
+                pad = torch.zeros(
+                    kv_a_pad_size,
+                    *loaded_weight.shape[1:],
+                    dtype=loaded_weight.dtype,
+                    device=loaded_weight.device,
+                )
+                loaded_weight = torch.cat([loaded_weight, pad], dim=0)
+
+            for param_name, weight_name, shard_id in stacked_params_mapping:
+                if weight_name not in name:
+                    continue
+                # We have mlp.experts[0].gate_proj in the checkpoint.
+                # Since we handle the experts below in expert_params_mapping,
+                # we need to skip here BEFORE we update the name, otherwise
+                # name will be updated to mlp.experts[0].gate_up_proj, which
+                # will then be updated below in expert_params_mapping
+                # for mlp.experts[0].gate_gate_up_proj, which breaks load.
+                if ("mlp.experts." in name) and name not in params_dict:
+                    continue
+                name_mapped = name.replace(weight_name, param_name)
+                # QKV fusion: skip if fused module doesn't exist in model
+                if param_name == ".fused_qkv_a_proj" and name_mapped not in params_dict:
+                    continue
+                name = name_mapped
+                # Skip loading extra bias for GPTQ models.
+                if name.endswith(".bias") and name not in params_dict:
+                    continue
+                if is_pp_missing_parameter(name, self):
+                    continue
+                param = params_dict[name]
+                weight_loader = param.weight_loader
+                weight_loader(param, loaded_weight, shard_id)
+                break
+            else:
+                for idx, (
+                    param_name,
+                    weight_name,
+                    expert_id,
+                    expert_shard_id,
+                ) in enumerate(expert_params_mapping):
+                    if weight_name not in name:
+                        continue
+                    name = name.replace(weight_name, param_name)
+                    if is_pp_missing_parameter(name, self):
+                        continue
+                    param = params_dict[name]
+                    weight_loader = param.weight_loader
+                    weight_loader(
+                        param,
+                        loaded_weight,
+                        name,
+                        expert_id=expert_id,
+                        shard_id=expert_shard_id,
+                    )
+                    break
+                else:
+                    # Skip loading extra bias for GPTQ models.
+                    if (
+                        name.endswith(".bias")
+                        and name not in params_dict
+                        and not self.config.is_linear_attn
+                    ):  # noqa: E501
+                        continue
+                    # Remapping the name of FP8 kv-scale.
+                    remapped_name = maybe_remap_kv_scale_name(name, params_dict)
+                    if remapped_name is None:
+                        continue
+                    name = remapped_name
+                    if is_pp_missing_parameter(name, self):
+                        continue
+
+                    param = params_dict[name]
+                    weight_loader = getattr(
+                        param, "weight_loader", default_weight_loader
+                    )
+                    weight_loader(param, loaded_weight, **kwargs)
+            loaded_params.add(name)
+        return loaded_params
+
+
+class Glm5NextForCausalLM(
+    nn.Module, HasInnerState, SupportsPP, MixtureOfExperts, IsHybrid
+):
+    def __init__(self, *, vllm_config: VllmConfig, prefix: str = ""):
+        super().__init__()
+        self.model_config = vllm_config.model_config
+        self.vllm_config = vllm_config
+        self.config = self.model_config.hf_config
+        quant_config = vllm_config.quant_config
+        self.quant_config = quant_config
+        self.model = Glm5NextModel(
+            vllm_config=vllm_config, prefix=maybe_prefix(prefix, "model")
+        )
+        if get_pp_group().is_last_rank:
+            self.lm_head = ParallelLMHead(
+                self.config.vocab_size,
+                self.config.hidden_size,
+                quant_config=quant_config,
+                prefix=maybe_prefix(prefix, "lm_head"),
+            )
+        else:
+            self.lm_head = PPMissingLayer()
+        logit_scale = getattr(self.config, "logit_scale", 1.0)
+        self.logits_processor = LogitsProcessor(
+            self.config.vocab_size, scale=logit_scale
+        )
+
+    def embed_input_ids(self, input_ids: torch.Tensor) -> torch.Tensor:
+        return self.model.embed_input_ids(input_ids)
+
+    def forward(
+        self,
+        input_ids: torch.Tensor | None,
+        positions: torch.Tensor,
+        intermediate_tensors: IntermediateTensors | None = None,
+        inputs_embeds: torch.Tensor | None = None,
+        **kwargs,
+    ) -> torch.Tensor | IntermediateTensors:
+        hidden_states = self.model(
+            input_ids, positions, intermediate_tensors, inputs_embeds, **kwargs
+        )
+        return hidden_states
+
+    @classmethod
+    def get_mamba_state_dtype_from_config(
+        cls,
+        vllm_config: "VllmConfig",
+    ) -> tuple[torch.dtype, torch.dtype]:
+        return MambaStateDtypeCalculator.kda_state_dtype(
+            vllm_config.model_config.dtype, vllm_config.cache_config.mamba_cache_dtype
+        )
+
+    @classmethod
+    def get_mamba_state_shape_from_config(
+        cls, vllm_config: "VllmConfig"
+    ) -> tuple[tuple[int, int], tuple[int, int, int]]:
+        parallel_config = vllm_config.parallel_config
+        hf_config = vllm_config.model_config.hf_config
+        tp_size = parallel_config.tensor_parallel_size
+        num_spec = (
+            vllm_config.speculative_config.num_speculative_tokens
+            if vllm_config.speculative_config
+            else 0
+        )
+        return MambaStateShapeCalculator.kda_state_shape(
+            tp_size,
+            hf_config.linear_num_heads,
+            hf_config.linear_head_dim,
+            conv_kernel_size=hf_config.linear_conv_kernel_dim,
+            num_spec=num_spec,
+        )
+
+    @classmethod
+    def get_mamba_state_copy_func(
+        cls,
+    ) -> tuple[
+        MambaStateCopyFunc, MambaStateCopyFunc, MambaStateCopyFunc, MambaStateCopyFunc
+    ]:
+        return MambaStateCopyFuncCalculator.kda_state_copy_func()
+
+    def compute_logits(
+        self,
+        hidden_states: torch.Tensor,
+    ) -> torch.Tensor | None:
+        logits = self.logits_processor(self.lm_head, hidden_states)
+        return logits
+
+    def load_weights(self, weights: Iterable[tuple[str, torch.Tensor]]) -> set[str]:
+        loader = AutoWeightsLoader(
+            self,
+            skip_prefixes=(["lm_head."] if self.config.tie_word_embeddings else None),
+        )
+        return loader.load_weights(weights)
+
+
+@MULTIMODAL_REGISTRY.register_processor(
+    Glm5NextMultiModalProcessor,
+    info=Glm5NextProcessingInfo,
+    dummy_inputs=Glm4vDummyInputsBuilder,
+)
+class Glm5NextForConditionalGeneration(
+    Glm4vForConditionalGeneration, HasInnerState, IsHybrid
+):
+    # The text model (KDA + dense-MLA + MoE) is a hybrid mamba model. The
+    # multimodal wrapper must declare the same interfaces so vLLM treats it as
+    # hybrid (auto-aligns mamba/attention block sizes, sizes the mamba state
+    # cache); the mamba-state classmethods delegate to the text model.
+    has_inner_state: ClassVar[Literal[True]] = True
+    is_hybrid: ClassVar[Literal[True]] = True
+
+    # NOTE: weight-prefix mapping is inherited from Glm4vForConditionalGeneration
+    # (``model.visual.`` -> ``visual.``, ``model.language_model.`` ->
+    # ``language_model.model.``, ``lm_head.`` -> ``language_model.lm_head.``),
+    # matching the GLM-OCR / GLM-4V serialization convention. If the real
+    # checkpoint's safetensors keys differ (e.g. ``language_model.model.`` with
+    # no outer ``model.``), override ``hf_to_vllm_mapper`` accordingly.
+
+    @classmethod
+    def get_mamba_state_dtype_from_config(cls, vllm_config: VllmConfig):
+        from .model import Glm5NextForCausalLM
+
+        return Glm5NextForCausalLM.get_mamba_state_dtype_from_config(vllm_config)
+
+    @classmethod
+    def get_mamba_state_shape_from_config(cls, vllm_config: VllmConfig):
+        from .model import Glm5NextForCausalLM
+
+        return Glm5NextForCausalLM.get_mamba_state_shape_from_config(vllm_config)
+
+    @classmethod
+    def get_mamba_state_copy_func(cls):
+        from .model import Glm5NextForCausalLM
+
+        return Glm5NextForCausalLM.get_mamba_state_copy_func()
+
+    def __init__(self, *, vllm_config: VllmConfig, prefix: str = ""):
+        super(Glm4vForConditionalGeneration, self).__init__()
+        config = vllm_config.model_config.hf_config
+        multimodal_config = vllm_config.model_config.multimodal_config
+        assert multimodal_config is not None
+
+        self.config = config
+        self.model_config = vllm_config.model_config
+        self.multimodal_config = multimodal_config
+        self.use_data_parallel = multimodal_config.mm_encoder_tp_mode == "data"
+        self.is_multimodal_pruning_enabled = (
+            multimodal_config.is_multimodal_pruning_enabled()
+        )
+
+        with self._mark_tower_model(vllm_config, {"image", "video"}):
+            self.visual = Glm5NextVisionTransformer(
+                config.text_config,
+                config.vision_config,
+                # Read eps from the VISION sub-config, not the top-level
+                # `config.rms_norm_eps`: Glm5NextConfig.__getattribute__ mirrors
+                # the latter onto text_config (1e-5), silently ignoring the
+                # vision tower's own (1e-6) rms_norm_eps.
+                norm_eps=config.vision_config.rms_norm_eps,
+                # Vision tower ships BF16 weights in this fp8 checkpoint (no
+                # weight_scale_inv for visual.*), so it must NOT inherit the
+                # global fp8 quant_config -- doing so incorrectly quantizes
+                # the tower
+                # and yields NaN image features. Mirrors the MLA/KDA proj
+                # pattern (quant_config=None for BF16 submodules).
+                quant_config=None,
+                prefix=maybe_prefix(prefix, "visual"),
+            )
+
+        with self._mark_language_model(vllm_config):
+            self.language_model = init_vllm_registered_model(
+                vllm_config=vllm_config,
+                hf_config=config.text_config,
+                prefix=maybe_prefix(prefix, "language_model"),
+                architectures=["Glm5NextForCausalLM"],
+            )
+
+        # Glm5NextForCausalLM does not implement make_empty_intermediate_tensors,
+        # so pipeline parallelism is gated off (consistent with the text-only
+        # model) and we intentionally do not alias it here.
+
+    def get_encoder_cudagraph_config(self):
+        # The forked vision tower (multimodal.py) has no abs-pos embeddings, so its
+        # prepare_encoder_metadata does not produce "pos_embeds". Drop it from the
+        # buffer_keys inherited from Glm4vForConditionalGeneration so encoder
+        # CUDA-graph capture/replay does not expect a buffer that is never filled.
+        config = super().get_encoder_cudagraph_config()
+        config.buffer_keys = [k for k in config.buffer_keys if k != "pos_embeds"]
+        return config
+
+
+def get_spec_layer_idx_from_weight_name(
+    config: Glm5NextConfig, weight_name: str
+) -> int | None:
+    if hasattr(config, "num_nextn_predict_layers") and (
+        config.num_nextn_predict_layers > 0
+    ):
+        layer_idx = config.num_hidden_layers
+        for i in range(config.num_nextn_predict_layers):
+            if weight_name.startswith(
+                f"model.layers.{layer_idx + i}."
+            ) or weight_name.startswith(f"layers.{layer_idx + i}."):
+                return layer_idx + i
+    return None
+
+
+def _try_load_fp8_indexer_wk(name, tensor, buf, params_dict, loaded_params):
+    if "indexer.wk." not in name or "wk_weights" in name:
+        return False
+    is_weight = name.endswith(".weight") and tensor.dtype == torch.float8_e4m3fn
+    is_scale = "weight_scale_inv" in name
+    if not is_weight and not is_scale:
+        return False
+    layer_prefix = name.rsplit(".wk.", 1)[0]
+    entry = buf.setdefault(layer_prefix, {})
+    entry["weight" if is_weight else "scale"] = tensor
+    if "weight" not in entry or "scale" not in entry:
+        return True
+
+    weight_fp8, scale_inv = entry["weight"], entry["scale"]
+    del buf[layer_prefix]
+    block_size = weight_fp8.shape[1] // scale_inv.shape[1]
+    weight_bf16 = scaled_dequantize(
+        weight_fp8,
+        scale_inv,
+        group_shape=GroupShape(block_size, block_size),
+        out_dtype=torch.bfloat16,
+    )
+
+    fused_name = f"{layer_prefix}.wk_weights_proj.weight"
+    param = params_dict[fused_name]
+    param.weight_loader(param, weight_bf16, 0)
+    loaded_params.add(fused_name)
+    return True
+
+
+def _dequant_fp8_block(
+    weight_fp8: torch.Tensor,
+    scale_inv: torch.Tensor,
+    block_size: int = 128,
+) -> torch.Tensor:
+    """Dequantize a block-FP8 (e4m3) weight with per-block scale to BF16.
+
+    Unlike ``scaled_dequantize`` this tolerates a non-divisible (partial last
+    block) shape by zero-padding to a multiple of ``block_size`` before the
+    scale broadcast and trimming back afterwards (e.g. kv_a_proj_with_mqa is
+    576 rows = 4*128 + 64).
+    """
+    out_dim, in_dim = weight_fp8.shape
+    pad_out = (-out_dim) % block_size
+    pad_in = (-in_dim) % block_size
+    w = weight_fp8
+    if pad_out or pad_in:
+        w = torch.nn.functional.pad(w, (0, pad_in, 0, pad_out))
+    # scale_inv is (ceil(out/block), ceil(in/block)); broadcast to (out, in).
+    s = scale_inv.to(torch.float32)
+    s_full = s.repeat_interleave(block_size, dim=0).repeat_interleave(block_size, dim=1)
+    out = (w.to(torch.float32) * s_full).to(torch.bfloat16)
+    return out[:out_dim, :in_dim].contiguous()
+
+
+# FP8 checkpoint projections that the MODEL keeps in BF16, so the block-FP8
+# (weight + weight_scale_inv) must be dequantized to BF16 on load.
+# Maps checkpoint proj-suffix -> (buffer key, model target base, fused shard id
+# or None for a direct projection, whether NoPE rope-padding applies).
+_FP8_ATTN_PROJS = {
+    ".q_a_proj.": ("q_a", "fused_qkv_a_proj", 0, False),
+    ".kv_a_proj_with_mqa.": ("kv_a", "fused_qkv_a_proj", 1, True),
+    ".q_b_proj.": ("q_b", "q_b_proj", None, False),
+    ".o_proj.": ("o_proj", "o_proj", None, False),
+}
+
+
+def _try_load_fp8_attn_proj(
+    name,
+    tensor,
+    buf,
+    params_dict,
+    loaded_params,
+    kv_a_pad_size: int,
+) -> bool:
+    """Dequantize FP8 q_a_proj / kv_a_proj_with_mqa / o_proj to BF16 on load.
+
+    The FP8 checkpoint stores these as block-FP8 (weight + weight_scale_inv),
+    but the model holds them in BF16 (``fused_qkv_a_proj`` is always BF16 via
+    DeepSeekV2FusedQkvAProjLinear; ``o_proj`` is excluded by
+    modules_to_not_convert). When the model target is BF16 (no
+    ``weight_scale_inv`` param) we dequantize; otherwise we return False so the
+    normal stacked/direct path loads the FP8 tensor as-is.
+    """
+    matched = None
+    for suffix, info in _FP8_ATTN_PROJS.items():
+        if suffix in name:
+            matched = (suffix, info)
+            break
+    if matched is None:
+        return False
+    suffix, (key, target_base, shard_id, is_kva) = matched
+    is_weight = name.endswith(".weight") and tensor.dtype == torch.float8_e4m3fn
+    is_scale = "weight_scale_inv" in name
+    if not is_weight and not is_scale:
+        return False
+
+    layer_prefix = name.rsplit(suffix, 1)[0]
+    target_w = f"{layer_prefix}.{target_base}.weight"
+    target_s = f"{layer_prefix}.{target_base}.weight_scale_inv"
+    # If the model actually kept this projection in FP8, let the normal path
+    # handle it (it has a weight_scale_inv param).
+    if target_s in params_dict:
+        return False
+
+    entry = buf.setdefault(layer_prefix, {}).setdefault(key, {})
+    entry["weight" if is_weight else "scale"] = tensor
+    if "weight" not in entry or "scale" not in entry:
+        return True
+
+    weight_fp8, scale_inv = entry["weight"], entry["scale"]
+    buf[layer_prefix].pop(key, None)
+    block_size = weight_fp8.shape[1] // scale_inv.shape[1]
+    weight_bf16 = _dequant_fp8_block(weight_fp8, scale_inv, block_size)
+    # NoPE: pad kv_a rope portion (kv_lora_rank -> kv_lora_rank + qk_rope_head_dim).
+    if is_kva and kv_a_pad_size > 0:
+        pad = torch.zeros(
+            kv_a_pad_size,
+            weight_bf16.shape[1],
+            dtype=weight_bf16.dtype,
+            device=weight_bf16.device,
+        )
+        weight_bf16 = torch.cat([weight_bf16, pad], dim=0)
+
+    param = params_dict[target_w]
+    if shard_id is None:
+        param.weight_loader(param, weight_bf16)
+    else:
+        param.weight_loader(param, weight_bf16, shard_id)
+    loaded_params.add(target_w)
+    return True