bullerwins/GLM-5.3-Flash-exl3-4bpw-ablit

GLM-5.3-Flash EXL3 4bpw — Keys/Dealign o_proj transplant

An EXL3 derivative of brandonmusic/GLM-5.3-Flash-tr3-4bpw, using the L15–43 + MTP L45 attention-output tensor selection shipped by drowzeys/keys-GLM-5.3-Flash-NVFP4-ablit-l15-43-mtp-l45.

EXL3 expert weights, Keys/Dealign BF16 attention-output projections. This is not an NVFP4 checkpoint, a fresh quantization, or a runtime refusal-direction projection. Layers 0–14 and 44 remain from the EXL3 parent.

Evaluation summary: the fixed 50-question GSM8K sanity check remained in the same ballpark, but fidelity to the BF16 teacher decreased: KLD increased 2.77× and realized-token perplexity increased 3.9%. The donor's Refusal32 suite, run unmodified against this derivative, reproduced their result: 32/32 bypass, 0 refusals, 0 garbled, 0 empty (greedy, thinking off; regex classifier, single run).

Model overview

Property	Description
Architecture	GLM-5.3-Flash; hybrid KDA/DSA attention with mHC
Hidden dimension	4096
Layers	45 main layers plus native MTP layer 45; zero-based indices
Routed experts	288, with top-8 routing
Quantization	Uniform-K4 EXL3/TR3 routed-expert quantization inherited from the parent
Edited weights	30 BF16 self_attn.o_proj.weight tensors
Preserved weights	All weights outside the selected attention-output tensors
Checkpoint layout	120 safetensors shards; approximately 175.64 GB of shard data
Runtime requirement	A compatible custom EXL3 runtime; not stock vLLM
Evaluation scope	Text quality comparison plus limited vision, tool, streaming and DFlash2 functional checks; video untested

Sources and provenance

Component	Source	Contribution
EXL3 checkpoint body	Brandon Music / ShapleyMCG	Quantized experts, non-selected weights, configuration, tokenizer, index and encoder closure
Tensor donor and selection recipe	Keys / drowzeys	BF16 output projections for L15–43 and MTP L45
Donor tensor origin	Dealign	Edited attention-output tensors incorporated into Keys
Keys donor body	RedHatAI GLM-5.3-Flash NVFP4	Donor lineage only; its NVFP4 experts are not copied into this EXL3 model
Upstream model	Z.ai GLM-5.3-Flash / BF16	Original model and architecture

Pinned source checkpoint revisions:

EXL3 parent: aba59d2175e1ee2887ae0ae1300ba848b1deed84
Keys donor:  80b6d18d77e3020f2384597081d405f19893f101

Upstream method and implementation references:

• Brandon Music's EXL3 repository
• Keys' transplant repository
• Keys METHOD.md at the pinned donor revision

Methodology

Exact tensor transplant

The following BF16 tensors were copied byte-for-byte from the Keys donor:

model.language_model.layers.{L}.self_attn.o_proj.weight
L = 15, 16, ..., 43, 45

This is 30 tensors, not 31. Layer 44 is excluded.

Weight group	Treatment
L15–43 attention output projections	Copy from Keys donor
MTP L45 attention output projection	Copy from Keys donor
L0–14 and L44 attention output projections	Preserve EXL3 parent
Experts, routers, QKV, embeddings, norms, vision and all other weights	Preserve EXL3 parent
Configuration, tokenizer and tensor index	Preserve EXL3 parent

The donor's METHOD.md and ABLIT_META.json specify a tensor transplant, not a rank-one projection. Its included refusal-direction .pt files are historical experiments and were not applied. No training, direction estimation, calibration, dequantization or requantization was performed.

The checkpoints use different expert packing formats, but the selected output projections are already BF16 in both. KDA output projections have shape [4096, 8192]; DSA and MTP output projections have shape [4096, 16384]. Exactly 2,617,245,696 bytes of tensor payload were transplanted.

Why preserve the anchors?

The Keys recipe preserves early layers and excludes L44. Its authors identified a large Dealign L44 change as a potential garbling risk in their NVFP4 experiments. This derivative follows the shipped selection rather than the older L15–45 recipe that included L44.

For the pinned checkpoints used here, L44 is byte-identical between the donor and EXL3 parent. Selected-tensor relative Frobenius deltas, measured in FP64, range from 0.041501 to 0.187423. Early layers remain from the EXL3 parent, including those with small donor/parent differences.

Integrity verification

• All 120 parent shards and 11 donor shards were hash-verified against their pinned Hugging Face content addresses.
• Every edited output shard was reread and verified.
• Every transplanted tensor matches its donor hash.
• All weight bytes outside the 30 selected tensor ranges match the EXL3 parent.
• Tensor index/header correspondence, configuration and EXL3 storage ABI were preserved and checked across 150,226 serialized tensors, including quantization auxiliaries.
• The parent's encoder source-closure verifier, reproducibility/r10/verify_bundle.py, passed for 41 files.
• 47 unit tests passed for the build, launcher and evaluation tooling.

The transplant was performed without modifying the source checkpoints. A reproduction must resolve tensor offsets from each checkpoint's own safetensors headers: do not reuse NVFP4 shard offsets for EXL3 files. Verify the complete output before making it available for inference.

Integrity verification establishes which bytes changed; the Refusal32 run below provides limited behavioral evidence, not a general guarantee of unchanged model behavior.

Evaluation results

The parent and derivative were evaluated using matching runtime settings and hardware. These results were measured for this comparison, not inherited from either source model card.

Measurement	Original EXL3	This derivative
Mean teacher KLD, 25 windows / 51,175 positions	0.024754893	0.068454542
Teacher top-1 agreement, 25 windows	95.34%	91.90%
Realized-token NLL, 25 windows	1.064222869	1.102432383
Mean teacher KLD, clean 17 windows / 34,799 positions	0.025328698	0.073521263
GSM8K first 50, zero-shot greedy	48/50	49/50
First 5 within that GSM8K subset	5/5	5/5
GSM8K API errors / truncated answers	0 / 0	0 / 0

Interpretation

No GSM8K question regressed in the fixed sample. The derivative answered one additional question correctly, but this small difference does not establish a general accuracy improvement.

Full-panel KLD increased 2.765×, teacher top-1 agreement decreased 3.44 percentage points, and realized-token perplexity increased 3.895%. The clean-17 subset shows the same direction of drift.

The results support retained capability on a limited sanity check, not unchanged fidelity, full-benchmark equivalence, or a safety claim. KLD measures changes in the output distribution; it is not itself a task-accuracy score.

Fidelity protocol

• Teacher dataset: GLM-5.3-Flash BF16 Teacher Logits, revision 95f4fdd94bf29989db2e0d1054e4931f55edb6aa.
• BF16 reference model revision: a6c167b62691b2bac901344b65cb651a70f53e43.
• All 25 sealed final windows, each containing 2048 input tokens and all 2047 next-token prediction positions. Teacher logits and token identities were verified. These windows were not used for fitting or tuning.
• Full-vocabulary prompt logits were captured before top-k selection. Input IDs and contiguous absolute row alignment were checked; row r predicts token r+1.
• The parent's published FP64 scoring kernel was reused to compute KL(teacher || student) in nats. The 24 padded columns were excluded, reducing stored vocabulary 154880 to real vocabulary 154856 before FP64 log-softmax and accumulation.
• Clean-17 uses the parent's published calibration-overlap exclusions; that overlap audit was not independently repeated.

GSM8K protocol

The evaluation used the official OpenAI GSM8K test set at revision 3101c7d5072418e28b9008a6636bde82a006892c:

• Fixed first 50 questions, identical zero-shot prompts for both models.
• Thinking enabled; greedy decoding; seed 42; maximum 2048 output tokens.
• Prompts requested step-by-step reasoning and a final #### numeric answer.
• Only explicit final numeric answers in content were graded. API failures and truncated responses counted as failures.

This is not the full GSM8K benchmark. The first-5 subtotal is part of the same evaluation, not an exact reproduction of the upstream stochastic five-question DFlash acceptance probe.

Evaluation configuration

The comparison used two NVIDIA GB10 GPUs (SM121) with a compatible custom vLLM EXL3 runtime based on MiaAI-Lab's implementation at source revision 6599585dd5ab0b5f1f68e84914f48825727ad1b3.

Both models used tensor parallelism 2, FP8 KV cache, eager execution, prefix caching disabled, speculation disabled, no additional runtime abliteration, text-only inference, context length 8192, one sequence and 2048-token prefill chunks. The E2 EXL3 path was used (EXL3_FAT_GROUPED=0).

Each model/window was measured once in the paired panel evaluation. Cold-run variance and formal statistical equivalence were not established. Because speculation was disabled, these quality scores do not evaluate the edited MTP head.

Additional functional checks

Separate functional validation with native MTP k=2 passed:

• 12 benign API smoke checks.
• Reasoning-output parsing and tool-call parsing.
• Retrieval from a 7463-token prompt.

These checks exercised the edited MTP layer but were not controlled MTP acceptance or quality benchmarks. They do not establish full-context or throughput performance.

A subsequent shared-profile test ran the original and derivative sequentially with DFlash2 k=7, vision enabled, TP2, FP8 KV and CUDA graphs. Both passed 10/10 functional checks:

Check	Original EXL3	This derivative
Arithmetic, JSON and Chinese answers	Pass	Pass
Reasoning parsing and tool calls	Pass	Pass
Vision: identify red/blue halves of a synthetic image	Pass	Pass
Retrieval from a 9,128-token prompt	Pass	Pass
Three concurrent requests	Pass	Pass
Streaming output	Pass	Pass
Active DFlash drafting and token acceptance	Pass	Pass

The intended checkpoint was verified on both ranks, with the same API model name and runtime settings for both variants. No OOM kills or container restarts were observed during these checks. The profile had a 1,000,000-token configured context ceiling, but no full-1M-context request was tested. Positive draft and accepted-token counters establish that speculation was active, not a controlled DFlash acceptance-rate or speed benchmark. The synthetic image test is a vision smoke check, not a comprehensive multimodal quality evaluation.

Upstream claims versus this model's results

Upstream evidence	Scope
EXL3 model-card KLD 0.02455456425	Published full-25 decoded-model evidence for the parent, not this derivative. The matched parent baseline measured here is 0.024754893.
EXL3 v75 FP8 KLD 0.02461059122	Five runs of one complete 2047-position window, not five full 25-window panels.
EXL3 speed, capacity, DFlash acceptance and vision results	Hardware- and runtime-specific upstream results; not inherited by this derivative.
Keys “32/32 bypass, 0 refuse, 0 garble”	Donor authors' NVFP4 result. Reproduced on this EXL3 derivative: 32/32 bypass, 0 refuse, 0 garble with the donor's unmodified suite (single run, regex classifier, greedy, thinking off).

Matching tensor edits across differently quantized expert bodies does not establish identical refusal behavior.

Usage and compatibility

Use a runtime that explicitly supports the parent's EXL3/TR3/MCG format. The parent documents a custom SM120/B12X path, while MiaAI-Lab provides a separate SM121 implementation. Follow the runtime's hardware-specific installation and launch instructions. Stock vLLM compatibility is not claimed.

Important configuration considerations:

• Do not apply another abliteration pass. The weights are already edited. If the runtime exposes ABLIT, set ABLIT=0. An older L15–45 runtime recipe includes L44 and would apply a second, different modification.
• Keep the supplied EXL3 quantization configuration. Do not relabel the checkpoint compressed-tensors or modelopt_fp4, and do not use the donor's NVFP4-specific loading recipe for these weights.
• Native MTP is retained. External DFlash2 is a separate optional checkpoint with its own compatibility and licensing requirements; it was not used in the paired quality evaluation.
• Chat templates and reasoning parsers affect behavior. The donor's thinking-off template is a serving overlay, not part of the weight edit. Its authors warn that their stock template can ignore enable_thinking=false. Keep template choices explicit rather than assuming a flag alone disables reasoning.

Limitations

• Refusal evidence is a single regex-classified 32-prompt run (donor's suite, unmodified); no human-rated or multi-run refusal evaluation.
• GSM8K results cover only a fixed 50-question subset, not general capability.
• Distributional fidelity measurably decreased after the transplant.
• No full-128K or full-1M context stress test, controlled throughput benchmark, comprehensive vision/video evaluation, MTP acceptance-quality benchmark or controlled DFlash2 performance benchmark was performed for this derivative.
• Retaining a weight component or configuration setting does not establish its functional quality in every runtime.
• Results may vary with hardware, kernels, KV-cache precision, templates, sampling configuration and speculative decoding.

License and acknowledgements

This derivative retains the EXL3 parent's ShapleyMCG License 1.0:

• Parent checkpoint license
• ShapleyMCG license
• Parent third-party notices

The donor's MIT declaration does not relicense the combined EXL3 checkpoint as MIT. Retain the applicable license files, third-party notices, encoder closure and bundled third-party license texts when redistributing the model. Review all applicable upstream terms before use or redistribution.

Credits:

• Brandon Music / ShapleyMCG — EXL3 checkpoint, encoder reproducibility closure, teacher-logit assets and fidelity evaluation harness.
• Keys / drowzeys — anchored tensor selection and documented transplant method; Dealign — edited attention-output tensors; RedHatAI — donor checkpoint body; Z.ai — original GLM model.
• OrcaRouter — the donor's format guide, not a checkpoint body copied here; Blackfrost Research — the donor's historical direction reference, not a projection applied to this derivative.
• MiaAI-Lab, Local Inference Lab / B12X, vLLM, turboderp / ExLlamaV3, and the contributors named in the retained third-party notices — runtime and quantization foundations.
• malaiwah — fidelity-suite reference documentation; OpenAI — GSM8K.

These acknowledgements identify provenance, not endorsement.

Responsible use

Treat the model as potentially having reduced safeguards. Apply appropriate application-level controls, access restrictions and human review. No safety certification is claimed; refusal evidence is a single automated 32-prompt run with a regex classifier, not a safety evaluation.

Preserve and comply with applicable upstream access and responsible-use terms, including the donor's Responsible Use Agreement.