---
library_name: transformers
license: apache-2.0
base_model: Qwen/Qwen3-30B-A3B-Base
tags:
- qwen3
- mixture-of-experts
- quantization-research
- shapleymcg
---

# Qwen3-30B-A3B ShapleyMCG K3/K4 validation reconstruction

This repository is the **exact expanded BF16 validation model** used for the
reported WikiText KLD experiment. It installs the selected K3/K4 reconstructed
expert matrices into the original Transformers checkpoint so the measured
student logits can be independently reproduced.

## Important format statement

The allocation has a logical average rate of **3.5 bits per MoE expert-weight
element** (9,216 K3 and 9,216 K4
matrix choices). This repository stores those reconstructions in BF16 and is
therefore about the size of the BF16 base model. It is **not** a compact
3.5-bpw checkpoint and is **not runtime-qualified**. The current official BTX
format couples gate/up bit rates, while this experiment selects gate and up
independently; publishing it as official BTX would misrepresent the measured
model.

## Exact KLD result

Against sealed BF16 teacher logits on 2,047 next-token positions from the
immutable WikiText-2-raw-v1 test-prefix window:

| Metric | Value |
|---|---:|
| Mean KLD | 0.0550610707425 |
| Median KLD | 0.0129405570572 |
| P95 KLD | 0.246331153046 |
| Maximum KLD | 3.09338690461 |

Allocation SHA256: `10bb3b71a2258500b281e3b37d57657e5e0ecf9318bb9c1a3fc6dbd2326e63c4`  
KLD report seal: `f1ac3903116141faa18bf2f8253ebd875730451f8c0a23645b366fd5517a072b`  
Independent verification seal: `b866e70b5e48433b3d9a014531acb0c460b3788198f3f8855830b6e58fd93fe6`

## Controlled causal-allocation result

The historical and causal arms use the same exact `Qwen3-30B-A3B-Base`
revision, candidate reconstructions, 9,216 K3 plus 9,216 K4 matrix choices,
sealed BF16 teacher, token IDs, SDPA attention implementation, and four-layer
reanchoring cadence. Only the exact-rate expert-matrix allocation changes.

| Exact-3.5 routed-expert allocation | Mean KLD | Top-1 agreement |
|---|---:|---:|
| Historical Hessian/router allocation | 0.0624957262898 | 0.90131900342 |
| **Aumann–Shapley/Fisher causal allocation** | **0.0550610707425** | **0.912066438691** |

The causal allocation lowers KLD by **11.8963%**
and gains **1.0747 percentage points** in
top-1 agreement. It changes 9,392 of the 18,432
matrix choices without changing the exact logical rate or stored candidate
payload budget. The independent float64 replay differs by at most
`9.04e-13` per token.

## Same-parent 20,480-position WikiText replication

The broader panel follows ExLlamaV3's published WikiText construction (ten
consecutive 2,048-token rows), but both arms and the BF16 teacher are executed
with Transformers SDPA. It is therefore a matched internal replication—not a
claim that Transformers SDPA and ExLlamaV3-native attention are numerically
identical.

| Exact-3.5 routed-expert allocation | Mean KLD | Top-1 agreement |
|---|---:|---:|
| Historical Hessian/router allocation | 0.049088886473 | 0.908203125 |
| **Aumann–Shapley/Fisher causal allocation** | **0.0452937027269** | **0.910888671875** |

The causal allocation lowers 20k-panel KLD by
**7.7312%** and gains
**0.2686 percentage points** in
top-1 agreement.
Independent float64 replay has maximum per-token differences of
`0` for the causal arm and
`0` for the historical
arm. The complete teacher/student logits, per-token vectors, reports, manifests,
and verification artifacts are retained in the reproducibility dataset.

The full candidates, calibration/Hessian artifacts, teacher and student logits,
token window, manifests, hashes, and receipts are published in
[`brandonmusic/shapleymcg-qwen3-30b-a3b-reproducibility`](https://huggingface.co/datasets/brandonmusic/shapleymcg-qwen3-30b-a3b-reproducibility).
The executable pipeline and methodology are in
[`brandonmmusic-max/shapleymcg`](https://github.com/brandonmmusic-max/shapleymcg).

## Post-trained replication and complete result index

The same full causal allocator was subsequently run on the post-trained
`Qwen/Qwen3-30B-A3B` parent using its separately fitted, retained MCG candidate
inventory. On a matched 20,480-position eager WikiText panel at exact 3.5
routed-expert BPW, it lowered KLD from the predecessor routed-p2 allocation's
`0.0411226321813` to `0.0403685517455` (**1.833736%**) and raised top-1 agreement
from `0.924267578125` to `0.925683593750`. Independent float64 replay had zero
maximum tokenwise difference.

The allocation also transferred to TurboDerp's unchanged exact-3.5 expert
candidate pool, lowering its matched KLD from `0.0302749179770` to
`0.0292690766473` (**3.322359%**) and improving top-1 agreement by
**0.273438 percentage points**. The separately produced MCG pool measured
`0.0457931025429` under the same allocation and body scope. That comparison
does not isolate the codec: calibration corpus and hidden-state policy,
rotation/scaling choices, procedural selection, and numeric encoder
implementation also differ.

The stricter allocator-only test reconstructs both arms from the identical
published TurboDerp K3/K4 candidates at exact 3.5 expert BPW. TurboDerp
v0.0.1's carried-surplus rule measured KLD `0.0339415351804`; full
Aumann–Shapley/Fisher measured `0.0292690766473`, a **13.766197%** reduction,
with a **0.600586 percentage-point** top-1 gain and all 10 evaluation rows
favorable. This proves an allocator-only advantage over the v0.0.1 rule on the
common reconstructed candidate bytes; it is not a comparison against a native
TurboDerp 3.5 checkpoint, which was not published for this model.

This model repository remains the Base-parent validation reconstruction; the
post-trained logits and provenance are in
[`brandonmusic/shapleymcg-qwen3-30b-a3b-posttrained-reproducibility`](https://huggingface.co/datasets/brandonmusic/shapleymcg-qwen3-30b-a3b-posttrained-reproducibility/tree/main/full-causal-v1).
The GitHub
[`complete results ledger`](https://github.com/brandonmmusic-max/shapleymcg/blob/main/docs/QWEN_COMPLETE_RESULTS_LEDGER.md)
and
[`method specification`](https://github.com/brandonmmusic-max/shapleymcg/blob/main/docs/SHAPLEYMCG_METHOD.md)
distinguish parent, allocator generation, codec, dense-weight scope, evaluator,
and token panel for every headline number.

## Loading

```python
from transformers import AutoModelForCausalLM, AutoTokenizer

model_id = "brandonmusic/Qwen3-30B-A3B-ShapleyMCG-K34-Validation-Reconstruction"
tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(model_id, torch_dtype="auto", device_map="auto")
```

## Attribution

- Base model: the Qwen team, [`Qwen/Qwen3-30B-A3B-Base`](https://huggingface.co/Qwen/Qwen3-30B-A3B-Base).
- EXL3/TRELLIS codec lineage: turboderp and ExLlamaV3 contributors.
- Aumann–Shapley quantization precedent: Joshua Hill,
  [`Saturation Makes Quantization Error Additive`](https://arxiv.org/abs/2607.12266),
  and NVIDIA Model Optimizer PR #2183.
- Routed Fisher/Jacobian attribution, exact-rate reconciliation, and MCG
  integration are described and attributed in the GitHub methodology and
  sealed dataset card.
- Experiment, integration, and publication: Brandon Music / ShapleyMCG.