Per-Tensor Quantization Arms — Methodology & Principles

Companion technical document for Confucius4-R2T2-Q4_K_M-GGUF. Details the general per-tensor quantization arms methodology, --keep-type semantics, cross-lingual validation protocols, and core architectural rules.

1. Why Per-Tensor Arms?

Standard model quantization tools apply uniform presets (e.g. standard Q4_K_M or Q5_K_M) across entire layers or architectures. For many speech models, however, uniform quantization either fails catastrophic quality checks or leaves substantial memory optimization on the table.

Building quantization arms means creating experimental variants that differ in exactly one block or layer type, isolating the quality, memory, and speed impact of each architectural component:

Isolating which blocks represent non-negotiable precision floors.
Finding which blocks can be aggressively quantized without measurable loss.
Measuring real kernel execution speed across mixed-precision representations.

2. Converter Overrides & Mechanics

Per-tensor control requires specifying precision at the individual weight level:

Base Type: Sets the baseline default type for all quantizable weights in the graph.
Explicit Overrides (--keep-type <tensor>=<type>): Selectively overrides the precision of specific tensors.
Source Naming Contract: Override rules must target the exact source parameter names (e.g. thinker.model.layers.0.mlp.down_proj.weight), not transformed runtime names.
Asserting Tensor Census: Always assert a strict tensor census per architectural block before and after conversion to prevent silent fallbacks or unmatched patterns:
- Tower: 147 tensors
- Attention (q, k, v, o_proj): 112 tensors
- Gate / Up (gate_proj, up_proj): 56 tensors
- Down (down_proj): 28 tensors
- Embeddings: 1 tensor
Command-Line Limits: When passing dozens of tensor overrides on Windows systems, drive conversion scripts from bash or JSON specifications rather than cmd.exe to avoid the 8191-character command-line length truncation limit.

3. Verification Protocol: Verify What Was Built

Never infer a model arm's composition from the conversion arguments alone—conversion flags can fail silently without throwing errors. Always verify the resulting GGUF by inspecting the tensor header table:

Verify that all 28 down_proj weights are assigned their target type.
Confirm that embed_tokens was successfully overridden from the default F16 pin to the desired low-bit type.
Confirm that sensitive encoder layers retain their target mixed precision.

4. Cross-Lingual Validation Protocol

A critical lesson learned during quantization validation:

English and Chinese Are Not a Screen

Speech models like R2T2 are trained with massive data allocations for English and Chinese, making them the model's most resilient input distributions.

Multiple quantization configurations matched the Q8_0 reference baseline on English (jfk) and Chinese (zh), but completely broke down on non-target languages.
For example, naive Q5_K matched Q8_0 on English and Chinese, but silently flipped to English when fed Russian audio.
Other experimental configurations passed English and Chinese tests, but emitted English translations when given German audio.

The Non-Target Probe Principle

Always evaluate quantization quality against languages with smaller training footprints (e.g. German, French, Russian). If a model switches languages or drops tokens on secondary languages, its representation space has degraded.

Automated Detection

Rather than manually inspecting hours of transcripts, automated validation checks should monitor:

Detected Language Output: Grep the model runtime's detected language tag (detected-language:). A switch from de or fr to en immediately identifies representation collapse.
Three-Tier Word Diffing:
- Raw: Exact matching including punctuation and casing differences (benign).
- Strict: Lowercase, punctuation-stripped matching.
- Loose: Diacritics folded (NFD normalization). Flags true content drift (omitted, substituted, or hallucinated words).
Empty Output Detection: Low-bit matmuls can collapse abruptly to empty strings. Ensure output files are confirmed non-empty before running text diffs.

5. Architectural Principles That Generalize

The findings from the R2T2 quantization campaign reveal principles that apply broadly to modern transformer and speech architectures:

Quantization Error is Cumulative, Not Per-Tensor: There is no fixed precision floor for a block in isolation. A block at Q4_K may work perfectly alone, but cause collapse when another block is also reduced. The total error budget is shared network-wide.
Residual Stream Writes Compound Strictly: Blocks that write directly into the residual stream (down_proj) compound numerical error through every subsequent layer. Consequently, down_proj imposes the strictest precision requirement (Q6_K for R2T2).
Non-Linearities Bound Error Propagation: Projections that feed into bounded activation functions (such as SwiGLU in gate_proj and up_proj) tolerate lower bitwidths (Q4_K) because the non-linearity bounds error growth.
Lookup Tables Do Not Compound: Token embeddings (embed_tokens) are accessed via table lookup. Their quantization error is introduced once per token and does not propagate recurrently. Aggressively quantizing embeddings (even to Q2_K) saves substantial memory with virtually no degradation.
Low-Bit Matmul Cliff: Matmul quantization does not degrade gracefully below Q4_K. Q3_K matmuls in R2T2 produced immediate empty output cliffs.
Speed Does Not Track Bit Width Alone: Kernel optimization quality dominates hardware throughput. Q5_K performed ~20% slower than Q8_0 due to non-vectorized paths, whereas highly optimized Q4_K and Q6_K routines provided significant speedups.