JThomas-CoE commited on
Commit
fc3e9c8
·
verified ·
1 Parent(s): 71117ff

Fix tilde approximations: ~ -> proper Unicode ≈

Browse files
Files changed (1) hide show
  1. README.md +11 -11
README.md CHANGED
@@ -24,9 +24,9 @@ library_name: gguf
24
  # Gemma4 College of Experts — Python Specialist
25
 
26
  **Base model:** [google/gemma-4-26b-it](https://huggingface.co/google/gemma-4-26b-it)
27
- **Architecture:** MoE — 26B total / ~4B active parameters (1 shared expert + 8 routed from a pool of 128 per MoE layer, 30 MoE layers)
28
  **Method:** Activation-directed expert surgery — 128 → 64 experts per layer (50% reduction)
29
- **Quantizations:** Q4_K_M (~9.7 GB on disk) · Q8_0 (~18.4 GB on disk)
30
  **HF repos:** `JThomas-CoE/coe-gemma4-python-hc-14b-a4b-q4` · `JThomas-CoE/coe-gemma4-python-hc-14b-a4b-q8`
31
  **Ollama tags:** `coe-gemma4-python-14b-a4b:q4` · `coe-gemma4-python-14b-a4b:q8`
32
  **Hub:** [JThomas-CoE on HuggingFace](https://huggingface.co/JThomas-CoE)
@@ -93,15 +93,15 @@ Temperature 0.6 is strongly recommended. Higher temperatures (≥ 0.8) materiall
93
 
94
  Both variants use the **same pruned expert mask** — the same 64 experts are retained per layer in both cases. The difference is quantization precision for the retained weights.
95
 
96
- **Q4_K_M** loads ~12.3 GB VRAM at 16k context (~7.1 GB less than the Q4 parent). It is the right choice for:
97
  - Routine completions, single-function generation, code review
98
  - Users on 16 GB VRAM cards
99
  - High-throughput multi-turn sessions
100
 
101
- **Q8_0** loads ~21.0 GB VRAM at 16k context. It is the right choice for:
102
  - Harder, trickier single problems where answer quality matters more than speed
103
  - Multi-step algorithmic challenges, competitive programming problems
104
- - Contexts where the extra ~8 GB VRAM is available
105
 
106
  ### Empirical Q4 vs Q8 Coding Quality Comparison (LiveCodeBench)
107
 
@@ -125,10 +125,10 @@ These models are produced by **activation-directed expert surgery** applied to t
125
 
126
  | Configuration | VRAM (16k ctx) | VRAM (64k ctx) | Active params |
127
  |---|---|---|---|
128
- | Gemma4-26B parent (Q4_K_M) | 19.4 GB | 20.5 GB | ~4B |
129
- | Gemma4-26B parent (Q8_0) | 37.5 GB | 38.6 GB | ~4B |
130
- | Specialist K=64 (Q4_K_M) | **12.3 GB** | **13.4 GB** | ~4B |
131
- | Specialist K=64 (Q8_0) | **21.0 GB** | **22.1 GB** | ~4B |
132
  | Q4 savings vs Q4 parent | **7.1 GB (37%)** | **7.1 GB (35%)** | unchanged |
133
  | Q8 savings vs Q8 parent | **16.5 GB (44%)** | **16.5 GB (43%)** | unchanged |
134
 
@@ -164,7 +164,7 @@ The Python corpus combines:
164
 
165
  Corpus size: approximately 892,000 tokens across the python.stdlib, python.scientific, python.data, python.async_web, and python.algorithms sub-corpora. Profiling was run on the full parent model with router hooks capturing per-token expert selections across all 30 MoE layers.
166
 
167
- **Corpus size considerations.** Choosing how much material to include for activation profiling involves two competing pressures. On one side, a corpus that is too small or too narrow may fail to activate the full set of experts that are genuinely relevant to the domain: rare but important concepts may appear in too few tokens to accumulate statistically reliable activation counts, leaving their associated experts underweighted or excluded from the mask. On the other side, a corpus that grows too large — particularly if expansion is driven by including only tangentially related material to hit a token budget — risks diluting the activation signal. At ~892k tokens the Python corpus is the largest in the CoE specialist family and the dilution risk is most live here: the breadth of sub-corpora (stdlib usage, scientific computing, data engineering, async web patterns, algorithms) was chosen to reflect the genuine breadth of Python practice, but each sub-corpus was capped and reviewed for domain relevance to avoid blurring the expert cluster toward general programming rather than Python-specific reasoning. A more rigorous data-driven approach — one that measures the dispersion or entropy of the emerging activation cluster after each corpus increment and uses that as a stopping criterion — would provide principled feedback to arrest growth at the point of diminishing domain focus. This remains an area of future work.
168
 
169
  ### Step 2 — 3D Histogram Collection
170
 
@@ -193,7 +193,7 @@ This is the frequency-weighted mean router confidence — how often the expert i
193
 
194
  **Pass 2 — Structural whitelist enforcement:** A set of experts identified as those experts that have an average activation rank of less than 2 and a minimum number of activations of 10 or more regardless of overall utility ranking. These are then swapped into the model if they are not included already by utility rank by swapping with existing included experts with low average rank and low utility. Applied to ensure high-confidence structural experts are never displaced by marginally scoring domain specialists.
195
 
196
- **Pass 3 — CoT/reasoning arbitrage:** Experts that activate strongly on domain-agnostic logic/reasoning chain-of-thought traces are swapped into the mask. Applied at a cap of 3 swaps per layer; ~85–95 total swaps across 30 layers.
197
 
198
  ### Step 5 — GGUF Surgery
199
 
 
24
  # Gemma4 College of Experts — Python Specialist
25
 
26
  **Base model:** [google/gemma-4-26b-it](https://huggingface.co/google/gemma-4-26b-it)
27
+ **Architecture:** MoE — 26B total / 4B active parameters (1 shared expert + 8 routed from a pool of 128 per MoE layer, 30 MoE layers)
28
  **Method:** Activation-directed expert surgery — 128 → 64 experts per layer (50% reduction)
29
+ **Quantizations:** Q4_K_M (9.7 GB on disk) · Q8_0 (18.4 GB on disk)
30
  **HF repos:** `JThomas-CoE/coe-gemma4-python-hc-14b-a4b-q4` · `JThomas-CoE/coe-gemma4-python-hc-14b-a4b-q8`
31
  **Ollama tags:** `coe-gemma4-python-14b-a4b:q4` · `coe-gemma4-python-14b-a4b:q8`
32
  **Hub:** [JThomas-CoE on HuggingFace](https://huggingface.co/JThomas-CoE)
 
93
 
94
  Both variants use the **same pruned expert mask** — the same 64 experts are retained per layer in both cases. The difference is quantization precision for the retained weights.
95
 
96
+ **Q4_K_M** loads 12.3 GB VRAM at 16k context (7.1 GB less than the Q4 parent). It is the right choice for:
97
  - Routine completions, single-function generation, code review
98
  - Users on 16 GB VRAM cards
99
  - High-throughput multi-turn sessions
100
 
101
+ **Q8_0** loads 21.0 GB VRAM at 16k context. It is the right choice for:
102
  - Harder, trickier single problems where answer quality matters more than speed
103
  - Multi-step algorithmic challenges, competitive programming problems
104
+ - Contexts where the extra 8 GB VRAM is available
105
 
106
  ### Empirical Q4 vs Q8 Coding Quality Comparison (LiveCodeBench)
107
 
 
125
 
126
  | Configuration | VRAM (16k ctx) | VRAM (64k ctx) | Active params |
127
  |---|---|---|---|
128
+ | Gemma4-26B parent (Q4_K_M) | 19.4 GB | 20.5 GB | 4B |
129
+ | Gemma4-26B parent (Q8_0) | 37.5 GB | 38.6 GB | 4B |
130
+ | Specialist K=64 (Q4_K_M) | **12.3 GB** | **13.4 GB** | 4B |
131
+ | Specialist K=64 (Q8_0) | **21.0 GB** | **22.1 GB** | 4B |
132
  | Q4 savings vs Q4 parent | **7.1 GB (37%)** | **7.1 GB (35%)** | unchanged |
133
  | Q8 savings vs Q8 parent | **16.5 GB (44%)** | **16.5 GB (43%)** | unchanged |
134
 
 
164
 
165
  Corpus size: approximately 892,000 tokens across the python.stdlib, python.scientific, python.data, python.async_web, and python.algorithms sub-corpora. Profiling was run on the full parent model with router hooks capturing per-token expert selections across all 30 MoE layers.
166
 
167
+ **Corpus size considerations.** Choosing how much material to include for activation profiling involves two competing pressures. On one side, a corpus that is too small or too narrow may fail to activate the full set of experts that are genuinely relevant to the domain: rare but important concepts may appear in too few tokens to accumulate statistically reliable activation counts, leaving their associated experts underweighted or excluded from the mask. On the other side, a corpus that grows too large — particularly if expansion is driven by including only tangentially related material to hit a token budget — risks diluting the activation signal. At 892k tokens the Python corpus is the largest in the CoE specialist family and the dilution risk is most live here: the breadth of sub-corpora (stdlib usage, scientific computing, data engineering, async web patterns, algorithms) was chosen to reflect the genuine breadth of Python practice, but each sub-corpus was capped and reviewed for domain relevance to avoid blurring the expert cluster toward general programming rather than Python-specific reasoning. A more rigorous data-driven approach — one that measures the dispersion or entropy of the emerging activation cluster after each corpus increment and uses that as a stopping criterion — would provide principled feedback to arrest growth at the point of diminishing domain focus. This remains an area of future work.
168
 
169
  ### Step 2 — 3D Histogram Collection
170
 
 
193
 
194
  **Pass 2 — Structural whitelist enforcement:** A set of experts identified as those experts that have an average activation rank of less than 2 and a minimum number of activations of 10 or more regardless of overall utility ranking. These are then swapped into the model if they are not included already by utility rank by swapping with existing included experts with low average rank and low utility. Applied to ensure high-confidence structural experts are never displaced by marginally scoring domain specialists.
195
 
196
+ **Pass 3 — CoT/reasoning arbitrage:** Experts that activate strongly on domain-agnostic logic/reasoning chain-of-thought traces are swapped into the mask. Applied at a cap of 3 swaps per layer; 85–95 total swaps across 30 layers.
197
 
198
  ### Step 5 — GGUF Surgery
199