Text Generation
GGUF
English
Mixture of Experts
mixture-of-experts
domain-specialist
expert-surgery
hand-curated
college-of-experts
ollama
python
programming
code-generation
Instructions to use JThomas-CoE/coe-gemma4-python-hc-14b-a4b-q8_0 with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Notebooks
- Google Colab
- Kaggle
- Local Apps Settings
- llama.cpp
How to use JThomas-CoE/coe-gemma4-python-hc-14b-a4b-q8_0 with llama.cpp:
Install (macOS, Linux)
curl -LsSf https://llama.app/install.sh | sh # Start a local OpenAI-compatible server with a web UI: llama serve -hf JThomas-CoE/coe-gemma4-python-hc-14b-a4b-q8_0:Q8_0 # Run inference directly in the terminal: llama cli -hf JThomas-CoE/coe-gemma4-python-hc-14b-a4b-q8_0:Q8_0
Install from WinGet (Windows)
winget install llama.cpp # Start a local OpenAI-compatible server with a web UI: llama serve -hf JThomas-CoE/coe-gemma4-python-hc-14b-a4b-q8_0:Q8_0 # Run inference directly in the terminal: llama cli -hf JThomas-CoE/coe-gemma4-python-hc-14b-a4b-q8_0:Q8_0
Use pre-built binary
# Download pre-built binary from: # https://github.com/ggerganov/llama.cpp/releases # Start a local OpenAI-compatible server with a web UI: ./llama-server -hf JThomas-CoE/coe-gemma4-python-hc-14b-a4b-q8_0:Q8_0 # Run inference directly in the terminal: ./llama-cli -hf JThomas-CoE/coe-gemma4-python-hc-14b-a4b-q8_0:Q8_0
Build from source code
git clone https://github.com/ggerganov/llama.cpp.git cd llama.cpp cmake -B build cmake --build build -j --target llama-server llama-cli # Start a local OpenAI-compatible server with a web UI: ./build/bin/llama-server -hf JThomas-CoE/coe-gemma4-python-hc-14b-a4b-q8_0:Q8_0 # Run inference directly in the terminal: ./build/bin/llama-cli -hf JThomas-CoE/coe-gemma4-python-hc-14b-a4b-q8_0:Q8_0
Use Docker
docker model run hf.co/JThomas-CoE/coe-gemma4-python-hc-14b-a4b-q8_0:Q8_0
- LM Studio
- Jan
- vLLM
How to use JThomas-CoE/coe-gemma4-python-hc-14b-a4b-q8_0 with vLLM:
Install from pip and serve model
# Install vLLM from pip: pip install vllm # Start the vLLM server: vllm serve "JThomas-CoE/coe-gemma4-python-hc-14b-a4b-q8_0" # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:8000/v1/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "JThomas-CoE/coe-gemma4-python-hc-14b-a4b-q8_0", "prompt": "Once upon a time,", "max_tokens": 512, "temperature": 0.5 }'Use Docker
docker model run hf.co/JThomas-CoE/coe-gemma4-python-hc-14b-a4b-q8_0:Q8_0
- Ollama
How to use JThomas-CoE/coe-gemma4-python-hc-14b-a4b-q8_0 with Ollama:
ollama run hf.co/JThomas-CoE/coe-gemma4-python-hc-14b-a4b-q8_0:Q8_0
- Unsloth Studio
How to use JThomas-CoE/coe-gemma4-python-hc-14b-a4b-q8_0 with Unsloth Studio:
Install Unsloth Studio (macOS, Linux, WSL)
curl -fsSL https://unsloth.ai/install.sh | sh # Run unsloth studio unsloth studio -H 0.0.0.0 -p 8888 # Then open http://localhost:8888 in your browser # Search for JThomas-CoE/coe-gemma4-python-hc-14b-a4b-q8_0 to start chatting
Install Unsloth Studio (Windows)
irm https://unsloth.ai/install.ps1 | iex # Run unsloth studio unsloth studio -H 0.0.0.0 -p 8888 # Then open http://localhost:8888 in your browser # Search for JThomas-CoE/coe-gemma4-python-hc-14b-a4b-q8_0 to start chatting
Using HuggingFace Spaces for Unsloth
# No setup required # Open https://huggingface.co/spaces/unsloth/studio in your browser # Search for JThomas-CoE/coe-gemma4-python-hc-14b-a4b-q8_0 to start chatting
- Docker Model Runner
How to use JThomas-CoE/coe-gemma4-python-hc-14b-a4b-q8_0 with Docker Model Runner:
docker model run hf.co/JThomas-CoE/coe-gemma4-python-hc-14b-a4b-q8_0:Q8_0
- Lemonade
How to use JThomas-CoE/coe-gemma4-python-hc-14b-a4b-q8_0 with Lemonade:
Pull the model
# Download Lemonade from https://lemonade-server.ai/ lemonade pull JThomas-CoE/coe-gemma4-python-hc-14b-a4b-q8_0:Q8_0
Run and chat with the model
lemonade run user.coe-gemma4-python-hc-14b-a4b-q8_0-Q8_0
List all available models
lemonade list
- Atomic Chat
Fix tilde approximations: ~ -> proper Unicode ≈
Browse files
README.md
CHANGED
|
@@ -24,9 +24,9 @@ library_name: gguf
|
|
| 24 |
# Gemma4 College of Experts — Python Specialist
|
| 25 |
|
| 26 |
**Base model:** [google/gemma-4-26b-it](https://huggingface.co/google/gemma-4-26b-it)
|
| 27 |
-
**Architecture:** MoE — 26B total /
|
| 28 |
**Method:** Activation-directed expert surgery — 128 → 64 experts per layer (50% reduction)
|
| 29 |
-
**Quantizations:** Q4_K_M (
|
| 30 |
**HF repos:** `JThomas-CoE/coe-gemma4-python-hc-14b-a4b-q4` · `JThomas-CoE/coe-gemma4-python-hc-14b-a4b-q8`
|
| 31 |
**Ollama tags:** `coe-gemma4-python-14b-a4b:q4` · `coe-gemma4-python-14b-a4b:q8`
|
| 32 |
**Hub:** [JThomas-CoE on HuggingFace](https://huggingface.co/JThomas-CoE)
|
|
@@ -93,15 +93,15 @@ Temperature 0.6 is strongly recommended. Higher temperatures (≥ 0.8) materiall
|
|
| 93 |
|
| 94 |
Both variants use the **same pruned expert mask** — the same 64 experts are retained per layer in both cases. The difference is quantization precision for the retained weights.
|
| 95 |
|
| 96 |
-
**Q4_K_M** loads
|
| 97 |
- Routine completions, single-function generation, code review
|
| 98 |
- Users on 16 GB VRAM cards
|
| 99 |
- High-throughput multi-turn sessions
|
| 100 |
|
| 101 |
-
**Q8_0** loads
|
| 102 |
- Harder, trickier single problems where answer quality matters more than speed
|
| 103 |
- Multi-step algorithmic challenges, competitive programming problems
|
| 104 |
-
- Contexts where the extra
|
| 105 |
|
| 106 |
### Empirical Q4 vs Q8 Coding Quality Comparison (LiveCodeBench)
|
| 107 |
|
|
@@ -125,10 +125,10 @@ These models are produced by **activation-directed expert surgery** applied to t
|
|
| 125 |
|
| 126 |
| Configuration | VRAM (16k ctx) | VRAM (64k ctx) | Active params |
|
| 127 |
|---|---|---|---|
|
| 128 |
-
| Gemma4-26B parent (Q4_K_M) | 19.4 GB | 20.5 GB |
|
| 129 |
-
| Gemma4-26B parent (Q8_0) | 37.5 GB | 38.6 GB |
|
| 130 |
-
| Specialist K=64 (Q4_K_M) | **12.3 GB** | **13.4 GB** |
|
| 131 |
-
| Specialist K=64 (Q8_0) | **21.0 GB** | **22.1 GB** |
|
| 132 |
| Q4 savings vs Q4 parent | **7.1 GB (37%)** | **7.1 GB (35%)** | unchanged |
|
| 133 |
| Q8 savings vs Q8 parent | **16.5 GB (44%)** | **16.5 GB (43%)** | unchanged |
|
| 134 |
|
|
@@ -164,7 +164,7 @@ The Python corpus combines:
|
|
| 164 |
|
| 165 |
Corpus size: approximately 892,000 tokens across the python.stdlib, python.scientific, python.data, python.async_web, and python.algorithms sub-corpora. Profiling was run on the full parent model with router hooks capturing per-token expert selections across all 30 MoE layers.
|
| 166 |
|
| 167 |
-
**Corpus size considerations.** Choosing how much material to include for activation profiling involves two competing pressures. On one side, a corpus that is too small or too narrow may fail to activate the full set of experts that are genuinely relevant to the domain: rare but important concepts may appear in too few tokens to accumulate statistically reliable activation counts, leaving their associated experts underweighted or excluded from the mask. On the other side, a corpus that grows too large — particularly if expansion is driven by including only tangentially related material to hit a token budget — risks diluting the activation signal. At
|
| 168 |
|
| 169 |
### Step 2 — 3D Histogram Collection
|
| 170 |
|
|
@@ -193,7 +193,7 @@ This is the frequency-weighted mean router confidence — how often the expert i
|
|
| 193 |
|
| 194 |
**Pass 2 — Structural whitelist enforcement:** A set of experts identified as those experts that have an average activation rank of less than 2 and a minimum number of activations of 10 or more regardless of overall utility ranking. These are then swapped into the model if they are not included already by utility rank by swapping with existing included experts with low average rank and low utility. Applied to ensure high-confidence structural experts are never displaced by marginally scoring domain specialists.
|
| 195 |
|
| 196 |
-
**Pass 3 — CoT/reasoning arbitrage:** Experts that activate strongly on domain-agnostic logic/reasoning chain-of-thought traces are swapped into the mask. Applied at a cap of 3 swaps per layer;
|
| 197 |
|
| 198 |
### Step 5 — GGUF Surgery
|
| 199 |
|
|
|
|
| 24 |
# Gemma4 College of Experts — Python Specialist
|
| 25 |
|
| 26 |
**Base model:** [google/gemma-4-26b-it](https://huggingface.co/google/gemma-4-26b-it)
|
| 27 |
+
**Architecture:** MoE — 26B total / ≈4B active parameters (1 shared expert + 8 routed from a pool of 128 per MoE layer, 30 MoE layers)
|
| 28 |
**Method:** Activation-directed expert surgery — 128 → 64 experts per layer (50% reduction)
|
| 29 |
+
**Quantizations:** Q4_K_M (≈9.7 GB on disk) · Q8_0 (≈18.4 GB on disk)
|
| 30 |
**HF repos:** `JThomas-CoE/coe-gemma4-python-hc-14b-a4b-q4` · `JThomas-CoE/coe-gemma4-python-hc-14b-a4b-q8`
|
| 31 |
**Ollama tags:** `coe-gemma4-python-14b-a4b:q4` · `coe-gemma4-python-14b-a4b:q8`
|
| 32 |
**Hub:** [JThomas-CoE on HuggingFace](https://huggingface.co/JThomas-CoE)
|
|
|
|
| 93 |
|
| 94 |
Both variants use the **same pruned expert mask** — the same 64 experts are retained per layer in both cases. The difference is quantization precision for the retained weights.
|
| 95 |
|
| 96 |
+
**Q4_K_M** loads ≈12.3 GB VRAM at 16k context (≈7.1 GB less than the Q4 parent). It is the right choice for:
|
| 97 |
- Routine completions, single-function generation, code review
|
| 98 |
- Users on 16 GB VRAM cards
|
| 99 |
- High-throughput multi-turn sessions
|
| 100 |
|
| 101 |
+
**Q8_0** loads ≈21.0 GB VRAM at 16k context. It is the right choice for:
|
| 102 |
- Harder, trickier single problems where answer quality matters more than speed
|
| 103 |
- Multi-step algorithmic challenges, competitive programming problems
|
| 104 |
+
- Contexts where the extra ≈8 GB VRAM is available
|
| 105 |
|
| 106 |
### Empirical Q4 vs Q8 Coding Quality Comparison (LiveCodeBench)
|
| 107 |
|
|
|
|
| 125 |
|
| 126 |
| Configuration | VRAM (16k ctx) | VRAM (64k ctx) | Active params |
|
| 127 |
|---|---|---|---|
|
| 128 |
+
| Gemma4-26B parent (Q4_K_M) | 19.4 GB | 20.5 GB | ≈4B |
|
| 129 |
+
| Gemma4-26B parent (Q8_0) | 37.5 GB | 38.6 GB | ≈4B |
|
| 130 |
+
| Specialist K=64 (Q4_K_M) | **12.3 GB** | **13.4 GB** | ≈4B |
|
| 131 |
+
| Specialist K=64 (Q8_0) | **21.0 GB** | **22.1 GB** | ≈4B |
|
| 132 |
| Q4 savings vs Q4 parent | **7.1 GB (37%)** | **7.1 GB (35%)** | unchanged |
|
| 133 |
| Q8 savings vs Q8 parent | **16.5 GB (44%)** | **16.5 GB (43%)** | unchanged |
|
| 134 |
|
|
|
|
| 164 |
|
| 165 |
Corpus size: approximately 892,000 tokens across the python.stdlib, python.scientific, python.data, python.async_web, and python.algorithms sub-corpora. Profiling was run on the full parent model with router hooks capturing per-token expert selections across all 30 MoE layers.
|
| 166 |
|
| 167 |
+
**Corpus size considerations.** Choosing how much material to include for activation profiling involves two competing pressures. On one side, a corpus that is too small or too narrow may fail to activate the full set of experts that are genuinely relevant to the domain: rare but important concepts may appear in too few tokens to accumulate statistically reliable activation counts, leaving their associated experts underweighted or excluded from the mask. On the other side, a corpus that grows too large — particularly if expansion is driven by including only tangentially related material to hit a token budget — risks diluting the activation signal. At ≈892k tokens the Python corpus is the largest in the CoE specialist family and the dilution risk is most live here: the breadth of sub-corpora (stdlib usage, scientific computing, data engineering, async web patterns, algorithms) was chosen to reflect the genuine breadth of Python practice, but each sub-corpus was capped and reviewed for domain relevance to avoid blurring the expert cluster toward general programming rather than Python-specific reasoning. A more rigorous data-driven approach — one that measures the dispersion or entropy of the emerging activation cluster after each corpus increment and uses that as a stopping criterion — would provide principled feedback to arrest growth at the point of diminishing domain focus. This remains an area of future work.
|
| 168 |
|
| 169 |
### Step 2 — 3D Histogram Collection
|
| 170 |
|
|
|
|
| 193 |
|
| 194 |
**Pass 2 — Structural whitelist enforcement:** A set of experts identified as those experts that have an average activation rank of less than 2 and a minimum number of activations of 10 or more regardless of overall utility ranking. These are then swapped into the model if they are not included already by utility rank by swapping with existing included experts with low average rank and low utility. Applied to ensure high-confidence structural experts are never displaced by marginally scoring domain specialists.
|
| 195 |
|
| 196 |
+
**Pass 3 — CoT/reasoning arbitrage:** Experts that activate strongly on domain-agnostic logic/reasoning chain-of-thought traces are swapped into the mask. Applied at a cap of 3 swaps per layer; ≈85–95 total swaps across 30 layers.
|
| 197 |
|
| 198 |
### Step 5 — GGUF Surgery
|
| 199 |
|