Instructions to use MrFuzzihead/Qwen3.8-35B-A3B-Distill-APEX-GGUF with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Notebooks
Google Colab
Kaggle
Local Apps Settings

How to use MrFuzzihead/Qwen3.8-35B-A3B-Distill-APEX-GGUF with llama.cpp:

Install (macOS, Linux)

curl -LsSf https://llama.app/install.sh | sh
# Start a local OpenAI-compatible server with a web UI:
llama serve -hf MrFuzzihead/Qwen3.8-35B-A3B-Distill-APEX-GGUF:Q4_K_M
# Run inference directly in the terminal:
llama cli -hf MrFuzzihead/Qwen3.8-35B-A3B-Distill-APEX-GGUF:Q4_K_M

Install from WinGet (Windows)

winget install llama.cpp
# Start a local OpenAI-compatible server with a web UI:
llama serve -hf MrFuzzihead/Qwen3.8-35B-A3B-Distill-APEX-GGUF:Q4_K_M
# Run inference directly in the terminal:
llama cli -hf MrFuzzihead/Qwen3.8-35B-A3B-Distill-APEX-GGUF:Q4_K_M

Use pre-built binary

# Download pre-built binary from:
# https://github.com/ggerganov/llama.cpp/releases
# Start a local OpenAI-compatible server with a web UI:
./llama-server -hf MrFuzzihead/Qwen3.8-35B-A3B-Distill-APEX-GGUF:Q4_K_M
# Run inference directly in the terminal:
./llama-cli -hf MrFuzzihead/Qwen3.8-35B-A3B-Distill-APEX-GGUF:Q4_K_M

Build from source code

git clone https://github.com/ggerganov/llama.cpp.git
cd llama.cpp
cmake -B build
cmake --build build -j --target llama-server llama-cli
# Start a local OpenAI-compatible server with a web UI:
./build/bin/llama-server -hf MrFuzzihead/Qwen3.8-35B-A3B-Distill-APEX-GGUF:Q4_K_M
# Run inference directly in the terminal:
./build/bin/llama-cli -hf MrFuzzihead/Qwen3.8-35B-A3B-Distill-APEX-GGUF:Q4_K_M

Use Docker

docker model run hf.co/MrFuzzihead/Qwen3.8-35B-A3B-Distill-APEX-GGUF:Q4_K_M

LM Studio
Jan

vLLM

How to use MrFuzzihead/Qwen3.8-35B-A3B-Distill-APEX-GGUF with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "MrFuzzihead/Qwen3.8-35B-A3B-Distill-APEX-GGUF"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "MrFuzzihead/Qwen3.8-35B-A3B-Distill-APEX-GGUF",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker

docker model run hf.co/MrFuzzihead/Qwen3.8-35B-A3B-Distill-APEX-GGUF:Q4_K_M

Ollama
How to use MrFuzzihead/Qwen3.8-35B-A3B-Distill-APEX-GGUF with Ollama:
```
ollama run hf.co/MrFuzzihead/Qwen3.8-35B-A3B-Distill-APEX-GGUF:Q4_K_M
```
Unsloth Desktop

How to use MrFuzzihead/Qwen3.8-35B-A3B-Distill-APEX-GGUF with Pi:

Start the llama.cpp server

# Install llama.cpp:
brew install llama.cpp
# Start a local OpenAI-compatible server:
llama serve -hf MrFuzzihead/Qwen3.8-35B-A3B-Distill-APEX-GGUF:Q4_K_M

Configure the model in Pi

# Install Pi:
npm install -g @earendil-works/pi-coding-agent
# Add to ~/.pi/agent/models.json:
{
  "providers": {
    "llama-cpp": {
      "baseUrl": "http://localhost:8080/v1",
      "api": "openai-completions",
      "apiKey": "none",
      "models": [
        {
          "id": "MrFuzzihead/Qwen3.8-35B-A3B-Distill-APEX-GGUF:Q4_K_M"
        }
      ]
    }
  }
}

Run Pi

# Start Pi in your project directory:
pi

Docker Model Runner
How to use MrFuzzihead/Qwen3.8-35B-A3B-Distill-APEX-GGUF with Docker Model Runner:
```
docker model run hf.co/MrFuzzihead/Qwen3.8-35B-A3B-Distill-APEX-GGUF:Q4_K_M
```

Lemonade

How to use MrFuzzihead/Qwen3.8-35B-A3B-Distill-APEX-GGUF with Lemonade:

Pull the model

# Download Lemonade from https://lemonade-server.ai/
lemonade pull MrFuzzihead/Qwen3.8-35B-A3B-Distill-APEX-GGUF:Q4_K_M

Run and chat with the model

lemonade run user.Qwen3.8-35B-A3B-Distill-APEX-GGUF-Q4_K_M

List all available models

lemonade list

Hermes Agent

How to use MrFuzzihead/Qwen3.8-35B-A3B-Distill-APEX-GGUF with Hermes Agent:

Start the llama.cpp server

# Install llama.cpp:
brew install llama.cpp
# Start a local OpenAI-compatible server:
llama serve -hf MrFuzzihead/Qwen3.8-35B-A3B-Distill-APEX-GGUF:Q4_K_M

Configure Hermes

# Install Hermes:
curl -fsSL https://hermes-agent.nousresearch.com/install.sh | bash
hermes setup
# Point Hermes at the local server:
hermes config set model.provider custom
hermes config set model.base_url http://127.0.0.1:8080/v1
hermes config set model.default MrFuzzihead/Qwen3.8-35B-A3B-Distill-APEX-GGUF:Q4_K_M

Run Hermes

hermes

Atomic Chat

OpenClaw

How to use MrFuzzihead/Qwen3.8-35B-A3B-Distill-APEX-GGUF with OpenClaw:

Start the llama.cpp server

# Install llama.cpp:
brew install llama.cpp
# Start a local OpenAI-compatible server:
llama serve -hf MrFuzzihead/Qwen3.8-35B-A3B-Distill-APEX-GGUF:Q4_K_M

Configure OpenClaw

# Install OpenClaw:
npm install -g openclaw@latest
# Register the local server and set it as the default model:
openclaw onboard --non-interactive --mode local \
  --auth-choice custom-api-key \
  --custom-base-url http://127.0.0.1:8080/v1 \
  --custom-model-id "MrFuzzihead/Qwen3.8-35B-A3B-Distill-APEX-GGUF:Q4_K_M" \
  --custom-provider-id llama-cpp \
  --custom-compatibility openai \
  --custom-text-input \
  --accept-risk \
  --skip-health

Run OpenClaw

openclaw agent --local --agent main --message "Hello from Hugging Face"

Qwen3.8-35B-A3B-Distill APEX GGUF

APEX (Adaptive Precision for EXpert models) quantizations of empero-ai/Qwen3.8-35B-A3B-Distill, covering every deployment scenario from maximum quality down to ~10 GB.

Quantized by: MrFuzzihead
Method: APEX — a MoE-aware mixed-precision quantization technique for llama.cpp, by the LocalAI team
More APEX quants: MrFuzzihead's APEX collection

APEX assigns quantization precision per tensor role (routed expert / shared expert / attention / SSM) and per layer position (edge layers high, middle layers compressed), exploiting the sparsity of Mixture-of-Experts models. On the APEX reference model (Qwen3.5-35B-A3B), APEX tiers match or beat Q8_0 quality at ~60% of the size — see the APEX technical report for details.

Note on the hardware-compatibility widget: Hugging Face detects quantization variants from standard quant names (Q4_K_M etc.) in the filename. APEX tiers are mixed-precision and use APEX names, so the widget cannot classify them (this also affects other APEX repos). Use the table below for sizes; mmproj and MTP files are never classified by the widget by design.

Files

File	Size	Best for
`Qwen3.8-35B-A3B-Distill-APEX-Quality.gguf`	21.2 GB	lowest-perplexity tier, 24 GB GPUs
`Qwen3.8-35B-A3B-Distill-APEX-I-Quality.gguf`	21.2 GB	best accuracy tier (recommended)
`Qwen3.8-35B-A3B-Distill-APEX-Balanced.gguf`	25.3	general purpose, serving
`Qwen3.8-35B-A3B-Distill-APEX-I-Balanced.gguf`	25.3	general purpose, lower KL divergence
`Qwen3.8-35B-A3B-Distill-APEX-Compact.gguf`	16.5	16–24 GB GPUs
`Qwen3.8-35B-A3B-Distill-APEX-I-Compact.gguf`	16.5	best accuracy at this size
`Qwen3.8-35B-A3B-Distill-APEX-Mini.gguf`	13.5	12–16 GB VRAM
`Qwen3.8-35B-A3B-Distill-APEX-I-Micro.gguf`	10.9	smallest footprint (IQ1_M experts)
`Qwen3.8-35B-A3B-Distill-APEX-I-Nano.gguf`	11.8	low tier with higher-fidelity experts
`mmproj-BF16.gguf`	0.88 GB	vision projector, full precision
`mmproj-Q8_0.gguf`	0.60 GB	vision projector, half the size
`mtp-Qwen3.8-35B-A3B-Distill-BF16.gguf`	3.74 GB	MTP draft head, full precision
`mtp-Qwen3.8-35B-A3B-Distill-Q8_0.gguf`	1.99 GB	MTP draft head, quantized (recommended)
`mtp-Qwen3.8-35B-A3B-Distill-Q4_K_M.gguf`	1.26 GB	MTP draft head, smallest / fastest load

Sizes are in GB (2³⁰ bytes), as displayed by the Hub. Full-GPU offload needs the file size plus the mmproj (if you use vision) plus the MTP head (if you use speculative decoding) plus KV cache and overhead.

I- files are quantized with an importance matrix built from diverse calibration data (chat, code, reasoning, tool-calling, agentic traces — no Wikipedia). In APEX evaluations, I-variants trade a negligible perplexity increase for better downstream accuracy and lower KL divergence. The Mini, Micro and Nano tiers use the diverse imatrix by design (per the APEX method), which is why they ship as I--named files (I-Micro, I-Nano) with no separate non-imatrix version.

The model

Qwen3.8-35B-A3B-Distill is Empero's distillation of the Qwen3.8 frontier teacher models (Qwen3.8 2.4T A95B and Qwen3.8 Flash Next) into the Qwen3.6-35B-A3B Mixture-of-Experts architecture — off-policy SFT on curated teacher chain-of-thought traces spanning math, code, general reasoning, instruction following, and tool use. The goal: bring frontier-scale reasoning behavior into a sparse 35B model that activates only 3B parameters per token.

~35B parameters, Mixture-of-Experts: 256 routed experts, 8 active per token, plus a shared expert.
40 layers, hybrid attention: ten cycles of three Gated DeltaNet (linear-attention) layers followed by one Gated Attention layer (qwen3_5_moe architecture, same family as Qwen3.6). Only 10 of 40 layers use full attention, so KV-cache overhead is minimal.
Multimodal base, text-only fine-tune: the vision tower from the Qwen3.6-35B-A3B base is retained in the checkpoint but was not touched or evaluated by this distillation; 262,144-token native context.
Reasoning: always-on distilled chain-of-thought — every response opens with a <think> block learned directly from Qwen3.8 teacher traces, rather than an adaptive effort toggle.
Tool calling: native Qwen3.6 function-calling spec, no wrapper or tool-specific fine-tune required.
Multi-Token Prediction: trained with an MTP head for self-speculative decoding, shipped here as separate MTP GGUFs.
License: Apache-2.0.

How APEX quantizes Qwen3.8-35B-A3B-Distill

Routed experts are ~94% of the weights here, but only 8 of 256 fire for any given token, so they tolerate lower precision than the parts every token passes through. Attention + SSM are only ~~2.7% of the weights but dense, and the shared expert (~~0.4%) is always active and heavy-tailed. APEX classifies each tensor by role and applies a layer-wise precision gradient — edge layers keep higher precision, middle layers compress harder:

Tier	Routed experts	Shared expert	Attention + SSM	Size
Quality / I-Quality	Q6_K edges (L0–4, L35–39) → Q5_K near-edge (L5–9, L30–34) → IQ4_XS middle (L10–29)	Q8_0	Q6_K	21.2 GB
Balanced / I-Balanced	Q6_K edges → Q5_K (L5–34)	Q8_0	Q6_K	not yet uploaded
Compact / I-Compact	Q4_K edges → Q3_K (L5–34)	Q6_K	Q4_K	not yet uploaded
Mini	Q3_K edges (L0–9, L30–39) → IQ2_S middle (L10–29)	Q5_K edges / Q4_K middle	Q4_K near edges, Q3_K elsewhere	not yet uploaded
I-Micro	Q3_K edges → IQ2_XS near-edge → IQ1_M middle	Q5_K edges / Q4_K middle	Q4_K near edges, Q3_K elsewhere	not yet uploaded
I-Nano	Q3_K edges → IQ2_S near-edge → IQ2_XXS middle	Q5_K edges / Q4_K middle	Q4_K near edges, Q3_K elsewhere	not yet uploaded

Embeddings and the LM head (~3% of weights) are kept at Q6_K (Q3_K on Mini) to preserve output-logit fidelity. Routers and norms stay F32. Because routing decisions use full-precision gates, quantization noise in inactive experts never affects the output.

For the tier configs and reproduction scripts, see the APEX repository.

Usage

Needs a recent llama.cpp build with qwen35moe (Qwen3.5 / Qwen3.6 A3B family) support. The low tiers additionally use IQ1_M / IQ2_XS / IQ2_S / IQ2_XXS expert formats, and the BF16 mmproj/MTP files need current BF16 support — when in doubt, use the latest release.

# text
llama-cli -m Qwen3.8-35B-A3B-Distill-APEX-I-Quality.gguf -p "Your prompt" -ngl 99

# OpenAI-compatible server (tool calls + reasoning via --jinja)
llama-server -m Qwen3.8-35B-A3B-Distill-APEX-I-Quality.gguf \
  --mmproj mmproj-BF16.gguf \
  --jinja --host 0.0.0.0 --port 8080 -ngl 99

# vision, standalone
llama-mtmd-cli -m Qwen3.8-35B-A3B-Distill-APEX-I-Quality.gguf \
  --mmproj mmproj-Q8_0.gguf \
  --image image.png -p "What is in this image?" -ngl 99

# self-speculative decoding with the MTP draft head
llama-server -m Qwen3.8-35B-A3B-Distill-APEX-I-Quality.gguf \
  --model-draft mtp-Qwen3.8-35B-A3B-Distill-Q8_0.gguf \
  --jinja --host 0.0.0.0 --port 8080 -ngl 99

The draft-model flag name/support for MTP-based self-speculative decoding varies by llama.cpp version (--model-draft / -md) — check your build's --help if it's rejected.

The GGUF embeds the official chat template, including the always-on thinking format and the native Qwen3.6 function-calling spec. Every response opens with a <think>...</think> block; parse and strip it for end users.

Recommended sampling parameters (from the empero-ai model card):

temperature: 0.6
top_p: 0.95
top_k: 20
Allow generous output length — 16,384 tokens recommended, since the model's chain-of-thought precedes the final answer.

Choosing an mmproj

mmproj-BF16.gguf (0.88 GB) is the full-precision projector; mmproj-Q8_0.gguf (0.60 GB) is the quantized one. Either pairs with any APEX tier above. If you are not using images or video, skip the mmproj entirely.

Choosing an MTP head

Three MTP draft heads are provided for self-speculative decoding: mtp-Qwen3.8-35B-A3B-Distill-BF16.gguf (3.74 GB, full precision), mtp-Qwen3.8-35B-A3B-Distill-Q8_0.gguf (1.99 GB, recommended balance), and mtp-Qwen3.8-35B-A3B-Distill-Q4_K_M.gguf (1.26 GB, smallest / fastest to load). Any pairs with any APEX tier above. If you are not using speculative decoding, skip the MTP file entirely.

Notes

MTP head: unlike some quants in this family, the Qwen3.6-35B-A3B base (and this distillation) was trained with Multi-Token Prediction, so a working MTP head is shipped here as separate GGUFs rather than baked into the main file.
Benchmarks: no perplexity or accuracy benchmarks were run on these specific files. The published APEX numbers are from the reference Qwen3.5-35B-A3B evaluation in the APEX repository; see the base model card for Qwen3.8-35B-A3B-Distill's own benchmark results against the Qwen3.6-35B-A3B base.
License: Apache-2.0, same as the base model.

Credits

Base model: empero-ai/Qwen3.8-35B-A3B-Distill by Empero (Apache-2.0), distilled from Qwen3.8 teachers onto Qwen/Qwen3.6-35B-A3B (Alibaba Qwen team)
Quantization method: APEX by the LocalAI team — built on llama.cpp
Quants: MrFuzzihead