Instructions to use avlp12/Qwen3.5-35B-A3B-Alis-Ultra-Slim-GGUF with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use avlp12/Qwen3.5-35B-A3B-Alis-Ultra-Slim-GGUF with llama-cpp-python:

# !pip install llama-cpp-python

from llama_cpp import Llama

llm = Llama.from_pretrained(
	repo_id="avlp12/Qwen3.5-35B-A3B-Alis-Ultra-Slim-GGUF",
	filename="Qwen3.5-35B-A3B-Alis-Ultra-Slim.gguf",
)

llm.create_chat_completion(
	messages = [
		{
			"role": "user",
			"content": "What is the capital of France?"
		}
	]
)

Notebooks
Google Colab
Kaggle
Local Apps Settings

llama.cpp

How to use avlp12/Qwen3.5-35B-A3B-Alis-Ultra-Slim-GGUF with llama.cpp:

Install (macOS, Linux)

curl -LsSf https://llama.app/install.sh | sh
# Start a local OpenAI-compatible server with a web UI:
llama serve -hf avlp12/Qwen3.5-35B-A3B-Alis-Ultra-Slim-GGUF
# Run inference directly in the terminal:
llama cli -hf avlp12/Qwen3.5-35B-A3B-Alis-Ultra-Slim-GGUF

Install from WinGet (Windows)

winget install llama.cpp
# Start a local OpenAI-compatible server with a web UI:
llama serve -hf avlp12/Qwen3.5-35B-A3B-Alis-Ultra-Slim-GGUF
# Run inference directly in the terminal:
llama cli -hf avlp12/Qwen3.5-35B-A3B-Alis-Ultra-Slim-GGUF

Use pre-built binary

# Download pre-built binary from:
# https://github.com/ggerganov/llama.cpp/releases
# Start a local OpenAI-compatible server with a web UI:
./llama-server -hf avlp12/Qwen3.5-35B-A3B-Alis-Ultra-Slim-GGUF
# Run inference directly in the terminal:
./llama-cli -hf avlp12/Qwen3.5-35B-A3B-Alis-Ultra-Slim-GGUF

Build from source code

git clone https://github.com/ggerganov/llama.cpp.git
cd llama.cpp
cmake -B build
cmake --build build -j --target llama-server llama-cli
# Start a local OpenAI-compatible server with a web UI:
./build/bin/llama-server -hf avlp12/Qwen3.5-35B-A3B-Alis-Ultra-Slim-GGUF
# Run inference directly in the terminal:
./build/bin/llama-cli -hf avlp12/Qwen3.5-35B-A3B-Alis-Ultra-Slim-GGUF

Use Docker

docker model run hf.co/avlp12/Qwen3.5-35B-A3B-Alis-Ultra-Slim-GGUF

LM Studio
Jan

vLLM

How to use avlp12/Qwen3.5-35B-A3B-Alis-Ultra-Slim-GGUF with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "avlp12/Qwen3.5-35B-A3B-Alis-Ultra-Slim-GGUF"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "avlp12/Qwen3.5-35B-A3B-Alis-Ultra-Slim-GGUF",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker

docker model run hf.co/avlp12/Qwen3.5-35B-A3B-Alis-Ultra-Slim-GGUF

Ollama
How to use avlp12/Qwen3.5-35B-A3B-Alis-Ultra-Slim-GGUF with Ollama:
```
ollama run hf.co/avlp12/Qwen3.5-35B-A3B-Alis-Ultra-Slim-GGUF
```

Unsloth Studio

How to use avlp12/Qwen3.5-35B-A3B-Alis-Ultra-Slim-GGUF with Unsloth Studio:

Install Unsloth Studio (macOS, Linux, WSL)

curl -fsSL https://unsloth.ai/install.sh | sh
# Run unsloth studio
unsloth studio -H 0.0.0.0 -p 8888
# Then open http://localhost:8888 in your browser
# Search for avlp12/Qwen3.5-35B-A3B-Alis-Ultra-Slim-GGUF to start chatting

Install Unsloth Studio (Windows)

irm https://unsloth.ai/install.ps1 | iex
# Run unsloth studio
unsloth studio -H 0.0.0.0 -p 8888
# Then open http://localhost:8888 in your browser
# Search for avlp12/Qwen3.5-35B-A3B-Alis-Ultra-Slim-GGUF to start chatting

Using HuggingFace Spaces for Unsloth

# No setup required
# Open https://huggingface.co/spaces/unsloth/studio in your browser
# Search for avlp12/Qwen3.5-35B-A3B-Alis-Ultra-Slim-GGUF to start chatting

How to use avlp12/Qwen3.5-35B-A3B-Alis-Ultra-Slim-GGUF with Pi:

Start the llama.cpp server

# Install llama.cpp:
brew install llama.cpp
# Start a local OpenAI-compatible server:
llama serve -hf avlp12/Qwen3.5-35B-A3B-Alis-Ultra-Slim-GGUF

Configure the model in Pi

# Install Pi:
npm install -g @mariozechner/pi-coding-agent
# Add to ~/.pi/agent/models.json:
{
  "providers": {
    "llama-cpp": {
      "baseUrl": "http://localhost:8080/v1",
      "api": "openai-completions",
      "apiKey": "none",
      "models": [
        {
          "id": "avlp12/Qwen3.5-35B-A3B-Alis-Ultra-Slim-GGUF"
        }
      ]
    }
  }
}

Run Pi

# Start Pi in your project directory:
pi

Hermes Agent new

How to use avlp12/Qwen3.5-35B-A3B-Alis-Ultra-Slim-GGUF with Hermes Agent:

Start the llama.cpp server

# Install llama.cpp:
brew install llama.cpp
# Start a local OpenAI-compatible server:
llama serve -hf avlp12/Qwen3.5-35B-A3B-Alis-Ultra-Slim-GGUF

Configure Hermes

# Install Hermes:
curl -fsSL https://hermes-agent.nousresearch.com/install.sh | bash
hermes setup
# Point Hermes at the local server:
hermes config set model.provider custom
hermes config set model.base_url http://127.0.0.1:8080/v1
hermes config set model.default avlp12/Qwen3.5-35B-A3B-Alis-Ultra-Slim-GGUF

Run Hermes

hermes

Atomic Chat new

OpenClaw new

How to use avlp12/Qwen3.5-35B-A3B-Alis-Ultra-Slim-GGUF with OpenClaw:

Start the llama.cpp server

# Install llama.cpp:
brew install llama.cpp
# Start a local OpenAI-compatible server:
llama serve -hf avlp12/Qwen3.5-35B-A3B-Alis-Ultra-Slim-GGUF

Configure OpenClaw

# Install OpenClaw:
npm install -g openclaw@latest
# Register the local server and set it as the default model:
openclaw onboard --non-interactive --mode local \
  --auth-choice custom-api-key \
  --custom-base-url http://127.0.0.1:8080/v1 \
  --custom-model-id "avlp12/Qwen3.5-35B-A3B-Alis-Ultra-Slim-GGUF" \
  --custom-provider-id llama-cpp \
  --custom-compatibility openai \
  --custom-text-input \
  --accept-risk \
  --skip-health

Run OpenClaw

openclaw agent --local --agent main --message "Hello from Hugging Face"

Docker Model Runner
How to use avlp12/Qwen3.5-35B-A3B-Alis-Ultra-Slim-GGUF with Docker Model Runner:
```
docker model run hf.co/avlp12/Qwen3.5-35B-A3B-Alis-Ultra-Slim-GGUF
```

Lemonade

How to use avlp12/Qwen3.5-35B-A3B-Alis-Ultra-Slim-GGUF with Lemonade:

Pull the model

# Download Lemonade from https://lemonade-server.ai/
lemonade pull avlp12/Qwen3.5-35B-A3B-Alis-Ultra-Slim-GGUF

Run and chat with the model

lemonade run user.Qwen3.5-35B-A3B-Alis-Ultra-Slim-GGUF-{{QUANT_TAG}}

List all available models

lemonade list

Alis Ultra Slim — Qwen3.5-35B-A3B GGUF (12.87 GiB)

Part of the Qwen3.5-35B-A3B · Alis builds collection.

MoE-aware mixed-precision quantization. Best quality under 13 GiB.

Pure quantization — zero fine-tuning. This is a direct quantization of the official Qwen3.5-35B-A3B weights. No merge, no fine-tune, no LoRA, no RLHF modifications. The original model's capabilities are preserved exactly as Qwen released them — only the numerical precision is optimized.

PPL 7.009 — beats APEX Mini on both perplexity (7.009 vs 7.048) and HellaSwag (78.00% vs 76.75%). The best stock-llama.cpp Qwen3.5-35B GGUF under 13 GiB.

Also available: Alis Ultra (14.26 GiB, PPL 6.96, HellaSwag 78.50%) — for maximum quality when memory allows.

Quick Start

# Download
hf download avlp12/Qwen3.5-35B-A3B-Alis-Ultra-Slim-GGUF \
    Qwen3.5-35B-A3B-Alis-Ultra-Slim.gguf --local-dir ./model

# Chat
llama-cli -m ./model/Qwen3.5-35B-A3B-Alis-Ultra-Slim.gguf \
    --conversation -ngl 99

# Server
llama-server -m ./model/Qwen3.5-35B-A3B-Alis-Ultra-Slim.gguf \
    --host 0.0.0.0 --port 8080 -ngl 99 \
    --temp 0.6 --top-p 0.95 --top-k 20 \
    -fa on --jinja

Compatible with stock llama.cpp, llama-server, and any GGUF-compatible backend. No custom forks required.

Alis Ultra Slim vs Ultra — Which One?

	Ultra Slim (this repo)	Ultra (link)
Size	12.87 GiB (3.19 BPW)	14.26 GiB (3.53 BPW)
PPL	7.009 ± 0.045	6.955 ± 0.045
HellaSwag	78.00%	78.50%
Base type	Q2_K	Q3_K_M
Best for	16 GiB constrained (M4, RTX 4060 Ti 16GB)	24 GiB+ devices (M4 Pro/Max, RTX 4090)
Key advantage	Best quality under 13 GiB	F16-matching accuracy

The core difference: Ultra Slim uses Q2_K as the base quantization type, keeping middle-layer gate/up experts at the K-quant floor (~2.6 BPW). Ultra uses Q3_K_M as the base, which allows llama.cpp's imatrix to auto-promote sensitive tensors — providing an extra quality buffer at the cost of 1.4 GiB.

Both variants share identical treatment of critical tensors: shared experts at Q8_0, attention at Q4_K/Q5_K, SSM at Q6_K, and ffn_down_exps at Q3_K.

Why Ultra Slim over APEX Mini?

	Alis Ultra Slim	APEX Mini
Size	12.87 GiB	12.33 GiB
PPL	7.009	7.048
HellaSwag	78.00%	76.75%
Compatibility	stock llama.cpp	stock llama.cpp

Ultra Slim is 0.54 GiB larger but wins on both PPL (-0.04) and HellaSwag (+1.25%p). The accuracy gap is significant — 78.00% vs 76.75% is well outside the confidence interval overlap.

Benchmarks

All measurements on M3 Ultra Mac Studio (512 GB, 80 GPU cores). PPL on wikitext-2-raw test set, context 512, 580 chunks. HellaSwag 0-shot, 400 tasks. All values directly measured by us on the same hardware.

Quality

Model	Size (GiB)	BPW	PPL	HellaSwag
F16 (baseline)	64.60	16.00	6.537	78.50%
Unsloth Q3_K_M (Dynamic)	15.22	3.77	6.779	78.50%
Alis Ultra	14.26	3.53	6.955	78.50%
Alis Ultra Slim	12.87	3.19	7.009	78.00%
APEX Mini	12.33	3.06	7.048	76.75%
Unsloth IQ2_XXS	9.91	2.46	7.519	77.00%

Key Takeaways

Alis Ultra Slim beats APEX Mini on both PPL (7.009 vs 7.048) and HellaSwag (78.00% vs 76.75%)
HellaSwag 78.00% is only 0.50%p below F16 baseline (78.50%) — remarkable for 5× compression
Even Unsloth IQ2_XXS (77.00%) outscores APEX Mini (76.75%) on HellaSwag despite being 2.4 GiB smaller

Speed (M3 Ultra, llama-bench)

Model	pp512 (tok/s)	tg128 (tok/s)
Alis Ultra (14.26 GiB)	2,239 ± 8.58	85.62 ± 1.22
Alis Ultra Slim (12.87 GiB)	2,204 ± 7.39	81.95 ± 0.76

Quantization Strategy

Tensor Classification & Layer Gradient

Based on APEX's MoE tensor role analysis and Unsloth's 121-configuration KL divergence study:

Tensor	Edge (L0-4, L35-39)	Middle (L5-34)
ffn_down_exps (most sensitive)	q3_K	q3_K
ffn_gate/up_exps	q3_K	q2_K (base)
Shared experts (every-token path)	Q8_0	Q8_0
Attention Q/K	q4_K	q4_K
Attention V	q5_K	q5_K
Attention O	q4_K	q4_K
SSM out	Q6_K	Q6_K
Embedding	Q4_K	—
Output	Q5_K	—

Design Principles

K-quant over IQ for MoE experts. Routed expert weights have near-Gaussian distributions (kurtosis 3.41). K-quant block-scaling outperforms IQ codebooks designed for heavy-tailed distributions.
Protect ffn_down, compress gate/up. ffn_down_exps is consistently the most sensitive expert tensor. We keep it at Q3_K while allowing gate/up to drop to Q2_K in middle layers.
Edge-layer gradient. Layers 0-4 and 35-39 (nearest to embedding/output) get higher precision.
Shared experts are sacred. With kurtosis 13.10 (4× routed experts) and 100% activation rate, shared experts stay at Q8_0.

Imatrix

14,062 chunks from 76,447 calibration samples across 6 domains:

General instruction (Alpaca, 52K) — broad language coverage
Math reasoning (GSM8K, 7.4K) — numerical precision
Wikipedia (wikitext-2-raw train, ~15K) — factual knowledge
Korean language (1,000) — multilingual preservation
Code (500) — syntax structure
Tool-call JSON (200) — structured output accuracy

Vision Support

This GGUF contains text weights only (733 tensors). Qwen3.5-35B-A3B is a multimodal model, but the vision encoder must be loaded separately as an mmproj file.

Setup

# 1. Download mmproj (choose one — F16 recommended for quality/size balance)
hf download unsloth/Qwen3.5-35B-A3B-GGUF mmproj-F16.gguf --local-dir ./model
# Alternatives: mmproj-BF16.gguf (903 MB) or mmproj-F32.gguf (1.79 GB)

# 2. Run with vision — Server mode
llama-server \
    -m ./model/Qwen3.5-35B-A3B-Alis-Ultra-Slim.gguf \
    --mmproj ./model/mmproj-F16.gguf \
    --host 0.0.0.0 --port 8080 -ngl 99 \
    -fa on --jinja

# 3. Run with vision — Interactive CLI
llama-mtmd-cli \
    -m ./model/Qwen3.5-35B-A3B-Alis-Ultra-Slim.gguf \
    --mmproj ./model/mmproj-F16.gguf \
    -ngl 99 --jinja

Memory Impact

The mmproj-F16 adds ~899 MiB to VRAM usage. Total for Ultra Slim + mmproj-F16: ~13.8 GiB — still fits in 16 GiB devices with room for KV cache.

Compatibility Notes

llama.cpp: Full support via --mmproj flag (llama-server, llama-mtmd-cli)
Ollama: Not currently supported — Qwen3.5 GGUF requires separate mmproj files which Ollama does not handle
LM Studio: Check for Qwen3.5 VLM support in your version
mmproj files are interchangeable across all Qwen3.5-35B-A3B quantizations (Alis, Unsloth, APEX, etc.)

Thinking Mode

Qwen3.5 supports thinking/non-thinking. To disable:

--chat-template-kwargs '{"enable_thinking":false}'

Technical Details

Base Model: Qwen/Qwen3.5-35B-A3B (35B total, 3B active per token, 256 experts, 8 active + 1 shared)
Architecture: Qwen3.5-MoE with Gated Delta Networks
Quantization Tool: llama.cpp build 8770, --tensor-type per-layer overrides
Imatrix: 14,062 chunks, PPL 4.5318 on calibration data
Source: F16 GGUF → converted from HuggingFace BF16