Instructions to use jc-builds/Qwen3.5-9B-Q4_K_M-GGUF with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Notebooks
Google Colab
Kaggle
Local Apps Settings

How to use jc-builds/Qwen3.5-9B-Q4_K_M-GGUF with llama.cpp:

Install (macOS, Linux)

curl -LsSf https://llama.app/install.sh | sh
# Start a local OpenAI-compatible server with a web UI:
llama serve -hf jc-builds/Qwen3.5-9B-Q4_K_M-GGUF:Q4_K_M
# Run inference directly in the terminal:
llama cli -hf jc-builds/Qwen3.5-9B-Q4_K_M-GGUF:Q4_K_M

Install from WinGet (Windows)

winget install llama.cpp
# Start a local OpenAI-compatible server with a web UI:
llama serve -hf jc-builds/Qwen3.5-9B-Q4_K_M-GGUF:Q4_K_M
# Run inference directly in the terminal:
llama cli -hf jc-builds/Qwen3.5-9B-Q4_K_M-GGUF:Q4_K_M

Use pre-built binary

# Download pre-built binary from:
# https://github.com/ggerganov/llama.cpp/releases
# Start a local OpenAI-compatible server with a web UI:
./llama-server -hf jc-builds/Qwen3.5-9B-Q4_K_M-GGUF:Q4_K_M
# Run inference directly in the terminal:
./llama-cli -hf jc-builds/Qwen3.5-9B-Q4_K_M-GGUF:Q4_K_M

Build from source code

git clone https://github.com/ggerganov/llama.cpp.git
cd llama.cpp
cmake -B build
cmake --build build -j --target llama-server llama-cli
# Start a local OpenAI-compatible server with a web UI:
./build/bin/llama-server -hf jc-builds/Qwen3.5-9B-Q4_K_M-GGUF:Q4_K_M
# Run inference directly in the terminal:
./build/bin/llama-cli -hf jc-builds/Qwen3.5-9B-Q4_K_M-GGUF:Q4_K_M

Use Docker

docker model run hf.co/jc-builds/Qwen3.5-9B-Q4_K_M-GGUF:Q4_K_M

LM Studio
Jan

vLLM

How to use jc-builds/Qwen3.5-9B-Q4_K_M-GGUF with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "jc-builds/Qwen3.5-9B-Q4_K_M-GGUF"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "jc-builds/Qwen3.5-9B-Q4_K_M-GGUF",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker

docker model run hf.co/jc-builds/Qwen3.5-9B-Q4_K_M-GGUF:Q4_K_M

Ollama
How to use jc-builds/Qwen3.5-9B-Q4_K_M-GGUF with Ollama:
```
ollama run hf.co/jc-builds/Qwen3.5-9B-Q4_K_M-GGUF:Q4_K_M
```
Unsloth Desktop

How to use jc-builds/Qwen3.5-9B-Q4_K_M-GGUF with Pi:

Start the llama.cpp server

# Install llama.cpp:
brew install llama.cpp
# Start a local OpenAI-compatible server:
llama serve -hf jc-builds/Qwen3.5-9B-Q4_K_M-GGUF:Q4_K_M

Configure the model in Pi

# Install Pi:
npm install -g @earendil-works/pi-coding-agent
# Add to ~/.pi/agent/models.json:
{
  "providers": {
    "llama-cpp": {
      "baseUrl": "http://localhost:8080/v1",
      "api": "openai-completions",
      "apiKey": "none",
      "models": [
        {
          "id": "jc-builds/Qwen3.5-9B-Q4_K_M-GGUF:Q4_K_M"
        }
      ]
    }
  }
}

Run Pi

# Start Pi in your project directory:
pi

Docker Model Runner
How to use jc-builds/Qwen3.5-9B-Q4_K_M-GGUF with Docker Model Runner:
```
docker model run hf.co/jc-builds/Qwen3.5-9B-Q4_K_M-GGUF:Q4_K_M
```

Lemonade

How to use jc-builds/Qwen3.5-9B-Q4_K_M-GGUF with Lemonade:

Pull the model

# Download Lemonade from https://lemonade-server.ai/
lemonade pull jc-builds/Qwen3.5-9B-Q4_K_M-GGUF:Q4_K_M

Run and chat with the model

lemonade run user.Qwen3.5-9B-Q4_K_M-GGUF-Q4_K_M

List all available models

lemonade list

Hermes Agent

How to use jc-builds/Qwen3.5-9B-Q4_K_M-GGUF with Hermes Agent:

Start the llama.cpp server

# Install llama.cpp:
brew install llama.cpp
# Start a local OpenAI-compatible server:
llama serve -hf jc-builds/Qwen3.5-9B-Q4_K_M-GGUF:Q4_K_M

Configure Hermes

# Install Hermes:
curl -fsSL https://hermes-agent.nousresearch.com/install.sh | bash
hermes setup
# Point Hermes at the local server:
hermes config set model.provider custom
hermes config set model.base_url http://127.0.0.1:8080/v1
hermes config set model.default jc-builds/Qwen3.5-9B-Q4_K_M-GGUF:Q4_K_M

Run Hermes

hermes

Atomic Chat

OpenClaw

How to use jc-builds/Qwen3.5-9B-Q4_K_M-GGUF with OpenClaw:

Start the llama.cpp server

# Install llama.cpp:
brew install llama.cpp
# Start a local OpenAI-compatible server:
llama serve -hf jc-builds/Qwen3.5-9B-Q4_K_M-GGUF:Q4_K_M

Configure OpenClaw

# Install OpenClaw:
npm install -g openclaw@latest
# Register the local server and set it as the default model:
openclaw onboard --non-interactive --mode local \
  --auth-choice custom-api-key \
  --custom-base-url http://127.0.0.1:8080/v1 \
  --custom-model-id "jc-builds/Qwen3.5-9B-Q4_K_M-GGUF:Q4_K_M" \
  --custom-provider-id llama-cpp \
  --custom-compatibility openai \
  --custom-text-input \
  --accept-risk \
  --skip-health

Run OpenClaw

openclaw agent --local --agent main --message "Hello from Hugging Face"

jc-builds commited on Mar 3

Commit

22f69b9

verified ·

1 Parent(s): aa6527a

Upload README.md with huggingface_hub

Browse files

Files changed (1) hide show

README.md +199 -0

README.md ADDED Viewed

	@@ -0,0 +1,199 @@

+---
+license: apache-2.0
+language:
+  - en
+  - zh
+  - ja
+  - ko
+  - fr
+  - de
+  - es
+  - pt
+  - ru
+  - ar
+  - it
+  - tr
+  - nl
+  - hi
+tags:
+  - qwen
+  - qwen3.5
+  - gguf
+  - on-device
+  - ios
+  - mobile
+  - thinking
+  - hybrid-attention
+  - deltanet
+model_name: Qwen3.5-9B Q4_K_M GGUF
+base_model: Qwen/Qwen3.5-9B
+quantized_by: jc-builds
+pipeline_tag: text-generation
+---
+# Qwen3.5-9B Q4_K_M GGUF
+> **4-bit quantized GGUF** of [Qwen/Qwen3.5-9B](https://huggingface.co/Qwen/Qwen3.5-9B) optimized for **on-device iOS inference** via [llama.cpp](https://github.com/ggml-org/llama.cpp). The most capable model you can run on an iPhone.
+| Property | Value |
+|---|---|
+| **Parameters** | 9 billion |
+| **Quantization** | Q4_K_M (4-bit, medium quality) |
+| **File Size** | 5.3 GB |
+| **Context Window** | 262,144 tokens (native) |
+| **Architecture** | Hybrid Gated DeltaNet + Attention |
+| **License** | Apache 2.0 |
+| **Languages** | 201 languages/dialects |
+## Key Features
+- **Best-in-Class On-Device AI**: Matches or beats models 9-13x its size
+- **Thinking Mode**: `<think>...</think>` chain-of-thought reasoning
+- **Hybrid Architecture**: Gated DeltaNet + Attention for efficient, high-quality inference
+- **Natively Multimodal**: Trained with early vision fusion
+- **Massive Context**: 262K native, extendable to 1M+ with YaRN
+## Benchmarks
+### Qwen3.5-9B vs Models 9-13x Larger
+![9B vs Large Models](9b_vs_large.png)
+The 9B model **beats Qwen3-80B** on GPQA Diamond (81.7 vs 77.2), IFEval (91.5 vs 88.9), and HMMT math (83.2 vs 73.7). It also **outperforms GPT-OSS-120B** on MMLU-Pro (82.5 vs 80.8) and GPQA Diamond (81.7 vs 80.1).
+### MMLU-Pro: Size Class Comparison
+| Model | Params | MMLU-Pro |
+|---|---|---|
+| **Qwen3.5-9B** | **9B** | **82.5** |
+| Qwen3-30B | 30B | 80.9 |
+| GPT-OSS-120B | 120B | 80.8 |
+| Qwen3.5-4B | 4B | 79.1 |
+| Gemma2-9B | 9B | ~55* |
+| Phi-4-mini | 3.8B | 52.8 |
+### On-Device Inference Speed
+![Speed Comparison](speed.png)
+### Vision Capabilities
+![Vision Benchmarks](vision.png)
+The 9B model outperforms the dedicated Qwen3-VL-30B (3x its size) on MMMU, MMMU-Pro, MathVision, OmniDocBench, and VideoMME.
+### Full Benchmark Table
+| Benchmark | Qwen3.5-9B | Qwen3-30B | Qwen3-80B | GPT-OSS-120B |
+|---|---|---|---|---|
+| **MMLU-Pro** | 82.5 | 80.9 | 82.7 | 80.8 |
+| **MMLU-Redux** | 91.1 | 91.4 | 92.5 | 91.0 |
+| **GPQA Diamond** | 81.7 | 73.4 | 77.2 | 80.1 |
+| **IFEval** | 91.5 | 88.9 | 88.9 | - |
+| **HMMT Feb 25** | 83.2 | 63.1 | 73.7 | 76.7 |
+| **HMMT Nov 25** | 82.9 | 73.8 | 81.2 | 81.8 |
+| **LiveCodeBench v6** | 65.6 | 66.0 | 68.7 | 82.7 |
+| **BFCL-V4 (Tool Use)** | 66.1 | 42.4 | - | - |
+| **C-Eval** | 88.2 | 87.4 | 89.7 | 76.2 |
+| **SuperGPQA** | 58.2 | 56.8 | 60.8 | 54.6 |
+### Vision Benchmarks
+| Benchmark | Qwen3.5-9B | Qwen3-VL-30B | GPT-5-Nano |
+|---|---|---|---|
+| **MMMU** | 78.4 | 76.0 | 75.8 |
+| **MMMU-Pro** | 70.1 | 63.0 | 57.2 |
+| **MathVision** | 78.9 | 65.7 | 62.2 |
+| **OmniDocBench** | 87.7 | 86.8 | 55.9 |
+| **VideoMME** | 84.5 | 79.9 | 71.7 |
+| **OSWorld** | 41.8 | 30.6 | - |
+## Device Compatibility
+| Device | RAM | Compatible | Speed |
+|---|---|---|---|
+| iPhone 16 Pro Max | 8 GB | Yes | ~22-28 tok/s |
+| iPhone 16 Pro | 8 GB | Yes | ~20-25 tok/s |
+| iPhone 16 / 15 Pro | 8 GB | Possible (tight) | ~15-20 tok/s |
+| iPhone 15 and older | 6 GB | Not recommended | - |
+| iPad Pro (M-series) | 8-16 GB | Yes | ~25-40 tok/s |
+| Mac (Apple Silicon) | 16+ GB | Yes | ~30-50 tok/s |
+> **Note**: The 9B model at 5.3 GB requires devices with 8 GB+ RAM. For older devices, use the [Qwen3.5-4B](https://huggingface.co/jc-builds/Qwen3.5-4B-Q4_K_M-GGUF) instead.
+## Usage
+### With llama.cpp
+```bash
+# Download
+huggingface-cli download jc-builds/Qwen3.5-9B-Q4_K_M-GGUF Qwen3.5-9B-Q4_K_M.gguf
+# Run (with thinking mode)
+./llama-cli -m Qwen3.5-9B-Q4_K_M.gguf -p "Prove that there are infinitely many primes." -ngl 99
+```
+### With Ollama
+```bash
+ollama run qwen3.5:9b
+```
+### In HaploAI (iOS)
+This model is available directly in the [HaploAI](https://apps.apple.com/app/haploai/id6503772307) iOS app (v1.18+). Download it from the model selection page.
+## Prompt Format
+Uses ChatML format:
+```
+<|im_start|>system
+You are a helpful assistant.<|im_end|>
+<|im_start|>user
+Hello!<|im_end|>
+<|im_start|>assistant
+```
+### Thinking Mode
+```
+<|im_start|>assistant
+<think>
+Let me reason through this carefully...
+First, assume there are finitely many primes p1, p2, ..., pn.
+Consider N = p1 * p2 * ... * pn + 1.
+N is not divisible by any pi, so either N is prime or has a prime factor not in our list.
+This contradicts our assumption.
+</think>
+There are infinitely many primes. Here is Euclid's classic proof...
+```
+## Architecture Details
+Qwen3.5 introduces a **hybrid Gated DeltaNet + Gated Attention** architecture:
+- **3:1 ratio**: 3 layers of Gated DeltaNet (linear attention) per 1 layer of full softmax attention
+- **32 total layers**: 8 blocks x (3 DeltaNet + 1 Attention)
+- **Hidden dimension**: 4,096
+- **Near-constant memory**: DeltaNet layers maintain bounded memory
+- **GQA**: 16 query heads, 4 KV heads for attention layers
+- **FFN intermediate**: 12,288
+- **RoPE**: `theta=10,000,000` with YaRN extension
+## Why Qwen3.5-9B?
+This model represents a paradigm shift in on-device AI:
+1. **9B params that beat 80B**: On GPQA Diamond, IFEval, and math benchmarks
+2. **Hybrid attention is the future**: DeltaNet layers provide near-constant memory, enabling huge context on mobile
+3. **Natively multimodal**: No separate vision encoder needed for basic image understanding
+4. **201 languages**: Broadest language support in its class
+5. **Apache 2.0**: Fully open, commercially usable
+## Credits
+- **Original model**: [Qwen/Qwen3.5-9B](https://huggingface.co/Qwen/Qwen3.5-9B) by Alibaba Cloud
+- **GGUF conversion**: [Unsloth](https://huggingface.co/unsloth/Qwen3.5-9B-GGUF)
+- **Quantization**: Q4_K_M via llama.cpp
+- **Optimized for iOS**: [jc-builds](https://huggingface.co/jc-builds)