Instructions to use larryliu0820/Qwen3-0.6B-Q8_0-ExecuTorch-GGML with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use larryliu0820/Qwen3-0.6B-Q8_0-ExecuTorch-GGML with llama-cpp-python:

# !pip install llama-cpp-python

from llama_cpp import Llama

llm = Llama.from_pretrained(
	repo_id="larryliu0820/Qwen3-0.6B-Q8_0-ExecuTorch-GGML",
	filename="Qwen3-0.6B-Q8_0.gguf",
)

llm.create_chat_completion(
	messages = [
		{
			"role": "user",
			"content": "What is the capital of France?"
		}
	]
)

Notebooks
Google Colab
Kaggle
Local Apps Settings

llama.cpp

How to use larryliu0820/Qwen3-0.6B-Q8_0-ExecuTorch-GGML with llama.cpp:

Install (macOS, Linux)

curl -LsSf https://llama.app/install.sh | sh
# Start a local OpenAI-compatible server with a web UI:
llama serve -hf larryliu0820/Qwen3-0.6B-Q8_0-ExecuTorch-GGML:Q8_0
# Run inference directly in the terminal:
llama cli -hf larryliu0820/Qwen3-0.6B-Q8_0-ExecuTorch-GGML:Q8_0

Install from WinGet (Windows)

winget install llama.cpp
# Start a local OpenAI-compatible server with a web UI:
llama serve -hf larryliu0820/Qwen3-0.6B-Q8_0-ExecuTorch-GGML:Q8_0
# Run inference directly in the terminal:
llama cli -hf larryliu0820/Qwen3-0.6B-Q8_0-ExecuTorch-GGML:Q8_0

Use pre-built binary

# Download pre-built binary from:
# https://github.com/ggerganov/llama.cpp/releases
# Start a local OpenAI-compatible server with a web UI:
./llama-server -hf larryliu0820/Qwen3-0.6B-Q8_0-ExecuTorch-GGML:Q8_0
# Run inference directly in the terminal:
./llama-cli -hf larryliu0820/Qwen3-0.6B-Q8_0-ExecuTorch-GGML:Q8_0

Build from source code

git clone https://github.com/ggerganov/llama.cpp.git
cd llama.cpp
cmake -B build
cmake --build build -j --target llama-server llama-cli
# Start a local OpenAI-compatible server with a web UI:
./build/bin/llama-server -hf larryliu0820/Qwen3-0.6B-Q8_0-ExecuTorch-GGML:Q8_0
# Run inference directly in the terminal:
./build/bin/llama-cli -hf larryliu0820/Qwen3-0.6B-Q8_0-ExecuTorch-GGML:Q8_0

Use Docker

docker model run hf.co/larryliu0820/Qwen3-0.6B-Q8_0-ExecuTorch-GGML:Q8_0

LM Studio
Jan

vLLM

How to use larryliu0820/Qwen3-0.6B-Q8_0-ExecuTorch-GGML with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "larryliu0820/Qwen3-0.6B-Q8_0-ExecuTorch-GGML"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "larryliu0820/Qwen3-0.6B-Q8_0-ExecuTorch-GGML",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker

docker model run hf.co/larryliu0820/Qwen3-0.6B-Q8_0-ExecuTorch-GGML:Q8_0

Ollama
How to use larryliu0820/Qwen3-0.6B-Q8_0-ExecuTorch-GGML with Ollama:
```
ollama run hf.co/larryliu0820/Qwen3-0.6B-Q8_0-ExecuTorch-GGML:Q8_0
```

Unsloth Studio

How to use larryliu0820/Qwen3-0.6B-Q8_0-ExecuTorch-GGML with Unsloth Studio:

Install Unsloth Studio (macOS, Linux, WSL)

curl -fsSL https://unsloth.ai/install.sh | sh
# Run unsloth studio
unsloth studio -H 0.0.0.0 -p 8888
# Then open http://localhost:8888 in your browser
# Search for larryliu0820/Qwen3-0.6B-Q8_0-ExecuTorch-GGML to start chatting

Install Unsloth Studio (Windows)

irm https://unsloth.ai/install.ps1 | iex
# Run unsloth studio
unsloth studio -H 0.0.0.0 -p 8888
# Then open http://localhost:8888 in your browser
# Search for larryliu0820/Qwen3-0.6B-Q8_0-ExecuTorch-GGML to start chatting

Using HuggingFace Spaces for Unsloth

# No setup required
# Open https://huggingface.co/spaces/unsloth/studio in your browser
# Search for larryliu0820/Qwen3-0.6B-Q8_0-ExecuTorch-GGML to start chatting

How to use larryliu0820/Qwen3-0.6B-Q8_0-ExecuTorch-GGML with Pi:

Start the llama.cpp server

# Install llama.cpp:
brew install llama.cpp
# Start a local OpenAI-compatible server:
llama serve -hf larryliu0820/Qwen3-0.6B-Q8_0-ExecuTorch-GGML:Q8_0

Configure the model in Pi

# Install Pi:
npm install -g @mariozechner/pi-coding-agent
# Add to ~/.pi/agent/models.json:
{
  "providers": {
    "llama-cpp": {
      "baseUrl": "http://localhost:8080/v1",
      "api": "openai-completions",
      "apiKey": "none",
      "models": [
        {
          "id": "larryliu0820/Qwen3-0.6B-Q8_0-ExecuTorch-GGML:Q8_0"
        }
      ]
    }
  }
}

Run Pi

# Start Pi in your project directory:
pi

Hermes Agent new

How to use larryliu0820/Qwen3-0.6B-Q8_0-ExecuTorch-GGML with Hermes Agent:

Start the llama.cpp server

# Install llama.cpp:
brew install llama.cpp
# Start a local OpenAI-compatible server:
llama serve -hf larryliu0820/Qwen3-0.6B-Q8_0-ExecuTorch-GGML:Q8_0

Configure Hermes

# Install Hermes:
curl -fsSL https://hermes-agent.nousresearch.com/install.sh | bash
hermes setup
# Point Hermes at the local server:
hermes config set model.provider custom
hermes config set model.base_url http://127.0.0.1:8080/v1
hermes config set model.default larryliu0820/Qwen3-0.6B-Q8_0-ExecuTorch-GGML:Q8_0

Run Hermes

hermes

Atomic Chat new

OpenClaw new

How to use larryliu0820/Qwen3-0.6B-Q8_0-ExecuTorch-GGML with OpenClaw:

Start the llama.cpp server

# Install llama.cpp:
brew install llama.cpp
# Start a local OpenAI-compatible server:
llama serve -hf larryliu0820/Qwen3-0.6B-Q8_0-ExecuTorch-GGML:Q8_0

Configure OpenClaw

# Install OpenClaw:
npm install -g openclaw@latest
# Register the local server and set it as the default model:
openclaw onboard --non-interactive --mode local \
  --auth-choice custom-api-key \
  --custom-base-url http://127.0.0.1:8080/v1 \
  --custom-model-id "larryliu0820/Qwen3-0.6B-Q8_0-ExecuTorch-GGML:Q8_0" \
  --custom-provider-id llama-cpp \
  --custom-compatibility openai \
  --custom-text-input \
  --accept-risk \
  --skip-health

Run OpenClaw

openclaw agent --local --agent main --message "Hello from Hugging Face"

Docker Model Runner
How to use larryliu0820/Qwen3-0.6B-Q8_0-ExecuTorch-GGML with Docker Model Runner:
```
docker model run hf.co/larryliu0820/Qwen3-0.6B-Q8_0-ExecuTorch-GGML:Q8_0
```

Lemonade

How to use larryliu0820/Qwen3-0.6B-Q8_0-ExecuTorch-GGML with Lemonade:

Pull the model

# Download Lemonade from https://lemonade-server.ai/
lemonade pull larryliu0820/Qwen3-0.6B-Q8_0-ExecuTorch-GGML:Q8_0

Run and chat with the model

lemonade run user.Qwen3-0.6B-Q8_0-ExecuTorch-GGML-Q8_0

List all available models

lemonade list

Qwen3-0.6B-Q8_0-ExecuTorch-GGML / README.md

larryliu0820

Add model card

610be8f verified 4 months ago

preview code

Raw

History Blame

5.94 kB

	---
	library_name: executorch
	license: apache-2.0
	base_model: Qwen/Qwen3-0.6B
	tags:
	- executorch
	- ggml
	- metal
	- cuda
	- on-device
	- quantized
	- q8_0
	language:
	- en
	pipeline_tag: text-generation
	---

	# Qwen3-0.6B Q8_0 — ExecuTorch + GGML Backend

	Qwen3-0.6B exported to ExecuTorch `.pte` format with the ggml backend and Q8_0 quantization.
	Runs on Metal (Apple Silicon) and CUDA (NVIDIA GPUs) using the same `.pte` file.

	Includes the original `.gguf` file for llama.cpp baseline comparison.

	## Performance

	### Decode throughput (tok/s, tg128)

	\| Platform \| executorch-ggml (.pte) \| llama.cpp (.gguf) \| % of llama.cpp \|
	\|---\|---\|---\|---\|
	\| NVIDIA A100-SXM4-40GB \| 411 \| 380 \| 108% \|
	\| Apple M4 Max \| 300 \| 300 \| 100% \|

	### Prefill throughput (tok/s, pp5)

	\| Platform \| executorch-ggml (.pte) \| llama.cpp (.gguf) \|
	\|---\|---\|---\|
	\| NVIDIA A100-SXM4-40GB \| 1012 \| 1518 \|
	\| Apple M4 Max \| 640 \| 800 \|

	### Per-step breakdown (decode, steady state)

	\| Phase \| Metal (M4 Max) \| CUDA (A100) \|
	\|---\|---\|---\|
	\| build_graph \| 0.0 ms (cached) \| 0.0 ms (cached) \|
	\| sched_alloc \| 0.0 ms (cached) \| 0.0 ms (cached) \|
	\| compute \| 0.4 ms (async) \| 0.4 ms (async) \|
	\| output sync \| 2.9 ms \| 2.0 ms \|
	\| total \| ~3.3 ms \| ~2.4 ms \|

	Graph: 602 nodes, 1 split (all ops on GPU).

	## Files

	\| File \| Size \| Description \|
	\|---\|---\|---\|
	\| `qwen3_q8_0.pte` \| 762 MB \| ExecuTorch model (Q8_0 weights, F32 KV cache) \|
	\| `Qwen3-0.6B-Q8_0.gguf` \| 610 MB \| GGUF model for llama.cpp baseline \|

	## Quick Start

	### 1. Build

	```bash
	git clone https://github.com/anthropics/executorch-ggml
	cd executorch-ggml
	git submodule update --init --recursive
	```

	Metal (macOS):
	```bash
	cmake -B build_native \
	-DEXECUTORCH_GGML_BUILD_LLAMA_RUNNER=ON \
	-DCMAKE_BUILD_TYPE=Release
	cmake --build build_native --target benchmark_llm --parallel 16
	```

	CUDA:
	```bash
	cmake -B build_native \
	-DEXECUTORCH_GGML_BUILD_LLAMA_RUNNER=ON \
	-DGGML_CUDA=ON \
	-DCMAKE_CUDA_ARCHITECTURES=80 \
	-DCMAKE_BUILD_TYPE=Release
	cmake --build build_native --target benchmark_llm --parallel 16
	```

	### 2. Download

	```bash
	pip install huggingface_hub
	python -c "
	from huggingface_hub import snapshot_download
	snapshot_download('larryliu0820/Qwen3-0.6B-Q8_0-ExecuTorch-GGML',
	local_dir='models/qwen3')
	"
	```

	### 3. Benchmark

	Run executorch-ggml:
	```bash
	./build_native/benchmark/benchmark_llm \
	models/qwen3/qwen3_q8_0.pte \
	--n-decode 128 --prompt-len 5
	```

	Run llama.cpp baseline (for comparison):
	```bash
	# Build llama-bench
	cd third-party/llama.cpp
	cmake -B build -DGGML_METAL=ON -DCMAKE_BUILD_TYPE=Release # or -DGGML_CUDA=ON
	cmake --build build --target llama-bench --parallel 16
	cd ../..

	# Benchmark
	third-party/llama.cpp/build/bin/llama-bench \
	-m models/qwen3/Qwen3-0.6B-Q8_0.gguf \
	-ngl 99 -p 5 -n 128 -r 5
	```

	### 4. Profiling

	```bash
	# Per-call timing breakdown
	GGML_PERF_LOG=1 ./build_native/benchmark/benchmark_llm \
	models/qwen3/qwen3_q8_0.pte --n-decode 32

	# Per-op timing (adds sync overhead — use for relative comparison only)
	GGML_PROFILE=1 ./build_native/benchmark/benchmark_llm \
	models/qwen3/qwen3_q8_0.pte --n-decode 5
	```

	## Export Pipeline

	The `.pte` was exported with all optimization passes:

	```python
	from executorch_ggml.modules.rms_norm import swap_rms_norm
	from executorch_ggml.passes.fold_rms_norm_weights import fold_rms_norm_weights
	from executorch_ggml.passes.fuse_rope_pass import fuse_rope_in_graph
	from executorch_ggml.passes.strip_gqa_expand_pass import strip_gqa_expand

	# 1. Module swaps (before export)
	swap_rms_norm(model) # 113 RMSNorm modules fused
	fold_rms_norm_weights(model) # 57 weight MULs eliminated

	# 2. Export
	ep = exportable.export()["model"]

	# 3. AOT graph passes
	fuse_rope_in_graph(ep.graph_module, head_dim=128, freq_base=1e6) # 56 RoPE fused
	strip_gqa_expand(ep.graph_module) # GQA REPEATs eliminated

	# 4. Edge lowering with Q8_0 quantization
	edge_mgr = to_edge_transform_and_lower(
	ep,
	partitioner=[GgmlPartitioner(quant_config=GgmlQuantConfig())],
	compile_config=EdgeCompileConfig(_check_ir_validity=False, _skip_dim_order=True),
	transform_passes=[ReplaceCopyOpsPass(), RemoveGraphAssertsPass()],
	constant_methods=exportable.metadata,
	)
	```

	See `runner/export_qwen3_q8_optimized.py` for the full script.

	## Optimizations Applied

	\| Optimization \| Effect \| Nodes \|
	\|---\|---\|---\|
	\| Fused RMSNorm (`swap_rms_norm`) \| 8 ops/norm → 1 \| -24% \|
	\| Fused RoPE (`fuse_rope_in_graph`) \| 9 ops/Q,K → 1 \| -24% \|
	\| GQA strip (`strip_gqa_expand`) \| Remove expand/repeat \| -10% \|
	\| RMSNorm weight fold \| Absorb weight into downstream linear \| -4% \|
	\| Mask conversion cache \| Deduplicate across 28 layers \| -8% \|
	\| RESHAPE collapse \| `RESHAPE(RESHAPE(x))` → `RESHAPE(x)` \| -13% \|
	\| PERMUTE composition \| `PERMUTE(PERMUTE(x))` → identity \| included \|
	\| SiLU-gate fusion \| `x * sigmoid(x) * up` → `swiglu_split` \| -9% \|
	\| Graph cache (default on) \| Skip rebuild on same shape \| 0 ms build \|
	\| Mutable KV cache \| `ggml_set_rows` on GPU, no CPU fallback \| 1 split \|

	## Environment Variables

	\| Variable \| Values \| Description \|
	\|---\|---\|---\|
	\| `GGML_PERF_LOG` \| `1` \| Per-call timing breakdown \|
	\| `GGML_PROFILE` \| `1` \| Per-op timing (adds sync overhead) \|
	\| `GGML_NO_GRAPH_CACHE` \| `1` \| Disable graph caching (debug) \|
	\| `GGML_DEBUG_DUMP` \| `<path>` \| Per-node tensor stats \|
	\| `GGML_SKIP_OUTPUT_COPY` \| `1` \| Skip logits GPU→CPU copy (CUDA only) \|

	## Model Details

	- Base model: [Qwen/Qwen3-0.6B](https://huggingface.co/Qwen/Qwen3-0.6B)
	- Parameters: 596M
	- Architecture: 28 layers, 16 attention heads, 8 KV heads, head_dim=128
	- Quantization: Q8_0 (weights only; KV cache is F32)
	- Max sequence length: 128 (exported)
	- Framework: [ExecuTorch](https://github.com/pytorch/executorch) with ggml backend