Instructions to use larryliu0820/Qwen3-0.6B-Q8_0-ExecuTorch-GGML with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use larryliu0820/Qwen3-0.6B-Q8_0-ExecuTorch-GGML with llama-cpp-python:

# !pip install llama-cpp-python

from llama_cpp import Llama

llm = Llama.from_pretrained(
	repo_id="larryliu0820/Qwen3-0.6B-Q8_0-ExecuTorch-GGML",
	filename="Qwen3-0.6B-Q8_0.gguf",
)

llm.create_chat_completion(
	messages = [
		{
			"role": "user",
			"content": "What is the capital of France?"
		}
	]
)

Notebooks
Google Colab
Kaggle
Local Apps Settings

llama.cpp

How to use larryliu0820/Qwen3-0.6B-Q8_0-ExecuTorch-GGML with llama.cpp:

Install (macOS, Linux)

curl -LsSf https://llama.app/install.sh | sh
# Start a local OpenAI-compatible server with a web UI:
llama serve -hf larryliu0820/Qwen3-0.6B-Q8_0-ExecuTorch-GGML:Q8_0
# Run inference directly in the terminal:
llama cli -hf larryliu0820/Qwen3-0.6B-Q8_0-ExecuTorch-GGML:Q8_0

Install from WinGet (Windows)

winget install llama.cpp
# Start a local OpenAI-compatible server with a web UI:
llama serve -hf larryliu0820/Qwen3-0.6B-Q8_0-ExecuTorch-GGML:Q8_0
# Run inference directly in the terminal:
llama cli -hf larryliu0820/Qwen3-0.6B-Q8_0-ExecuTorch-GGML:Q8_0

Use pre-built binary

# Download pre-built binary from:
# https://github.com/ggerganov/llama.cpp/releases
# Start a local OpenAI-compatible server with a web UI:
./llama-server -hf larryliu0820/Qwen3-0.6B-Q8_0-ExecuTorch-GGML:Q8_0
# Run inference directly in the terminal:
./llama-cli -hf larryliu0820/Qwen3-0.6B-Q8_0-ExecuTorch-GGML:Q8_0

Build from source code

git clone https://github.com/ggerganov/llama.cpp.git
cd llama.cpp
cmake -B build
cmake --build build -j --target llama-server llama-cli
# Start a local OpenAI-compatible server with a web UI:
./build/bin/llama-server -hf larryliu0820/Qwen3-0.6B-Q8_0-ExecuTorch-GGML:Q8_0
# Run inference directly in the terminal:
./build/bin/llama-cli -hf larryliu0820/Qwen3-0.6B-Q8_0-ExecuTorch-GGML:Q8_0

Use Docker

docker model run hf.co/larryliu0820/Qwen3-0.6B-Q8_0-ExecuTorch-GGML:Q8_0

LM Studio
Jan

vLLM

How to use larryliu0820/Qwen3-0.6B-Q8_0-ExecuTorch-GGML with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "larryliu0820/Qwen3-0.6B-Q8_0-ExecuTorch-GGML"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "larryliu0820/Qwen3-0.6B-Q8_0-ExecuTorch-GGML",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker

docker model run hf.co/larryliu0820/Qwen3-0.6B-Q8_0-ExecuTorch-GGML:Q8_0

Ollama
How to use larryliu0820/Qwen3-0.6B-Q8_0-ExecuTorch-GGML with Ollama:
```
ollama run hf.co/larryliu0820/Qwen3-0.6B-Q8_0-ExecuTorch-GGML:Q8_0
```

Unsloth Studio

How to use larryliu0820/Qwen3-0.6B-Q8_0-ExecuTorch-GGML with Unsloth Studio:

Install Unsloth Studio (macOS, Linux, WSL)

curl -fsSL https://unsloth.ai/install.sh | sh
# Run unsloth studio
unsloth studio -H 0.0.0.0 -p 8888
# Then open http://localhost:8888 in your browser
# Search for larryliu0820/Qwen3-0.6B-Q8_0-ExecuTorch-GGML to start chatting

Install Unsloth Studio (Windows)

irm https://unsloth.ai/install.ps1 | iex
# Run unsloth studio
unsloth studio -H 0.0.0.0 -p 8888
# Then open http://localhost:8888 in your browser
# Search for larryliu0820/Qwen3-0.6B-Q8_0-ExecuTorch-GGML to start chatting

Using HuggingFace Spaces for Unsloth

# No setup required
# Open https://huggingface.co/spaces/unsloth/studio in your browser
# Search for larryliu0820/Qwen3-0.6B-Q8_0-ExecuTorch-GGML to start chatting

How to use larryliu0820/Qwen3-0.6B-Q8_0-ExecuTorch-GGML with Pi:

Start the llama.cpp server

# Install llama.cpp:
brew install llama.cpp
# Start a local OpenAI-compatible server:
llama serve -hf larryliu0820/Qwen3-0.6B-Q8_0-ExecuTorch-GGML:Q8_0

Configure the model in Pi

# Install Pi:
npm install -g @mariozechner/pi-coding-agent
# Add to ~/.pi/agent/models.json:
{
  "providers": {
    "llama-cpp": {
      "baseUrl": "http://localhost:8080/v1",
      "api": "openai-completions",
      "apiKey": "none",
      "models": [
        {
          "id": "larryliu0820/Qwen3-0.6B-Q8_0-ExecuTorch-GGML:Q8_0"
        }
      ]
    }
  }
}

Run Pi

# Start Pi in your project directory:
pi

Hermes Agent new

How to use larryliu0820/Qwen3-0.6B-Q8_0-ExecuTorch-GGML with Hermes Agent:

Start the llama.cpp server

# Install llama.cpp:
brew install llama.cpp
# Start a local OpenAI-compatible server:
llama serve -hf larryliu0820/Qwen3-0.6B-Q8_0-ExecuTorch-GGML:Q8_0

Configure Hermes

# Install Hermes:
curl -fsSL https://hermes-agent.nousresearch.com/install.sh | bash
hermes setup
# Point Hermes at the local server:
hermes config set model.provider custom
hermes config set model.base_url http://127.0.0.1:8080/v1
hermes config set model.default larryliu0820/Qwen3-0.6B-Q8_0-ExecuTorch-GGML:Q8_0

Run Hermes

hermes

Atomic Chat new

OpenClaw new

How to use larryliu0820/Qwen3-0.6B-Q8_0-ExecuTorch-GGML with OpenClaw:

Start the llama.cpp server

# Install llama.cpp:
brew install llama.cpp
# Start a local OpenAI-compatible server:
llama serve -hf larryliu0820/Qwen3-0.6B-Q8_0-ExecuTorch-GGML:Q8_0

Configure OpenClaw

# Install OpenClaw:
npm install -g openclaw@latest
# Register the local server and set it as the default model:
openclaw onboard --non-interactive --mode local \
  --auth-choice custom-api-key \
  --custom-base-url http://127.0.0.1:8080/v1 \
  --custom-model-id "larryliu0820/Qwen3-0.6B-Q8_0-ExecuTorch-GGML:Q8_0" \
  --custom-provider-id llama-cpp \
  --custom-compatibility openai \
  --custom-text-input \
  --accept-risk \
  --skip-health

Run OpenClaw

openclaw agent --local --agent main --message "Hello from Hugging Face"

Docker Model Runner
How to use larryliu0820/Qwen3-0.6B-Q8_0-ExecuTorch-GGML with Docker Model Runner:
```
docker model run hf.co/larryliu0820/Qwen3-0.6B-Q8_0-ExecuTorch-GGML:Q8_0
```

Lemonade

How to use larryliu0820/Qwen3-0.6B-Q8_0-ExecuTorch-GGML with Lemonade:

Pull the model

# Download Lemonade from https://lemonade-server.ai/
lemonade pull larryliu0820/Qwen3-0.6B-Q8_0-ExecuTorch-GGML:Q8_0

Run and chat with the model

lemonade run user.Qwen3-0.6B-Q8_0-ExecuTorch-GGML-Q8_0

List all available models

lemonade list

Qwen3-0.6B-Q8_0-ExecuTorch-GGML / README.md

larryliu0820

Add model card

610be8f verified 4 months ago

5.94 kB

library_name: executorch
license: apache-2.0
base_model: Qwen/Qwen3-0.6B
tags:
  - executorch
  - ggml
  - metal
  - cuda
  - on-device
  - quantized
  - q8_0
language:
  - en
pipeline_tag: text-generation

Qwen3-0.6B Q8_0 — ExecuTorch + GGML Backend

Qwen3-0.6B exported to ExecuTorch .pte format with the ggml backend and Q8_0 quantization. Runs on Metal (Apple Silicon) and CUDA (NVIDIA GPUs) using the same .pte file.

Includes the original .gguf file for llama.cpp baseline comparison.

Performance

Decode throughput (tok/s, tg128)

Platform	executorch-ggml (.pte)	llama.cpp (.gguf)	% of llama.cpp
NVIDIA A100-SXM4-40GB	411	380	108%
Apple M4 Max	300	300	100%

Prefill throughput (tok/s, pp5)

Platform	executorch-ggml (.pte)	llama.cpp (.gguf)
NVIDIA A100-SXM4-40GB	1012	1518
Apple M4 Max	640	800

Per-step breakdown (decode, steady state)

Phase	Metal (M4 Max)	CUDA (A100)
build_graph	0.0 ms (cached)	0.0 ms (cached)
sched_alloc	0.0 ms (cached)	0.0 ms (cached)
compute	0.4 ms (async)	0.4 ms (async)
output sync	2.9 ms	2.0 ms
total	~3.3 ms	~2.4 ms

Graph: 602 nodes, 1 split (all ops on GPU).

Files

File	Size	Description
`qwen3_q8_0.pte`	762 MB	ExecuTorch model (Q8_0 weights, F32 KV cache)
`Qwen3-0.6B-Q8_0.gguf`	610 MB	GGUF model for llama.cpp baseline

Quick Start

1. Build

git clone https://github.com/anthropics/executorch-ggml
cd executorch-ggml
git submodule update --init --recursive

Metal (macOS):

cmake -B build_native \
    -DEXECUTORCH_GGML_BUILD_LLAMA_RUNNER=ON \
    -DCMAKE_BUILD_TYPE=Release
cmake --build build_native --target benchmark_llm --parallel 16

CUDA:

cmake -B build_native \
    -DEXECUTORCH_GGML_BUILD_LLAMA_RUNNER=ON \
    -DGGML_CUDA=ON \
    -DCMAKE_CUDA_ARCHITECTURES=80 \
    -DCMAKE_BUILD_TYPE=Release
cmake --build build_native --target benchmark_llm --parallel 16

2. Download

pip install huggingface_hub
python -c "
from huggingface_hub import snapshot_download
snapshot_download('larryliu0820/Qwen3-0.6B-Q8_0-ExecuTorch-GGML',
                  local_dir='models/qwen3')
"

3. Benchmark

Run executorch-ggml:

./build_native/benchmark/benchmark_llm \
    models/qwen3/qwen3_q8_0.pte \
    --n-decode 128 --prompt-len 5

Run llama.cpp baseline (for comparison):

# Build llama-bench
cd third-party/llama.cpp
cmake -B build -DGGML_METAL=ON -DCMAKE_BUILD_TYPE=Release  # or -DGGML_CUDA=ON
cmake --build build --target llama-bench --parallel 16
cd ../..

# Benchmark
third-party/llama.cpp/build/bin/llama-bench \
    -m models/qwen3/Qwen3-0.6B-Q8_0.gguf \
    -ngl 99 -p 5 -n 128 -r 5

4. Profiling

# Per-call timing breakdown
GGML_PERF_LOG=1 ./build_native/benchmark/benchmark_llm \
    models/qwen3/qwen3_q8_0.pte --n-decode 32

# Per-op timing (adds sync overhead — use for relative comparison only)
GGML_PROFILE=1 ./build_native/benchmark/benchmark_llm \
    models/qwen3/qwen3_q8_0.pte --n-decode 5

Export Pipeline

The .pte was exported with all optimization passes:

from executorch_ggml.modules.rms_norm import swap_rms_norm
from executorch_ggml.passes.fold_rms_norm_weights import fold_rms_norm_weights
from executorch_ggml.passes.fuse_rope_pass import fuse_rope_in_graph
from executorch_ggml.passes.strip_gqa_expand_pass import strip_gqa_expand

# 1. Module swaps (before export)
swap_rms_norm(model)           # 113 RMSNorm modules fused
fold_rms_norm_weights(model)   # 57 weight MULs eliminated

# 2. Export
ep = exportable.export()["model"]

# 3. AOT graph passes
fuse_rope_in_graph(ep.graph_module, head_dim=128, freq_base=1e6)  # 56 RoPE fused
strip_gqa_expand(ep.graph_module)  # GQA REPEATs eliminated

# 4. Edge lowering with Q8_0 quantization
edge_mgr = to_edge_transform_and_lower(
    ep,
    partitioner=[GgmlPartitioner(quant_config=GgmlQuantConfig())],
    compile_config=EdgeCompileConfig(_check_ir_validity=False, _skip_dim_order=True),
    transform_passes=[ReplaceCopyOpsPass(), RemoveGraphAssertsPass()],
    constant_methods=exportable.metadata,
)

See runner/export_qwen3_q8_optimized.py for the full script.

Optimizations Applied

Optimization	Effect	Nodes
Fused RMSNorm (`swap_rms_norm`)	8 ops/norm → 1	-24%
Fused RoPE (`fuse_rope_in_graph`)	9 ops/Q,K → 1	-24%
GQA strip (`strip_gqa_expand`)	Remove expand/repeat	-10%
RMSNorm weight fold	Absorb weight into downstream linear	-4%
Mask conversion cache	Deduplicate across 28 layers	-8%
RESHAPE collapse	`RESHAPE(RESHAPE(x))` → `RESHAPE(x)`	-13%
PERMUTE composition	`PERMUTE(PERMUTE(x))` → identity	included
SiLU-gate fusion	`x * sigmoid(x) * up` → `swiglu_split`	-9%
Graph cache (default on)	Skip rebuild on same shape	0 ms build
Mutable KV cache	`ggml_set_rows` on GPU, no CPU fallback	1 split

Environment Variables

Variable	Values	Description
`GGML_PERF_LOG`	`1`	Per-call timing breakdown
`GGML_PROFILE`	`1`	Per-op timing (adds sync overhead)
`GGML_NO_GRAPH_CACHE`	`1`	Disable graph caching (debug)
`GGML_DEBUG_DUMP`	`<path>`	Per-node tensor stats
`GGML_SKIP_OUTPUT_COPY`	`1`	Skip logits GPU→CPU copy (CUDA only)

Model Details

Base model: Qwen/Qwen3-0.6B
Parameters: 596M
Architecture: 28 layers, 16 attention heads, 8 KV heads, head_dim=128
Quantization: Q8_0 (weights only; KV cache is F32)
Max sequence length: 128 (exported)
Framework: ExecuTorch with ggml backend