---
library_name: executorch
license: apache-2.0
base_model: Qwen/Qwen3-0.6B
tags:
  - executorch
  - ggml
  - metal
  - cuda
  - on-device
  - quantized
  - q8_0
language:
  - en
pipeline_tag: text-generation
---

# Qwen3-0.6B Q8_0 — ExecuTorch + GGML Backend

Qwen3-0.6B exported to ExecuTorch `.pte` format with the **ggml backend** and Q8_0 quantization.
Runs on **Metal** (Apple Silicon) and **CUDA** (NVIDIA GPUs) using the same `.pte` file.

Includes the original `.gguf` file for llama.cpp baseline comparison.

## Performance

### Decode throughput (tok/s, tg128)

| Platform | executorch-ggml (.pte) | llama.cpp (.gguf) | % of llama.cpp |
|---|---|---|---|
| **NVIDIA A100-SXM4-40GB** | **411** | 380 | **108%** |
| **Apple M4 Max** | **300** | 300 | **100%** |

### Prefill throughput (tok/s, pp5)

| Platform | executorch-ggml (.pte) | llama.cpp (.gguf) |
|---|---|---|
| **NVIDIA A100-SXM4-40GB** | 1012 | 1518 |
| **Apple M4 Max** | 640 | 800 |

### Per-step breakdown (decode, steady state)

| Phase | Metal (M4 Max) | CUDA (A100) |
|---|---|---|
| build_graph | 0.0 ms (cached) | 0.0 ms (cached) |
| sched_alloc | 0.0 ms (cached) | 0.0 ms (cached) |
| compute | 0.4 ms (async) | 0.4 ms (async) |
| output sync | 2.9 ms | 2.0 ms |
| **total** | **~3.3 ms** | **~2.4 ms** |

Graph: 602 nodes, 1 split (all ops on GPU).

## Files

| File | Size | Description |
|---|---|---|
| `qwen3_q8_0.pte` | 762 MB | ExecuTorch model (Q8_0 weights, F32 KV cache) |
| `Qwen3-0.6B-Q8_0.gguf` | 610 MB | GGUF model for llama.cpp baseline |

## Quick Start

### 1. Build

```bash
git clone https://github.com/anthropics/executorch-ggml
cd executorch-ggml
git submodule update --init --recursive
```

**Metal (macOS):**
```bash
cmake -B build_native \
    -DEXECUTORCH_GGML_BUILD_LLAMA_RUNNER=ON \
    -DCMAKE_BUILD_TYPE=Release
cmake --build build_native --target benchmark_llm --parallel 16
```

**CUDA:**
```bash
cmake -B build_native \
    -DEXECUTORCH_GGML_BUILD_LLAMA_RUNNER=ON \
    -DGGML_CUDA=ON \
    -DCMAKE_CUDA_ARCHITECTURES=80 \
    -DCMAKE_BUILD_TYPE=Release
cmake --build build_native --target benchmark_llm --parallel 16
```

### 2. Download

```bash
pip install huggingface_hub
python -c "
from huggingface_hub import snapshot_download
snapshot_download('larryliu0820/Qwen3-0.6B-Q8_0-ExecuTorch-GGML',
                  local_dir='models/qwen3')
"
```

### 3. Benchmark

**Run executorch-ggml:**
```bash
./build_native/benchmark/benchmark_llm \
    models/qwen3/qwen3_q8_0.pte \
    --n-decode 128 --prompt-len 5
```

**Run llama.cpp baseline (for comparison):**
```bash
# Build llama-bench
cd third-party/llama.cpp
cmake -B build -DGGML_METAL=ON -DCMAKE_BUILD_TYPE=Release  # or -DGGML_CUDA=ON
cmake --build build --target llama-bench --parallel 16
cd ../..

# Benchmark
third-party/llama.cpp/build/bin/llama-bench \
    -m models/qwen3/Qwen3-0.6B-Q8_0.gguf \
    -ngl 99 -p 5 -n 128 -r 5
```

### 4. Profiling

```bash
# Per-call timing breakdown
GGML_PERF_LOG=1 ./build_native/benchmark/benchmark_llm \
    models/qwen3/qwen3_q8_0.pte --n-decode 32

# Per-op timing (adds sync overhead — use for relative comparison only)
GGML_PROFILE=1 ./build_native/benchmark/benchmark_llm \
    models/qwen3/qwen3_q8_0.pte --n-decode 5
```

## Export Pipeline

The `.pte` was exported with all optimization passes:

```python
from executorch_ggml.modules.rms_norm import swap_rms_norm
from executorch_ggml.passes.fold_rms_norm_weights import fold_rms_norm_weights
from executorch_ggml.passes.fuse_rope_pass import fuse_rope_in_graph
from executorch_ggml.passes.strip_gqa_expand_pass import strip_gqa_expand

# 1. Module swaps (before export)
swap_rms_norm(model)           # 113 RMSNorm modules fused
fold_rms_norm_weights(model)   # 57 weight MULs eliminated

# 2. Export
ep = exportable.export()["model"]

# 3. AOT graph passes
fuse_rope_in_graph(ep.graph_module, head_dim=128, freq_base=1e6)  # 56 RoPE fused
strip_gqa_expand(ep.graph_module)  # GQA REPEATs eliminated

# 4. Edge lowering with Q8_0 quantization
edge_mgr = to_edge_transform_and_lower(
    ep,
    partitioner=[GgmlPartitioner(quant_config=GgmlQuantConfig())],
    compile_config=EdgeCompileConfig(_check_ir_validity=False, _skip_dim_order=True),
    transform_passes=[ReplaceCopyOpsPass(), RemoveGraphAssertsPass()],
    constant_methods=exportable.metadata,
)
```

See `runner/export_qwen3_q8_optimized.py` for the full script.

## Optimizations Applied

| Optimization | Effect | Nodes |
|---|---|---|
| Fused RMSNorm (`swap_rms_norm`) | 8 ops/norm → 1 | -24% |
| Fused RoPE (`fuse_rope_in_graph`) | 9 ops/Q,K → 1 | -24% |
| GQA strip (`strip_gqa_expand`) | Remove expand/repeat | -10% |
| RMSNorm weight fold | Absorb weight into downstream linear | -4% |
| Mask conversion cache | Deduplicate across 28 layers | -8% |
| RESHAPE collapse | `RESHAPE(RESHAPE(x))` → `RESHAPE(x)` | -13% |
| PERMUTE composition | `PERMUTE(PERMUTE(x))` → identity | included |
| SiLU-gate fusion | `x * sigmoid(x) * up` → `swiglu_split` | -9% |
| **Graph cache** (default on) | Skip rebuild on same shape | 0 ms build |
| **Mutable KV cache** | `ggml_set_rows` on GPU, no CPU fallback | 1 split |

## Environment Variables

| Variable | Values | Description |
|---|---|---|
| `GGML_PERF_LOG` | `1` | Per-call timing breakdown |
| `GGML_PROFILE` | `1` | Per-op timing (adds sync overhead) |
| `GGML_NO_GRAPH_CACHE` | `1` | Disable graph caching (debug) |
| `GGML_DEBUG_DUMP` | `<path>` | Per-node tensor stats |
| `GGML_SKIP_OUTPUT_COPY` | `1` | Skip logits GPU→CPU copy (CUDA only) |

## Model Details

- **Base model**: [Qwen/Qwen3-0.6B](https://huggingface.co/Qwen/Qwen3-0.6B)
- **Parameters**: 596M
- **Architecture**: 28 layers, 16 attention heads, 8 KV heads, head_dim=128
- **Quantization**: Q8_0 (weights only; KV cache is F32)
- **Max sequence length**: 128 (exported)
- **Framework**: [ExecuTorch](https://github.com/pytorch/executorch) with ggml backend