Instructions to use a4lg/Qwen3.5-35B-A3B-MTP-ONLY-GGUF with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use a4lg/Qwen3.5-35B-A3B-MTP-ONLY-GGUF with llama-cpp-python:

# !pip install llama-cpp-python

from llama_cpp import Llama

llm = Llama.from_pretrained(
	repo_id="a4lg/Qwen3.5-35B-A3B-MTP-ONLY-GGUF",
	filename="Qwen3.5-35B-A3B-MTP-ONLY-BF16.gguf",
)

llm.create_chat_completion(
	messages = "No input example has been defined for this model task."
)

Notebooks
Google Colab
Kaggle
Local Apps Settings

llama.cpp

How to use a4lg/Qwen3.5-35B-A3B-MTP-ONLY-GGUF with llama.cpp:

Install (macOS, Linux)

curl -LsSf https://llama.app/install.sh | sh
# Start a local OpenAI-compatible server with a web UI:
llama serve -hf a4lg/Qwen3.5-35B-A3B-MTP-ONLY-GGUF:Q4_K_M
# Run inference directly in the terminal:
llama cli -hf a4lg/Qwen3.5-35B-A3B-MTP-ONLY-GGUF:Q4_K_M

Install from WinGet (Windows)

winget install llama.cpp
# Start a local OpenAI-compatible server with a web UI:
llama serve -hf a4lg/Qwen3.5-35B-A3B-MTP-ONLY-GGUF:Q4_K_M
# Run inference directly in the terminal:
llama cli -hf a4lg/Qwen3.5-35B-A3B-MTP-ONLY-GGUF:Q4_K_M

Use pre-built binary

# Download pre-built binary from:
# https://github.com/ggerganov/llama.cpp/releases
# Start a local OpenAI-compatible server with a web UI:
./llama-server -hf a4lg/Qwen3.5-35B-A3B-MTP-ONLY-GGUF:Q4_K_M
# Run inference directly in the terminal:
./llama-cli -hf a4lg/Qwen3.5-35B-A3B-MTP-ONLY-GGUF:Q4_K_M

Build from source code

git clone https://github.com/ggerganov/llama.cpp.git
cd llama.cpp
cmake -B build
cmake --build build -j --target llama-server llama-cli
# Start a local OpenAI-compatible server with a web UI:
./build/bin/llama-server -hf a4lg/Qwen3.5-35B-A3B-MTP-ONLY-GGUF:Q4_K_M
# Run inference directly in the terminal:
./build/bin/llama-cli -hf a4lg/Qwen3.5-35B-A3B-MTP-ONLY-GGUF:Q4_K_M

Use Docker

docker model run hf.co/a4lg/Qwen3.5-35B-A3B-MTP-ONLY-GGUF:Q4_K_M

LM Studio
Jan
Ollama
How to use a4lg/Qwen3.5-35B-A3B-MTP-ONLY-GGUF with Ollama:
```
ollama run hf.co/a4lg/Qwen3.5-35B-A3B-MTP-ONLY-GGUF:Q4_K_M
```

Unsloth Studio

How to use a4lg/Qwen3.5-35B-A3B-MTP-ONLY-GGUF with Unsloth Studio:

Install Unsloth Studio (macOS, Linux, WSL)

curl -fsSL https://unsloth.ai/install.sh | sh
# Run unsloth studio
unsloth studio -H 0.0.0.0 -p 8888
# Then open http://localhost:8888 in your browser
# Search for a4lg/Qwen3.5-35B-A3B-MTP-ONLY-GGUF to start chatting

Install Unsloth Studio (Windows)

irm https://unsloth.ai/install.ps1 | iex
# Run unsloth studio
unsloth studio -H 0.0.0.0 -p 8888
# Then open http://localhost:8888 in your browser
# Search for a4lg/Qwen3.5-35B-A3B-MTP-ONLY-GGUF to start chatting

Using HuggingFace Spaces for Unsloth

# No setup required
# Open https://huggingface.co/spaces/unsloth/studio in your browser
# Search for a4lg/Qwen3.5-35B-A3B-MTP-ONLY-GGUF to start chatting

How to use a4lg/Qwen3.5-35B-A3B-MTP-ONLY-GGUF with Pi:

Start the llama.cpp server

# Install llama.cpp:
brew install llama.cpp
# Start a local OpenAI-compatible server:
llama serve -hf a4lg/Qwen3.5-35B-A3B-MTP-ONLY-GGUF:Q4_K_M

Configure the model in Pi

# Install Pi:
npm install -g @mariozechner/pi-coding-agent
# Add to ~/.pi/agent/models.json:
{
  "providers": {
    "llama-cpp": {
      "baseUrl": "http://localhost:8080/v1",
      "api": "openai-completions",
      "apiKey": "none",
      "models": [
        {
          "id": "a4lg/Qwen3.5-35B-A3B-MTP-ONLY-GGUF:Q4_K_M"
        }
      ]
    }
  }
}

Run Pi

# Start Pi in your project directory:
pi

Hermes Agent new

How to use a4lg/Qwen3.5-35B-A3B-MTP-ONLY-GGUF with Hermes Agent:

Start the llama.cpp server

# Install llama.cpp:
brew install llama.cpp
# Start a local OpenAI-compatible server:
llama serve -hf a4lg/Qwen3.5-35B-A3B-MTP-ONLY-GGUF:Q4_K_M

Configure Hermes

# Install Hermes:
curl -fsSL https://hermes-agent.nousresearch.com/install.sh | bash
hermes setup
# Point Hermes at the local server:
hermes config set model.provider custom
hermes config set model.base_url http://127.0.0.1:8080/v1
hermes config set model.default a4lg/Qwen3.5-35B-A3B-MTP-ONLY-GGUF:Q4_K_M

Run Hermes

hermes

Atomic Chat new

OpenClaw new

How to use a4lg/Qwen3.5-35B-A3B-MTP-ONLY-GGUF with OpenClaw:

Start the llama.cpp server

# Install llama.cpp:
brew install llama.cpp
# Start a local OpenAI-compatible server:
llama serve -hf a4lg/Qwen3.5-35B-A3B-MTP-ONLY-GGUF:Q4_K_M

Configure OpenClaw

# Install OpenClaw:
npm install -g openclaw@latest
# Register the local server and set it as the default model:
openclaw onboard --non-interactive --mode local \
  --auth-choice custom-api-key \
  --custom-base-url http://127.0.0.1:8080/v1 \
  --custom-model-id "a4lg/Qwen3.5-35B-A3B-MTP-ONLY-GGUF:Q4_K_M" \
  --custom-provider-id llama-cpp \
  --custom-compatibility openai \
  --custom-text-input \
  --accept-risk \
  --skip-health

Run OpenClaw

openclaw agent --local --agent main --message "Hello from Hugging Face"

Docker Model Runner
How to use a4lg/Qwen3.5-35B-A3B-MTP-ONLY-GGUF with Docker Model Runner:
```
docker model run hf.co/a4lg/Qwen3.5-35B-A3B-MTP-ONLY-GGUF:Q4_K_M
```

Lemonade

How to use a4lg/Qwen3.5-35B-A3B-MTP-ONLY-GGUF with Lemonade:

Pull the model

# Download Lemonade from https://lemonade-server.ai/
lemonade pull a4lg/Qwen3.5-35B-A3B-MTP-ONLY-GGUF:Q4_K_M

Run and chat with the model

lemonade run user.Qwen3.5-35B-A3B-MTP-ONLY-GGUF-Q4_K_M

List all available models

lemonade list

MTP-only GGUF subset of Qwen3.5/3.6 Medium/Large models

This is a supplement for Qwen3.5-35B-A3B-based models (including fine-tunes and abliterated models) without MTP tensors.

This repository contains an MTP-only subset of Qwen/Qwen3.5-35B-A3B which provides the draft model for speculative decoding in the GGUF format.

It accelerates token generation using speculative decoding with the draft model from the original Qwen model. In most cases, this is sufficient to accelerate Qwen-based derivative models even if this draft model is not trained from them.

Note that however, the performance metrics heavily depend on the derivative model you use, your machine and your MTP settings.

Benchmark it before blindly trusting it.

Using this Model

It can be used in two ways:

As a separate draft model file (Method 1)
As a donor for grafting the draft model into a Qwen-based model (Method 2; Recommended)

Method 1: Separate Draft Model File

It is easy to begin with but memory-inefficient as it does not share some tensors with the original model.

If you find the draft model can accelerate a Qwen-based model you use, grafting the draft model (Method 2) is recommended (note: switching to Method 2 may slightly change the acceptance rate).

If you use llama-server, you may configure like this:

llama-server \
    --model       Qwen3.6-27B-finetune-Q4_K_M.gguf \
    --model-draft Qwen3.6-27B-MTP-ONLY-Q4_K_M.gguf \
    ... \
    --spec-type draft-mtp
    --spec-draft-n-max 4

--model specifies the original Qwen-based model and new --model-draft specifies a file from this repository.

You also need --spec-type draft-mtp to enable the draft model.

Once the draft model is enabled, you may configure the rest of MTP options as you like (in this example, custom --spec-draft-n-max is specified).

Method 2: Grafting the Draft Model (Using this as a Donor)

This is recommended.

First, download convert.py written by @buzz to transplant MTP tensors. You may also need to install some dependencies required by this script.

Then, you can run this script like:

# ./convert.py INPUT MTP OUTPUT
python3 convert.py \
    Qwen3.6-27B-finetune-Q4_K_M.gguf \
    Qwen3.6-27B-MTP-ONLY-Q4_K_M.gguf \
    Qwen3.6-27B-finetune-Q4_K_M+MTP.gguf

The second argument of convert.py is a GGUF file (donor) from this repository.

Once the grafted GGUF file is created, you may use this like a regular Qwen model with embedded draft model.

This is an example for llama-server users.

llama-server \
    --model Qwen3.6-27B-finetune-Q4_K_M+MTP.gguf \
    ... \
    --spec-type draft-mtp
    --spec-draft-n-max 4

The output of convert.py must be specified as the model file name (DO NOT use --model-draft in this case).

--spec-type draft-mtp enables the draft model transplanted into the main one.

Additional Quantization (`Q4_K_M`, `Q5_K_M`, `Q6_K` and `Q8_0`)

Quantized GGUF files are provided so that deploying the draft model easier.

To maximize the accuracy, tensor types are based on Unsloth quants (see the table below):

This Repo	Unsloth
`Q4_K_M`	`UD-Q4_K_XL`
`Q5_K_M`	`UD-Q5_K_XL`
`Q6_K`	`UD-Q6_K_XL`
`Q8_0`	`UD-Q8_K_XL`

It is not required to match the quantization level. For instance, you may pair Q6_K-quantized draft model with the Q4_K_S-quantized main model.

Conversion Process

Tools: llama.cpp (b9837)
With: Patched conversion/base.py
The if block right after # verify tensor name presence and identify potentially missing files is commented out.

This modification is performed because the author of this repository downloaded only a subset of the full Qwen model while the original convert_hf_to_gguf.py expects the full model.

The --mtp option of convert_hf_to_gguf.py is the crucial part of this conversion process because this option does exactly what the author expects: create an MTP-only GGUF subset.

For additional quantization, the llama-quantize tool (llama.cpp) is used:

Without --imatrix
With custom --tensor-type-file to port tensor types from Unsloth's quants

License and Copyright

For all GGUF files under this repository, the license terms of the original Qwen model (Apache License version 2.0) applies (as the author of this repository did not perform any changes significant enough for own copyright):

Copyright 2026 Alibaba Cloud

No NOTICE files are attached in the original model.

This README file is licensed under the terms of CC-BY-4.0.

Copyright 2026 a4lg.

Links: Sources

The Original Model: Qwen/Qwen3.5-35B-A3B
Tensor Types (Unsloth): unsloth/Qwen3.5-35B-A3B-MTP-GGUF

Links: All MTP Subsets: Qwen3.5/3.6

Downloads last month: 1,177

GGUF

Model size

2B params

Architecture

qwen35moe

Hardware compatibility

4-bit

5-bit

6-bit

8-bit

16-bit

Inference Providers NEW

This model isn't deployed by any Inference Provider. 🙋 Ask for provider support

Model tree for a4lg/Qwen3.5-35B-A3B-MTP-ONLY-GGUF

Base model

Qwen/Qwen3.5-35B-A3B-Base

Finetuned

Qwen/Qwen3.5-35B-A3B

Quantized

(276)

this model

Collection including a4lg/Qwen3.5-35B-A3B-MTP-ONLY-GGUF

MTP-only GGUF subsets: Qwen3.5/3.6

Collection

7 items • Updated 18 days ago • 1

MTP-only GGUF subset of Qwen3.5/3.6 Medium/Large models

Using this Model

Method 1: Separate Draft Model File

Method 2: Grafting the Draft Model (Using this as a Donor)

Additional Quantization (Q4_K_M, Q5_K_M, Q6_K and Q8_0)

Conversion Process

License and Copyright

Links: Sources

Links: All MTP Subsets: Qwen3.5/3.6

Model tree for a4lg/Qwen3.5-35B-A3B-MTP-ONLY-GGUF

Collection including a4lg/Qwen3.5-35B-A3B-MTP-ONLY-GGUF

Additional Quantization (`Q4_K_M`, `Q5_K_M`, `Q6_K` and `Q8_0`)