Instructions to use QQZ2026/Qwen3.6-27B-NVIDIA-NVFP4-no-MTP-GGUF with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Notebooks
Google Colab
Kaggle
Local Apps Settings

How to use QQZ2026/Qwen3.6-27B-NVIDIA-NVFP4-no-MTP-GGUF with llama.cpp:

Install (macOS, Linux)

curl -LsSf https://llama.app/install.sh | sh
# Start a local OpenAI-compatible server with a web UI:
llama serve -hf QQZ2026/Qwen3.6-27B-NVIDIA-NVFP4-no-MTP-GGUF:NVFP4
# Run inference directly in the terminal:
llama cli -hf QQZ2026/Qwen3.6-27B-NVIDIA-NVFP4-no-MTP-GGUF:NVFP4

Install from WinGet (Windows)

winget install llama.cpp
# Start a local OpenAI-compatible server with a web UI:
llama serve -hf QQZ2026/Qwen3.6-27B-NVIDIA-NVFP4-no-MTP-GGUF:NVFP4
# Run inference directly in the terminal:
llama cli -hf QQZ2026/Qwen3.6-27B-NVIDIA-NVFP4-no-MTP-GGUF:NVFP4

Use pre-built binary

# Download pre-built binary from:
# https://github.com/ggerganov/llama.cpp/releases
# Start a local OpenAI-compatible server with a web UI:
./llama-server -hf QQZ2026/Qwen3.6-27B-NVIDIA-NVFP4-no-MTP-GGUF:NVFP4
# Run inference directly in the terminal:
./llama-cli -hf QQZ2026/Qwen3.6-27B-NVIDIA-NVFP4-no-MTP-GGUF:NVFP4

Build from source code

git clone https://github.com/ggerganov/llama.cpp.git
cd llama.cpp
cmake -B build
cmake --build build -j --target llama-server llama-cli
# Start a local OpenAI-compatible server with a web UI:
./build/bin/llama-server -hf QQZ2026/Qwen3.6-27B-NVIDIA-NVFP4-no-MTP-GGUF:NVFP4
# Run inference directly in the terminal:
./build/bin/llama-cli -hf QQZ2026/Qwen3.6-27B-NVIDIA-NVFP4-no-MTP-GGUF:NVFP4

Use Docker

docker model run hf.co/QQZ2026/Qwen3.6-27B-NVIDIA-NVFP4-no-MTP-GGUF:NVFP4

LM Studio
Jan

vLLM

How to use QQZ2026/Qwen3.6-27B-NVIDIA-NVFP4-no-MTP-GGUF with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "QQZ2026/Qwen3.6-27B-NVIDIA-NVFP4-no-MTP-GGUF"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "QQZ2026/Qwen3.6-27B-NVIDIA-NVFP4-no-MTP-GGUF",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker

docker model run hf.co/QQZ2026/Qwen3.6-27B-NVIDIA-NVFP4-no-MTP-GGUF:NVFP4

Ollama
How to use QQZ2026/Qwen3.6-27B-NVIDIA-NVFP4-no-MTP-GGUF with Ollama:
```
ollama run hf.co/QQZ2026/Qwen3.6-27B-NVIDIA-NVFP4-no-MTP-GGUF:NVFP4
```
Unsloth Desktop

How to use QQZ2026/Qwen3.6-27B-NVIDIA-NVFP4-no-MTP-GGUF with Pi:

Start the llama.cpp server

# Install llama.cpp:
brew install llama.cpp
# Start a local OpenAI-compatible server:
llama serve -hf QQZ2026/Qwen3.6-27B-NVIDIA-NVFP4-no-MTP-GGUF:NVFP4

Configure the model in Pi

# Install Pi:
npm install -g @earendil-works/pi-coding-agent
# Add to ~/.pi/agent/models.json:
{
  "providers": {
    "llama-cpp": {
      "baseUrl": "http://localhost:8080/v1",
      "api": "openai-completions",
      "apiKey": "none",
      "models": [
        {
          "id": "QQZ2026/Qwen3.6-27B-NVIDIA-NVFP4-no-MTP-GGUF:NVFP4"
        }
      ]
    }
  }
}

Run Pi

# Start Pi in your project directory:
pi

Docker Model Runner
How to use QQZ2026/Qwen3.6-27B-NVIDIA-NVFP4-no-MTP-GGUF with Docker Model Runner:
```
docker model run hf.co/QQZ2026/Qwen3.6-27B-NVIDIA-NVFP4-no-MTP-GGUF:NVFP4
```

Lemonade

How to use QQZ2026/Qwen3.6-27B-NVIDIA-NVFP4-no-MTP-GGUF with Lemonade:

Pull the model

# Download Lemonade from https://lemonade-server.ai/
lemonade pull QQZ2026/Qwen3.6-27B-NVIDIA-NVFP4-no-MTP-GGUF:NVFP4

Run and chat with the model

lemonade run user.Qwen3.6-27B-NVIDIA-NVFP4-no-MTP-GGUF-NVFP4

List all available models

lemonade list

Hermes Agent

How to use QQZ2026/Qwen3.6-27B-NVIDIA-NVFP4-no-MTP-GGUF with Hermes Agent:

Start the llama.cpp server

# Install llama.cpp:
brew install llama.cpp
# Start a local OpenAI-compatible server:
llama serve -hf QQZ2026/Qwen3.6-27B-NVIDIA-NVFP4-no-MTP-GGUF:NVFP4

Configure Hermes

# Install Hermes:
curl -fsSL https://hermes-agent.nousresearch.com/install.sh | bash
hermes setup
# Point Hermes at the local server:
hermes config set model.provider custom
hermes config set model.base_url http://127.0.0.1:8080/v1
hermes config set model.default QQZ2026/Qwen3.6-27B-NVIDIA-NVFP4-no-MTP-GGUF:NVFP4

Run Hermes

hermes

Atomic Chat

OpenClaw

How to use QQZ2026/Qwen3.6-27B-NVIDIA-NVFP4-no-MTP-GGUF with OpenClaw:

Start the llama.cpp server

# Install llama.cpp:
brew install llama.cpp
# Start a local OpenAI-compatible server:
llama serve -hf QQZ2026/Qwen3.6-27B-NVIDIA-NVFP4-no-MTP-GGUF:NVFP4

Configure OpenClaw

# Install OpenClaw:
npm install -g openclaw@latest
# Register the local server and set it as the default model:
openclaw onboard --non-interactive --mode local \
  --auth-choice custom-api-key \
  --custom-base-url http://127.0.0.1:8080/v1 \
  --custom-model-id "QQZ2026/Qwen3.6-27B-NVIDIA-NVFP4-no-MTP-GGUF:NVFP4" \
  --custom-provider-id llama-cpp \
  --custom-compatibility openai \
  --custom-text-input \
  --accept-risk \
  --skip-health

Run OpenClaw

openclaw agent --local --agent main --message "Hello from Hugging Face"

Qwen3.6-27B NVIDIA NVFP4 no-MTP for 16GB GPUs

Reproducible scripts and deployment notes for a no-MTP derivative of utautako/Qwen3.6-27B-NVIDIA-NVFP4-MTP-GGUF.

The single embedded MTP / next-token-prediction layer is physically removed with llama-quantize in COPY mode. The remaining model tensors and quantization are preserved. This saves 263,314,336 bytes (251.12 MiB) and made a verified full-GPU, 65,536-token deployment possible on one RTX 5060 Ti 16GB using quantized KV cache.

Provenance and scope

Base model: Qwen/Qwen3.6-27B
NVIDIA checkpoint: nvidia/Qwen3.6-27B-NVFP4
GGUF source conversion: utautako/Qwen3.6-27B-NVIDIA-NVFP4-MTP-GGUF
Modification and RTX 5060 Ti validation: Wilson Zhang / wilsonzhang2

This is not a fresh NVFP4 quantization and is not an official Qwen, NVIDIA, utautako, or llama.cpp release. It is a modified GGUF derivative optimized for a tight 16GB VRAM budget.

What changed

The source GGUF declares:

qwen35.block_count = 65
qwen35.nextn_predict_layers = 1

The build script:

prunes layer 64, the embedded MTP layer;
sets qwen35.block_count to 64;
sets qwen35.nextn_predict_layers to 0;
copies all retained tensors without requantizing them.

Expected output:

File: Qwen3.6-27B-NVIDIA-NVFP4-no-MTP.gguf
Size: 15,484,336,608 bytes (14.4209 GiB)
SHA256: 8e319bca3cd808c2cb61e616a124390979c5f6732b293217b4b89fd826b72d7a

Verified configuration

Item	Verified value
GPU	NVIDIA GeForce RTX 5060 Ti 16GB
Driver	580.173.02
llama.cpp	build 10023, commit `b3c9d1b84`
Context	65,536 tokens
GPU offload	all layers
KV cache	`q4_0` K + `q4_0` V
Parallel slots	1
Batch / micro-batch	512 / 64
Observed VRAM after load	approximately 15,814 MiB of 16,311 MiB
Short non-thinking generation	approximately 24.9 tok/s; single local test, not a standardized benchmark
Controlled TTFT benchmark	not yet published

The test establishes that the configuration loads and serves requests. It does not establish identical behavior on every 16GB card, driver, llama.cpp revision, display-attached GPU, or operating system.

Build the derivative

Requirements:

the upstream MTP GGUF;
a recent llama-quantize supporting --prune-layers and --override-kv;
enough free disk space for the source, temporary output, and final output;
sufficient RAM or swap for the copy operation.

scripts/build-no-mtp.sh \
  /path/to/Qwen3.6-27B-NVIDIA-NVFP4-MTP.gguf \
  /path/to/Qwen3.6-27B-NVIDIA-NVFP4-no-MTP.gguf

The script refuses to overwrite an existing destination and validates the expected output size and SHA256.

Run at 64K on one 16GB GPU

MODEL=/path/to/Qwen3.6-27B-NVIDIA-NVFP4-no-MTP.gguf \
LLAMA_SERVER=/path/to/llama-server \
scripts/run-64k.sh

Equivalent core command:

llama-server \
  --model Qwen3.6-27B-NVIDIA-NVFP4-no-MTP.gguf \
  --ctx-size 65536 \
  --n-gpu-layers all \
  --split-mode none \
  --flash-attn on \
  --cache-type-k q4_0 \
  --cache-type-v q4_0 \
  --batch-size 512 \
  --ubatch-size 64 \
  --parallel 1 \
  --threads 7 \
  --fit off \
  --spec-type none \
  --jinja

A sample systemd unit is included at systemd/qwen27b.service.

Why no MTP?

The upstream GGUF includes one MTP layer. Merely disabling speculative decoding does not necessarily remove that layer from the model file or VRAM budget. The physical removal saves about 251 MiB. On the verified RTX 5060 Ti configuration, that margin was the difference between a practical 64K full-GPU deployment and an over-constrained setup.

On GPUs with more VRAM, the original MTP build may be faster and is generally preferable when its draft layer and draft KV cache fit without forcing CPU offload.

Limitations

Text inference was tested; multimodal inference and an mmproj were not validated here.
No standardized quality regression suite has been run specifically against the no-MTP derivative. Removing a speculative draft layer should not change the retained main-model logits, but users should validate their workloads.
64K allocation is extremely tight. Desktop display usage, another CUDA process, larger batches, multiple slots, or different compute buffers can cause OOM.
The published performance number is a local observation, not a cross-platform benchmark.

License and attribution

The upstream model and conversion repositories identify Apache-2.0 as their license. This repository includes the Apache License 2.0 text and an attribution notice. Preserve upstream model-card terms and attribution when redistributing the GGUF.

The scripts and documentation authored in this repository are also released under Apache-2.0. See LICENSE, NOTICE, and ATTRIBUTION.md.

Downloads last month: 215

GGUF

Model size

27B params

Architecture

qwen35

Hardware compatibility

4-bit

Model tree for QQZ2026/Qwen3.6-27B-NVIDIA-NVFP4-no-MTP-GGUF

Base model

Qwen/Qwen3.6-27B

Quantized

(715)

this model