Instructions to use Jianqiao1/Qwen3.6-27B-Architect-Polaris2-Fable-B-F451-MoQ-GGUF with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Notebooks
Google Colab
Kaggle
Local Apps Settings

How to use Jianqiao1/Qwen3.6-27B-Architect-Polaris2-Fable-B-F451-MoQ-GGUF with llama.cpp:

Install (macOS, Linux)

curl -LsSf https://llama.app/install.sh | sh
# Start a local OpenAI-compatible server with a web UI:
llama serve -hf Jianqiao1/Qwen3.6-27B-Architect-Polaris2-Fable-B-F451-MoQ-GGUF:BF16
# Run inference directly in the terminal:
llama cli -hf Jianqiao1/Qwen3.6-27B-Architect-Polaris2-Fable-B-F451-MoQ-GGUF:BF16

Install from WinGet (Windows)

winget install llama.cpp
# Start a local OpenAI-compatible server with a web UI:
llama serve -hf Jianqiao1/Qwen3.6-27B-Architect-Polaris2-Fable-B-F451-MoQ-GGUF:BF16
# Run inference directly in the terminal:
llama cli -hf Jianqiao1/Qwen3.6-27B-Architect-Polaris2-Fable-B-F451-MoQ-GGUF:BF16

Use pre-built binary

# Download pre-built binary from:
# https://github.com/ggerganov/llama.cpp/releases
# Start a local OpenAI-compatible server with a web UI:
./llama-server -hf Jianqiao1/Qwen3.6-27B-Architect-Polaris2-Fable-B-F451-MoQ-GGUF:BF16
# Run inference directly in the terminal:
./llama-cli -hf Jianqiao1/Qwen3.6-27B-Architect-Polaris2-Fable-B-F451-MoQ-GGUF:BF16

Build from source code

git clone https://github.com/ggerganov/llama.cpp.git
cd llama.cpp
cmake -B build
cmake --build build -j --target llama-server llama-cli
# Start a local OpenAI-compatible server with a web UI:
./build/bin/llama-server -hf Jianqiao1/Qwen3.6-27B-Architect-Polaris2-Fable-B-F451-MoQ-GGUF:BF16
# Run inference directly in the terminal:
./build/bin/llama-cli -hf Jianqiao1/Qwen3.6-27B-Architect-Polaris2-Fable-B-F451-MoQ-GGUF:BF16

Use Docker

docker model run hf.co/Jianqiao1/Qwen3.6-27B-Architect-Polaris2-Fable-B-F451-MoQ-GGUF:BF16

LM Studio
Jan

vLLM

How to use Jianqiao1/Qwen3.6-27B-Architect-Polaris2-Fable-B-F451-MoQ-GGUF with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "Jianqiao1/Qwen3.6-27B-Architect-Polaris2-Fable-B-F451-MoQ-GGUF"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "Jianqiao1/Qwen3.6-27B-Architect-Polaris2-Fable-B-F451-MoQ-GGUF",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker

docker model run hf.co/Jianqiao1/Qwen3.6-27B-Architect-Polaris2-Fable-B-F451-MoQ-GGUF:BF16

Ollama
How to use Jianqiao1/Qwen3.6-27B-Architect-Polaris2-Fable-B-F451-MoQ-GGUF with Ollama:
```
ollama run hf.co/Jianqiao1/Qwen3.6-27B-Architect-Polaris2-Fable-B-F451-MoQ-GGUF:BF16
```

Unsloth Studio

How to use Jianqiao1/Qwen3.6-27B-Architect-Polaris2-Fable-B-F451-MoQ-GGUF with Unsloth Studio:

Install Unsloth Studio (macOS, Linux, WSL)

curl -fsSL https://unsloth.ai/install.sh | sh
# Run unsloth studio
unsloth studio -H 0.0.0.0 -p 8888
# Then open http://localhost:8888 in your browser
# Search for Jianqiao1/Qwen3.6-27B-Architect-Polaris2-Fable-B-F451-MoQ-GGUF to start chatting

Install Unsloth Studio (Windows)

irm https://unsloth.ai/install.ps1 | iex
# Run unsloth studio
unsloth studio -H 0.0.0.0 -p 8888
# Then open http://localhost:8888 in your browser
# Search for Jianqiao1/Qwen3.6-27B-Architect-Polaris2-Fable-B-F451-MoQ-GGUF to start chatting

Using HuggingFace Spaces for Unsloth

# No setup required
# Open https://huggingface.co/spaces/unsloth/studio in your browser
# Search for Jianqiao1/Qwen3.6-27B-Architect-Polaris2-Fable-B-F451-MoQ-GGUF to start chatting

How to use Jianqiao1/Qwen3.6-27B-Architect-Polaris2-Fable-B-F451-MoQ-GGUF with Pi:

Start the llama.cpp server

# Install llama.cpp:
brew install llama.cpp
# Start a local OpenAI-compatible server:
llama serve -hf Jianqiao1/Qwen3.6-27B-Architect-Polaris2-Fable-B-F451-MoQ-GGUF:BF16

Configure the model in Pi

# Install Pi:
npm install -g @mariozechner/pi-coding-agent
# Add to ~/.pi/agent/models.json:
{
  "providers": {
    "llama-cpp": {
      "baseUrl": "http://localhost:8080/v1",
      "api": "openai-completions",
      "apiKey": "none",
      "models": [
        {
          "id": "Jianqiao1/Qwen3.6-27B-Architect-Polaris2-Fable-B-F451-MoQ-GGUF:BF16"
        }
      ]
    }
  }
}

Run Pi

# Start Pi in your project directory:
pi

Hermes Agent new

How to use Jianqiao1/Qwen3.6-27B-Architect-Polaris2-Fable-B-F451-MoQ-GGUF with Hermes Agent:

Start the llama.cpp server

# Install llama.cpp:
brew install llama.cpp
# Start a local OpenAI-compatible server:
llama serve -hf Jianqiao1/Qwen3.6-27B-Architect-Polaris2-Fable-B-F451-MoQ-GGUF:BF16

Configure Hermes

# Install Hermes:
curl -fsSL https://hermes-agent.nousresearch.com/install.sh | bash
hermes setup
# Point Hermes at the local server:
hermes config set model.provider custom
hermes config set model.base_url http://127.0.0.1:8080/v1
hermes config set model.default Jianqiao1/Qwen3.6-27B-Architect-Polaris2-Fable-B-F451-MoQ-GGUF:BF16

Run Hermes

hermes

Atomic Chat new

OpenClaw new

How to use Jianqiao1/Qwen3.6-27B-Architect-Polaris2-Fable-B-F451-MoQ-GGUF with OpenClaw:

Start the llama.cpp server

# Install llama.cpp:
brew install llama.cpp
# Start a local OpenAI-compatible server:
llama serve -hf Jianqiao1/Qwen3.6-27B-Architect-Polaris2-Fable-B-F451-MoQ-GGUF:BF16

Configure OpenClaw

# Install OpenClaw:
npm install -g openclaw@latest
# Register the local server and set it as the default model:
openclaw onboard --non-interactive --mode local \
  --auth-choice custom-api-key \
  --custom-base-url http://127.0.0.1:8080/v1 \
  --custom-model-id "Jianqiao1/Qwen3.6-27B-Architect-Polaris2-Fable-B-F451-MoQ-GGUF:BF16" \
  --custom-provider-id llama-cpp \
  --custom-compatibility openai \
  --custom-text-input \
  --accept-risk \
  --skip-health

Run OpenClaw

openclaw agent --local --agent main --message "Hello from Hugging Face"

Docker Model Runner
How to use Jianqiao1/Qwen3.6-27B-Architect-Polaris2-Fable-B-F451-MoQ-GGUF with Docker Model Runner:
```
docker model run hf.co/Jianqiao1/Qwen3.6-27B-Architect-Polaris2-Fable-B-F451-MoQ-GGUF:BF16
```

Lemonade

How to use Jianqiao1/Qwen3.6-27B-Architect-Polaris2-Fable-B-F451-MoQ-GGUF with Lemonade:

Pull the model

# Download Lemonade from https://lemonade-server.ai/
lemonade pull Jianqiao1/Qwen3.6-27B-Architect-Polaris2-Fable-B-F451-MoQ-GGUF:BF16

Run and chat with the model

lemonade run user.Qwen3.6-27B-Architect-Polaris2-Fable-B-F451-MoQ-GGUF-BF16

List all available models

lemonade list

Jianqiao1 commited on 4 days ago

Commit

95999d8

verified ·

1 Parent(s): df8d515

Upload README.md with huggingface_hub

Browse files

Files changed (1) hide show

README.md +151 -0

README.md CHANGED Viewed

@@ -1,3 +1,154 @@
 ---
 license: apache-2.0
 ---

 ---
 license: apache-2.0
+base_model: nightmedia/Qwen3.6-27B-Architect-Polaris2-Fable-B-F451
+base_model_relation: quantized
+pipeline_tag: text-generation
+tags:
+  - gguf
+  - llama.cpp
+  - qwen3.6
+  - quantized
+  - moq
+  - mtp
+  - nvfp4
 ---
+# Qwen3.6-27B Architect Polaris2 Fable B F451 - MTP MoQ GGUF
+Tensor-aware mixed-precision GGUF quantizations of [nightmedia/Qwen3.6-27B-Architect-Polaris2-Fable-B-F451](https://huggingface.co/nightmedia/Qwen3.6-27B-Architect-Polaris2-Fable-B-F451), including the model's MTP tensors.
+This repository provides several MoQ size/quality points, an experimental NVFP4 variant, the c128 importance matrix used during quantization, and the complete interactive evaluation report.
+## Quantization method
+MoQ uses tensor-aware mixed quantization instead of assigning one quantization type to the entire model.
+- Weight tensors are grouped by their structural role and sensitivity.
+- Per-group KLD sensitivity studies are used to decide which groups require higher precision.
+- Less sensitive groups use more compact formats, while sensitive groups are protected with higher-precision formats.
+- A 128-chunk importance matrix is incorporated into the quantization process.
+- The large MTP tensors are explicitly retained at high precision.
+- Every output was audited against its tensor recipe: all 866 tensor names, shapes, and quantization types matched.
+The `MTP-MoQ-NVFP4` variant uses NVFP4 for selected tensor groups while retaining the rest of the mixed-precision policy. On an RTX 5090 it improves short-prompt prefill throughput substantially, although the regular MoQ-4.8 model provides better KLD quality at the same file size.
+No single quant is best for every use case:
+- `MTP-MoQ-4.6` is a compact balanced option.
+- `MTP-MoQ-4.8` is the recommended quality/size point around 16.15 GB.
+- `MTP-MoQ-5.1` provides the best quality in the regular MoQ series.
+- `MTP-MoQ-NVFP4` is intended for testing NVFP4 acceleration, especially on Blackwell GPUs.
+## Files
+Sizes are decimal GB. Payload BPW is calculated from the actual GGUF tensor payload and excludes the GGUF header.
+| File | Size GB | Payload BPW | PPL | Mean KLD | p999 KLD |
+|---|---:|---:|---:|---:|---:|
+| `Qwen3.6-27B-Architect-Polaris2-Fable-B-F451-MTP-MoQ-3.2.gguf` | 10.810 | 3.1621 | 6.805998 | 0.097714 | 2.751078 |
+| `Qwen3.6-27B-Architect-Polaris2-Fable-B-F451-MTP-MoQ-3.6.gguf` | 12.134 | 3.5500 | 6.661694 | 0.069665 | 1.796203 |
+| `Qwen3.6-27B-Architect-Polaris2-Fable-B-F451-MTP-MoQ-3.8.gguf` | 12.840 | 3.7566 | 6.518863 | 0.044993 | 1.291825 |
+| `Qwen3.6-27B-Architect-Polaris2-Fable-B-F451-MTP-MoQ-4.1.gguf` | 14.266 | 4.1740 | 6.447097 | 0.027318 | 0.807045 |
+| `Qwen3.6-27B-Architect-Polaris2-Fable-B-F451-MTP-MoQ-4.3.gguf` | 15.002 | 4.3897 | 6.421027 | 0.019253 | 0.591232 |
+| `Qwen3.6-27B-Architect-Polaris2-Fable-B-F451-MTP-MoQ-4.6.gguf` | 15.242 | 4.4599 | 6.392239 | 0.015145 | 0.505663 |
+| `Qwen3.6-27B-Architect-Polaris2-Fable-B-F451-MTP-MoQ-4.8.gguf` | 16.150 | 4.7258 | 6.379456 | 0.012836 | 0.423877 |
+| `Qwen3.6-27B-Architect-Polaris2-Fable-B-F451-MTP-MoQ-4.9.gguf` | 16.524 | 4.8353 | 6.384560 | 0.012218 | 0.402534 |
+| `Qwen3.6-27B-Architect-Polaris2-Fable-B-F451-MTP-MoQ-5.1.gguf` | 17.452 | 5.1069 | 6.371239 | 0.009613 | 0.325677 |
+| `Qwen3.6-27B-Architect-Polaris2-Fable-B-F451-MTP-MoQ-NVFP4.gguf` | 16.150 | 4.7258 | 6.397915 | 0.015615 | 0.518097 |
+Additional files:
+- `Qwen3.6-27B-Architect-Polaris2-Fable-B-F451-BF16.gguf`: BF16 GGUF reference model.
+- `Qwen3.6-27B-Architect-Polaris2-Fable-B-F451.c128.imatrix.gguf`: the 128-chunk importance matrix used for these quantizations.
+- `Jianqiao1_MoQ_vs_DavidAU_GGUF_report.html`: self-contained interactive report with zoomable and pannable charts.
+## Quality evaluation
+All quality results were measured under the same conditions:
+- WikiText-2 raw test set
+- Context length 512
+- 580 evaluation chunks
+- KLD calculated against logits from the BF16 model
+- BF16 reference PPL: `6.320035`
+Lower PPL, Mean KLD, and p999 KLD are better.
+### MoQ results
+| Model | Size GB | Payload BPW | PPL | Mean KLD | p999 KLD |
+|---|---:|---:|---:|---:|---:|
+| MTP-MoQ-3.2 | 10.810 | 3.1621 | 6.805998 | 0.097714 | 2.751078 |
+| MTP-MoQ-3.6 | 12.134 | 3.5500 | 6.661694 | 0.069665 | 1.796203 |
+| MTP-MoQ-3.8 | 12.840 | 3.7566 | 6.518863 | 0.044993 | 1.291825 |
+| MTP-MoQ-4.1 | 14.266 | 4.1740 | 6.447097 | 0.027318 | 0.807045 |
+| MTP-MoQ-4.3 | 15.002 | 4.3897 | 6.421027 | 0.019253 | 0.591232 |
+| MTP-MoQ-4.6 | 15.242 | 4.4599 | 6.392239 | 0.015145 | 0.505663 |
+| MTP-MoQ-4.8 | 16.150 | 4.7258 | 6.379456 | 0.012836 | 0.423877 |
+| MTP-MoQ-4.9 | 16.524 | 4.8353 | 6.384560 | 0.012218 | 0.402534 |
+| MTP-MoQ-5.1 | 17.452 | 5.1069 | 6.371239 | 0.009613 | 0.325677 |
+| MTP-MoQ-NVFP4 | 16.150 | 4.7258 | 6.397915 | 0.015615 | 0.518097 |
+### Comparison with DavidAU Fable Fusion 711 GGUF
+The following DavidAU GGUF files were evaluated locally using the same BF16 reference logits and the same evaluation settings.
+| DavidAU quant | Size GB | Payload BPW | PPL | Mean KLD | p999 KLD |
+|---|---:|---:|---:|---:|---:|
+| IQ2_M | 12.125 | 3.5471 | 7.100246 | 0.140480 | 3.807066 |
+| IQ3_M | 14.532 | 4.2520 | 6.545219 | 0.047823 | 1.487676 |
+| IQ4_XS | 17.034 | 4.9846 | 6.387851 | 0.013953 | 0.514973 |
+| Q4_K_S | 17.537 | 5.1321 | 6.378748 | 0.013965 | 0.492694 |
+| IQ4_NL | 17.753 | 5.1952 | 6.386419 | 0.013701 | 0.492569 |
+| Q4_K_M | 18.499 | 5.4135 | 6.365021 | 0.011404 | 0.412386 |
+| Q5_K_S | 20.631 | 6.0379 | 6.341159 | 0.005466 | 0.211335 |
+| Q5_K_M | 21.182 | 6.1993 | 6.337103 | 0.004809 | 0.179674 |
+| Q6_K | 24.034 | 7.0343 | 6.324763 | 0.001467 | 0.059772 |
+### Near-size comparison
+Positive percentages mean the MoQ model is smaller or has a lower metric.
+| MoQ model | DavidAU model | Size difference | Mean KLD lower | p999 KLD lower | PPL lower |
+|---|---|---:|---:|---:|---:|
+| MTP-MoQ-3.6 | IQ2_M | -0.08% | 50.41% | 52.82% | 6.18% |
+| MTP-MoQ-4.1 | IQ3_M | 1.83% | 42.88% | 45.75% | 1.50% |
+| MTP-MoQ-4.8 | IQ4_XS | 5.19% | 8.01% | 17.69% | 0.13% |
+| MTP-MoQ-5.1 | Q4_K_S | 0.49% | 31.16% | 33.90% | 0.12% |
+## Quality charts
+### Model size vs p999 KLD
+![Model size vs p999 KLD](p999_kld.png)
+### Model size vs Mean KLD
+![Model size vs Mean KLD](mean_kld.png)
+### Model size vs WikiText-2 PPL
+![Model size vs WikiText-2 PPL](ppl.png)
+For interactive zooming, panning, series selection, and detailed hover values, download and open [`Jianqiao1_MoQ_vs_DavidAU_GGUF_report.html`](Jianqiao1_MoQ_vs_DavidAU_GGUF_report.html).
+## RTX 5090 performance
+Throughput was measured with f16 KV cache and three repetitions. `MTP-MoQ-NVFP4` and `MTP-MoQ-4.8` have exactly the same 16.150 GB file size.
+| Model | pp512 tok/s | tg128 tok/s | pg32768,256 tok/s |
+|---|---:|---:|---:|
+| MTP-MoQ-NVFP4 | 3256.97 | 78.92 | 2377.72 |
+| MTP-MoQ-4.8 | 2669.45 | 78.65 | 2356.68 |
+| MTP-MoQ-4.6 | 2731.86 | 82.87 | 2400.43 |
+| NVFP4 vs MoQ-4.8 | +22.01% | +0.34% | +0.89% |
+## License
+This repository is released under the Apache License 2.0. Users should also review and comply with the terms of the original model.
+## Acknowledgements
+- Thanks to [nightmedia](https://huggingface.co/nightmedia) for the original Qwen3.6-27B Architect Polaris2 Fable B F451 model.
+- Thanks to DavidAU for the Fable Fusion 711 GGUF series used in the comparison.
+- Thanks to the [Qwen team](https://huggingface.co/Qwen) for Qwen3.6.