README.md

metadata

base_model: Qwen/Qwen3.6-35B-A3B
license: apache-2.0
language:
  - en
tags:
  - qwen
  - qwen3
  - gguf
  - quantization
  - llama-cpp
  - local-llm
  - 12gb-vram
  - rtx3060
library_name: llama.cpp

Qwen3.6-35B-A3B - 12GB VRAM Target Quant

Overview

This is an experimental GGUF quantization of Qwen3.6-35B-A3B, created with a different goal than most traditional quant releases.

Rather than targeting a specific quantization level or quality metric, this model targets a specific VRAM footprint. The objective was simple:

Produce the highest-quality quantization possible that fits comfortably within a 12GB graphics card.

The primary target hardware is the popular 12GB NVIDIA RTX 3060, although the model should also be suitable for other GPUs with similar memory capacity.

This quantization was produced using Thireus' Quant Assign GGUF tools, allowing custom bit allocation across tensors rather than relying on a standard preset quantization scheme.

Design Goals

• Fit within approximately 12GB of VRAM.
• Maximize model quality within that size constraint.
• Leave a small amount of free VRAM for:
  • Windows desktop operation
  • Background applications
  • Driver overhead
  • Typical inference workloads
• Explore what is achievable when optimizing for a memory budget rather than a quantization target.

Important Notes

This release is primarily an experiment.

No special effort was made to optimize for:

• Extremely large context windows
• Maximum throughput
• Benchmark performance
• Quantization uniformity

The focus was simply:

"What is the best-quality Qwen3.6-35B-A3B quant that will reasonably fit on a 12GB GPU?"

As such, your results may vary depending on:

• Backend
• Context size
• GPU driver version
• Inference engine
• Additional VRAM consumption from other applications

Hardware Target

Recommended:

• NVIDIA RTX 3060 12GB
• NVIDIA RTX 4070 12GB
• AMD GPUs with comparable VRAM capacity

The quant was intentionally made slightly smaller than the theoretical 12GB limit to provide some headroom for real-world usage.

Quantization

Generated using:

• Thireus' Quant Assign GGUF tools

Quantization strategy:

• Custom tensor allocation
• Size-constrained design
• Quality prioritized within a fixed memory budget

Intended Use

This model may be useful for users who:

• Have a 12GB GPU and want the largest practical Qwen3.6 model possible.
• Prefer larger models at aggressive mixed precision over smaller models at higher precision.
• Enjoy experimenting with custom quantization strategies.
• Run local inference via llama.cpp or compatible GGUF runtimes.

Disclaimer

This is an experimental quantization and should not be considered an official release from the Qwen team.

Performance, quality, and memory usage will vary depending on your inference setup.

Feedback, benchmark results, and comparisons against standard quantizations are welcome.

Credits

• Base model: Qwen3.6-35B-A3B
• Quantization tooling: Thireus' Quant Assign GGUF tools
• GGUF ecosystem: llama.cpp and contributors

Why This Exists

Most GGUF releases answer the question:

"How much quality can I get at X bits?"

This release attempts to answer a different question:

"How much quality can I get in 12GB?"