Instructions to use hghdomer/qwen3.5-0.8b-q1_0-qat with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use hghdomer/qwen3.5-0.8b-q1_0-qat with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("image-text-to-text", model="hghdomer/qwen3.5-0.8b-q1_0-qat")
messages = [
    {
        "role": "user",
        "content": [
            {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"},
            {"type": "text", "text": "What animal is on the candy?"}
        ]
    },
]
pipe(text=messages)

# pip install -U transformers accelerate
# Load model directly
from transformers import AutoProcessor, AutoModelForMultimodalLM

processor = AutoProcessor.from_pretrained("hghdomer/qwen3.5-0.8b-q1_0-qat")
model = AutoModelForMultimodalLM.from_pretrained("hghdomer/qwen3.5-0.8b-q1_0-qat", device_map="auto")
messages = [
    {
        "role": "user",
        "content": [
            {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"},
            {"type": "text", "text": "What animal is on the candy?"}
        ]
    },
]
inputs = processor.apply_chat_template(
	messages,
	add_generation_prompt=True,
	tokenize=True,
	return_dict=True,
	return_tensors="pt",
).to(model.device)

outputs = model.generate(**inputs, max_new_tokens=256)
print(processor.decode(outputs[0][inputs["input_ids"].shape[-1]:]))

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use hghdomer/qwen3.5-0.8b-q1_0-qat with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "hghdomer/qwen3.5-0.8b-q1_0-qat"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "hghdomer/qwen3.5-0.8b-q1_0-qat",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Use Docker

docker model run hf.co/hghdomer/qwen3.5-0.8b-q1_0-qat

SGLang

How to use hghdomer/qwen3.5-0.8b-q1_0-qat with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "hghdomer/qwen3.5-0.8b-q1_0-qat" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "hghdomer/qwen3.5-0.8b-q1_0-qat",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "hghdomer/qwen3.5-0.8b-q1_0-qat" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "hghdomer/qwen3.5-0.8b-q1_0-qat",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Docker Model Runner
How to use hghdomer/qwen3.5-0.8b-q1_0-qat with Docker Model Runner:
```
docker model run hf.co/hghdomer/qwen3.5-0.8b-q1_0-qat
```

You need to agree to share your contact information to access this model

This repository is publicly accessible, but you have to accept the conditions to access its files and content.

Qwen3.5-0.8B, 1-bit Q1_0 (QAT, round 3)

This is Qwen/Qwen3.5-0.8B (revision 2fc06364) with all 150 text-decoder weight matrices (497,025,024 weights) in llama.cpp Q1_0 format: one sign bit per weight plus an fp16 mean|w| scale per 128 inputs. It was trained by quantization-aware distillation from the bf16 original.

It is a research artifact for the Overlay project, which is replicating Lamb Labs' 1-bit "Larry" and compiling models into silicon tiles.

Files

File	What
`model-0000*.safetensors` + index	Standard HF weights. The 1-bit matrices are stored as their dequantized values (sign × scale) in bf16. Loads with `Qwen3_5ForConditionalGeneration.from_pretrained`.
`q1_0.safetensors` (67 MB)	The packed 1-bit content. For each quantized matrix: `<name>.signs` uint8 `[out, in/8]` (little-endian bits, 1 = positive) and `<name>.scales` fp16 `[out, in/128]`.
`export.json`	Export manifest with SHA-256 of each file

Not quantized (bf16, as in the original):

the tied embedding / LM head (254M parameters);
the vision tower and the MTP head;
all norms, conv1d, A_log, dt_bias, in_proj_a, in_proj_b.

Training

Three rounds, 290M tokens of forward-KL distillation over the full vocabulary from the frozen bf16 teacher.

Round 1: 150M tokens, 50% FineWeb-Edu and 50% chat (UltraChat-200k plus Tülu-3 persona instruction-following).
Rounds 2 and 3: 40M and 100M tokens. 15% of the data is on-policy: the student's own answers, scored by the teacher.
Optimizer: straight-through estimator onto fp32 latent weights, AdamW, peak LR 5e-5 then 2.5e-5.
Hardware: 1× A100 / H100.

Results (vs. the bf16 teacher)

	This model	Teacher
Top-1 next-token agreement, wikitext-2 / chat	67.1% / 74.2%	100%
Perplexity, wikitext-2 / chat	29.4 / 10.85	18.6 / 8.8
IFEval prompt-strict, greedy (541 prompts, thinking off)	29.4%	54.3%
IFEval prompt-strict, repetition penalty 1.1	31.1%	50.8%
IFEval instruction-strict, repetition penalty 1.1	42.3%	62.1%

Rounding the same model to 1 bit without training gives 0.4% top-1 agreement.

Usage notes

Use a repetition penalty of about 1.1. Under greedy decoding the model falls into repetition loops on about half of IFEval's prompts; the penalty cuts that to 17%.
Chat template with thinking off: enable_thinking=False.
Expect large quality loss. It keeps only about 57% of the original's instruction-following. Small models lose much more at 1 bit than large ones, so this is not a drop-in replacement for the original.

Verification

The export reproduces the training-time forward exactly: max difference 0.0 on all 150 matrices.
q1_0.safetensors unpacks to exactly the HF weights (150/150 matrices).
Local CPU greedy output matches the GPU evaluation output.

License: Apache-2.0, inherited from Qwen3.5-0.8B.

Downloads last month: -

Safetensors

Model size

0.9B params

Tensor type

BF16

F32

Model tree for hghdomer/qwen3.5-0.8b-q1_0-qat

Base model

Qwen/Qwen3.5-0.8B-Base

Finetuned

Qwen/Qwen3.5-0.8B

Quantized

(336)

this model