Instructions to use lew96123/qwen3.5-0.8b-custom-packed-turboquant_mse-true-uniform-1bit with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use lew96123/qwen3.5-0.8b-custom-packed-turboquant_mse-true-uniform-1bit with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("image-text-to-text", model="lew96123/qwen3.5-0.8b-custom-packed-turboquant_mse-true-uniform-1bit")
messages = [
    {
        "role": "user",
        "content": [
            {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"},
            {"type": "text", "text": "What animal is on the candy?"}
        ]
    },
]
pipe(text=messages)

# Load model directly
from transformers import AutoProcessor, AutoModelForMultimodalLM

processor = AutoProcessor.from_pretrained("lew96123/qwen3.5-0.8b-custom-packed-turboquant_mse-true-uniform-1bit")
model = AutoModelForMultimodalLM.from_pretrained("lew96123/qwen3.5-0.8b-custom-packed-turboquant_mse-true-uniform-1bit", device_map="auto")
messages = [
    {
        "role": "user",
        "content": [
            {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"},
            {"type": "text", "text": "What animal is on the candy?"}
        ]
    },
]
inputs = processor.apply_chat_template(
	messages,
	add_generation_prompt=True,
	tokenize=True,
	return_dict=True,
	return_tensors="pt",
).to(model.device)

outputs = model.generate(**inputs, max_new_tokens=40)
print(processor.decode(outputs[0][inputs["input_ids"].shape[-1]:]))

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use lew96123/qwen3.5-0.8b-custom-packed-turboquant_mse-true-uniform-1bit with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "lew96123/qwen3.5-0.8b-custom-packed-turboquant_mse-true-uniform-1bit"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "lew96123/qwen3.5-0.8b-custom-packed-turboquant_mse-true-uniform-1bit",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Use Docker

docker model run hf.co/lew96123/qwen3.5-0.8b-custom-packed-turboquant_mse-true-uniform-1bit

SGLang

How to use lew96123/qwen3.5-0.8b-custom-packed-turboquant_mse-true-uniform-1bit with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "lew96123/qwen3.5-0.8b-custom-packed-turboquant_mse-true-uniform-1bit" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "lew96123/qwen3.5-0.8b-custom-packed-turboquant_mse-true-uniform-1bit",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "lew96123/qwen3.5-0.8b-custom-packed-turboquant_mse-true-uniform-1bit" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "lew96123/qwen3.5-0.8b-custom-packed-turboquant_mse-true-uniform-1bit",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Docker Model Runner
How to use lew96123/qwen3.5-0.8b-custom-packed-turboquant_mse-true-uniform-1bit with Docker Model Runner:
```
docker model run hf.co/lew96123/qwen3.5-0.8b-custom-packed-turboquant_mse-true-uniform-1bit
```

lew96123/qwen3.5-0.8b-custom-packed-turboquant_mse-true-uniform-1bit

This repository is a standalone quantized release of Qwen/Qwen3.5-0.8B published by lew96123. The previous model card was only a short summary; this one is intentionally detailed so the repo page itself explains what was stored, how it was produced, what was evaluated, and what trade-offs showed up.

Release snapshot

Base model: Qwen/Qwen3.5-0.8B
Loader kind / pipeline tag: image-text-to-text / image-text-to-text
Primary target bit-width: 1
Effective quantized bit-widths actually present in the checkpoint: 1-bit (265 tensors)
Quantization method: turboquant_mse
Quantized tensors: 265
Passthrough tensors: 223
Original storage: 1.63 GiB (1,746,882,752 bytes)
Stored artifact size: 106.25 MiB (111,412,032 bytes)
Bytes saved: 1.52 GiB (1,635,470,720 bytes)
Compression ratio: 15.679x
Base perplexity: 18.2902
Quantized perplexity: 1320406.8612
Perplexity delta vs base: +1320388.5710 (72192.006x of base)
Mean last-token logit cosine similarity: -0.060028
Evaluation dataset role: sanity-test

Artifact inventory

File	Purpose
`turboquant_weights.safetensors`	Packed checkpoint payload containing quantized weights (and any passthrough tensors stored directly).
`quant_manifest.json`	Full manifest with per-parameter metadata, storage prefixes, tensor shapes, and effective bit-width decisions.
`eval_summary.json`	Serialized evaluation metrics for the base model and the quantized artifact.
`sample_generations.json`	Deterministic sample generations generated from the stored prompts in the pipeline config.
`load_quantized.py`	Self-contained Python loader that reconstructs the quantized model from this repository alone.
`requirements.txt`	Minimal dependency hints for loading this artifact.
`config.json`, tokenizer files, generation config, and related support files	Upstream support files copied so the repo can be loaded without separately downloading missing config assets.

Quantization design

This release uses TurboQuant: Online Vector Quantization with Near-optimal Distortion Rate (arXiv:2504.19874, Zandieh et al. 2025) -- online vector quantization with near-optimal distortion rate. Each weight row is treated as a d-dimensional vector and quantized using a random rotation followed by optimal scalar quantization on the induced Beta distribution coordinates (Lloyd-Max codebook).

Bit-width target for the main release: 1
Quantizer: turboquant_mse
Seed: 42
Grid size for Lloyd-Max codebook estimation: 8193
Lloyd-Max refinement iterations: 96
Quantized scalar values covered by the manifest: 873,250,816
Passthrough scalar values covered by the manifest: 187,968

Allocator policy

Allocator-driven mixed precision is disabled for this artifact, so the release stays uniform apart from any protected-weight rules.

Mixed-precision allocator: disabled
Allocation method: uniform
Allowed effective bit-widths: 1
Allocation target: None
Calibration text count: 0
Max calibration windows: None
Promoted tensors from allocator: 0

Paper context and claim boundaries

Primary reference for the quantizer idea: TurboQuant: Online Vector Quantization with Near-optimal Distortion Rate (arXiv:2504.19874, Apr 2025; Amir Zandieh, Majid Daliri, Majid Hadian, Vahab Mirrokni).
This repo adapts TurboQuant-style vector quantization to an offline full-weight checkpoint PTQ flow.
The cited TurboQuant paper's strongest empirical evidence is on vector / KV-cache quantization and nearest-neighbor search, so this artifact should be read as a paper-inspired adaptation, not as a full reproduction of the paper's published experimental setup.
Related literature that may improve later iterations, but is not implemented in this artifact:
- SpinQuant (arXiv:2405.16406) -- learned rotations for PTQ.
- APTQ (arXiv:2402.14866) and BAQ (arXiv:2506.05664) -- sensitivity-based mixed precision / bit allocation.
- GuidedQuant (arXiv:2505.07004), LeanQuant (arXiv:2407.10032), and D^2Quant (arXiv:2602.02546) -- stronger sub-4-bit weight-only PTQ strategies.
- PolarQuant (arXiv:2603.29078) and ParoQuant (arXiv:2511.10645) -- newer rotation / preprocessing directions for low-bit compression.

Protected-weight policy

Some tensors are intentionally treated as sensitive and do not follow the same policy as the main body weights.

Matching exact names:
No exact-name protected tensors were configured.
Matching name fragments:
No substring-based protected tensors were configured.
Effective behavior: Protected weights remain passthrough tensors instead of being quantized.

Protected tensors quantized in this artifact

No protected tensors were quantized in this run.

Protected tensors kept as passthrough

No protected tensors were left as passthrough due to the protection rules.

Mixed-precision tensors introduced by the protection rule

No tensors were promoted above the primary release bit-width.

Storage accounting

Quantity	Value
Original bytes	1,746,882,752
Stored bytes	111,412,032
Bytes saved	1,635,470,720
Compression ratio	15.679x
Quantized tensor count	265
Passthrough tensor count	223
Quantized scalar values	873,250,816
Passthrough scalar values	187,968

Parameter coverage

The table below shows the largest tensors recorded in the manifest. This is helpful when auditing whether the storage budget is dominated by a few very large matrices or by a broad spread of medium-sized tensors.

Tensor	Kind	Effective bit-width	Shape	Values
`model.language_model.embed_tokens.weight`	quantized	1	`248320x1024`	254,279,680
`model.visual.merger.linear_fc1.weight`	quantized	1	`3072x3072`	9,437,184
`model.language_model.layers.0.linear_attn.in_proj_qkv.weight`	quantized	1	`6144x1024`	6,291,456
`model.language_model.layers.1.linear_attn.in_proj_qkv.weight`	quantized	1	`6144x1024`	6,291,456
`model.language_model.layers.10.linear_attn.in_proj_qkv.weight`	quantized	1	`6144x1024`	6,291,456
`model.language_model.layers.12.linear_attn.in_proj_qkv.weight`	quantized	1	`6144x1024`	6,291,456
`model.language_model.layers.13.linear_attn.in_proj_qkv.weight`	quantized	1	`6144x1024`	6,291,456
`model.language_model.layers.14.linear_attn.in_proj_qkv.weight`	quantized	1	`6144x1024`	6,291,456
`model.language_model.layers.16.linear_attn.in_proj_qkv.weight`	quantized	1	`6144x1024`	6,291,456
`model.language_model.layers.17.linear_attn.in_proj_qkv.weight`	quantized	1	`6144x1024`	6,291,456
`model.language_model.layers.18.linear_attn.in_proj_qkv.weight`	quantized	1	`6144x1024`	6,291,456
`model.language_model.layers.2.linear_attn.in_proj_qkv.weight`	quantized	1	`6144x1024`	6,291,456

Evaluation protocol

The dataset below is used in this repo as a sanity / regression test set only. It is useful for catching catastrophic degradation and comparing close variants under a fixed quick pass, but it is not enough to support broad quality claims.

Item	Value
Dataset	`wikitext:wikitext-2-raw-v1`
Dataset repo	`wikitext`
Dataset config	`wikitext-2-raw-v1`
Dataset role in this repo	`sanity-test`
Split	`test`
Max context length	512
Sliding-window stride	256
Max eval windows	full pass
Logit comparison prompts	4
Sample generation prompts	3
Generation mode	deterministic greedy decoding
Max generated tokens	64

Metric comparison

Metric	Base model	Quantized model	Notes
Average NLL	2.906366	14.093450	Lower is better
Perplexity	18.2902	1320406.8612	Lower is better
Tokens scored	295,893	295,893	Sliding-window effective tokens
Windows scored	1,160	1,160	Reported when dataset evaluation is used
Mean last-token logit cosine similarity	N/A	-0.060028	Higher is better

How to load the artifact

from load_quantized import load_quantized_model, load_tokenizer

model, manifest = load_quantized_model(".", device="cuda")
tokenizer = load_tokenizer(".")

prompt = "Explain why quantization can reduce storage requirements."
rendered = tokenizer.apply_chat_template(
    [{"role": "user", "content": prompt}],
    tokenize=False,
    add_generation_prompt=True,
) if hasattr(tokenizer, "apply_chat_template") else prompt

batch = tokenizer(rendered, return_tensors="pt").to(model.device)
output = model.generate(**batch, max_new_tokens=64, do_sample=False)
print(tokenizer.decode(output[0], skip_special_tokens=True))

Sample generations

Inline sample generations are omitted for this release because the fixed prompt set produced severely degraded outputs under this quantization setting (perplexity ratio vs base: 72192.0x, mean last-token logit cosine: -0.060). The deterministic outputs are still preserved in sample_generations.json; when available, the unsanitized decode is retained in the raw_response field for auditability.

Reproducibility configuration

The pipeline fields below are the key knobs that shaped this artifact.

{
  "base_model_id": "Qwen/Qwen3.5-0.8B",
  "loader_kind": "image-text-to-text",
  "quantization_method": "turboquant_mse",
  "primary_bit_width": 1,
  "effective_quantized_bit_widths": [
    1
  ],
  "seed": 42,
  "grid_size": 8193,
  "iterations": 96,
  "eval_dataset": "wikitext:wikitext-2-raw-v1",
  "eval_dataset_role": "sanity-test",
  "eval_stride": 256,
  "max_eval_length": 512,
  "enable_sensitivity_allocation": false,
  "allocator_name": "disabled",
  "allocation_method": "uniform",
  "allocation_allowed_bit_widths": [],
  "calibration_texts": [],
  "max_generation_tokens": 64,
  "protected_weight_exact_names": [],
  "protected_weight_name_fragments": []
}

Limitations and caveats

This is not an official Qwen release.
Evaluation here is deliberately lightweight and release-focused; it is not a substitute for a broad benchmark suite.
The default Wikitext setup in this repo is a quick regression check, not a publication-grade benchmark.
The packed artifact is reconstructed into floating-point tensors at load time, so this repo is mainly a reproducible storage/research artifact rather than a kernel-level optimized inference backend.
Vision / multimodal behavior was not benchmarked separately in this run even if the upstream model supports those paths.
Strong 1-bit weight-quality claims would be misleading for this code path today; treat 1-bit results as exploratory unless broader benchmarks say otherwise.
If you are comparing against other low-bit methods, check that the protection rules, dataset slice, and evaluation window cap are aligned; otherwise apples and oranges will sneak onto the spreadsheet.

Provenance

Base model: Qwen/Qwen3.5-0.8B
Base license: apache-2.0
Upload account: lew96123
Artifact repo id: lew96123/qwen3.5-0.8b-custom-packed-turboquant_mse-true-uniform-1bit
Manifest source of truth: quant_manifest.json
Evaluation metrics source of truth: eval_summary.json
Sample outputs source of truth: sample_generations.json

Downloads last month: 10

Model tree for lew96123/qwen3.5-0.8b-custom-packed-turboquant_mse-true-uniform-1bit

Base model

Qwen/Qwen3.5-0.8B-Base

Finetuned

Qwen/Qwen3.5-0.8B

Finetuned

(397)

this model

Papers for lew96123/qwen3.5-0.8b-custom-packed-turboquant_mse-true-uniform-1bit

PolarQuant: Optimal Gaussian Weight Quantization via Hadamard Rotation for LLM Compression

Paper • 2603.29078 • Published Mar 30 • 1