--- license: apache-2.0 base_model: - Qwen/Qwen3-VL-8B-Instruct base_model_relation: quantized library_name: transformers pipeline_tag: image-text-to-text tags: - sdnq - qwen3_vl - int8 - static-quantization - quantized - boogu-image - text-encoder --- # Qwen3-VL-8B-Instruct SDNQ INT8 static SDNQ static weight-only **int8** quantization of [Qwen/Qwen3-VL-8B-Instruct](https://huggingface.co/Qwen/Qwen3-VL-8B-Instruct) (snapshot `0c351dd01ed87e9c1b53cbc748cba10e6187ff3b`). Built as the instruction encoder (`mllm`) for the Boogu Image 0.1 pipelines: the `mllm/` shipped in [WaveCut/Boogu-Image-0.1-Turbo-SDNQ-uint4-static](https://huggingface.co/WaveCut/Boogu-Image-0.1-Turbo-SDNQ-uint4-static) (and the Edit variant) is the stock Qwen3-VL-8B-Instruct byte for byte, so this repo is a drop-in replacement that cuts the encoder's host RAM from ~17.5 GiB (bf16) to ~9.4 GiB on disk / in RAM. It is a complete Qwen3-VL-8B-Instruct checkpoint (processor + tokenizer included) and loads with plain `transformers` after `import sdnq`. ## Recipe - SDNQ `0.2.5` static quantization, `weights_dtype=int8`, `group_size=0` (auto), `dequantize_fp32=True`. - Kept in bf16: embeddings, `lm_head`, `multi_modal_projector`, vision `patch_embed` and every module below SDNQ's size floor. - Quantized layers: `{"int8": 368}`; quantized params: `{"int8": 7517270016}`; saved size 9.37 GiB (bf16 source 16.33 GiB). - Save/load round trip verified through `sdnq.loader.load_sdnq_model` (all parameters bit-identical); encoder `last_hidden_state` vs bf16 on a mixed RU/EN/ZH prompt: {"cos_mean": 0.9716747403144836, "cos_min": 0.7031210064888, "max_abs": 30.71875, "rel_l2": 0.24023021757602692}. - Runtime used for all numbers below: torch `2.10.0+cu130`, transformers `4.57.1`, diffusers `0.39.0.dev0`, sdnq `0.2.5`, `NVIDIA GeForce RTX 4090` (RunPod, disposable pod), Python 3.12.3. ## Boogu Image 0.1 Turbo A/B (same seed, bf16 encoder vs this encoder) Pipeline `BooguImageTurboPipeline` from `WaveCut/Boogu-Image-0.1-Turbo-SDNQ-uint4-static` @ `422ac37ad305169703fd8f847ef76418cab8363f` (SDNQ uint4 transformer), 1024x1024, 4 DMD steps, `text_guidance_scale=1.0`, `image_guidance_scale=1.0`, `empty_instruction_guidance_scale=0.0`, `dmd_conditioning_sigma=0.001`, `max_sequence_length=1280`, sequential CPU offload, SDNQ quantized matmul on for the transformer. 12 prompts (EN/RU/ZH, typography, diagrams, long prompt); image metrics are against the bf16-encoder image of the same prompt and seed, embedding metrics compare the encoder output that is fed to the transformer. Encoder rows ending in `-wo` run **weight-only** (quantized weights dequantized per layer on the GPU, `use_quantized_matmul=False`); rows without the suffix also quantize the activations to int8 for the matmul, which is measurably worse for this encoder (Qwen3-VL activation outliers) and is not the recommended mode. | Encoder | Gen mean s | Encode mean s | Peak VRAM MiB | Host RSS after load GiB | RSS anon GiB | PSNR | SSIM | LPIPS | Embed cos | Embed rel-L2 | |---|---:|---:|---:|---:|---:|---:|---:|---:|---:|---:| | bf16 (reference) | 20.01 | 1.27 | 2446 | 1.33 | 0.83 | - | - | - | - | - | | **int8-wo** (this repo) | 18.86 | 1.65 | 2446 | 1.34 | 0.83 | 20.40 | 0.7507 | 0.1837 | 0.99901 | 0.03786 | | int8 | 19.48 | 3.52 | 2446 | 1.34 | 0.83 | 14.66 | 0.6070 | 0.3689 | 0.96518 | 0.23915 | | uint4-svd32 | 18.85 | 3.23 | 2446 | 1.58 | 0.92 | 13.72 | 0.5794 | 0.4101 | 0.91081 | 0.36603 | | uint4-svd32-wo | 18.80 | 1.87 | 2446 | 1.57 | 0.91 | 13.90 | 0.5969 | 0.3834 | 0.92782 | 0.32357 | Host RSS is the whole pipeline process (transformer + VAE + encoder) after load with sequential offload, measured from `/proc/self/status`. ### Per prompt (int8-wo) | Prompt | PSNR | SSIM | LPIPS | Embed cos | Embed rel-L2 | |---|---:|---:|---:|---:|---:| | 00_vase | 34.89 | 0.9766 | 0.0138 | 0.99899 | 0.03695 | | 01_portrait | 18.32 | 0.6910 | 0.2393 | 0.99904 | 0.03652 | | 02_abstract | 19.80 | 0.6543 | 0.2766 | 0.99887 | 0.03938 | | 03_typography_en | 17.92 | 0.7703 | 0.1275 | 0.99906 | 0.03706 | | 04_text_ru | 18.21 | 0.7628 | 0.1471 | 0.99918 | 0.03715 | | 05_diagram | 17.15 | 0.8007 | 0.2020 | 0.99893 | 0.03835 | | 06_product | 21.88 | 0.8377 | 0.2001 | 0.99894 | 0.03795 | | 07_anime | 14.41 | 0.5586 | 0.2380 | 0.99891 | 0.03882 | | 08_landscape | 24.23 | 0.8055 | 0.2121 | 0.99893 | 0.03843 | | 09_multiobject | 19.96 | 0.7162 | 0.1882 | 0.99903 | 0.03771 | | 10_long | 20.10 | 0.6254 | 0.2218 | 0.99933 | 0.03286 | | 11_zh | 17.93 | 0.8093 | 0.1378 | 0.99894 | 0.04319 | [![Contact sheet](comparison/contact_sheet.jpg)](comparison/contact_sheet.jpg) Pairwise images (bf16 | quantized) are in `comparison/`. ## Usage ```python import torch import sdnq # registers the SDNQ quantizer with transformers from transformers import AutoProcessor, Qwen3VLForConditionalGeneration repo = "WaveCut/Qwen3-VL-8B-Instruct-SDNQ-int8-static" processor = AutoProcessor.from_pretrained(repo) model = Qwen3VLForConditionalGeneration.from_pretrained(repo, dtype=torch.bfloat16, device_map="cuda").eval() ``` Loader used by the aifarm draw service (weights stay on CPU, sequential offload moves layers per forward): ```python from sdnq.loader import load_sdnq_model from huggingface_hub import snapshot_download # model_cls=None: sdnq 0.2.5 crashes on an explicit transformers class (fixed in 0.2.6); # use_quantized_matmul=False: weight-only, keeps the encoder output at cos ~0.999 vs bf16. mllm = load_sdnq_model(snapshot_download(repo), model_cls=None, dtype=torch.bfloat16, device="cpu", use_quantized_matmul=False) ``` ## Files - `model-*.safetensors`, `config.json` (with `quantization_config`), `quantization_config.json` - SDNQ checkpoint. - `quantization_summary.json` - recipe, layer counts, storage, save/load checks. - `benchmark/summary.json`, `benchmark/*.metrics.json` - raw A/B metrics; `benchmark/prompts.json` - the prompt set. - `comparison/` - pairwise images and the contact sheet. - `SHA256SUMS` - checksums of the checkpoint files.