--- license: other license_name: swift-open-license-1.0 library_name: transformers pipeline_tag: image-text-to-text base_model: ukisai/Swift-Qwen3.8-27b base_model_relation: quantized tags: - qwen3_8 - qwen3_5 - fp8 - compressed-tensors - sglang - speculative-decoding - dflash - reasoning - efficient-thinking - conversational --- # Swift-Qwen3.8-27B-FP8 FP8 quantization of [`ukisai/Swift-Qwen3.8-27b`](https://huggingface.co/ukisai/Swift-Qwen3.8-27b). This is an independent community quantization and is **not an official UkisAI release**. The goal of this checkpoint is to preserve the behavior of Swift-Qwen3.8-27B while reducing VRAM requirements and enabling high-throughput inference with SGLang, including speculative decoding with the model's native MTP head or an external DFlash2 draft model. ## Model summary - Upstream model: `ukisai/Swift-Qwen3.8-27b` - Architecture: `Qwen3_5ForConditionalGeneration` - Quantization format: `compressed-tensors` - Quantization scheme: `FP8_BLOCK` - Weight block size: `128 x 128` - Activations: dynamic FP8 - Activation group size: `128` - Quantizer: `llmcompressor 0.13.0` - Declared context length: `262,144` - Checkpoint size: approximately `29 GB` - Native MTP components: retained The checkpoint was produced from the BF16 Swift-Qwen3.8-27B model rather than requantizing an already quantized derivative. ## Quantization details The quantization process used the official Qwen3.8 FP8 configuration as a reference for the block-FP8 layout. Checkpoint audit: | Item | Count | |---|---:| | Total checkpoint tensors | 1,199 | | 2D weight tensors | 617 | | FP8 quantization candidates | 407 | | Effectively excluded / preserved modules | 626 | | Incompatible FP8 candidates after validation | 0 | Matrices that are not compatible with the required `128 x 128` block structure were preserved instead of being forcibly quantized. The following classes of tensors were intentionally preserved where appropriate: - embeddings - `lm_head` - normalization parameters - non-2D weights - matrices whose dimensions are incompatible with the FP8 block layout The native MTP layers are retained. Compatible MTP projection matrices are quantized to FP8, while incompatible components remain unquantized. ## Validation Validated locally on: - NVIDIA RTX PRO 6000 Blackwell 96 GB - SGLang `0.5.19.dev135+ga4ffb996d` - `compressed-tensors 0.18.0` - CUDA-capable Linux deployment - single-GPU tensor parallelism (`TP=1`) SGLang successfully loads the checkpoint as: ```text type=Qwen3_5ForConditionalGeneration quant=compressed-tensors ``` Observed target-model weight memory during loading: ```text 28.47 GB ``` OpenAI-compatible `/v1/chat/completions` inference was validated successfully. Multimodal inference has not yet been separately benchmarked for this quantized checkpoint. ## Performance All measurements below are local measurements from a single NVIDIA RTX PRO 6000 Blackwell 96 GB GPU. They are intended to document this deployment, not to serve as standardized cross-model benchmarks. ### Fixed 4,096-token generation Same prompt and generation setup for all configurations: | Configuration | Median throughput | |---|---:| | Swift FP8, target model only | 50.45 tok/s | | Swift FP8 + native NEXTN/MTP | 103.38 tok/s | | Swift FP8 + DFlash2, 8 draft tokens | **132.62 tok/s** | Measured DFlash2 runs: ```text 131.24 tok/s 132.65 tok/s 132.62 tok/s median: 132.62 tok/s ``` Compared with target-only generation, DFlash2 produced approximately **2.63x** higher output throughput in this test. Compared with native NEXTN/MTP, DFlash2 was approximately **28% faster**. The DFlash2 draft model is external and is **not included in this repository**. ## Reasoning-heavy agent test A separate local A/B test used the same diagnostic prompt, sampling parameters, seed, and reasoning setting for both systems. The prompt asked the model to diagnose an intermittently slow OpenAI-compatible inference deployment with high GPU utilization, large KV cache, speculative decoding, variable context sizes and concurrency-sensitive latency. | Metric | Previous Qwen FP8 production | Swift FP8 + DFlash2 | |---|---:|---:| | Wall time | 193.48 s | **139.63 s** | | Prompt tokens | 229 | 229 | | Reasoning tokens | 14,838 | **10,716** | | Completion tokens | 22,848 | **16,170** | | Finish reason | stop | stop | | Effective completion throughput | 118.09 tok/s | 115.80 tok/s | Observed change: - wall-clock time: approximately **-27.8%** - reasoning tokens: approximately **-27.8%** - completion tokens: approximately **-29.2%** The main benefit in this test was not higher raw per-token throughput. Swift reached a similarly useful diagnostic answer with substantially fewer reasoning and completion tokens. This is a local workload test and should not be interpreted as a standardized quality benchmark. ## SGLang usage ### Basic serving ```bash python -m sglang.launch_server \ --model-path /path/to/Swift-Qwen3.8-27B-FP8 \ --served-model-name Swift-Qwen3.8-27B-FP8 \ --host 0.0.0.0 \ --port 30000 \ --attention-backend flashinfer \ --reasoning-parser qwen3 \ --tool-call-parser qwen3_coder ``` ### Native NEXTN / MTP speculative decoding The retained native MTP head can be used with SGLang: ```bash python -m sglang.launch_server \ --model-path /path/to/Swift-Qwen3.8-27B-FP8 \ --served-model-name Swift-Qwen3.8-27B-FP8 \ --host 0.0.0.0 \ --port 30000 \ --attention-backend flashinfer \ --reasoning-parser qwen3 \ --tool-call-parser qwen3_coder \ --speculative-algorithm NEXTN \ --speculative-num-steps 3 \ --speculative-eagle-topk 1 \ --speculative-num-draft-tokens 4 ``` In the validated SGLang build, NEXTN is internally represented through the EAGLE speculative-decoding path. ### DFlash2 speculative decoding Best local throughput was obtained with a compatible external DFlash2 draft checkpoint: ```bash python -m sglang.launch_server \ --model-path /path/to/Swift-Qwen3.8-27B-FP8 \ --served-model-name Swift-Qwen3.8-27B-FP8 \ --host 0.0.0.0 \ --port 30000 \ --attention-backend flashinfer \ --reasoning-parser qwen3 \ --tool-call-parser qwen3_coder \ --speculative-algorithm DFLASH \ --speculative-draft-model-path /path/to/Qwen3.8-27B-DFlash2 \ --speculative-num-draft-tokens 8 ``` The DFlash2 weights are not redistributed in this repository. Because speculative decoding verifies proposed tokens against the target model, the external draft model affects acceptance rate and speed rather than replacing the target model's token distribution. ## Notes This repository contains the quantized target checkpoint only. It does not include: - a DFlash2 draft checkpoint - the original BF16 Swift checkpoint - SGLang runtime binaries or containers Performance depends heavily on GPU architecture, SGLang version, attention backend, context length, concurrency, KV-cache configuration and speculative decoding parameters. ## License This checkpoint is derived from: [`ukisai/Swift-Qwen3.8-27b`](https://huggingface.co/ukisai/Swift-Qwen3.8-27b) and follows the **Swift Open License v1.0** applicable to the upstream model. Please refer to the upstream repository and its license text for the authoritative licensing terms. No additional rights to the upstream model are granted by this repository. ## Attribution Original model: - UkisAI - `ukisai/Swift-Qwen3.8-27b` FP8 conversion, validation and local performance measurements for this repository were performed independently by the repository maintainer. ## Citation For the underlying Swift model, please cite or reference the upstream project: ```bibtex @misc{swift-qwen3.8-27b, title = {Swift-Qwen3.8-27B}, author = {UkisAI}, year = {2026}, url = {https://huggingface.co/ukisai/Swift-Qwen3.8-27b} } ```