--- base_model: trohrbaugh/Qwen3.8-27B-heretic-ara library_name: transformers pipeline_tag: text-generation language: - en tags: - qwen - qwen3.8 - heretic - gptq - int4 - w4a16 - g128 - compressed-tensors - marlin - fp8 - w4a8 - vllm - blackwell - local-llm --- # Qwen3.8-27B-Heretic-GPTQ-W4A16-G128-Marlin A GPTQ INT4 quantized derivative of: **`trohrbaugh/Qwen3.8-27B-heretic-ara`** This build was created for efficient local inference with vLLM, with particular emphasis on fitting a capable 27B-class model onto a 24 GB GPU while retaining a large context window and strong decode performance. The model is stored using the `compressed-tensors` `pack-quantized` format and is compatible with vLLM's Marlin WNA16 inference path. On NVIDIA Blackwell SM120, the currently recommended runtime configuration uses the same INT4 checkpoint with **dynamic FP8 Marlin input activations** via: ```bash VLLM_MARLIN_INPUT_DTYPE=fp8 ``` This improves long-context prefill performance without changing the stored checkpoint format. --- ## Model Summary | Property | Value | |---|---| | Upstream model | `trohrbaugh/Qwen3.8-27B-heretic-ara` | | Quantization algorithm | GPTQ | | Stored weight precision | INT4 | | Checkpoint quantization | W4A16 | | Group size | 128 | | Symmetric | Yes | | Quantization strategy | Per-group | | Storage format | `compressed-tensors` | | Packed format | `pack-quantized` | | Default vLLM kernel | Marlin WNA16 | | Recommended Blackwell runtime | Marlin W4 + dynamic FP8 activations | | Tested GPU | NVIDIA RTX PRO 4000 Blackwell 24 GB | | Compute capability | SM120 | | Tested context | 76,800 tokens | | KV cache | FP8 | | Primary runtime | vLLM | --- ## Quantization This model was quantized using a GPTQ recipe and saved using the `compressed-tensors` format. The included `recipe.yaml` records the quantization configuration. ### GPTQ Configuration ```yaml GPTQModifier: config_groups: group_128: targets: - Linear weights: num_bits: 4 type: int symmetric: true group_size: 128 strategy: group block_structure: null dynamic: false actorder: weight scale_dtype: null zp_dtype: null observer: memoryless_minmax observer_kwargs: {} input_activations: null output_activations: null targets: - Linear ignore: - lm_head bypass_divisibility_checks: false requires_calibration_data: true block_size: 128 dampening_frac: 0.01 actorder: static offload_hessians: false ``` The resulting model configuration records: ```text quant_method: compressed-tensors format: pack-quantized quantization_status: compressed num_bits: 4 group_size: 128 strategy: group symmetric: true type: int ``` Some linear-attention components and `lm_head` are excluded from INT4 quantization, as recorded in the final `quantization_config` contained in `config.json`. The checkpoint itself is best described as: **GPTQ W4A16 INT4, Group Size 128, stored using compressed-tensors, Marlin-compatible.** The optional FP8 runtime mode described below does **not** change the checkpoint into a separately quantized W4A8 model. --- ## Marlin / vLLM This model has been tested with vLLM. For compatible quantized layers, vLLM selected: ```text MarlinLinearKernel for CompressedTensorsWNA16 ``` Marlin is the optimized inference kernel used at runtime. It is not the model's storage format. ```text Model storage: compressed-tensors / pack-quantized Checkpoint quantization: GPTQ W4A16 INT4 G128 Default runtime kernel: Marlin WNA16 ``` ### Recommended Blackwell Runtime On NVIDIA Blackwell SM120, improved prefill performance was measured using: ```bash VLLM_MARLIN_INPUT_DTYPE=fp8 ``` This keeps the model's INT4 weights unchanged while dynamically quantizing Marlin input activations to FP8 during inference. Conceptually: ```text Stored checkpoint: INT4 W4 / Group Size 128 Standard runtime: INT4 weights + BF16 activations | v Marlin W4A16 Recommended SM120 runtime: INT4 weights + dynamic FP8 activations | v Marlin W4A8-FP8 runtime path ``` The term **W4A8-FP8** in this model card refers to the runtime execution path only. It does not describe a different on-disk quantization format. --- ## Tested Hardware ```text GPU: NVIDIA RTX PRO 4000 Blackwell VRAM: 24 GB Compute capability: SM120 Runtime: vLLM Checkpoint quantization: GPTQ W4A16 G128 Storage: compressed-tensors Kernel: Marlin Recommended activation runtime: FP8 KV cache: FP8 Prefix caching: Enabled Maximum configured context: 76,800 tokens Maximum sequences: 1 Maximum batched prefill tokens: 8,192 ``` The tested configuration successfully processed 64,000-token cold-prefill requests while retaining a configured maximum context of 76,800 tokens. --- ## Blackwell Performance The following measurements were collected on an NVIDIA RTX PRO 4000 Blackwell using the same checkpoint. ### Cold TTFT | Context | Standard Marlin W4A16 | Marlin W4A8-FP8 Runtime | Reduction | |---:|---:|---:|---:| | 2,048 | 1.485 s | 0.947 s | 36.2% | | 16,384 | 9.808 s | 7.251 s | 26.1% | | 32,768 | 12.957 s | 9.776 s | 24.6% | | 64,000* | 51.704 s | 39.346 s | 23.9% | \*The 64K comparison used a unique-prefix cold-prefill workload to avoid prefix-cache reuse between requests. ### Effective 64K Prefill Throughput ```text Standard W4A16: ~1,238 tokens/s W4A8-FP8 runtime: ~1,627 tokens/s ``` This represents approximately a **31% increase in effective cold-prefill throughput** for the tested 64K workload. ### Decode Performance Three 512-token generation runs using the recommended FP8 activation runtime produced: ```text Run 1: 33.69 tokens/s Run 2: 33.65 tokens/s Run 3: 33.54 tokens/s Mean: ~33.63 tokens/s ``` The FP8 Marlin activation path therefore provided a substantial reduction in long-context prefill latency without a material loss in decode throughput in this test environment. Benchmark results should not be assumed to generalize to different GPUs, drivers, CUDA versions, vLLM builds, prompts, context structures, or concurrency levels. --- ## Performance Profiling Profiling of the standard W4A16 runtime at approximately 64K context showed that the dominant GPU execution costs were: ```text Marlin quantized GEMMs: ~69% FlashInfer attention: ~23% GDN / linear attention: small relative contribution Host-to-device transfers: negligible ``` This indicated that the primary long-context prefill bottleneck was the W4A16 Marlin GEMM path rather than PCIe transfer overhead, KV-cache management, or GDN execution. Using dynamic FP8 Marlin activations reduced the measured 64K cold-prefill TTFT from approximately: ```text 51.704 seconds ``` to: ```text 39.346 seconds ``` on the tested SM120 system. --- ## Intended Use This model is intended for legitimate research, development, and experimentation, including: - Local LLM inference - AI research - LLM evaluation - Agentic AI experimentation - Software engineering - Programming assistance - Model robustness evaluation The model's reduced refusal behaviour may make it particularly useful for controlled research where excessive model refusals interfere with legitimate analysis. --- ## Responsible Use and High-Risk Systems This model should not be used to facilitate malicious cyber activity, including unauthorized access, exploitation, credential theft, malware deployment, destructive actions, or other activity intended to compromise systems, networks, data, or users without authorization. Use of this model does not provide authorization to test, access, modify, or interfere with third-party systems. Users are responsible for obtaining appropriate permission and for complying with applicable laws, regulations, contracts, organizational policies, and rules of engagement. This model should also not be relied upon as the sole decision-making or control component in high-risk or risk-averse environments where errors, hallucinations, or unexpected behaviour could cause significant harm. Examples include: - Medical diagnosis, treatment, triage, or clinical decision support - Defence, military, weapons, targeting, mission-critical, or safety-critical systems - Critical infrastructure control - Aviation, transport, industrial control, or other safety-critical systems - Emergency response and life-safety systems - High-impact legal, financial, or regulatory decision-making If the model is used in or around such environments, it should be treated as an assistive research or analysis tool only, with appropriate human oversight, independent verification, access controls, testing, logging, and fail-safe mechanisms. Model outputs should not be treated as authoritative, deterministic, or safe by default. --- ## Model Behaviour This is a Heretic-derived model and may exhibit fewer refusals than conventional instruction-tuned models. That does not imply that model outputs are: - Correct - Safe - Legal - Ethical - Reliable - Appropriate for production use The model may generate incorrect, misleading, offensive, unsafe, or otherwise undesirable output. Outputs should be independently reviewed before use in consequential environments. --- ## Quantization Considerations Quantization trades some numerical precision for substantially reduced memory requirements and improved deployment efficiency. Compared with the original higher-precision model, this build may exhibit differences in: - Reasoning quality - Long-context behaviour - Numerical accuracy - Code generation - Rare-token prediction - Tool-use reliability - Instruction following - Generation consistency The optional FP8 activation runtime introduces an additional numerical precision trade-off compared with the standard BF16 activation path. Basic deterministic sanity testing on the tested SM120 system produced coherent output, but users should independently evaluate task-specific model quality and accuracy before relying on this configuration. Users requiring maximum model fidelity should evaluate the original higher-precision upstream model. --- ## Prefix Caching The tested vLLM configuration enables: ```bash --enable-prefix-caching ``` This can reduce repeated prefill work when requests reuse a substantial common prompt prefix, particularly for long agent system prompts, persistent personas, tool schemas, repeated instructions, and multi-turn conversations. Cold-prefill benchmark measurements described above used unique prompt prefixes where stated to avoid incorrectly attributing prefix-cache reuse to kernel performance. --- ## Model Lineage ```text Qwen3.8-27B | v trohrbaugh/Qwen3.8-27B-heretic-ara | v GPTQ calibration | v INT4 weight quantization | v W4A16 / Group Size 128 | v compressed-tensors / pack-quantized | v vLLM / Marlin | +-----------------------------+ | | v v BF16 activation runtime Dynamic FP8 activation runtime Marlin W4A16 Marlin W4A8-FP8 Recommended on tested SM120 GPU ``` --- ## Repository Contents ```text README.md chat_template.jinja config.json generation_config.json model.safetensors recipe.yaml tokenizer.json tokenizer_config.json ``` The included `recipe.yaml` records the GPTQ quantization recipe used for this build. --- ## Quantization Metadata The final `config.json` contains: ```json { "format": "pack-quantized", "quant_method": "compressed-tensors", "quantization_status": "compressed" } ``` The primary weight configuration is: ```json { "num_bits": 4, "type": "int", "symmetric": true, "group_size": 128, "strategy": "group", "dynamic": false, "actorder": "weight" } ``` The quantization recipe identifies GPTQ through `GPTQModifier`. Therefore the checkpoint itself should be described as: **GPTQ INT4 W4A16 G128 stored in compressed-tensors format and compatible with vLLM's Marlin WNA16 inference path.** On tested NVIDIA Blackwell SM120 hardware, the recommended runtime uses the same checkpoint with dynamic FP8 Marlin input activations. --- ## Compatibility Primary tested runtime: - vLLM Expected baseline use is with runtimes supporting: - `compressed-tensors` - Packed INT4 weights - W4A16 inference For the tested Blackwell-optimized runtime, support is additionally required for: - Marlin - Dynamic FP8 Marlin input activations - NVIDIA SM120 - `VLLM_MARLIN_INPUT_DTYPE=fp8` Marlin acceleration and FP8 runtime support depend on runtime version, CUDA environment, GPU architecture, and kernel availability. Other runtimes may load the checkpoint without supporting the same optimized execution path. --- ## Attribution Quantization and packaging: **GreyArea128** Upstream model: **`trohrbaugh/Qwen3.8-27B-heretic-ara`** Credit belongs to the respective authors and organizations responsible for the upstream model, architecture, training, modifications, and supporting software. --- ## Licensing This repository is a derivative model. Users must review and comply with the licenses, notices, terms, and usage requirements applicable to: 1. `trohrbaugh/Qwen3.8-27B-heretic-ara` 2. The underlying Qwen3.8 model 3. Any other upstream components from which those models were derived 4. Relevant software used to quantize or serve the model This model card does not replace or override upstream license obligations. No additional rights are granted beyond those available under the applicable upstream licenses. --- ## Disclaimer This model and repository are provided **as-is**, without warranty of any kind. The author does not guarantee accuracy, reliability, security, fitness for a particular purpose, regulatory compliance, or suitability for production deployment. Benchmark results are specific to the tested environment and should not be interpreted as guaranteed performance on other systems. Users are responsible for evaluating the model and its outputs for their intended application. --- ## Acknowledgements Thanks to the developers and communities behind: - Qwen - The upstream Heretic model - Hugging Face - llm-compressor - compressed-tensors - vLLM - Marlin - FlashInfer