Qwen3.8-27B-Heretic-GPTQ-W4A16-G128-Marlin

A GPTQ INT4 quantized derivative of:

trohrbaugh/Qwen3.8-27B-heretic-ara

This build was created for efficient local inference with vLLM, with particular emphasis on fitting a capable 27B-class model onto a 24 GB GPU while retaining a large context window and strong decode performance.

The model is stored using the compressed-tensors pack-quantized format and is compatible with vLLM's Marlin WNA16 inference path.

On NVIDIA Blackwell SM120, the currently recommended runtime configuration uses the same INT4 checkpoint with dynamic FP8 Marlin input activations via:

VLLM_MARLIN_INPUT_DTYPE=fp8

This improves long-context prefill performance without changing the stored checkpoint format.

---

Model Summary

Property	Value
Upstream model	trohrbaugh/Qwen3.8-27B-heretic-ara
Quantization algorithm	GPTQ
Stored weight precision	INT4
Checkpoint quantization	W4A16
Group size	128
Symmetric	Yes
Quantization strategy	Per-group
Storage format	compressed-tensors
Packed format	pack-quantized
Default vLLM kernel	Marlin WNA16
Recommended Blackwell runtime	Marlin W4 + dynamic FP8 activations
Tested GPU	NVIDIA RTX PRO 4000 Blackwell 24 GB
Compute capability	SM120
Tested context	76,800 tokens
KV cache	FP8
Primary runtime	vLLM

---

Quantization

This model was quantized using a GPTQ recipe and saved using the compressed-tensors format.

The included recipe.yaml records the quantization configuration.

GPTQ Configuration

GPTQModifier:
  config_groups:
    group_128:
      targets:
        - Linear
      weights:
        num_bits: 4
        type: int
        symmetric: true
        group_size: 128
        strategy: group
        block_structure: null
        dynamic: false
        actorder: weight
        scale_dtype: null
        zp_dtype: null
        observer: memoryless_minmax
        observer_kwargs: {}
      input_activations: null
      output_activations: null
  targets:
    - Linear
  ignore:
    - lm_head
  bypass_divisibility_checks: false
  requires_calibration_data: true
  block_size: 128
  dampening_frac: 0.01
  actorder: static
  offload_hessians: false

The resulting model configuration records:

quant_method: compressed-tensors
format: pack-quantized
quantization_status: compressed

num_bits: 4
group_size: 128
strategy: group
symmetric: true
type: int

Some linear-attention components and lm_head are excluded from INT4 quantization, as recorded in the final quantization_config contained in config.json.

The checkpoint itself is best described as:

GPTQ W4A16 INT4, Group Size 128, stored using compressed-tensors, Marlin-compatible.

The optional FP8 runtime mode described below does not change the checkpoint into a separately quantized W4A8 model.

---

Marlin / vLLM

This model has been tested with vLLM.

For compatible quantized layers, vLLM selected:

MarlinLinearKernel for CompressedTensorsWNA16

Marlin is the optimized inference kernel used at runtime. It is not the model's storage format.

Model storage:
compressed-tensors / pack-quantized

Checkpoint quantization:
GPTQ W4A16 INT4 G128

Default runtime kernel:
Marlin WNA16

Recommended Blackwell Runtime

On NVIDIA Blackwell SM120, improved prefill performance was measured using:

VLLM_MARLIN_INPUT_DTYPE=fp8

This keeps the model's INT4 weights unchanged while dynamically quantizing Marlin input activations to FP8 during inference.

Conceptually:

Stored checkpoint:
INT4 W4 / Group Size 128

Standard runtime:
INT4 weights + BF16 activations
        |
        v
Marlin W4A16

Recommended SM120 runtime:
INT4 weights + dynamic FP8 activations
        |
        v
Marlin W4A8-FP8 runtime path

The term W4A8-FP8 in this model card refers to the runtime execution path only.

It does not describe a different on-disk quantization format.

---

Tested Hardware

GPU: NVIDIA RTX PRO 4000 Blackwell
VRAM: 24 GB
Compute capability: SM120

Runtime: vLLM
Checkpoint quantization: GPTQ W4A16 G128
Storage: compressed-tensors
Kernel: Marlin
Recommended activation runtime: FP8
KV cache: FP8

Prefix caching: Enabled
Maximum configured context: 76,800 tokens
Maximum sequences: 1
Maximum batched prefill tokens: 8,192

The tested configuration successfully processed 64,000-token cold-prefill requests while retaining a configured maximum context of 76,800 tokens.

---

Blackwell Performance

The following measurements were collected on an NVIDIA RTX PRO 4000 Blackwell using the same checkpoint.

Cold TTFT

Context	Standard Marlin W4A16	Marlin W4A8-FP8 Runtime	Reduction
2,048	1.485 s	0.947 s	36.2%
16,384	9.808 s	7.251 s	26.1%
32,768	12.957 s	9.776 s	24.6%
64,000*	51.704 s	39.346 s	23.9%

*The 64K comparison used a unique-prefix cold-prefill workload to avoid prefix-cache reuse between requests.

Effective 64K Prefill Throughput

Standard W4A16:
~1,238 tokens/s

W4A8-FP8 runtime:
~1,627 tokens/s

This represents approximately a 31% increase in effective cold-prefill throughput for the tested 64K workload.

Decode Performance

Three 512-token generation runs using the recommended FP8 activation runtime produced:

Run 1: 33.69 tokens/s
Run 2: 33.65 tokens/s
Run 3: 33.54 tokens/s

Mean: ~33.63 tokens/s

The FP8 Marlin activation path therefore provided a substantial reduction in long-context prefill latency without a material loss in decode throughput in this test environment.

Benchmark results should not be assumed to generalize to different GPUs, drivers, CUDA versions, vLLM builds, prompts, context structures, or concurrency levels.

---

Performance Profiling

Profiling of the standard W4A16 runtime at approximately 64K context showed that the dominant GPU execution costs were:

Marlin quantized GEMMs:      ~69%
FlashInfer attention:        ~23%
GDN / linear attention:      small relative contribution
Host-to-device transfers:    negligible

This indicated that the primary long-context prefill bottleneck was the W4A16 Marlin GEMM path rather than PCIe transfer overhead, KV-cache management, or GDN execution.

Using dynamic FP8 Marlin activations reduced the measured 64K cold-prefill TTFT from approximately:

51.704 seconds

to:

39.346 seconds

on the tested SM120 system.

---

Intended Use

This model is intended for legitimate research, development, and experimentation, including:

• Local LLM inference
• AI research
• LLM evaluation
• Agentic AI experimentation
• Software engineering
• Programming assistance
• Model robustness evaluation

The model's reduced refusal behaviour may make it particularly useful for controlled research where excessive model refusals interfere with legitimate analysis.

---

Responsible Use and High-Risk Systems

This model should not be used to facilitate malicious cyber activity, including unauthorized access, exploitation, credential theft, malware deployment, destructive actions, or other activity intended to compromise systems, networks, data, or users without authorization.

Use of this model does not provide authorization to test, access, modify, or interfere with third-party systems. Users are responsible for obtaining appropriate permission and for complying with applicable laws, regulations, contracts, organizational policies, and rules of engagement.

This model should also not be relied upon as the sole decision-making or control component in high-risk or risk-averse environments where errors, hallucinations, or unexpected behaviour could cause significant harm.

Examples include:

• Medical diagnosis, treatment, triage, or clinical decision support
• Defence, military, weapons, targeting, mission-critical, or safety-critical systems
• Critical infrastructure control
• Aviation, transport, industrial control, or other safety-critical systems
• Emergency response and life-safety systems
• High-impact legal, financial, or regulatory decision-making

If the model is used in or around such environments, it should be treated as an assistive research or analysis tool only, with appropriate human oversight, independent verification, access controls, testing, logging, and fail-safe mechanisms.

Model outputs should not be treated as authoritative, deterministic, or safe by default.

---

Model Behaviour

This is a Heretic-derived model and may exhibit fewer refusals than conventional instruction-tuned models.

That does not imply that model outputs are:

• Correct
• Safe
• Legal
• Ethical
• Reliable
• Appropriate for production use

The model may generate incorrect, misleading, offensive, unsafe, or otherwise undesirable output.

Outputs should be independently reviewed before use in consequential environments.

---

Quantization Considerations

Quantization trades some numerical precision for substantially reduced memory requirements and improved deployment efficiency.

Compared with the original higher-precision model, this build may exhibit differences in:

• Reasoning quality
• Long-context behaviour
• Numerical accuracy
• Code generation
• Rare-token prediction
• Tool-use reliability
• Instruction following
• Generation consistency

The optional FP8 activation runtime introduces an additional numerical precision trade-off compared with the standard BF16 activation path.

Basic deterministic sanity testing on the tested SM120 system produced coherent output, but users should independently evaluate task-specific model quality and accuracy before relying on this configuration.

Users requiring maximum model fidelity should evaluate the original higher-precision upstream model.

---

Prefix Caching

The tested vLLM configuration enables:

--enable-prefix-caching

This can reduce repeated prefill work when requests reuse a substantial common prompt prefix, particularly for long agent system prompts, persistent personas, tool schemas, repeated instructions, and multi-turn conversations.

Cold-prefill benchmark measurements described above used unique prompt prefixes where stated to avoid incorrectly attributing prefix-cache reuse to kernel performance.

---

Model Lineage

Qwen3.8-27B
        |
        v
trohrbaugh/Qwen3.8-27B-heretic-ara
        |
        v
GPTQ calibration
        |
        v
INT4 weight quantization
        |
        v
W4A16 / Group Size 128
        |
        v
compressed-tensors / pack-quantized
        |
        v
vLLM / Marlin
        |
        +-----------------------------+
        |                             |
        v                             v
BF16 activation runtime       Dynamic FP8 activation runtime
Marlin W4A16                  Marlin W4A8-FP8
                              Recommended on tested SM120 GPU

---

Repository Contents

README.md
chat_template.jinja
config.json
generation_config.json
model.safetensors
recipe.yaml
tokenizer.json
tokenizer_config.json

The included recipe.yaml records the GPTQ quantization recipe used for this build.

---

Quantization Metadata

The final config.json contains:

{
  "format": "pack-quantized",
  "quant_method": "compressed-tensors",
  "quantization_status": "compressed"
}

The primary weight configuration is:

{
  "num_bits": 4,
  "type": "int",
  "symmetric": true,
  "group_size": 128,
  "strategy": "group",
  "dynamic": false,
  "actorder": "weight"
}

The quantization recipe identifies GPTQ through GPTQModifier.

Therefore the checkpoint itself should be described as:

GPTQ INT4 W4A16 G128 stored in compressed-tensors format and compatible with vLLM's Marlin WNA16 inference path.

On tested NVIDIA Blackwell SM120 hardware, the recommended runtime uses the same checkpoint with dynamic FP8 Marlin input activations.

---

Compatibility

Primary tested runtime:

• vLLM

Expected baseline use is with runtimes supporting:

• compressed-tensors
• Packed INT4 weights
• W4A16 inference

For the tested Blackwell-optimized runtime, support is additionally required for:

• Marlin
• Dynamic FP8 Marlin input activations
• NVIDIA SM120
• VLLM_MARLIN_INPUT_DTYPE=fp8

Marlin acceleration and FP8 runtime support depend on runtime version, CUDA environment, GPU architecture, and kernel availability.

Other runtimes may load the checkpoint without supporting the same optimized execution path.

---

Attribution

Quantization and packaging:

GreyArea128

Upstream model:

trohrbaugh/Qwen3.8-27B-heretic-ara

Credit belongs to the respective authors and organizations responsible for the upstream model, architecture, training, modifications, and supporting software.

---

Licensing

This repository is a derivative model.

Users must review and comply with the licenses, notices, terms, and usage requirements applicable to:

1. trohrbaugh/Qwen3.8-27B-heretic-ara
2. The underlying Qwen3.8 model
3. Any other upstream components from which those models were derived
4. Relevant software used to quantize or serve the model

This model card does not replace or override upstream license obligations.

No additional rights are granted beyond those available under the applicable upstream licenses.

---

Disclaimer

This model and repository are provided as-is, without warranty of any kind.

The author does not guarantee accuracy, reliability, security, fitness for a particular purpose, regulatory compliance, or suitability for production deployment.

Benchmark results are specific to the tested environment and should not be interpreted as guaranteed performance on other systems.

Users are responsible for evaluating the model and its outputs for their intended application.

---

Acknowledgements

Thanks to the developers and communities behind:

• Qwen
• The upstream Heretic model
• Hugging Face
• llm-compressor
• compressed-tensors
• vLLM
• Marlin
• FlashInfer