--- base_model: e12ex2/Foundation-Sec-8B-Instruct-heretic library_name: transformers pipeline_tag: text-generation language: - en tags: - foundation-sec - llama - heretic - gptq - int4 - w4a16 - g128 - gptqmodel - vllm - local-llm --- # Foundation-Sec-8B-Instruct-Heretic-GPTQ-W4A16-G128 A GPTQ INT4 quantized derivative of: **`e12ex2/Foundation-Sec-8B-Instruct-heretic`** This build was created for efficient local inference while retaining the behaviour of the upstream Foundation-Sec instruction-tuned Heretic derivative. The checkpoint is stored in standard GPTQ format and was quantized using `gptqmodel 7.3.2`. --- ## Model Summary | Property | Value | |---|---| | Upstream model | `e12ex2/Foundation-Sec-8B-Instruct-heretic` | | Quantization algorithm | GPTQ | | Quantizer | `gptqmodel 7.3.2` | | Weight precision | INT4 | | Runtime form | W4A16 | | Group size | 128 | | Symmetric | Yes | | Activation ordering | Disabled (`desc_act: false`) | | True sequential | Yes | | Checkpoint format | GPTQ | | Pack dtype | INT32 | | `lm_head` quantized | No | | Model size on disk | ~5.4 GB | | Primary intended runtime | vLLM / GPTQ-compatible runtimes | --- ## Quantization The final model configuration records: ```json { "bits": 4, "checkpoint_format": "gptq", "desc_act": false, "format": "gptq", "group_size": 128, "lm_head": false, "method": "gptq", "pack_dtype": "int32", "quant_method": "gptq", "sym": true } ``` The quantizer metadata records: ```text quantizer: gptqmodel:7.3.2 true_sequential: true damp_percent: 0.05 damp_auto_increment: 0.01 pack_impl: cpu act_group_aware: true ``` This model is therefore best described as: **GPTQ INT4 W4A16, Group Size 128, symmetric quantization, produced with GPTQModel 7.3.2.** --- ## Quantization Files The repository includes: ```text quantize_config.json quant_log.csv ``` These files are retained to make the quantization configuration and process more transparent. There is no `recipe.yaml` for this build. --- ## Repository Contents ```text README.md chat_template.jinja config.json generation_config.json model-00001-of-00002.safetensors model-00002-of-00002.safetensors model.safetensors.index.json quant_log.csv quantize_config.json tokenizer.json tokenizer_config.json ``` The model weights are split across two `safetensors` shards. --- ## Compatibility This model is intended for runtimes with GPTQ checkpoint support. Primary target runtime: - vLLM Other GPTQ-compatible runtimes may also work, depending on their support for the model architecture and quantization metadata. If your runtime provides multiple GPTQ execution kernels, benchmark them on your own hardware rather than assuming one backend will always be fastest. --- ## Responsible Use and High-Risk Systems This model should not be used to facilitate malicious cyber activity, including unauthorized access, exploitation, credential theft, malware deployment, destructive actions, or other activity intended to compromise systems, networks, data, or users without authorization. Use of this model does not provide authorization to test, access, modify, or interfere with third-party systems. Users are responsible for obtaining appropriate permission and for complying with applicable laws, regulations, contracts, organizational policies, and rules of engagement. This model should also not be relied upon as the sole decision-making or control component in high-risk or risk-averse environments where errors, hallucinations, or unexpected behaviour could cause significant harm. Examples include: - Medical diagnosis, treatment, triage, or clinical decision support - Defence, military, weapons, targeting, mission-critical, or safety-critical systems - Critical infrastructure control - Aviation, transport, industrial control, or other safety-critical systems - Emergency response and life-safety systems - High-impact legal, financial, or regulatory decision-making If the model is used in or around such environments, it should be treated as an assistive research or analysis tool only, with appropriate human oversight, independent verification, access controls, testing, logging, and fail-safe mechanisms. Model outputs should not be treated as authoritative, deterministic, or safe by default. --- ## Model Behaviour This is a Heretic-derived model and may exhibit fewer refusals than conventional instruction-tuned models. That does not imply that model outputs are: - Correct - Safe - Legal - Ethical - Reliable - Appropriate for production use The model may generate incorrect, misleading, offensive, unsafe, or otherwise undesirable output. Outputs should be independently reviewed before use in consequential environments. --- ## Quantization Considerations Quantization reduces memory requirements and can improve deployment efficiency, but may introduce some quality loss compared with the original higher-precision model. Possible differences may include: - Reasoning quality - Numerical accuracy - Code generation quality - Long-context behaviour - Instruction following - Generation consistency - Rare-token prediction Users requiring maximum fidelity should compare this build directly against the upstream higher-precision model. --- ## Attribution Quantization and packaging: **GreyArea128** Upstream model: **`e12ex2/Foundation-Sec-8B-Instruct-heretic`** Credit belongs to the respective authors and organizations responsible for the upstream model, architecture, training, modifications, and supporting software. --- ## Licensing This repository is a derivative model. Users should review and comply with the licenses, notices, terms, and usage requirements applicable to: 1. `e12ex2/Foundation-Sec-8B-Instruct-heretic` 2. The underlying Foundation-Sec model 3. Any upstream base model used in that lineage 4. GPTQModel and other software used to quantize or serve the model This model card does not replace or override upstream license obligations. No additional rights are granted beyond those available under the applicable upstream licenses. --- ## Disclaimer This model and repository are provided **as-is**, without warranty of any kind. The author does not guarantee accuracy, reliability, security, fitness for a particular purpose, regulatory compliance, or suitability for production deployment. Users are responsible for evaluating the model and its outputs for their own use case. --- ## Acknowledgements Thanks to the developers and communities behind: - Foundation AI / Foundation-Sec - The upstream Heretic derivative - Hugging Face - GPTQModel - vLLM - Transformers