Instructions to use GreyArea128/Foundation-Sec-8B-Instruct-Heretic-GPTQ-W4A16-G128 with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use GreyArea128/Foundation-Sec-8B-Instruct-Heretic-GPTQ-W4A16-G128 with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="GreyArea128/Foundation-Sec-8B-Instruct-Heretic-GPTQ-W4A16-G128")
messages = [
    {"role": "user", "content": "Who are you?"},
]
pipe(messages)

# Load model directly
from transformers import AutoTokenizer, AutoModelForCausalLM

tokenizer = AutoTokenizer.from_pretrained("GreyArea128/Foundation-Sec-8B-Instruct-Heretic-GPTQ-W4A16-G128")
model = AutoModelForCausalLM.from_pretrained("GreyArea128/Foundation-Sec-8B-Instruct-Heretic-GPTQ-W4A16-G128", device_map="auto")
messages = [
    {"role": "user", "content": "Who are you?"},
]
inputs = tokenizer.apply_chat_template(
	messages,
	add_generation_prompt=True,
	tokenize=True,
	return_dict=True,
	return_tensors="pt",
).to(model.device)

outputs = model.generate(**inputs, max_new_tokens=40)
print(tokenizer.decode(outputs[0][inputs["input_ids"].shape[-1]:]))

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use GreyArea128/Foundation-Sec-8B-Instruct-Heretic-GPTQ-W4A16-G128 with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "GreyArea128/Foundation-Sec-8B-Instruct-Heretic-GPTQ-W4A16-G128"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "GreyArea128/Foundation-Sec-8B-Instruct-Heretic-GPTQ-W4A16-G128",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker

docker model run hf.co/GreyArea128/Foundation-Sec-8B-Instruct-Heretic-GPTQ-W4A16-G128

SGLang

How to use GreyArea128/Foundation-Sec-8B-Instruct-Heretic-GPTQ-W4A16-G128 with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "GreyArea128/Foundation-Sec-8B-Instruct-Heretic-GPTQ-W4A16-G128" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "GreyArea128/Foundation-Sec-8B-Instruct-Heretic-GPTQ-W4A16-G128",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "GreyArea128/Foundation-Sec-8B-Instruct-Heretic-GPTQ-W4A16-G128" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "GreyArea128/Foundation-Sec-8B-Instruct-Heretic-GPTQ-W4A16-G128",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Docker Model Runner
How to use GreyArea128/Foundation-Sec-8B-Instruct-Heretic-GPTQ-W4A16-G128 with Docker Model Runner:
```
docker model run hf.co/GreyArea128/Foundation-Sec-8B-Instruct-Heretic-GPTQ-W4A16-G128
```

Foundation-Sec-8B-Instruct-Heretic-GPTQ-W4A16-G128

A GPTQ INT4 quantized derivative of:

e12ex2/Foundation-Sec-8B-Instruct-heretic

This build was created for efficient local inference while retaining the behaviour of the upstream Foundation-Sec instruction-tuned Heretic derivative.

The checkpoint is stored in standard GPTQ format and was quantized using gptqmodel 7.3.2.

Model Summary

Property	Value
Upstream model	`e12ex2/Foundation-Sec-8B-Instruct-heretic`
Quantization algorithm	GPTQ
Quantizer	`gptqmodel 7.3.2`
Weight precision	INT4
Runtime form	W4A16
Group size	128
Symmetric	Yes
Activation ordering	Disabled (`desc_act: false`)
True sequential	Yes
Checkpoint format	GPTQ
Pack dtype	INT32
`lm_head` quantized	No
Model size on disk	~5.4 GB
Primary intended runtime	vLLM / GPTQ-compatible runtimes

Quantization

The final model configuration records:

{
  "bits": 4,
  "checkpoint_format": "gptq",
  "desc_act": false,
  "format": "gptq",
  "group_size": 128,
  "lm_head": false,
  "method": "gptq",
  "pack_dtype": "int32",
  "quant_method": "gptq",
  "sym": true
}

The quantizer metadata records:

quantizer: gptqmodel:7.3.2
true_sequential: true
damp_percent: 0.05
damp_auto_increment: 0.01
pack_impl: cpu
act_group_aware: true

This model is therefore best described as:

GPTQ INT4 W4A16, Group Size 128, symmetric quantization, produced with GPTQModel 7.3.2.

Quantization Files

The repository includes:

quantize_config.json
quant_log.csv

These files are retained to make the quantization configuration and process more transparent.

There is no recipe.yaml for this build.

Repository Contents

README.md
chat_template.jinja
config.json
generation_config.json
model-00001-of-00002.safetensors
model-00002-of-00002.safetensors
model.safetensors.index.json
quant_log.csv
quantize_config.json
tokenizer.json
tokenizer_config.json

The model weights are split across two safetensors shards.

Compatibility

This model is intended for runtimes with GPTQ checkpoint support.

Primary target runtime:

vLLM

Other GPTQ-compatible runtimes may also work, depending on their support for the model architecture and quantization metadata.

If your runtime provides multiple GPTQ execution kernels, benchmark them on your own hardware rather than assuming one backend will always be fastest.

Responsible Use and High-Risk Systems

This model should not be used to facilitate malicious cyber activity, including unauthorized access, exploitation, credential theft, malware deployment, destructive actions, or other activity intended to compromise systems, networks, data, or users without authorization.

Use of this model does not provide authorization to test, access, modify, or interfere with third-party systems. Users are responsible for obtaining appropriate permission and for complying with applicable laws, regulations, contracts, organizational policies, and rules of engagement.

This model should also not be relied upon as the sole decision-making or control component in high-risk or risk-averse environments where errors, hallucinations, or unexpected behaviour could cause significant harm.

Examples include:

Medical diagnosis, treatment, triage, or clinical decision support
Defence, military, weapons, targeting, mission-critical, or safety-critical systems
Critical infrastructure control
Aviation, transport, industrial control, or other safety-critical systems
Emergency response and life-safety systems
High-impact legal, financial, or regulatory decision-making

If the model is used in or around such environments, it should be treated as an assistive research or analysis tool only, with appropriate human oversight, independent verification, access controls, testing, logging, and fail-safe mechanisms.

Model outputs should not be treated as authoritative, deterministic, or safe by default.

Model Behaviour

This is a Heretic-derived model and may exhibit fewer refusals than conventional instruction-tuned models.

That does not imply that model outputs are:

Correct
Safe
Legal
Ethical
Reliable
Appropriate for production use

The model may generate incorrect, misleading, offensive, unsafe, or otherwise undesirable output.

Outputs should be independently reviewed before use in consequential environments.

Quantization Considerations

Quantization reduces memory requirements and can improve deployment efficiency, but may introduce some quality loss compared with the original higher-precision model.

Possible differences may include:

Reasoning quality
Numerical accuracy
Code generation quality
Long-context behaviour
Instruction following
Generation consistency
Rare-token prediction

Users requiring maximum fidelity should compare this build directly against the upstream higher-precision model.

Attribution

Quantization and packaging:

GreyArea128

Upstream model:

e12ex2/Foundation-Sec-8B-Instruct-heretic

Credit belongs to the respective authors and organizations responsible for the upstream model, architecture, training, modifications, and supporting software.

Licensing

This repository is a derivative model.

Users should review and comply with the licenses, notices, terms, and usage requirements applicable to:

e12ex2/Foundation-Sec-8B-Instruct-heretic
The underlying Foundation-Sec model
Any upstream base model used in that lineage
GPTQModel and other software used to quantize or serve the model

This model card does not replace or override upstream license obligations.

No additional rights are granted beyond those available under the applicable upstream licenses.

Disclaimer

This model and repository are provided as-is, without warranty of any kind.

The author does not guarantee accuracy, reliability, security, fitness for a particular purpose, regulatory compliance, or suitability for production deployment.

Users are responsible for evaluating the model and its outputs for their own use case.