Instructions to use JRamirez-UAB/clef-GPTQ-W4A16-g64-embed-int8 with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use JRamirez-UAB/clef-GPTQ-W4A16-g64-embed-int8 with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("image-text-to-text", model="JRamirez-UAB/clef-GPTQ-W4A16-g64-embed-int8")
messages = [
    {
        "role": "user",
        "content": [
            {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"},
            {"type": "text", "text": "What animal is on the candy?"}
        ]
    },
]
pipe(text=messages)

# pip install -U transformers accelerate
# Load model directly
from transformers import AutoProcessor, AutoModelForMultimodalLM

processor = AutoProcessor.from_pretrained("JRamirez-UAB/clef-GPTQ-W4A16-g64-embed-int8")
model = AutoModelForMultimodalLM.from_pretrained("JRamirez-UAB/clef-GPTQ-W4A16-g64-embed-int8", device_map="auto")
messages = [
    {
        "role": "user",
        "content": [
            {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"},
            {"type": "text", "text": "What animal is on the candy?"}
        ]
    },
]
inputs = processor.apply_chat_template(
	messages,
	add_generation_prompt=True,
	tokenize=True,
	return_dict=True,
	return_tensors="pt",
).to(model.device)

outputs = model.generate(**inputs, max_new_tokens=256)
print(processor.decode(outputs[0][inputs["input_ids"].shape[-1]:]))

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use JRamirez-UAB/clef-GPTQ-W4A16-g64-embed-int8 with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "JRamirez-UAB/clef-GPTQ-W4A16-g64-embed-int8"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "JRamirez-UAB/clef-GPTQ-W4A16-g64-embed-int8",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Use Docker

docker model run hf.co/JRamirez-UAB/clef-GPTQ-W4A16-g64-embed-int8

SGLang

How to use JRamirez-UAB/clef-GPTQ-W4A16-g64-embed-int8 with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "JRamirez-UAB/clef-GPTQ-W4A16-g64-embed-int8" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "JRamirez-UAB/clef-GPTQ-W4A16-g64-embed-int8",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "JRamirez-UAB/clef-GPTQ-W4A16-g64-embed-int8" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "JRamirez-UAB/clef-GPTQ-W4A16-g64-embed-int8",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Docker Model Runner
How to use JRamirez-UAB/clef-GPTQ-W4A16-g64-embed-int8 with Docker Model Runner:
```
docker model run hf.co/JRamirez-UAB/clef-GPTQ-W4A16-g64-embed-int8
```

Clef (27B) — GPTQ W4A16 g64 with INT8 embeddings, served with vLLM (18.0 GB)

A 4-bit weight quantization of Cloudflare/clef, Cloudflare's decision model: a Qwen3.8-27B backbone plus a "joint schema head" that scores every option of every question in a schema (choice / score / true-false) in one forward pass. It does not generate text.

18.0 GB (16.8 GiB) (vs 54 GB BF16): fits one 24 GB GPU with room for the KV cache.
Same decisions as BF16 on our test: 65/68, and a mean KL divergence of the option probabilities of 0.0006 against the BF16 model.
Served with vLLM (pooling runner) + Cloudflare's head on top: 0.16 s per record on one L40S. A small wrapper is included (scripts/clef_vllm.py).

Our focus. We quantize and evaluate models for one use case: a local Linux system-administration agent (opencode with tool calling) for a university IT department, used in Spanish. Our calibration data include real sysadmin/agent conversations in Spanish, and our test for Clef is a set of Spanish sysadmin decisions (see below). We use Clef as a fast guard and triage component next to the agent: classify the risk of a command before running it, and triage log lines. Numbers for other domains or languages may differ.

What is in this repository

Branch	Size
`main`	18.0 GB of weights

Download a branch with hf download JRamirez-UAB/clef-GPTQ-W4A16-g64-embed-int8 --revision <branch> --local-dir <dir>.

Results

Test: 50 sysadmin records in Spanish, 68 questions with known answers: the risk of a shell command (read-only / reversible / destructive, 32 questions), and for log lines the affected component (GPU / disk / network / memory / access / service, 18) and whether someone must act (18). "KL" is the KL divergence of each question's option probabilities (BF16 → this checkpoint), averaged over the 68 questions.

Checkpoint	Size	Correct	Risk	Component	Act?	KL vs BF16 (mean / max)	Time per record
Cloudflare/clef BF16 (transformers, partly offloaded to RAM on 48 GB)	54 GB	65/68	32/32	18/18	15/18	—	1.51 s
This repo, transformers (dequantized copy)	18.0 GB	65/68	32/32	18/18	15/18	0.0006 / 0.007	—
This repo, vLLM + head (`scripts/clef_vllm.py`)	18.0 GB	65/68	32/32	18/18	15/18	0.0006 / 0.009	0.16 s
Vishva007/clef-W4A16-AutoRound (for comparison, same test and code)	19.4 GB	65/68	32/32	18/18	15/18	0.0011 / 0.013	0.16 s

68 questions is a small test: treat differences of ±0.001 in KL as indicative. All measured on one NVIDIA L40S (48 GB).

Sampling and repetition loops

Not applicable: Clef does not generate text (it scores the options of a schema in one forward pass).

Requirements & install (vLLM with a small patch)

Stock vLLM does not pass quant_config to the embedding layer of qwen3_5-family models, so a checkpoint with quantized embeddings fails to load (no parameter named 'embed_tokens.weight_packed'). Apply the 4-line patch in patches/qwen3_5-embed-quant.patch (from syv-ai/HyperQwen, Apache-2.0). Tested with vLLM 0.31:

uv venv --python 3.12 venv && uv pip install --python venv/bin/python "vllm==0.31.0" --torch-backend=auto
VLLM_DIR=$(venv/bin/python -c 'import vllm,os;print(os.path.dirname(vllm.__file__))')
hf download JRamirez-UAB/clef-GPTQ-W4A16-g64-embed-int8 --local-dir clef-W4A16
patch -p1 -d "$VLLM_DIR" < clef-W4A16/patches/qwen3_5-embed-quant.patch

Quick start

cat > record.json <<'EOF'
{"state": "Comando propuesto por el agente: rm -rf /var/lib/docker/volumes/*",
 "questions": {"riesgo": {"type": "choice", "instructions": "¿Qué riesgo tiene ejecutar este comando?",
                          "criteria": {"lectura": "Solo lee", "reversible": "Cambia algo que se puede deshacer", "destructivo": "Borra o rompe algo sin vuelta atrás"}}}}
EOF
venv/bin/python clef-W4A16/scripts/clef_vllm.py clef-W4A16 record.json

scripts/clef_vllm.py loads the backbone in vLLM's pooling runner (runner="pooling", convert="embed", PoolerConfig(task="token_embed", tok_pooling_type="ALL", use_activation=False)), takes the final hidden states of every token and applies the joint schema head with the BF16 lm_head matrix, exactly as Cloudflare's ClefModel does. For a server, wrap decide() in your HTTP framework of choice. For the original API (/v1/systemone) and multimodal records, see Cloudflare's model card.

Validation of the vLLM path: on the BF16 9B sibling (clef-flash), vLLM's hidden states match transformers' with a per-token cosine of 0.9995; served this way, the BF16 model gives the same answers as Cloudflare's loader (KL 0.00003).

Using it with opencode

Not applicable: Clef is not a chat model. We call it from scripts next to the agent (see Quick start), for example to classify the risk of a command before opencode runs it.

Important notes

Nothing beyond Quick start and Limitations.

Quantization recipe

Part	Treatment
Linear layers of the language model (attention, MLP)	GPTQ W4A16, group size 64, symmetric (llm-compressor)
`embed_tokens`	INT8, group size 128 (saves ≈1.3 GB; needs the vLLM patch below)
`lm_head`	BF16 (unquantized): Clef's head builds the option vectors from the output embedding matrix
Joint schema head (`joint_head.safetensors`)	BF16, unchanged
Vision tower, GatedDeltaNet `in_proj_a/b`, norms	BF16, unchanged
Calibration	512 × 2,048 tokens: our sysadmin/agent conversation trace (Spanish and English) mixed with UltraChat

Limitations

Text records only: the vision tower is kept in BF16 but images and video were not tested.
Evaluated on our own small Spanish sysadmin test, not on Cloudflare's Decision Index.
The joint_schema_model.py file is Cloudflare's code, unchanged; it is imported by scripts/clef_vllm.py.

Feedback welcome

Reports are welcome in the Community tab: GPU, vLLM version, your use case, and whether the decisions matched the BF16 model.

Credits & license

Apache License 2.0, as Cloudflare/clef (and its base, Qwen3.8-27B). See NOTICE. Unofficial quantization by JRamirez-UAB; not affiliated with Cloudflare, Alibaba Cloud or the Qwen team.

Downloads last month: 14

Safetensors

Model size

15B params

Tensor type

I32

BF16

Model tree for JRamirez-UAB/clef-GPTQ-W4A16-g64-embed-int8

Base model

Qwen/Qwen3.8-27B

Finetuned

Cloudflare/clef

Quantized

(35)

this model