barha/granite-switch-4.1-3b-preview-GGUF

Granite Switch 4.1 3B Preview - GGUF (bf16)

GGUF conversion of ibm-granite/granite-switch-4.1-3b-preview, for use with llama.cpp.

• Quantization: bf16 (no quantization loss, same precision as the original checkpoint)
• Converted with: convert_hf_to_gguf.py from llama.cpp (master)
• Original model card / adapters / usage: see ibm-granite/granite-switch-4.1-3b-preview

Files

File	Description
granite-switch-4.1-3b-preview-bf16.gguf	bf16 GGUF, includes base model weights and all 12 embedded LoRA adapters

Basic usage

llama-cli -m granite-switch-4.1-3b-preview-bf16.gguf -p "Hello"

With no control token, the model runs as the plain base chat model.

Adapter activation

Granite Switch embeds 12 adapters in one checkpoint. An in-graph router selects which adapter is active by detecting a control token in the prompt. The selection is per request (one adapter per generation) — the control token routes the whole sequence to a single adapter slot, then is substituted out before embedding so it does not corrupt the text.

To activate an adapter you place its control token in the prompt at the position the model was trained for. Placement differs by adapter type:

• LoRA adapters — the token goes at the very start, replacing the leading <|start_of_role|>. Adapters: citations, hallucination_detection, context-attribution.
• aLoRA adapters — the token is spliced right before the final assistant<|end_of_role|> generation prompt (replacing its <|start_of_role|>). Adapters: query_rewrite, query_clarification, answerability, factuality-detection, policy-guardrails, factuality-correction, guardian-core, uncertainty, requirement-check.

Adapter	Type	Control token	Output
citations	lora	<|citations|>	citation spans
hallucination_detection	lora	<|hallucination_detection|>	[{"r","f":faithful/partial/unfaithful/NA,"e"}]
context-attribution	lora	<|context-attribution|>	attribution spans
query_rewrite	alora	<|query_rewrite|>	rewritten query
query_clarification	alora	<|query_clarification|>	clarification or CLEAR
answerability	alora	<|answerability|>	answerable / unanswerable
factuality-detection	alora	<|factuality-detection|>	yes / no
policy-guardrails	alora	<|policy-guardrails|>	Yes / No / Ambiguous
factuality-correction	alora	<|factuality-correction|>	corrected text
guardian-core	alora	<|guardian-core|>	yes / no (risk)
uncertainty	alora	<|uncertainty|>	0..9 (certainty)
requirement-check	alora	<|requirement-check|>	yes / no

Example: hallucination_detection (LoRA — token at start)

<|hallucination_detection|>user<|end_of_role|>Tell me about the moon.<|end_of_text|>
<|start_of_role|>assistant<|end_of_role|>The moon is made of green cheese.<|end_of_text|>
<|start_of_role|>assistant<|end_of_role|>

Produces the adapter's structured JSON, e.g. [{"r": 0, "f": "unfaithful", "e": "..."}] instead of a chat reply.

Example: answerability (aLoRA — token before the assistant prompt)

<|start_of_role|>system<|end_of_role|>You are a helpful assistant with access to the following documents...
<documents>
{"doc_id": "1", "text": "The square root of 4 is 2."}
</documents>
...<|end_of_text|>
<|start_of_role|>user<|end_of_role|>What is the square root of 4?<|end_of_text|>
<|answerability|>assistant<|end_of_role|>

Produces answerable (or unanswerable for a question the documents don't cover).

The exact formats above are what the model's own chat template renders. The reliable way to build them is to render the template with adapter_name set — tokenizer.apply_chat_template(messages, documents=..., adapter_name="answerability", add_generation_prompt=True, tokenize=False) — rather than hand-constructing them.

Running the adapters with Ollama

Because adapter selection lives in the ggml graph and expects the control token already present in the prompt, the cleanest path through Ollama is a raw request where you supply the fully-rendered prompt yourself:

ollama create granite-switch-4.1-3b-preview -f Modelfile   # FROM the bf16 GGUF

curl http://localhost:11434/api/generate -d '{
  "model": "granite-switch-4.1-3b-preview",
  "raw": true,
  "prompt": "<|start_of_role|>user<|end_of_role|><|hallucination_detection|>The moon is made of green cheese.<|end_of_text|>\n<|start_of_role|>assistant<|end_of_role|>",
  "stream": false,
  "options": {"temperature": 0}
}'

raw: true bypasses Ollama's chat template so the control token reaches the model verbatim. (The normal /api/chat path works too if you embed the token in the message content, but it cannot reproduce the aLoRA boundary placement for you — raw mode is the faithful option.)

A current Ollama build already includes granite-switch support, so no patched Ollama is required — you can also drive the adapters with Mellea against stock Ollama, letting Mellea render the template and place the control tokens.

License

Apache 2.0, inherited from the base model.