---
license: apache-2.0
library_name: gguf
base_model: ibm-granite/granite-switch-4.1-3b-preview
tags:
- language
- granite-switch
- granite-4.1
- gguf
---

# Granite Switch 4.1 3B Preview - GGUF (bf16)

GGUF conversion of [ibm-granite/granite-switch-4.1-3b-preview](https://huggingface.co/ibm-granite/granite-switch-4.1-3b-preview), for use with [llama.cpp](https://github.com/ggml-org/llama.cpp).

- **Quantization:** bf16 (no quantization loss, same precision as the original checkpoint)
- **Converted with:** `convert_hf_to_gguf.py` from llama.cpp (master)
- **Original model card / adapters / usage:** see [ibm-granite/granite-switch-4.1-3b-preview](https://huggingface.co/ibm-granite/granite-switch-4.1-3b-preview)

## Files

| File | Description |
|---|---|
| `granite-switch-4.1-3b-preview-bf16.gguf` | bf16 GGUF, includes base model weights and all 12 embedded LoRA adapters |

## Basic usage

```bash
llama-cli -m granite-switch-4.1-3b-preview-bf16.gguf -p "Hello"
```

With no control token, the model runs as the plain base chat model.

## Adapter activation

Granite Switch embeds 12 adapters in one checkpoint. An in-graph router selects
which adapter is active by detecting a control token in the prompt. The selection
is **per request (one adapter per generation)** — the control token routes the whole
sequence to a single adapter slot, then is substituted out before embedding so it
does not corrupt the text.

To activate an adapter you place its control token in the prompt at the position
the model was trained for. Placement differs by adapter type:

- **LoRA adapters** — the token goes at the very start, replacing the leading
  `<|start_of_role|>`. Adapters: `citations`, `hallucination_detection`,
  `context-attribution`.
- **aLoRA adapters** — the token is spliced right before the final
  `assistant<|end_of_role|>` generation prompt (replacing its `<|start_of_role|>`).
  Adapters: `query_rewrite`, `query_clarification`, `answerability`,
  `factuality-detection`, `policy-guardrails`, `factuality-correction`,
  `guardian-core`, `uncertainty`, `requirement-check`.

| Adapter | Type | Control token | Output |
|---|---|---|---|
| citations | lora | `<\|citations\|>` | citation spans |
| hallucination_detection | lora | `<\|hallucination_detection\|>` | `[{"r","f":faithful/partial/unfaithful/NA,"e"}]` |
| context-attribution | lora | `<\|context-attribution\|>` | attribution spans |
| query_rewrite | alora | `<\|query_rewrite\|>` | rewritten query |
| query_clarification | alora | `<\|query_clarification\|>` | clarification or `CLEAR` |
| answerability | alora | `<\|answerability\|>` | `answerable` / `unanswerable` |
| factuality-detection | alora | `<\|factuality-detection\|>` | `yes` / `no` |
| policy-guardrails | alora | `<\|policy-guardrails\|>` | `Yes` / `No` / `Ambiguous` |
| factuality-correction | alora | `<\|factuality-correction\|>` | corrected text |
| guardian-core | alora | `<\|guardian-core\|>` | `yes` / `no` (risk) |
| uncertainty | alora | `<\|uncertainty\|>` | `0`..`9` (certainty) |
| requirement-check | alora | `<\|requirement-check\|>` | `yes` / `no` |

### Example: `hallucination_detection` (LoRA — token at start)

```
<|hallucination_detection|>user<|end_of_role|>Tell me about the moon.<|end_of_text|>
<|start_of_role|>assistant<|end_of_role|>The moon is made of green cheese.<|end_of_text|>
<|start_of_role|>assistant<|end_of_role|>
```
Produces the adapter's structured JSON, e.g.
`[{"r": 0, "f": "unfaithful", "e": "..."}]` instead of a chat reply.

### Example: `answerability` (aLoRA — token before the assistant prompt)

```
<|start_of_role|>system<|end_of_role|>You are a helpful assistant with access to the following documents...
<documents>
{"doc_id": "1", "text": "The square root of 4 is 2."}
</documents>
...<|end_of_text|>
<|start_of_role|>user<|end_of_role|>What is the square root of 4?<|end_of_text|>
<|answerability|>assistant<|end_of_role|>
```
Produces `answerable` (or `unanswerable` for a question the documents don't cover).

The exact formats above are what the model's own chat template renders. The
reliable way to build them is to render the template with `adapter_name` set —
`tokenizer.apply_chat_template(messages, documents=..., adapter_name="answerability",
add_generation_prompt=True, tokenize=False)` — rather than hand-constructing them.

## Running the adapters with Ollama

Because adapter selection lives in the ggml graph and expects the control token
already present in the prompt, the cleanest path through Ollama is a **raw** request
where you supply the fully-rendered prompt yourself:

```bash
ollama create granite-switch-4.1-3b-preview -f Modelfile   # FROM the bf16 GGUF
```

```bash
curl http://localhost:11434/api/generate -d '{
  "model": "granite-switch-4.1-3b-preview",
  "raw": true,
  "prompt": "<|start_of_role|>user<|end_of_role|><|hallucination_detection|>The moon is made of green cheese.<|end_of_text|>\n<|start_of_role|>assistant<|end_of_role|>",
  "stream": false,
  "options": {"temperature": 0}
}'
```

`raw: true` bypasses Ollama's chat template so the control token reaches the model
verbatim. (The normal `/api/chat` path works too if you embed the token in the
message content, but it cannot reproduce the aLoRA boundary placement for you —
raw mode is the faithful option.)

A current Ollama build already includes granite-switch support, so **no patched
Ollama is required** — you can also drive the adapters with
[Mellea](https://mellea.ai/) against stock Ollama, letting Mellea render the
template and place the control tokens.

## License

Apache 2.0, inherited from the base model.