---
base_model: Soofi-Project/Soofi-S-Instruct-Preview
license: other
language:
  - en
  - de
  - es
  - fr
  - it
library_name: gguf
pipeline_tag: text-generation
tags:
  - gguf
  - mamba-2
  - moe
  - soofi
  - llama.cpp
  - ollama
  - quantized
  - preview
quantized_by: Soofi-Project
---

# ⚠️ Closed Beta Release Disclaimer

> [!WARNING]
> **This model is a beta preview and a research artifact. It is not an open release.**

This checkpoint is an intermediate result, currently in a closed beta with already selected partners to work on an improved version. It is under development and subject to change based on partner feedback. Outputs, weights, and behavior may differ between checkpoints.

The final model will be released **openly under a permissive license, without gated access**. We will share access details as soon as it is ready.

**Lineage:** This checkpoint is based on [Soofi-Project/Soofi-S-Base](https://huggingface.co/Soofi-Project/Soofi-S-Base). Please refer to the base model card for architectural and training details.

---

# Soofi-S-Instruct-Preview-GGUF

GGUF quantizations of
[**Soofi-Project/Soofi-S-Instruct-Preview**](https://huggingface.co/Soofi-Project/Soofi-S-Instruct-Preview)
for use with **llama.cpp** and **Ollama**.

Converted from fp16 safetensors with `convert_hf_to_gguf.py` and quantized with
`llama-quantize` (llama.cpp).

> **Architecture support:** SOOFI-S is a custom hybrid Mamba-2/MoE model and
> ships with its own modeling code. GGUF conversion requires a build of
> llama.cpp that understands this architecture — verify against the actual
> checkpoint before relying on these quants.

## Available quantizations

| File | Quant | Bits/weight | Size (approx.) | Recommendation |
|---|---|---|---|---|
| `soofi-s-instruct-preview-Q8_0.gguf`  | Q8_0   | 8.5 | ~32 GB | Practically lossless, maximum quality |
| `soofi-s-instruct-preview-Q5_K_M.gguf`| Q5_K_M | 6.6 | ~25 GB | Good size/quality trade-off, recommended default |
| `soofi-s-instruct-preview-Q4_K_M.gguf`| Q4_K_M | ~5.5 | ~21 GB | Smallest, for tighter memory budgets |

> Sizes scale with the **total** 30B parameters (not the 3.5B active). Q4_K_M is
> an estimate; the others are measured.
>
> **No Q6_K:** this architecture's tensor columns (2688/1856/3712) are not
> divisible by 256, so every K-quant tensor falls back to a non-K type. For
> Q6_K that fallback is q8_0, making it ~as large as Q8_0 for no gain; Q5_K_M
> and Q4_K_M fall back to q5_1/q4_1 and still shrink.

## Usage with Ollama

**Directly from this repo** (select the quant level via the tag):

```bash
ollama run hf.co/Soofi-Project/Soofi-S-Instruct-Preview-GGUF:Q5_K_M
```

> For a **private** repo, Ollama needs to know your HF token; otherwise use the
> local Modelfile route.

**Locally with a Modelfile:**

Ollama uses Go templates and does **not** apply the Jinja `chat_template`
embedded in the GGUF, so supply a matching Go template yourself; otherwise
Ollama falls back to a generic one that may miss this model's ChatML format and
its non-thinking behaviour. Unlike the Isar/Rhine reasoning variants, the
Instruct chat template carries **no identity** in its default system prompt
(only a date) and is **non-thinking by default**, so the template below closes
the `<think>` block immediately:

```dockerfile
FROM ./soofi-s-instruct-preview-Q5_K_M.gguf
PARAMETER temperature 0.6
PARAMETER top_p 0.95
PARAMETER num_ctx 8192
PARAMETER stop "<|im_end|>"

TEMPLATE """{{- if .System }}<|im_start|>system
{{ .System }}<|im_end|>
{{ end }}
{{- range .Messages }}
{{- if eq .Role "user" }}<|im_start|>user
{{ .Content }}<|im_end|>
{{ else if eq .Role "assistant" }}<|im_start|>assistant
{{ .Content }}<|im_end|>
{{ end }}
{{- end }}<|im_start|>assistant
<think></think>"""
```

```bash
ollama create soofi-s-instruct:q5 -f Modelfile
ollama run soofi-s-instruct:q5
```

> **Thinking is disabled at the template level.** The prompt ends with a
> closed `<think></think>`, so the model answers directly instead of emitting a
> reasoning block — you do **not** need to turn thinking off in your client
> (e.g. Open WebUI). Without this (a generic template, or the `hf.co` route that
> leaves `<think>` open), the whole answer ends up inside the think block.
>
> **No identity system prompt here.** The Instruct chat template's default
> system prompt contains only a knowledge cutoff and date — no "You are Soofi …"
> text. If this variant still identifies as Soofi in vLLM, that identity is
> baked into the **weights** (and therefore survives quantization), not injected
> by the template. To *force* a fixed identity, add a `SYSTEM """…"""` block (see
> the Isar/Rhine GGUF cards for the wording).
>
> **Tool calling is intentionally omitted** from this Go template. The model's
> native tool format (`<tool_call><function=…>`) is not reproduced here, so
> Ollama's *native* function-calling path is unavailable. In Open WebUI, MCP
> tools still work via the prompt-based *Default* function-calling mode (Open
> WebUI handles tool calls in its own layer). For robust *native* tool calling
> with the original format, run the model under `llama-server --jinja` instead,
> which applies the GGUF's embedded Jinja template (identity **and** tools)
> verbatim — see the llama.cpp section below.

## Usage with llama.cpp

```bash
llama-cli -m soofi-s-instruct-preview-Q5_K_M.gguf -p "Explain AI sovereignty in one sentence."
# or as an OpenAI-compatible server:
llama-server -m soofi-s-instruct-preview-Q5_K_M.gguf --jinja --port 8080
```

> **Use `--jinja`.** It makes llama.cpp apply the GGUF's embedded chat template
> (the model's own Jinja), so the native **tool-calling** format works out of
> the box — no manual template needed. This is the recommended backend for
> native function calling (e.g. as an OpenAI endpoint in Open WebUI).

## Architecture note

This is a **hybrid Mixture-of-Experts** model designed from scratch: 23
Mamba-2/MoE layers + 6 attention layers, 128 routing experts + 1 shared expert
per MoE layer, 6 experts active per token (30B total / 3.5B active). During
quantization, router/gate and certain attention tensors may intentionally be
kept at higher precision; the same applies to the SSM/recurrent (Mamba-2)
tensors. A recent version of llama.cpp is recommended.

## Related models

- Base (bf16): [Soofi-Project/Soofi-S-Instruct-Preview](https://huggingface.co/Soofi-Project/Soofi-S-Instruct-Preview)
- FP8 quantization (vLLM): [Soofi-Project/Soofi-S-Instruct-Preview-FP8](https://huggingface.co/Soofi-Project/Soofi-S-Instruct-Preview-FP8)

## License & provenance

Released under a custom license ("Other"), following the base model
[Soofi-Project/Soofi-S-Instruct-Preview](https://huggingface.co/Soofi-Project/Soofi-S-Instruct-Preview).
TODO: mirror the full license text once the base model card defines it.