---
license: apache-2.0
library_name: openvino
pipeline_tag: text-generation
base_model:
- anthracite-org/magnum-v4-12b
tags:
- openvino
- int4
- intel-arc
- mistral
- roleplay
- creative-writing
- conversational
language:
- en
---

# Magnum v4 12B - OpenVINO INT4

OpenVINO INT4 conversion of [anthracite-org/magnum-v4-12b](https://huggingface.co/anthracite-org/magnum-v4-12b) for Intel CPU and GPU inference.

Anthracite created and trained Magnum v4 12B. Wondernutts performed only the OpenVINO conversion. Read and credit [Anthracite's original model card](https://huggingface.co/anthracite-org/magnum-v4-12b) for the authoritative training description, datasets, evaluations, safety information, and acknowledgements.

## About the original fine-tune

Magnum v4 12B is a full-parameter fine-tune of [Mistral-Nemo-Instruct-2407](https://huggingface.co/mistralai/Mistral-Nemo-Instruct-2407). Anthracite designed the Magnum series for character roleplay, story generation, and prose inspired by the Claude 3 Sonnet and Opus style. The source card reports two training epochs at a 32,768-token sequence length and credits Recursal/Featherless for compute support along with the Anthracite contributors who made the tune possible.

The bundled tokenizer uses Mistral Instruct formatting. A representative serialized exchange is:

```text
<s>[INST] SYSTEM MESSAGE
USER MESSAGE[/INST] ASSISTANT MESSAGE</s>[INST] USER MESSAGE[/INST]
```

Use the bundled chat template rather than manually concatenating tokens when possible.

## Conversion details

| Item | Value |
|---|---|
| Source | `anthracite-org/magnum-v4-12b` |
| Architecture | Text-only Mistral Nemo 12B |
| Weight compression | INT4 asymmetric, group size 128 |
| Packed constants | 560 unsigned INT4 weight constants; four unsigned INT8 fallbacks |
| OpenVINO binary payload | approximately 6.53 GiB |
| Included IRs | model, tokenizer, and detokenizer |

The source configuration advertises a much larger architectural context than the 32K training length stated by Anthracite. Long-context behavior and performance have not been independently validated for this conversion, so the architectural maximum should not be read as a tested-quality claim.

## Download

```bash
python -m pip install "openvino-genai==2026.2.1.0" "huggingface_hub>=1.0"
hf download Wondernutts/magnum-v4-12b-int4-ov --local-dir ./magnum-v4-12b-int4-ov
```

## Run with OpenVINO GenAI

```python
import openvino_genai as ov_genai

model_dir = "./magnum-v4-12b-int4-ov"
pipe = ov_genai.LLMPipeline(
    model_dir,
    "GPU",
    CACHE_DIR="./ov_cache",
    DYNAMIC_QUANTIZATION_GROUP_SIZE=128,
)

config = ov_genai.GenerationConfig()
config.max_new_tokens = 768
config.do_sample = True
config.temperature = 0.9
config.top_p = 0.95
config.repetition_penalty = 1.1

pipe.start_chat("You are an observant fantasy character. Stay in character and write vivid prose.")
print(pipe.generate(
    "The campfire dies as footsteps circle beyond the trees.",
    generation_config=config,
))
pipe.finish_chat()
```

Use `LLMPipeline`; this is a text-only model. `start_chat()` applies the bundled tokenizer template and retains KV-cache state across turns. Change `"GPU"` to `"CPU"` for CPU inference.

## Arc Pro B70 benchmark

Tested on one Intel Arc Pro B70 with prefix caching disabled. Each PP result is
the mean of two exact-token runs after one same-shape warmup. Decode used 128
forced output tokens.

| DQGS | 512 PP tok/s | 2K PP tok/s | 6K PP tok/s | Decode tok/s |
|---:|---:|---:|---:|---:|
| 0 | 5,095 | 4,481 | 3,897 | 72.05 |
| 128 | 5,894 | 6,268 | 5,134 | 72.37 |

`DQGS=128` improved PP by 15.7% at 512, 39.9% at 2K, and 31.8% at
6K. The generated-token hashes matched `DQGS=0` at every measured shape. A
separate 6,610-token coherence run produced the same 128-token SHA-256 under
both settings and passed all exact-detail and style checks.

## License and attribution

Apache-2.0, following [the source repository](https://huggingface.co/anthracite-org/magnum-v4-12b). Fine-tune and model credit: [Anthracite](https://huggingface.co/anthracite-org). The source card also credits Recursal/Featherless for compute sponsorship. OpenVINO conversion: Wondernutts.