Wondernutts/magnum-v4-12b-int4-ov

Magnum v4 12B - OpenVINO INT4

OpenVINO INT4 conversion of anthracite-org/magnum-v4-12b for Intel CPU and GPU inference.

Anthracite created and trained Magnum v4 12B. Wondernutts performed only the OpenVINO conversion. Read and credit Anthracite's original model card for the authoritative training description, datasets, evaluations, safety information, and acknowledgements.

About the original fine-tune

Magnum v4 12B is a full-parameter fine-tune of Mistral-Nemo-Instruct-2407. Anthracite designed the Magnum series for character roleplay, story generation, and prose inspired by the Claude 3 Sonnet and Opus style. The source card reports two training epochs at a 32,768-token sequence length and credits Recursal/Featherless for compute support along with the Anthracite contributors who made the tune possible.

The bundled tokenizer uses Mistral Instruct formatting. A representative serialized exchange is:

<s>[INST] SYSTEM MESSAGE
USER MESSAGE[/INST] ASSISTANT MESSAGE</s>[INST] USER MESSAGE[/INST]

Use the bundled chat template rather than manually concatenating tokens when possible.

Conversion details

Item	Value
Source	anthracite-org/magnum-v4-12b
Architecture	Text-only Mistral Nemo 12B
Weight compression	INT4 asymmetric, group size 128
Packed constants	560 unsigned INT4 weight constants; four unsigned INT8 fallbacks
OpenVINO binary payload	approximately 6.53 GiB
Included IRs	model, tokenizer, and detokenizer

The source configuration advertises a much larger architectural context than the 32K training length stated by Anthracite. Long-context behavior and performance have not been independently validated for this conversion, so the architectural maximum should not be read as a tested-quality claim.

Download

python -m pip install "openvino-genai==2026.2.1.0" "huggingface_hub>=1.0"
hf download Wondernutts/magnum-v4-12b-int4-ov --local-dir ./magnum-v4-12b-int4-ov

Run with OpenVINO GenAI

import openvino_genai as ov_genai

model_dir = "./magnum-v4-12b-int4-ov"
pipe = ov_genai.LLMPipeline(
    model_dir,
    "GPU",
    CACHE_DIR="./ov_cache",
    DYNAMIC_QUANTIZATION_GROUP_SIZE=128,
)

config = ov_genai.GenerationConfig()
config.max_new_tokens = 768
config.do_sample = True
config.temperature = 0.9
config.top_p = 0.95
config.repetition_penalty = 1.1

pipe.start_chat("You are an observant fantasy character. Stay in character and write vivid prose.")
print(pipe.generate(
    "The campfire dies as footsteps circle beyond the trees.",
    generation_config=config,
))
pipe.finish_chat()

Use LLMPipeline; this is a text-only model. start_chat() applies the bundled tokenizer template and retains KV-cache state across turns. Change "GPU" to "CPU" for CPU inference.

Arc Pro B70 benchmark

Tested on one Intel Arc Pro B70 with prefix caching disabled. Each PP result is the mean of two exact-token runs after one same-shape warmup. Decode used 128 forced output tokens.

DQGS	512 PP tok/s	2K PP tok/s	6K PP tok/s	Decode tok/s
0	5,095	4,481	3,897	72.05
128	5,894	6,268	5,134	72.37

DQGS=128 improved PP by 15.7% at 512, 39.9% at 2K, and 31.8% at 6K. The generated-token hashes matched DQGS=0 at every measured shape. A separate 6,610-token coherence run produced the same 128-token SHA-256 under both settings and passed all exact-detail and style checks.

License and attribution

Apache-2.0, following the source repository. Fine-tune and model credit: Anthracite. The source card also credits Recursal/Featherless for compute sponsorship. OpenVINO conversion: Wondernutts.