Wondernutts/Gemma-4-31B-StyleTune-int4-ov

Gemma-4 31B StyleTune OpenVINO INT4

Private experimental OpenVINO INT4 conversion of Gryphe/Gemma-4-31B-StyleTune.

Gryphe created and trained StyleTune. Wondernutts performed the OpenVINO conversion and graph preparation. See the source model card for the training method, source benchmarks, and credits.

Current status

Do not use this build as a persistent server yet. Testing on 2026-07-25 found a repeated-request failure in the OpenVINO GenAI VLM runtime:

• The plain VLMPipeline produces coherent output.
• After a request generates multiple tokens, later requests can fail with CL_OUT_OF_RESOURCES.
• The failure occurs with both the stock OpenVINO 2026.2 GPU plugin and the Wondernuttz custom plugin.
• Device memory was about 19.4 GB at failure on a 32 GB B70, so this was not VRAM exhaustion.
• Continuous batching avoids the crash, but the 6,620-token coherence test produced corrupted output and failed all four retrieval/style checks.

No persistent inference recipe or performance claim is published for this conversion until one path is both repeat-stable and coherent. The repository remains private for that reason.

Conversion details

Item	Value
Source	Gryphe/Gemma-4-31B-StyleTune
Architecture	Dense Gemma 4 31B conditional-generation VLM export
Language compression	INT4 asymmetric, group size 128, ratio 1.0
OpenVINO binary payload	About 18.11 GiB
Graph preparation	Precomputed RoPE lookup-table path
Important source property	Untied, retrained lm_head preserved

The repository contains the language model, text embedding models, per-layer embedding model, vision model, tokenizer, and detokenizer IRs.

Diagnostic environment

• Ubuntu 24.04.4 LTS
• Linux 7.0.0 with the xe driver
• Intel Arc Pro B70 32 GB
• Intel compute runtime 26.22.38646.6
• OpenVINO 2026.2.0
• OpenVINO GenAI 2026.2.0.0
• Python 3.12
• Wondernuttz custom GPU branch: arc-xe2-int4-2026.2

Download for development

python -m pip install \
  openvino==2026.2.0 \
  openvino-genai==2026.2.0.0 \
  "huggingface_hub>=1.19,<2"
hf auth login
hf download Wondernutts/Gemma-4-31B-StyleTune-int4-ov \
  --local-dir ./gemma4-31b-styletune-int4-ov

VLMPipeline is required by the exported graph, including for text-only use. The old card's continuous-batching recommendation has been withdrawn because repeat stability without coherence is not acceptable.

Next work

1. Isolate the VLM request-state fault in OpenVINO GenAI.
2. Test a language-only export that does not require the VLM pipeline.
3. Re-run repeated-request, fixed-token, and long-context coherence gates.
4. Publish throughput only after the same runtime passes all three gates.

License and attribution

Apache-2.0, following Gryphe's source repository and upstream Gemma. StyleTune authorship and training credit: Gryphe. OpenVINO conversion and Arc testing: Wondernutts.