Wondernutts/Shadow-Siren-26B-A4B-int4-ov

Gemma 4 26B-A4B — custom OpenVINO runtime update

• Tested hardware: one Intel Arc Pro B70, 32 GB VRAM, on Linux. No B50, Windows, CPU, or offload performance claim.
• Latest uncached 24K prefill: 4,841 → 5,653 tokens/s (+16.8%) in a matched Heretic-reference comparison; the clean deployment build confirmed 5,667 tokens/s. Prefix reuse was OFF, kernels were warmed, and PP means input tokens divided by time to first token. This is not a cache-hit benchmark or a 6K+ result at 24K.
• Shorter prompts / generation: the matched test retained about 7.4K PP at 6,622 input tokens and 112 output tokens/s on short prompts. These are reference-model figures, not guaranteed speeds for this exact fine-tune or every RP request.
• Memory: about 26.8 GiB sampled peak card usage at 24K with an 8 GiB KV pool. Download size is not total VRAM. A 12 GB minimum has not been established.
• Custom fork required for this gain: Wondernuttz/OpenVINO, wide-query commit, plus matching OpenVINO GenAI/tokenizers. This combines custom tuning with credited Intel/OpenVINO kernels and fixes; it is not an unmodified stock wheel.
• Inference: use the complete OpenVINO export with OpenVINO GenAI, not direct Transformers loading. The tested 26B/B70 profile uses PA, DQ128, U4 KV, cache8 GiB, batch16384 and one sequence. Set GEMMA_MIXED_512_TILE=wideq before Python starts, alongside MOE_USE_GROUPED_GEMM_PREFILL=1 and MOE_GROUPED_BINARY_LOOKUP=1. RP prefix caching may stay enabled. See full instructions and rollback.
• Quality / scope: all 17 matched Heretic output strings were identical; Chimera passed separate 24K recall and prefix-update checks. The clean kernel passed 24 repeated U4 reference tests with bit-identical repeats. The old baseline showed U4 variability, and an extra OFF/ON comparison gate did not pass; that caveat remains documented. This is bounded validation, not a universal intelligence guarantee.
• Context and installation: the live Heretic bots received the new kernel while keeping their existing 16K context and settings. Valid exports need no recompression. Rebuild custom runtime wheels/container to get the new kernel—git pull alone does not replace OpenVINO inside an image. No new prebuilt image is supplied. 131K RoPE LUT coverage is not validated 131K usable context; reserve output space.

This checkpoint: the headline result is the Heretic reference benchmark. This fine-tune has not received a fresh per-checkpoint wide-query qualification in this update; its existing conversion and model-specific profile are unchanged.

Earlier binary-lookup results (including 7,435 PP at 6,622 and 6,866 PP at 15,872) remain historical measurements with their original profiles. The wider-query result above is an additional cached/chunked-prefill improvement, not a replacement claim for every model or context size. Original model/merge credits, license and detailed historical notes below remain unchanged.

Detailed September runtime notes, configuration, and validation history

September 6 runtime update: faster long-prompt processing

Clean Release follow-up and rollout status

This checkpoint completed the clean Release 24K factual/boundary/prefix-cache gates and is included in the local runtime rollout. Manual RP review found broadly coherent prose, not perfect instruction following; see the per-model caveats.

The clean Release report records a matched Heretic 24K result of 2,711 → 4,771 PP tok/sec (+76%), the per-checkpoint decode/cache measurements, exact runtime hashes, and limitations. The kernel journal also recorded engine resets when the temporary canary server was terminated after its successful replies; shutdown/model-switch stability is not certified. The old runtime was retained for rollback. These updates change runtime documentation, not model weights or repository visibility.

The numbers below are reference-stack measurements on Gemma 4 26B A4B Heretic, not a benchmark of this exact fine-tune. This checkpoint belongs to the related Gemma 26B A4B MoE family; per-checkpoint speed, memory, and quality need their own validation.

The Wondernuttz OpenVINO fork update and full report adds an opt-in binary search for grouped-MoE token-row lookup and incorporates eight credited Intel upstream backports. No model reconversion or weight download is required to get this runtime optimization. It requires the patched, ABI-matched OpenVINO runtime; an environment variable alone cannot add it to a stock wheel.

Heretic reference input	Previous lookup	New lookup	Matched PP gain
6,622 tokens	6,028 PP tok/sec	7,435 PP tok/sec	23%
15,872 tokens	2,921 PP tok/sec	6,866 PP tok/sec	135% / 2.35x
24,576 tokens	2,735 PP tok/sec	4,767 PP tok/sec	74%

One Intel Arc Pro B70 (32 GB), DQ128, U4 KV, 8 GiB cache allocation, batch16384, one sequence, prefix reuse off. PP is input tokens divided by TTFT. At 24K, TTFT was ~5.15 seconds, long-context decode ~93 tok/sec, sampled VRAM ~26.5 GiB. Decode was essentially unchanged. The 24K matched pair produced identical output strings and passed 11 automated checks; this is bounded coherence evidence, not a universal intelligence or production-stability guarantee.

With prefix caching on, an exact 24K repeat reached first token in 173 ms. That is cache-hit latency, not raw prefill throughput. An updated-fact probe retrieved the corrected password but still answered earlier embedded questions, so strict output-only formatting was not perfect. Separate patched-only 32K probes also passed; do not automatically raise this fine-tune's serving cap to match a reference experiment.

To opt in, set these before pipeline creation, using the matching fork runtime:

export MOE_USE_GROUPED_GEMM_PREFILL=1
export MOE_GROUPED_BINARY_LOOKUP=1

The report includes the PA scheduler configuration, exact revisions, binary hash, matched methodology, limitations, and rollback switch. The published measurements use a Release/O3 binary with profiling capability compiled in but measurement counters disabled; clean-release deployment validation is separate. Native vision/audio are not validated by these text tests. Preserve your model's chat template, reasoning framing, and sampler.

Correction to older tuning attribution: MOE_MICRO_GEMM_N_HINT is bypassed by the active grouped path and must not be credited as a grouped tile optimization. Older benchmark/setup sections below document their historical builds, not updated limits or identical per-tune guarantees. The new lookup does not apply to dense Gemma 31B.

Original base-model, fine-tune, merge credits and licenses below are unchanged. Wondernuttz's contribution is the OpenVINO conversion/runtime work and testing.

OpenVINO INT4 AWQ - asymmetric 4-bit weight compression, verified group size 64.

Shadow Siren 26B-A4B - OpenVINO INT4 AWQ

OpenVINO INT4 conversion of Vortex5/Shadow-Siren-26B-A4B, optimized for local inference on Intel Arc GPUs and compatible OpenVINO devices.

This repository is a quantized deployment artifact. The original fine-tune and merge work belongs to Vortex5 and the creators of its component models; Wondernutts performed the OpenVINO conversion and does not claim authorship of the source model work.

Model and credits

Shadow Siren 26B-A4B is a Gemma 4 26B-A4B mixture-of-experts model intended for roleplay, creative writing, storytelling, and brainstorming. The source model combines work from:

• google/gemma-4-26B-A4B-it
• zerofata/G4-MeroMero-26B-A4B
• ReadyArt/Serenity-26B-A4B
• Gryphe/Pantheon-Reasoning-26B-A4B-1.1
• ReadyArt/Dark-Scarlett-v1.0-26B-A4B
• Darkhn/Gemma-4-26B-A4B-Animus-V14.1-FFT

See the Vortex5 source model card for the full description, merge history, intended use, and original credits.

OpenVINO conversion

• Weight format: asymmetric INT4
• Compression: AWQ, group size 64, ratio 1.0
• MoE routing: router layers excluded from INT4 weight compression
• Graph: multimodal Gemma 4 OpenVINO IR (VLMPipeline)
• Tokenizer and detokenizer OpenVINO artifacts included
• RoPE lookup-table optimization: 131,072 positions, clamped at index 131,071 (LUT131K patch)

The conversion was structurally verified after export: the LUT graph patch, tokenizer artifacts, router exclusion, and quantized weights were checked. A chat-formatted local generation smoke test also completed successfully. No throughput or benchmark score is claimed by this card.

Inference with OpenVINO GenAI

Install a recent OpenVINO GenAI build and Hugging Face Hub client:

pip install -U openvino-genai huggingface-hub

Download the full repository and point VLMPipeline at the local snapshot:

import openvino_genai as ov_genai
from huggingface_hub import snapshot_download

model_dir = snapshot_download("Wondernutts/Shadow-Siren-26B-A4B-int4-ov")

pipe = ov_genai.VLMPipeline(
    model_dir,
    "GPU",
    DYNAMIC_QUANTIZATION_GROUP_SIZE=128,
)

config = ov_genai.GenerationConfig()
config.max_new_tokens = 512
config.do_sample = True
config.temperature = 0.9
config.top_p = 0.95
config.repetition_penalty = 1.2

result = pipe.generate("Write a short scene in a moonlit inn.", generation_config=config)
print(result)

The repository includes the multimodal graph and vision embeddings. Native image input requires the multimodal VLMPipeline API and a compatible OpenVINO GenAI build.

Notes

• This is not a Transformers checkpoint; use OpenVINO/OpenVINO GenAI rather than AutoModelForCausalLM.
• The source model is creative/roleplay-oriented. Deployers remain responsible for prompts, outputs, applicable law, and platform policy.
• License: Apache-2.0, inherited from the source model. Review the upstream card and component licenses before redistribution or commercial deployment.