--- license: apache-2.0 base_model: - Vortex5/Shadow-Siren-26B-A4B base_model_relation: quantized tags: - openvino - int4 - awq - intel-arc - gemma4 - roleplay - conversational pipeline_tag: image-text-to-text --- ## Gemma 4 26B-A4B — custom OpenVINO runtime update - **Tested hardware:** one **Intel Arc Pro B70, 32 GB VRAM, on Linux**. No B50, Windows, CPU, or offload performance claim. - **Latest uncached 24K prefill:** **4,841 → 5,653 tokens/s (+16.8%)** in a matched Heretic-reference comparison; the clean deployment build confirmed **5,667 tokens/s**. Prefix reuse was OFF, kernels were warmed, and PP means input tokens divided by time to first token. This is not a cache-hit benchmark or a 6K+ result at 24K. - **Shorter prompts / generation:** the matched test retained about **7.4K PP at 6,622 input tokens** and **112 output tokens/s on short prompts**. These are reference-model figures, not guaranteed speeds for this exact fine-tune or every RP request. - **Memory:** about **26.8 GiB sampled peak card usage at 24K** with an 8 GiB KV pool. Download size is not total VRAM. A 12 GB minimum has not been established. - **Custom fork required for this gain:** [Wondernuttz/OpenVINO, wide-query commit](https://github.com/Wondernuttz/openvino/commit/80c431dfd426bcff914886d6f9a9cef83b44b0c4), plus matching OpenVINO GenAI/tokenizers. This combines custom tuning with credited Intel/OpenVINO kernels and fixes; it is not an unmodified stock wheel. - **Inference:** use the complete OpenVINO export with OpenVINO GenAI, not direct Transformers loading. The tested 26B/B70 profile uses PA, DQ128, U4 KV, cache8 GiB, batch16384 and one sequence. Set `GEMMA_MIXED_512_TILE=wideq` before Python starts, alongside `MOE_USE_GROUPED_GEMM_PREFILL=1` and `MOE_GROUPED_BINARY_LOOKUP=1`. RP prefix caching may stay enabled. See [full instructions and rollback](https://github.com/Wondernuttz/openvino/blob/80c431dfd426bcff914886d6f9a9cef83b44b0c4/WIDEQ_24K_20260906.md). - **Quality / scope:** all 17 matched Heretic output strings were identical; Chimera passed separate 24K recall and prefix-update checks. The clean kernel passed 24 repeated U4 reference tests with bit-identical repeats. The old baseline showed U4 variability, and an extra OFF/ON comparison gate did not pass; that caveat remains documented. This is bounded validation, not a universal intelligence guarantee. - **Context and installation:** the live Heretic bots received the new kernel while keeping their existing 16K context and settings. Valid exports need **no recompression**. Rebuild custom runtime wheels/container to get the new kernel—`git pull` alone does not replace OpenVINO inside an image. No new prebuilt image is supplied. 131K RoPE LUT coverage is not validated 131K usable context; reserve output space. **This checkpoint:** the headline result is the Heretic reference benchmark. This fine-tune has not received a fresh per-checkpoint wide-query qualification in this update; its existing conversion and model-specific profile are unchanged. Earlier binary-lookup results (including 7,435 PP at 6,622 and 6,866 PP at 15,872) remain historical measurements with their original profiles. The wider-query result above is an additional cached/chunked-prefill improvement, not a replacement claim for every model or context size. Original model/merge credits, license and detailed historical notes below remain unchanged.
Detailed September runtime notes, configuration, and validation history ## September 6 runtime update: faster long-prompt processing ### Clean Release follow-up and rollout status This checkpoint completed the clean Release 24K factual/boundary/prefix-cache gates and is included in the local runtime rollout. Manual RP review found broadly coherent prose, not perfect instruction following; see the per-model caveats. The [clean Release report](https://github.com/Wondernuttz/openvino/blob/7b27ac8eb88faec682d4c96dfba749b93df32285/WONDERNUTTZ_GEMMA4_RELEASE_GATE_20260906.md) records a matched Heretic 24K result of **2,711 → 4,771 PP tok/sec (+76%)**, the per-checkpoint decode/cache measurements, exact runtime hashes, and limitations. The kernel journal also recorded engine resets when the temporary canary server was terminated after its successful replies; shutdown/model-switch stability is not certified. The old runtime was retained for rollback. These updates change runtime documentation, not model weights or repository visibility. **The numbers below are reference-stack measurements on Gemma 4 26B A4B Heretic, not a benchmark of this exact fine-tune.** This checkpoint belongs to the related Gemma 26B A4B MoE family; per-checkpoint speed, memory, and quality need their own validation. The [Wondernuttz OpenVINO fork update and full report](https://github.com/Wondernuttz/openvino/blob/ca07f85b2de1d05f13980d8fdaa538f14b01c54e/WONDERNUTTZ_GEMMA4_PREFILL_20260906.md) adds an opt-in binary search for grouped-MoE token-row lookup and incorporates eight credited Intel upstream backports. **No model reconversion or weight download is required to get this runtime optimization.** It requires the patched, ABI-matched OpenVINO runtime; an environment variable alone cannot add it to a stock wheel. | Heretic reference input | Previous lookup | New lookup | Matched PP gain | |---:|---:|---:|---:| | 6,622 tokens | 6,028 PP tok/sec | 7,435 PP tok/sec | 23% | | 15,872 tokens | 2,921 PP tok/sec | 6,866 PP tok/sec | 135% / 2.35x | | 24,576 tokens | 2,735 PP tok/sec | 4,767 PP tok/sec | 74% | One Intel Arc Pro B70 (32 GB), DQ128, U4 KV, 8 GiB cache allocation, batch16384, one sequence, prefix reuse **off**. PP is input tokens divided by TTFT. At 24K, TTFT was ~5.15 seconds, long-context decode ~93 tok/sec, sampled VRAM ~26.5 GiB. Decode was essentially unchanged. The 24K matched pair produced identical output strings and passed 11 automated checks; this is bounded coherence evidence, not a universal intelligence or production-stability guarantee. With prefix caching **on**, an exact 24K repeat reached first token in 173 ms. That is cache-hit latency, not raw prefill throughput. An updated-fact probe retrieved the corrected password but still answered earlier embedded questions, so strict output-only formatting was not perfect. Separate patched-only 32K probes also passed; do not automatically raise this fine-tune's serving cap to match a reference experiment. To opt in, set these before pipeline creation, using the matching fork runtime: ```bash export MOE_USE_GROUPED_GEMM_PREFILL=1 export MOE_GROUPED_BINARY_LOOKUP=1 ``` The [report](https://github.com/Wondernuttz/openvino/blob/ca07f85b2de1d05f13980d8fdaa538f14b01c54e/WONDERNUTTZ_GEMMA4_PREFILL_20260906.md#reference-text-only-pipeline-configuration) includes the PA scheduler configuration, exact revisions, binary hash, matched methodology, limitations, and rollback switch. The published measurements use a Release/O3 binary with profiling capability compiled in but measurement counters disabled; clean-release deployment validation is separate. Native vision/audio are not validated by these text tests. Preserve your model's chat template, reasoning framing, and sampler. Correction to older tuning attribution: `MOE_MICRO_GEMM_N_HINT` is bypassed by the active grouped path and must not be credited as a grouped tile optimization. Older benchmark/setup sections below document their historical builds, not updated limits or identical per-tune guarantees. The new lookup does not apply to dense Gemma 31B. Original base-model, fine-tune, merge credits and licenses below are unchanged. Wondernuttz's contribution is the OpenVINO conversion/runtime work and testing.
> **OpenVINO INT4 AWQ** - asymmetric 4-bit weight compression, verified group size 64. # Shadow Siren 26B-A4B - OpenVINO INT4 AWQ OpenVINO INT4 conversion of [Vortex5/Shadow-Siren-26B-A4B](https://huggingface.co/Vortex5/Shadow-Siren-26B-A4B), optimized for local inference on Intel Arc GPUs and compatible OpenVINO devices. This repository is a quantized deployment artifact. The original fine-tune and merge work belongs to Vortex5 and the creators of its component models; Wondernutts performed the OpenVINO conversion and does not claim authorship of the source model work. ## Model and credits Shadow Siren 26B-A4B is a Gemma 4 26B-A4B mixture-of-experts model intended for roleplay, creative writing, storytelling, and brainstorming. The source model combines work from: - [google/gemma-4-26B-A4B-it](https://huggingface.co/google/gemma-4-26B-A4B-it) - [zerofata/G4-MeroMero-26B-A4B](https://huggingface.co/zerofata/G4-MeroMero-26B-A4B) - [ReadyArt/Serenity-26B-A4B](https://huggingface.co/ReadyArt/Serenity-26B-A4B) - [Gryphe/Pantheon-Reasoning-26B-A4B-1.1](https://huggingface.co/Gryphe/Pantheon-Reasoning-26B-A4B-1.1) - [ReadyArt/Dark-Scarlett-v1.0-26B-A4B](https://huggingface.co/ReadyArt/Dark-Scarlett-v1.0-26B-A4B) - [Darkhn/Gemma-4-26B-A4B-Animus-V14.1-FFT](https://huggingface.co/Darkhn/Gemma-4-26B-A4B-Animus-V14.1-FFT) See the [Vortex5 source model card](https://huggingface.co/Vortex5/Shadow-Siren-26B-A4B) for the full description, merge history, intended use, and original credits. ## OpenVINO conversion - Weight format: asymmetric INT4 - Compression: AWQ, group size 64, ratio 1.0 - MoE routing: router layers excluded from INT4 weight compression - Graph: multimodal Gemma 4 OpenVINO IR (`VLMPipeline`) - Tokenizer and detokenizer OpenVINO artifacts included - RoPE lookup-table optimization: 131,072 positions, clamped at index 131,071 (LUT131K patch) The conversion was structurally verified after export: the LUT graph patch, tokenizer artifacts, router exclusion, and quantized weights were checked. A chat-formatted local generation smoke test also completed successfully. No throughput or benchmark score is claimed by this card. ## Inference with OpenVINO GenAI Install a recent OpenVINO GenAI build and Hugging Face Hub client: ```bash pip install -U openvino-genai huggingface-hub ``` Download the full repository and point `VLMPipeline` at the local snapshot: ```python import openvino_genai as ov_genai from huggingface_hub import snapshot_download model_dir = snapshot_download("Wondernutts/Shadow-Siren-26B-A4B-int4-ov") pipe = ov_genai.VLMPipeline( model_dir, "GPU", DYNAMIC_QUANTIZATION_GROUP_SIZE=128, ) config = ov_genai.GenerationConfig() config.max_new_tokens = 512 config.do_sample = True config.temperature = 0.9 config.top_p = 0.95 config.repetition_penalty = 1.2 result = pipe.generate("Write a short scene in a moonlit inn.", generation_config=config) print(result) ``` The repository includes the multimodal graph and vision embeddings. Native image input requires the multimodal `VLMPipeline` API and a compatible OpenVINO GenAI build. ## Notes - This is not a Transformers checkpoint; use OpenVINO/OpenVINO GenAI rather than `AutoModelForCausalLM`. - The source model is creative/roleplay-oriented. Deployers remain responsible for prompts, outputs, applicable law, and platform policy. - License: Apache-2.0, inherited from the source model. Review the upstream card and component licenses before redistribution or commercial deployment.