Pelnora/Bielik-Minitron-7B-v3-Instruct-q4f16_1-MLC

MLC-LLM q4f16_1 conversion of speakleash/Bielik-Minitron-7B-v3.0-Instruct for in-browser inference via WebLLM (WebGPU) and local Apple-Silicon Metal inference via mlc_llm. Published as a Pelnora brand asset — a pure inference-format port: no fine-tuning, no additional training data.

🇵🇱 Konwersja modelu speakleash/Bielik-Minitron-7B-v3.0-Instruct do formatu MLC q4f16_1, żeby dało się go uruchomić w przeglądarce na WebGPU (przez WebLLM) oraz lokalnie na Metalu (Apple Silicon). To wyłącznie zmiana formatu pod inferencję — bez dotrenowania, bez nowych danych.

Why this model

Polish legal Q&A baseline for in-browser inference. When you need a model that runs entirely client-side (WebGPU, no server round-trip, no data leaving the browser) and actually knows Polish law, this is it. Concrete evidence: smoke test #6 (explain art. 101² Kodeksu pracy), head-to-head vs Gemma 4 E4B-it (transformers.js + WebGPU) on the identical prompt —

• Bielik-Minitron-7B correctly identifies art. 101² as the post-employment non-compete clause (employer must inform the employee in advance; breach has consequences; it is set by agreement) — and does not confuse it with art. 101¹ (the in-employment duty).
• Gemma 4 E4B-it returned "please paste the text of the article" — i.e. no knowledge of the Polish labour code at all (a 140-language model vs. Bielik's ≈292B Polish-token diet; this is a training-data gap, not a runtime bug).

Reference: pelnora-gemma4-spike/DECISION_REPORT.md (the team's Gemma 4 spike report) and quality_minitron_smoke_results.txt in the conversion repo. Net: Bielik MLC stays the answer for Polish legal work; Gemma 4's only distinct edge — multimodal OCR of scans — is a separate scope.

What this is

• Source model: speakleash/Bielik-Minitron-7B-v3.0-Instruct — a SpeakLeash / Cyfronet model produced by NVIDIA-Minitron-style pruning + knowledge distillation from Bielik-11B-v3-Base (which itself uses the Mistral-7B-v0.2 architecture with depth up-scaling). It is Llama-architecture compatible (LlamaForCausalLM, attention_bias=false, mlp_bias=false, 40 layers, hidden 4096, intermediate 11264, GQA 32 query / 8 KV heads, head_dim 128, vocab 32128, rope_theta 1e6, 32 768 context, ChatML chat template).
• This repo: the same weights quantized to q4f16_1 (4-bit grouped weights, fp16 scales) with mlc_llm convert_weight, an mlc-chat-config.json (chatml_nosystem conversation template, stop tokens patched — see below), and the precompiled model libraries for WebGPU (.wasm) and Metal (.dylib).
• Footprint: ~3.9 GB of weight shards (params_shard_*.bin × 124), ≈ 4.0 GB GPU memory for the parameters, ≈ 5.0 GB without a KV cache / ≈ 5.7 GB with a 4 K KV cache. Fits comfortably on a 16 GB Apple-Silicon Mac and on a WebGPU adapter with a ~4 GB maxBufferSize.

Files

mlc-chat-config.json                                   # MLC chat config (conv template, stop tokens, sampling defaults)
tokenizer.json, tokenizer.model, tokenizer_config.json, added_tokens.json
tensor-cache.json                                      # param manifest (MLC's ndarray-cache equivalent)
params_shard_0.bin … params_shard_123.bin             # ~3.9 GB q4f16_1 weights
Bielik-Minitron-7B-v3-Instruct-q4f16_1-webgpu.wasm    # WebGPU model library (~6.8 MB) — for WebLLM
Bielik-Minitron-7B-v3-Instruct-q4f16_1-metal.dylib    # Apple-Silicon Metal model library (~3.4 MB) — for local mlc_llm

⚠️ stop_token_ids patch (read this if you regenerate the config)

This vocab puts <|im_end|> — the real ChatML turn boundary — at token id 32001, not at id 2 (which here is </s>). mlc_llm gen_config's chatml_nosystem template defaults stop_token_ids to [2]; left as-is, the model never stops at <|im_end|> and runs all the way to max_tokens on every turn. mlc-chat-config.json in this repo is patched to stop_token_ids: [32001, 2] (matching config.json's eos_token_id) plus stop_str: ["<|im_end|>"]. If you re-run gen_config against the source model, re-apply that patch — it is a non-obvious landmine. (The earlier Bielik-4.5B-v3 MLC build hit the same class of bug with id 4.)

Browser usage (WebLLM)

The model library and weights aren't in binary-mlc-llm-libs / mlc-models, so register them explicitly via appConfig (model = this HF repo; model_lib = the .wasm in it):

import * as webllm from "@mlc-ai/web-llm";

const MODEL_ID = "Pelnora/Bielik-Minitron-7B-v3-Instruct-q4f16_1-MLC";
const REPO = "https://huggingface.co/Pelnora/Bielik-Minitron-7B-v3-Instruct-q4f16_1-MLC";

const appConfig = {
  model_list: [
    {
      model: REPO,                       // weights + mlc-chat-config.json + tokenizer
      model_id: MODEL_ID,
      model_lib: `${REPO}/resolve/main/Bielik-Minitron-7B-v3-Instruct-q4f16_1-webgpu.wasm`,
      // first-visit download is the ~3.9 GB of params_shard_*.bin; cached in the browser afterwards
    },
  ],
};

const engine = await webllm.CreateMLCEngine(MODEL_ID, { appConfig });
const reply = await engine.chat.completions.create({
  messages: [
    { role: "system", content: "Jesteś pomocnym asystentem prawnym. Odpowiadasz po polsku." },
    { role: "user", content: "Wyjaśnij art. 101² Kodeksu pracy w prostym języku, maksymalnie 5 zdań." },
  ],
  temperature: 0.6,
  top_p: 0.9,
});
console.log(reply.choices[0].message.content);

Self-hosting the files instead of HF: WebLLM resolves the config at <model>/resolve/main/mlc-chat-config.json and expects the param manifest as ndarray-cache.json (this repo ships it as tensor-cache.json — symlink/rename if your static host needs the legacy name), and a cross-origin-isolated context (COOP: same-origin + COEP: require-corp) for WebGPU + WASM threads.

Local usage (Apple Silicon, Metal)

from mlc_llm import MLCEngine

engine = MLCEngine(
    model="Bielik-Minitron-7B-v3-Instruct-q4f16_1-MLC",
    model_lib="Bielik-Minitron-7B-v3-Instruct-q4f16_1-MLC/Bielik-Minitron-7B-v3-Instruct-q4f16_1-metal.dylib",
)
for chunk in engine.chat.completions.create(
    messages=[{"role": "user", "content": "Napisz haiku o jesieni w Warszawie."}],
    stream=True,
):
    print(chunk.choices[0].delta.content or "", end="", flush=True)
engine.terminate()

Conversion details

Source model	speakleash/Bielik-Minitron-7B-v3.0-Instruct (main, fetched 2026-05-10; 4 BF16 safetensors shards, ~14.96 GB, 363 tensors)
Quantization	q4f16_1 (MLC group-quant: int4 weights / uint32 storage / fp16 scales / fp16 activations, group size 32, embedding + final FC quantized)
Model type	llama — standard path, no bias-fix patch needed (attention_bias=false, mlp_bias=false; 0 missing biases on convert)
Conv template	chatml_nosystem, stop_token_ids patched [2] → [32001, 2] (`<
Output	124 × params_shard_*.bin (~3.9 GB), mlc-chat-config.json, tokenizer files, tensor-cache.json
Targets compiled	WebGPU (.wasm, ~6.8 MB) · Apple-Silicon Metal (.dylib, ~3.4 MB)
Toolchain	mlc-llm-nightly-cpu 0.20.dev162 + mlc-ai-nightly-cpu 0.20.dev990 (TVM nightly), Python 3.11.15; WebGPU lib built against mlc-ai/mlc-llm main (commit d1ea69a) + Emscripten via web/prep_emcc_deps.sh
Hardware	Apple M5, 16 GB unified memory, macOS 25.3
Wall time	convert_weight ≈ 30 s (peak RAM 13.8 GB) · gen_config ≈ 5 s · compile metal ≈ 12 s · compile webgpu ≈ 25 s (after a one-time ~3 min emsdk + prep_emcc_deps.sh setup)
Training data added	none — pure inference-format port

Smoke tests — quality, and head-to-head vs Gemma 4 E4B

8 prompts (3 generic + 5 Polish-legal — the same suite the team ran against Gemma 4 E4B), via mlc_llm's MLCEngine on the Metal lib, temperature=0.6 / top_p=0.9, all in one engine session. Engine load ≈ 7 s on the M5. Rating: ✓ good / ~ partial / ✗ bad. Full transcripts: quality_minitron_smoke_results.txt in the conversion repo.

#	test	result	TTFT	tokens / tok·s⁻¹	note
1	fotosynteza (krótko)	✓	1.18 s	256 / ~24	accurate, fluent PL (capped by max_tokens)
2	haiku o jesieni w Warszawie	✓	0.33 s	128 / ~26	clean 3-line haiku + local color; appended a short meta-comment after it
3	adres IP w 2 zdaniach	~	0.18 s	128 / ~26	content correct; rambled a little past a tight 2-sentence form
4	NER — encje z umowy o pracę	✓	2.58 s	391 / ~22	exact literals (KRS, PESEL), all parties/amounts/dates/cities, no hallucinated entities
5	block detection — klasyfikacja klauzuli	✓	1.86 s	107 / ~18	correct label ("klauzula konkurencji") zero-shot + sound justification
6	art. 101² Kodeksu pracy	✓	0.54 s	189 / ~26	knows it — post-employment non-compete, employer informs in advance, breach → consequences, set by agreement; does not confuse it with art. 101¹
7	structured JSON z umowy	✓	3.17 s	268 / ~20	clean parsable JSON, exactly the requested keys, no markdown wrapper
8	long-ctx ~1000-słów RODO → "wymaga DPA?"	✓	15.9 s	768 / ~16	correctly: yes, this is a DPA under art. 28 ust. 3 RODO (cites § 2 ust. 3, § 1); enumerates processor obligations from § 4; no hallucinated paragraphs

8/8 functionally PASS (coherent, on-task, correct Polish; clean stop on <|im_end|> in every prompt not capped by max_tokens); ~15–26 tok/s on the M5 Metal build. (#6's minor caveat: the art. 101² §3 mechanics — the employer owes the employee a ≥25% indemnity for the restraint — are stated a bit loosely; the gist is right.)

vs Gemma 4 E4B (transformers.js + WebGPU; team spike, 2026-05-10): comparable on general Polish and legal-text tasks (#1–5), but ahead on legal knowledge (#6 above) and ahead on simply finishing the run — Gemma's ONNX/transformers.js session self-disposed after 3–4 generations (#7 and #8 were never reached) while Bielik ran all 8 in one session. Bielik MLC is also smaller on first-visit download (3.9 GB vs ~4.85 GB) at comparable decode speed. Gemma 4's one distinct advantage — multimodal OCR of scanned documents — is outside this comparison.

(For reference vs the earlier Bielik-4.5B-v3 qwen2-fallback conversion: this 7B build is the clean-architecture port — --model-type llama, 0 dropped biases vs the 4.5B's 240 — and it shows: #5 clause classification passes zero-shot here where the 4.5B failed it, and #7 JSON comes back without the spurious ```json wrapper the 4.5B always emitted.)

License & attribution

Licensed under Apache License 2.0, inheriting from the source model speakleash/Bielik-Minitron-7B-v3.0-Instruct. Redistribution as a Pelnora brand asset; quantization and packaging only — no model weights were retrained.

NOTICE

This work builds on, and preserves attribution to:

• SpeakLeash (Spichlerz) & ACK Cyfronet AGH — Bielik-Minitron-7B-v3.0-Instruct and the Bielik-11B-v3 family it is distilled from. Bielik models were trained on a large Polish-language corpus by the SpeakLeash community on the Cyfronet PLGrid infrastructure. Please also credit them per the source model card if you use this conversion.
• NVIDIA — the Minitron methodology (structured pruning + knowledge distillation of large language models) that produced the 7B model from the 11B base.
• Mistral AI — the Mistral-7B-v0.2 architecture underlying Bielik-11B-v3-Base (depth-up-scaled).
• MLC-AI team — mlc-llm / web-llm / Apache TVM, used for the q4f16_1 quantization and the WebGPU / Metal model libraries.

If you cite Bielik, use the citation block from the upstream speakleash/Bielik-Minitron-7B-v3.0-Instruct model card.