Menterium/translategemma-12b-it-onnx-int4-dml

TranslateGemma 12B — ONNX Runtime GenAI, int4, DirectML

ONNX Runtime GenAI build (int4 weights, fp16 activations, DirectML execution provider) of the text component of google/translategemma-12b-it, Google's translation-specialized Gemma 3 model covering 55 languages.

Built for local, fully offline translation in Cortility (Menterium IT Solutions GmbH); usable with any onnxruntime-genai client on DirectX-12-capable GPUs. The 4B sibling is published as translategemma-4b-it-onnx-int4-dml (and -cpu).

When this model is worth its size

8.8 GB on disk and roughly 8 GB of VRAM — three times the 4B build. That price is only worth paying for languages the 4B handles poorly. Per-language MetricX scores from the TranslateGemma technical report, Appendix A (lower is better, 0–25 MQM scale, en→xx on WMT24++):

Language	4B	12B	Worth switching
Hungarian	7.84	5.00	yes — the 4B is not usable here
Turkish	6.17	4.64	yes
Finnish	5.68	3.77	yes
Polish	5.64	4.58	yes
Czech	5.41	4.03	yes
Greek	4.66	3.34	worthwhile
Romanian	4.18	3.25	worthwhile
Ukrainian	4.16	3.29	worthwhile
Dutch, French, Portuguese, Nordic, Russian	2.8–3.3	2.0–2.5	marginal
German, Spanish, Italian	1.9–2.6	1.4–2.2	no — the 4B is already good

Our own spot check agrees with those numbers. Ten German sentences translated into Hungarian: the 4B produced five unusable outputs (invented words, wrong meanings), the 12B none. The same ten sentences in Spanish, French, Italian, Dutch and Portuguese: 27 of 50 outputs were byte-identical between the two models, the rest differed only in wording.

Note the benchmark direction: the published scores are en→xx. Translating from a non-English source (e.g. German → Hungarian) is likely somewhat worse, so treat these numbers as an optimistic bound.

Conversion notes

• Text decoder extracted from the multimodal checkpoint (Gemma3ForCausalLM, model type gemma3_text); the vision tower is not included.
• Converted with the onnxruntime-genai model builder (python -m onnxruntime_genai.models.builder -p int4 -e dml).
• Builder version must match the consuming runtime. Built with onnxruntime-genai 0.13.1, the version Cortility ships (Microsoft.ML.OnnxRuntimeGenAI.DirectML). A build made with 0.15.2 emits nodes the 0.13.1 DirectML provider cannot partition; loading then fails with "This session cannot use the graph capture feature as requested by the user as all compute graph nodes have not been partitioned to the DmlExecutionProvider". The model itself is fine — it simply cannot be loaded by the older runtime.
• luid stripped from genai_config.json. Newer builders pin the model to the exact GPU present at build time; this build ships [{"dml": {}}] so the host application selects the adapter at load time.
• genai_config.json ships with eos_token_id: [1, 106] so that <end_of_turn> terminates generation (the stock config only stopped on <eos>).

Prompt format

Gemma requires the <bos> token at the start of the prompt — without it the output degenerates. The tokenizer of onnxruntime-genai does not add it automatically:

<bos><start_of_turn>user
Translate the following text into English. Reply with ONLY the translation.

Text:
Wie heißt die Hauptstadt von Frankreich?<end_of_turn>
<start_of_turn>model

Greedy decoding is recommended for translation. onnxruntime-genai decodes greedily unless do_sample is set explicitly — setting temperature alone has no effect.

License

Gemma is provided under and subject to the Gemma Terms of Use found at ai.google.dev/gemma/terms. Use of this model is also subject to the Gemma Prohibited Use Policy. This repository redistributes a quantized derivative of TranslateGemma; all rights in the underlying model remain with Google.