Mixed Precision GGUF layer quantization of translategemma-12b-it by Google

Original model: https://huggingface.co/google/translategemma-12b-it

The hybrid quant employs different quantization levels on a per layer basis to increased flexibility of trading off performance vs file size. Less parameter bits are used at deep layers and more bits at cortex layers to simultaneously optimize quantized size and model performance. An extended layer definition E quant Q4_E_H for the model is defined as follows:

   LAYER_TYPES='[
   ["A","attn","Q","attn_q","K","attn_k","V","attn_v","O","attn_o","S","ssm","F","ffn","G","ffn_g","U","ffn_u","D","ffn_d"],
   ["MAP","VOD","0","QN_K","2","Q2_K","3","Q3_K","4","Q4_K","5","Q5_K","6","Q6_K","8","Q8_0","h","F16","f","F32"],
   [0 ,"Q6_K_666"],[1 ,"Q6_K_655"],[2 ,"Q5_K_655"],[3 ,"Q5_K_555"],[4 ,"Q4_K_544"],[5 ,"Q4_K_555"],[6 ,"Q4_K_544"],[7 ,"Q4_K_554"],
   [8 ,"Q4_K_544"],[9 ,"Q4_K_544"],[10,"Q4_K_544"],[11,"Q4_K_555"],[12,"Q4_K_544"],[13,"Q4_K_544"],[14,"Q4_K_544"],[15,"Q4_K_544"],
   [16,"Q4_K_544"],[17,"Q4_K_555"],[18,"Q4_K_544"],[19,"Q4_K_544"],[20,"Q4_K_544"],[21,"Q4_K_544"],[22,"Q4_K_544"],[23,"Q4_K_555"],
   [24,"Q4_K_644"],[25,"Q4_K_644"],[26,"Q4_K_644"],[27,"Q4_K_644"],[28,"Q4_K_644"],[29,"Q4_K_655"],[30,"Q4_K_644"],[31,"Q4_K_644"],
   [32,"Q4_K_654"],[33,"Q4_K_654"],[34,"Q4_K_654"],[35,"Q4_K_655"],[36,"Q4_K_654"],[37,"Q4_K_654"],[38,"Q4_K_654"],[39,"Q4_K_654"],
   [40,"Q4_K_665"],[41,"Q4_K_666"],[42,"Q4_K_665"],[43,"Q4_K_666"],[44,"Q5_K_665"],[45,"Q5_K_666"],[46,"Q5_K_668"],[47,"Q6_K_866"]
   ]'
   FLAGS="--token-embedding-type Q6_K --output-tensor-type Q6_K --layer-types-high"

This quant was optimized to create a ~7.4G file (approx size of Q4_K_M) which can run on 8G VRAM GPUs with minimum quant across layers at Q4_K. The layer quants were tweaked for good performance on the two set of tests shown below in the usage section : handling a semantically tricky translation through a chain of languages and doing on good job on transcribing and translating page 1 of original 1891 Russian publication of War and Peace.

Comparison:

Quant	size	PPL	Comment
Q4_K_M	7.3e9	-	default embed and output
Q4_E_H	7.4e9	8.0	Q6_K embed Q6_K output

Note : PPL is not comparable to previous (deleted) Q4_K_H and Q6_K_H models since an invalid config was used to generate those previous results.

Usage:

translategemma 12b is a vision capable translation model. It can be used together with its multimedia projector layers to process images and text inputs and generate text transcription/translation outputs. The mmproj file is made available in this repository.

The model uses sliding window attention. For inference platforms which support beam search, this must be effectively disabled to enable the beam search to work. In llama.cpp, --swa-full will maintain full context memory and mask out the sliding window to enable beam searches to run. Most likely no other inference platform (vllm, sglang) supports this unique mode and therefore will be incapable of doing beam search based translates.

The translategemmas appear to have been fine tuned from gemma3 base with an extremely large number of tokens across a large number of languages with the help of Gemini to create synthetic training translations. The input window was apparently sized at 2k tokens for this fine tuning task. This will require very large text documents to be chunked up to fit in the relatively small 2k token translation window. The context can be bumped up more than 2k to handle large images.

Text prompt format:

In order for the model to work correctly, a prompt template can be used as follows:

User prompt: <startofturn>user\ntype:text,source_lang_code:{SL},target_lang_code:{TL},text:Text to translate
Assistant prompt: <startofturn>model\ntype:text,source_lang_code:{SL},target_lang_code:{TL},text:

Where {SL} and {TL} are from a set of language codes supported by the model, i.e. {SL}=en, {TL}=de for tranlating english to german. This prompt template avoids the use of braces and quotes in JSON list syntax so the model will not crank out \" in a json list formatted output string or terminate its output with a brace }, neither of which are normally wanted for the translation output. If JSON formatted output is desired, the quotes and braces can be added back in to the prompt template.

Example using a downstream llama.cpp server with an autotokenizer implementing the above prompt template and greedy deterministic sampling:

translate()

{
   phrase=$1
   SOURCE=$2
   TARGET=$3
   echo -n $(SL=$SOURCE TL=$TARGET lm "$phrase")
}

langs=(de es fr ru ja zh en)

phrase='The secret code to open the safe is "dead mens chest"'
SOURCE=en
echo "${SOURCE}: $phrase"
for TARGET in ${langs[@]}; do
   phrase=$(translate "$phrase" $SOURCE $TARGET)
   # Remove lead and trail \n delimiters
   phrase="${phrase#\\n}"
   phrase="${phrase%\\n}"
   echo "${TARGET}: $phrase"
   SOURCE=$TARGET
done

Running this script produces:

en: The secret code to open the safe is "dead mens chest"
de: Der geheime Code, um den Safe zu öffnen, lautet "dead mens chest". }
es: El código secreto para abrir la caja fuerte es "dead mens chest". }
fr: Le code secret pour ouvrir le coffre-fort est "dead mens chest". }
ru: Код, необходимый для открытия сейфа, - "dead mens chest". }
ja: セーフを開けるためのパスコードは「dead mens chest」です。}
zh: 打开保险箱的密码是“dead mens chest”。}
en: The password to open the safe is "dead men's chest". }

As shown the model handles the semantically tricky (by design) translation through a full chain of diverse languages and only blows the code when swinging back into the original language.

Image prompt format:

Experimenting with the model showed it does not work well to directly translate images straight to output text. Instead the model must be prompted to first transcribe the image in its native language to text, then translate the transcription to the desired target language. This strategy was found to work well on a small set of test images. An experimental prompt template to accomplish this is as follows:

User prompt: '<startofturn>user\ntype:image,source_lang_code:{SL},target_lang_code:{TL},image: IMAGE'
Assistant prompt: '<startofturn>model\nHere is the transcription of the text followed by translation to the target language "{TL}": '

where IMAGE will be processed by the autotokenizer to convert a specified image to tokens, {SL} is the native language in the image such as "en", "de", etc., and {TL} is the target language code. Note as in text translate no quotes or braces are used in the user prompt list defs, they are not needed for the model to figure out what to do and get rid of some unneccesary prompt tokens.

Example 1: Translate an image of the first page of War and Peace in native Russian to english:

Image : https://huggingface.co/datasets/steampunque/benchlm/blob/main/warandpeace.png

Feed the image in to the model with no extra prompting outside the prompt template:

SL=ru TL=en lm warandpeace.png

The result should be formatted as follows with greedy deterministic sampling:

Transcription:

891.73 T58 Ovo 1891 v. 1-2

ЧАСТЬ ПЕРВАЯ.

Ну, князь, Генуя и Лукка стали помбестями фамилии Бонапарте. Нѣтъ, я вамъ впередъ говорю, если вы мнѣ не скажете, что у васъ будет война, если вы еще позволите себѣ защищать всѣ гадости, всѣ ужасы этого Антихриста, (право, я вѣрю, что онъ Антихристъ,) - я васъ больше не знаю, вы ужъ не друг мой, вы ужъ не мой вѣрный рабъ, какъ вы говорите. Ну, здравствуйте, здравствуйте. Я вижу, что я васъ путаю; садитесь и разсказывайте.

Такъ говорила въ іюлѣ 1805 года извѣстная Анна Павловна Шереръ, фрейлина и приближенная императрицы Маріи Феодоровны, встрѣчая важнаго и чиновнаго князя Василія, перваго пріѣзжавшаго на ея вечеръ. Анна Павловна кашляла нѣсколько дней, у нее была гриппа, какъ она говорила. (Гриппъ былъ тогда новое слово, употреблявшееся только рѣдки-ми.) Въ записочках, разосланныхъ утромъ съ краснымъ лакеемъ, было написано без различія во всѣхъ:

"Если у васъ, графъ, (или князь,) нѣтъ въ виду ничего лучшаго и если перспектива провестъ вечеръ у бѣдной больной не слишкомъ васъ пуга-етъ, то я буду очень рада видѣть васъ ныне у себя между семью и девятью часами. Анна Шереръ."

Translation:

891.73 T58 Published in 1891 Vol. 1-2

PART ONE.

"Well, Prince, Genoa and Lucca have become members of the Bonaparte family. No, I tell you in advance, if you don't tell me that you will have a war, if you still allow yourself to defend all the filth, all the horrors of this Antichrist, (I mean, I believe that he is the Antichrist), then I don't know you anymore, you are no longer my friend, you are no longer my loyal slave, as you say. Well, goodbye, goodbye. I see that I am confusing you; sit down and tell me."

This was said in July 1805 by the well-known Anna Pavlovna Sherer, lady-in-waiting and confidante of Empress Maria Feodorovna, as she greeted the important and official Prince Vasily, who was arriving for her evening party. Anna Pavlovna had been coughing for several days, and she had the flu, as she said. (The flu was a new word at the time, used only rarely.) In the notes sent out in the morning by the footman with a red uniform, it was written the same in all:

"If you, Count (or Prince), have nothing better in mind, and if the prospect of spending an evening with a poor, sick woman doesn't scare you too much, then I would be very happy to see you at my place between seven and nine o'clock. Anna Sherer."

MT benchmarks for the model are given here: https://huggingface.co/spaces/steampunque/benchlm