--- language: - ms - en - zh - ta --- # Malaysian Qwen 2.5 72B Instruct Continue finetuning https://huggingface.co/Qwen/Qwen2.5-72B-Instruct on highly curated 1.5B tokens Malaysian instruction dataset. ## Improvement 1. Support respond in Mandarin, Tamil, Jawi, Manglish, Johor, Kedah, Kelantan, Pahang, Perak, Sabah, Sarawak, Selangor, Negeri Sembilan and Terengganu. 2. Able to code in Mandarin, Tamil, Jawi, Manglish, Johor, Kedah, Kelantan, Pahang, Perak, Sabah, Sarawak, Selangor, Negeri Sembilan and Terengganu. 3. Multi-turn Malaysian context such as related to Malaysian Legislation, politics, religions and languages. ## Training session Finetune on [mesolitica/Malaysian-SFT](https://huggingface.co/datasets/mesolitica/Malaysian-SFT) to make the model understand Malaysian context. ## How we train 1. LoRA on `["q_proj", "k_proj", "v_proj", "o_proj", "gate_proj", "up_proj", "down_proj", "embed_tokens", "lm_head"]`. 2. 128 Rank with alpha 256, or alpha of 2.0 3. Multipacking 8192 context length with proper SDPA causal masking to prevent document contamination and also make sure proper position ids. 4. Chunk CCE loss for LoRA. 5. WanDB at https://wandb.ai/huseinzol05/lora-embedding-128-qwen2.5-72b-malaysian-8k?nw=nwuserhuseinzol05 Source code at https://github.com/mesolitica/malaya/tree/master/session/qwen2.5 ## Benchmark ### MalayMMLU #### Probability next tokens Based on 0-shot official MalayMMLU First token accuracy, ``` Model Accuracy shot by_letter category 0 Malaysian-Qwen2.5-72B-Instruct 81.620958 0shot True STEM 1 Malaysian-Qwen2.5-72B-Instruct 80.820611 0shot True Language 2 Malaysian-Qwen2.5-72B-Instruct 77.536860 0shot True Social science 3 Malaysian-Qwen2.5-72B-Instruct 76.900935 0shot True Others 4 Malaysian-Qwen2.5-72B-Instruct 82.730375 0shot True Humanities {'Social science': 6918, 'Language': 6288, 'Humanities': 4395, 'Others': 4169, 'STEM': 2443} Model : Malaysian-Qwen2.5-72B-Instruct Metric : first Shot : 0shot average accuracy 79.63490686821129 accuracy for STEM 81.62095783872289 accuracy for Language 80.82061068702289 accuracy for Social science 77.53686036426713 accuracy for Others 76.90093547613337 accuracy for Humanities 82.73037542662117 ``` While the original model, ``` Model Accuracy shot by_letter category 0 Qwen2.5-72B-Instruct 80.884159 0shot True STEM 1 Qwen2.5-72B-Instruct 79.103053 0shot True Language 2 Qwen2.5-72B-Instruct 75.802255 0shot True Social science 3 Qwen2.5-72B-Instruct 75.053970 0shot True Others 4 Qwen2.5-72B-Instruct 79.977247 0shot True Humanities {'Social science': 6918, 'Language': 6288, 'Humanities': 4395, 'Others': 4169, 'STEM': 2443} Model : Qwen2.5-72B-Instruct Metric : first Shot : 0shot average accuracy 77.80118118366167 accuracy for STEM 80.88415882112157 accuracy for Language 79.1030534351145 accuracy for Social science 75.80225498699046 accuracy for Others 75.05396977692492 accuracy for Humanities 79.97724687144482 ``` #### First token match using vLLM Based on 0-shot exact first token match using vLLM Guided Decoding, ``` Model Accuracy shot category 0 Malaysian-Qwen2.5-72B-Instruct 80.229226 0 STEM 1 Malaysian-Qwen2.5-72B-Instruct 78.101145 0 Language 2 Malaysian-Qwen2.5-72B-Instruct 75.252963 0 Social science 3 Malaysian-Qwen2.5-72B-Instruct 74.358359 0 Others 4 Malaysian-Qwen2.5-72B-Instruct 80.477816 0 Humanities Model : Malaysian-Qwen2.5-72B-Instruct Metric : full Shot : 0 average accuracy 77.28905959608475 accuracy for STEM 80.22922636103151 accuracy for Language 78.10114503816794 accuracy for Social science 75.25296328418618 accuracy for Others 74.35835931878148 accuracy for Humanities 80.4778156996587 ``` While the original model, ``` Model Accuracy shot category 0 Qwen2.5-72B-Instruct 81.129758 0 STEM 1 Qwen2.5-72B-Instruct 78.975827 0 Language 2 Qwen2.5-72B-Instruct 75.397514 0 Social science 3 Qwen2.5-72B-Instruct 75.077956 0 Others 4 Qwen2.5-72B-Instruct 79.954494 0 Humanities Model : Qwen2.5-72B-Instruct Metric : full Shot : 0 average accuracy 77.67728079957048 accuracy for STEM 81.12975849365534 accuracy for Language 78.97582697201018 accuracy for Social science 75.39751373229257 accuracy for Others 75.0779563444471 accuracy for Humanities 79.95449374288964 ``` ## Acknowledgement Special thanks to https://www.sns.com.my and Nvidia for 8x H100 node!