🇲🇲 Myanmar-Ghost-Instruct-LoRA

Myanmar Language Instruction-Tuned LLM based on Qwen2.5-Coder-1.5B-Instruct

A lightweight LoRA adapter for Myanmar language text generation and instruction following

---

📌 Model Overview

Myanmar-Ghost-Instruct-LoRA is a LoRA (Low-Rank Adaptation) adapter trained on Qwen/Qwen2.5-Coder-1.5B-Instruct to enhance Myanmar (Burmese) language understanding and generation capabilities.

Key Features

• 🐍 Lightweight: Only ~74MB (LoRA adapter)
• 🇲🇲 Myanmar-First: Optimized for Burmese text generation
• 💻 Code Capable: Base model retains code generation abilities
• ⚡ Fast Inference: Low-rank adaptation for efficient deployment
• 🔧 Easy Integration: Compatible with PEFT and Transformers libraries

Model Tree

Qwen/Qwen2.5-1.5B
└── Qwen/Qwen2.5-Coder-1.5B
    └── Qwen/Qwen2.5-Coder-1.5B-Instruct
        └── amkyawdev/Myanmar-Ghost-Instruct-LoRA ✅ (this model)

---

🚀 Quick Start

Using PEFT (Recommended)

from peft import PeftModel
from transformers import AutoModelForCausalLM, AutoTokenizer
import torch

# Load base model and tokenizer
base_model = AutoModelForCausalLM.from_pretrained(
    "Qwen/Qwen2.5-Coder-1.5B-Instruct",
    device_map="auto",
    torch_dtype=torch.float16,
    trust_remote_code=True
)

tokenizer = AutoTokenizer.from_pretrained(
    "Qwen/Qwen2.5-Coder-1.5B-Instruct",
    trust_remote_code=True
)

# Load LoRA adapter
model = PeftModel.from_pretrained(
    base_model, 
    "amkyawdev/Myanmar-Ghost-Instruct-LoRA"
)

# Generate text
messages = [
    {"role": "user", "content": "မြန်မာစာတစ်ပိုဒ် ရေးပါ။"}
]
text = tokenizer.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
inputs = tokenizer([text], return_tensors="pt").to(model.device)

outputs = model.generate(
    **inputs,
    max_new_tokens=512,
    temperature=0.7,
    top_p=0.9
)
print(tokenizer.decode(outputs[0], skip_special_tokens=True))

Using Transformers Pipeline

from transformers import pipeline

pipe = pipeline(
    "text-generation",
    model="amkyawdev/Myanmar-Ghost-Instruct-LoRA",
    model_kwargs={"device_map": "auto", "torch_dtype": "float16"}
)

messages = [
    {"role": "user", "content": "မြန်မာစာတစ်ပိုဒ် ရေးပါ။"}
]
output = pipe(messages, max_new_tokens=512, temperature=0.7)
print(output[0]["generated_text"])

Using vLLM

# Install vLLM
pip install vllm

# Start server
vllm serve "amkyawdev/Myanmar-Ghost-Instruct-LoRA" --dtype float16

# API call
curl -X POST "http://localhost:8000/v1/chat/completions" \
  -H "Content-Type: application/json" \
  --data '{
    "model": "amkyawdev/Myanmar-Ghost-Instruct-LoRA",
    "messages": [{"role": "user", "content": "မြန်မာစာတစ်ပိုဒ် ရေးပါ။"}]
  }'

---

📊 Technical Specifications

LoRA Configuration

Parameter	Value
PEFT Type	LORA
Rank (r)	16
Alpha	32
Dropout	0.05
Target Modules	q_proj, k_proj, v_proj, o_proj, gate_proj, up_proj, down_proj
Bias	none
Task Type	CAUSAL_LM

Tokenizer

Property	Value
Tokenizer Class	Qwen2Tokenizer
Model Max Length	32,768 tokens
Special Tokens	`<
Padding Token	`<

Adapter File Size

• adapter_model.safetensors: ~74 MB
• Total model size (with base): ~3-4 GB

---

🏋️ Training Details

Training Hyperparameters

Parameter	Value
Total Steps	200
Save Steps	100
Batch Size	1
Max Steps per Epoch	12,500 (estimated)
Training Epochs	1
Max Learning Rate	2e-4 (warmup)
Final Learning Rate	~5.2e-8
Training Framework	PEFT 0.19.1
Base Model	Qwen2.5-Coder-1.5B-Instruct

Training Progress

Step	Loss	Learning Rate	Grad Norm
1	12.01	0.0	6.93
50	~2.5	~1e-4	~3.0
100	~1.5	~5e-5	~2.5
200 (final)	1.91	5.2e-8	2.66

Available Checkpoints

• checkpoint-100/ - Model at step 100
• checkpoint-200/ - Final model at step 200

---

📚 Training Data

This model was trained on Myanmar language instruction datasets including:

• Myanmar V3 Clean Dataset (amkyawdev/myanmar-v3-clean)

  • ~878K samples
  • Cleaned and quality-filtered Myanmar text

• AMK Coder V3 Dataset V2 (amkyawdev/amk-coder-v3-dataset-v2)

  • ~1.01M samples
  • Code and natural language instruction pairs

---

🎯 Intended Uses

Direct Use Cases

• ✅ Myanmar language text generation
• ✅ Burmese language conversation
• ✅ Translation assistance (Myanmar ↔ other languages)
• ✅ Text summarization in Burmese
• ✅ Code generation assistance (preserved from base model)

Downstream Use Cases

• 🔧 Fine-tuning for specific Myanmar NLP tasks
• 🔧 Domain-specific applications (healthcare, legal, education)
• 🔧 Chatbot development for Burmese speakers
• 🔧 Research on low-resource language LLMs

Out-of-Scope Uses

• ⚠️ Medical or legal advice without human verification
• ⚠️ High-stakes decision-making systems
• ⚠️ Production systems without thorough evaluation
• ⚠️ Generating harmful or misleading content

---

⚠️ Bias, Risks, and Limitations

Technical Limitations

1. Model Size: 1.5B parameters may limit performance on complex tasks
2. Training Steps: Limited training (200 steps) may affect instruction-following quality
3. Token Limit: 32,768 context window
4. Resource Requirements: GPU recommended for inference

Sociotechnical Considerations

1. Language Coverage: Optimized primarily for Burmese; may vary for regional dialects
2. Cultural Bias: Training data may reflect specific cultural perspectives
3. Safety: As with any LLM, outputs should be verified before critical use

Recommendations

• Evaluate on your specific use case before production deployment
• Implement appropriate content filtering
• Provide human oversight for sensitive applications
• Consider fine-tuning for domain-specific tasks

---

📈 Evaluation

Evaluation Status

⚠️ Formal benchmark evaluation pending. The model has not been systematically evaluated on standard NLP benchmarks yet.

Recommended Evaluation Tasks

If you evaluate this model, consider the following benchmarks:

1. Myanmar NLP Tasks

   • Myanmar text classification
   • Sentiment analysis (Burmese)
   • Named entity recognition

2. General Language Tasks

   • MMLU (Multilingual Massive Multitask)
   • Hellaswag
   • TruthfulQA

3. Code Generation (inherited from base model)

   • HumanEval
   • MBPP

User Feedback

We welcome community feedback! Please share your evaluation results and use cases in the Discussions tab.

---

🔧 Merge and Deploy

Merge LoRA with Base Model

from peft import PeftModel
from transformers import AutoModelForCausalLM
import torch

base_model = AutoModelForCausalLM.from_pretrained(
    "Qwen/Qwen2.5-Coder-1.5B-Instruct",
    device_map="cpu",
    torch_dtype=torch.float32,
)
model = PeftModel.from_pretrained(base_model, "amkyawdev/Myanmar-Ghost-Instruct-LoRA")

# Merge adapter weights
merged_model = model.merge_and_unload()
merged_model.save_pretrained("merged-model")

Quantization for Deployment

# 4-bit quantization with GGUF
from transformers import AutoModelForCausalLM, BitsAndBytesConfig

quantization_config = BitsAndBytesConfig(
    load_in_4bit=True,
    bnb_4bit_use_double_quant=True,
    bnb_4bit_quant_type="nf4",
    bnb_4bit_compute_dtype=torch.float16
)

model = AutoModelForCausalLM.from_pretrained(
    "amkyawdev/Myanmar-Ghost-Instruct-LoRA",
    quantization_config=quantization_config,
    device_map="auto"
)

---

🌐 Related Models

Explore more models from the author:

Model	Description
Myanmar-Ghost-Instruct-GGUF	GGUF format for local inference
myanmar-ai-v3	Full model version
qwen2.5-myanmar-ai-adapter	Alternative adapter
amk-coder-v2	Coding-focused model

---

📞 Contact & Support

• Author: Aung Myo Kyaw (amkyawdev)
• Website: amkyaw-ai.vercel.app
• GitHub: github.com/AmkyawDev
• Demo: Myanmar AI V3 Demo

Framework Versions

• PEFT: 0.19.1
• Transformers: Compatible with latest version
• PyTorch: Recommended 2.0+

---

📄 License

This adapter is released under the Apache 2.0 License.

The base model Qwen/Qwen2.5-Coder-1.5B-Instruct is licensed by Alibaba Cloud and subject to its terms.

---

Made with ❤️ for the Myanmar AI community

This model card was created to improve transparency and reproducibility.