---
license: apache-2.0
datasets:
- iapp/rag_thai_laws
language:
- th
- en
base_model:
- ThaiLLM/ThaiLLM-8B
pipeline_tag: text-generation
tags:
- legal
library_name: transformers
---
# ThaiLLM-8B-ThaiLaw

A fine-tuned version of [ThaiLLM/ThaiLLM-8B](https://huggingface.co/ThaiLLM/ThaiLLM-8B) specialized for Thai legal documents and law-related tasks.
> Note : After testing I find that it hallucinate so badly that I can't even recommend anyone using this model.
> I promised that further model release will be better quality.
## Model Description

- **Base Model:** ThaiLLM/ThaiLLM-8B (Qwen3 architecture)
- **Training Data:** Thai legal documents from iapp/rag_thai_laws
- **Language:** Thai (th)
- **License:** Apache 2.0
- **Training Method:** QLoRA (4-bit quantization)

## Training Details

### Dataset
This model was fine-tuned exclusively on:
- **iapp/rag_thai_laws** (42,755 samples)
  - Thai law texts from the Office of the Council of State
  - Cleaned and processed dataset
  - Contains various types of Thai legal documents

### Hardware Configuration
- **GPU:** 2x NVIDIA RTX PRO 4000 (24GB VRAM each)
- **Training Time:** ~2 hours
- **Framework:** PyTorch with Unsloth optimizations

### Training Hyperparameters
- **Method:** QLoRA (4-bit quantization)
- **Max Sequence Length:** 2048
- **Batch Size:** 4 per device
- **Gradient Accumulation Steps:** 8 (effective batch size: 64)
- **LoRA Rank (r):** 16
- **LoRA Alpha:** 32
- **LoRA Dropout:** 0.05
- **Target Modules:** q_proj, k_proj, v_proj, o_proj, gate_proj, up_proj, down_proj
- **Learning Rate:** 2e-4
- **Embedding Learning Rate:** 1e-5
- **Epochs:** 1
- **Optimizer:** adamw_8bit
- **Warmup Ratio:** 0.05
- **Data Packing:** Enabled (2-3x speedup)
- **Gradient Checkpointing:** Enabled

### Training Procedure
The model was trained using Unsloth's efficient QLoRA implementation with the following optimizations:
- 4-bit quantization for memory efficiency
- Data packing for faster training on short texts
- Gradient checkpointing to reduce memory usage
- ChatML template for instruction-following format

## Usage

### Basic Usage

```python
from transformers import AutoModelForCausalLM, AutoTokenizer

model_name = "sthaps/ThaiLLM-8B-ThaiLaw"

model = AutoModelForCausalLM.from_pretrained(
    model_name,
    device_map="auto",
)
tokenizer = AutoTokenizer.from_pretrained(model_name)

# Example usage
messages = [
    {"role": "system", "content": "คุณเป็นผู้ช่วยด้านกฎหมายไทยที่เชี่ยวชาญ คุณต้องตอบคำถามเกี่ยวกับกฎหมายไทยอย่างถูกต้องและครบถ้วน"},
    {"role": "user", "content": "อธิบายเกี่ยวกับพระราชบัญญัติ"},
]

input_text = tokenizer.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
inputs = tokenizer(input_text, return_tensors="pt").to(model.device)

outputs = model.generate(**inputs, max_new_tokens=512, temperature=0.7, top_p=0.95)
response = tokenizer.decode(outputs[0], skip_special_tokens=True)
print(response)
```

### Using with Unsloth (Recommended)

```python
from unsloth import FastLanguageModel

model, tokenizer = FastLanguageModel.from_pretrained(
    model_name="sthaps/ThaiLLM-8B-ThaiLaw",
    max_seq_length=2048,
    dtype=None,
    load_in_4bit=True,
)

# Enable faster inference
FastLanguageModel.for_inference(model)

messages = [
    {"role": "system", "content": "คุณเป็นผู้ช่วยด้านกฎหมายไทยที่เชี่ยวชาญ"},
    {"role": "user", "content": "อธิบายเกี่ยวกับกฎหมายแรงงานไทย"},
]

input_text = tokenizer.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
inputs = tokenizer(input_text, return_tensors="pt").to(model.device)

outputs = model.generate(**inputs, max_new_tokens=512)
response = tokenizer.decode(outputs[0], skip_special_tokens=True)
print(response)
```

## Limitations

- **Training Data Scope:** The model was trained on a single dataset (iapp/rag_thai_laws) and may not represent all areas of Thai law
- **Accuracy:** While trained on legal documents, this is an AI model and may produce incorrect information
- **Professional Advice:** This model should not be used as a substitute for professional legal advice
- **Language:** Primarily optimized for Thai language legal documents
- **Updates:** The model does not have access to real-time legal updates or changes

## Model Architecture

- **Base Architecture:** ThaiLLM (8B parameters)
- **Fine-tuning Method:** QLoRA (Low-Rank Adaptation with 4-bit quantization)
- **Trainable Parameters:** ~0.5% of total parameters (LoRA adapters only)
- **Total Parameters:** ~8 billion

## Training Metrics

- **Training Samples:** 42,755
- **Training Steps:** ~1,336 (with packing)
- **Effective Batch Size:** 64
- **Training Loss:** Decreased steadily throughout training
- **Convergence:** Achieved in 1 epoch with packing

## Acknowledgments

- **Base Model:** [ThaiLLM/ThaiLLM-8B](https://huggingface.co/ThaiLLM/ThaiLLM-8B) by ThaiLLM team
- **Dataset:** [iapp/rag_thai_laws](https://huggingface.co/datasets/iapp/rag_thai_laws) by iAPP
- **Training Framework:** [Unsloth](https://github.com/unslothai/unsloth) for efficient fine-tuning