---
language:
- ar
license: apache-2.0
base_model: LiquidAI/LFM2-1.2B-RAG
tags:
- arabic
- rag
- question-answering
- fine-tuned
- Lora
- liquid
- extractive-qa
datasets:
- hsseinmz/arcd
library_name: transformers
pipeline_tag: question-answering
---

# LFM2-1.2B-RAG Arabic (LoRA Fine-tuned)

Fine-tuned version of [LiquidAI/LFM2-1.2B-RAG](https://huggingface.co/LiquidAI/LFM2-1.2B-RAG) for Arabic reading comprehension and question answering tasks using **LoRA (Low-Rank Adaptation)** technique.

## 📋 Model Description

This model specializes in extractive question answering for Arabic text. It has been fine-tuned using LoRA on the Arabic Reading Comprehension Dataset (ARCD) to improve its ability to answer questions based on provided context in Modern Standard Arabic.

**Key Features:**
- Optimized for Arabic extractive QA
- Context-based question answering
- Maintains faithfulness to source documents
- Efficient fine-tuning via LoRA (rank=16)

## 🎯 Intended Use

### Direct Use
- Arabic question answering systems
- RAG (Retrieval-Augmented Generation) applications for Arabic content
- Information extraction from Arabic documents
- Educational tools for Arabic reading comprehension

### Downstream Use
Can be further fine-tuned for:
- Domain-specific QA (medical, legal, financial)
- Multi-turn conversational QA
- Document summarization with Q&A

### Out-of-Scope Use
**Not recommended for:**
- Open-domain question answering without context
- Creative writing or content generation
- Translation tasks
- Code generation

## 🚀 How to Use

### Basic Usage

```python
import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

# Load model and tokenizer
model_id = "azeddinShr/LFM2-1.2B-RAG-ARABIC-LoRA"
model = AutoModelForCausalLM.from_pretrained(
    model_id,
    device_map="auto",
    torch_dtype=torch.bfloat16,
)
tokenizer = AutoTokenizer.from_pretrained(model_id)

# Prepare input
context = "نيوم هو مشروع ضخم في شمال غرب السعودية بتكلفة 500 مليار دولار."
question = "ما هي تكلفة مشروع نيوم؟"

prompt = f"استخدم السياق التالي للإجابة على السؤال:\n\n{context}\n\nالسؤال: {question}"

# Generate answer
messages = [{"role": "user", "content": prompt}]
input_ids = tokenizer.apply_chat_template(
    messages,
    add_generation_prompt=True,
    return_tensors="pt"
).to(model.device)

with torch.no_grad():
    outputs = model.generate(
        input_ids,
        max_new_tokens=150,
        temperature=0.0,
        do_sample=False,
        pad_token_id=tokenizer.eos_token_id
    )

answer = tokenizer.decode(outputs[0][input_ids.shape[1]:], skip_special_tokens=True)
print(answer)  # Output: 500 مليار دولار

```

## 📊 Training Details

### Training Data
- **Dataset:** [hsseinmz/arcd](https://huggingface.co/datasets/hsseinmz/arcd)
- **Training samples:** 693
- **Validation samples:** 351
- **Test samples:** 351
- **Language:** Modern Standard Arabic
- **Task:** Extractive question answering

### Training Procedure

**Fine-tuning method:** LoRA (Low-Rank Adaptation)

**Hyperparameters:**
- **Base model:** LiquidAI/LFM2-1.2B-RAG
- **Epochs:** 10
- **Batch size:** 16 (4 per device × 4 gradient accumulation)
- **Learning rate:** 2e-4
- **Optimizer:** AdamW (8-bit paged)
- **LR scheduler:** Cosine
- **Warmup steps:** 50
- **Weight decay:** 0.01
- **LoRA rank (r):** 16
- **LoRA alpha:** 32
- **LoRA dropout:** 0.05
- **Target modules:** w1, w2, w3, q_proj, k_proj, v_proj, out_proj, in_proj

**Training infrastructure:**
- Precision: bfloat16
- Gradient checkpointing: Enabled
- Framework: Hugging Face Transformers + PEFT + TRL


## 🔒 Ethical Considerations

- This model should not be used for generating misleading or false information
- Users should verify factual claims, especially for sensitive topics
- The model's responses reflect patterns in training data and may not represent complete or unbiased information

## 📜 Citation

If you use this model in your research or application, please cite:

```bibtex
@misc{lfm2-rag-arabic-lora,
  author = {Azeddin sahir},
  title = {LFM2-1.2B-RAG Arabic (LoRA Fine-tuned)},
  year = {2025},
  publisher = {Hugging Face},
  howpublished = {\url{https://huggingface.co/azeddinShr/lfm2-1.2b-arabic-qa-lora}}
}
```

## 👍🏻 Acknowledgments

- **Base Model:** [LiquidAI](https://www.liquid.ai/) for LFM2-1.2B-RAG
- **Dataset:** [ARCD](https://huggingface.co/datasets/hsseinmz/arcd) - Arabic Reading Comprehension Dataset
- **Framework:** Hugging Face Transformers, PEFT, TRL

## 📄 License

Same as based model

## 📧 Contact

For questions, issues, or collaboration opportunities, please open an issue in the model repository, contact via Hugging Face, or email me directly at [azdinsahir11@gmail.com](mailto:azdinsahir11@gmail.com).