luminaudoabba/llama3-naijaweb-merged

NaijaLLaMA-8B (QLoRA Adapter)

Overview

This repository contains the LoRA adapter weights for NaijaLLaMA-8B, a parameter-efficient fine-tuned variant of LLaMA-3.1-8B adapted for Nigerian English and Nigerian Pidgin contexts.

The base model used was:

unsloth/llama-3.1-8b-bnb-4bit

Fine-tuning was performed using QLoRA via the Unsloth framework and Hugging Face TRL.

⚠️ This repository contains LoRA adapter weights only, not the full merged model.

---

Motivation

Large language models are predominantly trained on Western-centric corpora and may underperform on culturally grounded Nigerian expressions and Pidgin syntax.

This work investigates whether conservative, low-rank QLoRA adaptation on a small Nigerian corpus (12,000 samples) can induce measurable contextual alignment under strict hardware constraints (single Tesla T4, 16GB VRAM).

This model accompanies the paper:

Low-resource Fine-Tuning of LLaMA-3.1 using QLoRA for Nigerian Language Contexts

---

Training Configuration

Dataset

• Source: saheedniyi/naijaweb
• Subset size: 12,000 training samples
• Validation set: 500 samples
• Composition (approximate):
  • Nigerian English (45%)
  • Nigerian Pidgin (30%)
  • Code-switched text (25%)

Hardware

• Platform: Kaggle
• GPU: Tesla T4 (16GB VRAM)

Fine-Tuning Method

• Technique: QLoRA (4-bit NF4 quantization)
• Target modules:
  • q_proj
  • k_proj
  • v_proj
  • o_proj

Hyperparameters

• LoRA rank (r): 64
• LoRA alpha (α): 16
• Dropout: 0
• Optimizer: AdamW (8-bit)
• Learning rate: 2e-4 (linear decay)
• Batch size: 2
• Gradient accumulation: 8
• Effective batch size: 16
• Epochs: 1
• Max sequence length: 1024
• Random seed: 42

---

Training Behavior

Training loss decreased from approximately 2.04 to 1.98 over one epoch, indicating stable optimization under conservative scaling.

Perplexity on the held-out validation subset reduced from 8.20 (base) to 7.38 (fine-tuned).

Note: This model is an exploratory academic artifact. Improvements are modest and primarily observed in culturally grounded prompts.

---

Usage

Load the base model and attach the adapters:

from unsloth import FastLanguageModel

model, tokenizer = FastLanguageModel.from_pretrained(
    model_name="unsloth/llama-3.1-8b-bnb-4bit",
    max_seq_length=1024,
    load_in_4bit=True,
)

model.load_adapter("luminaudoabba/llama3-naijaweb-merged")
model.enable_adapter_layers()