---
base_model: unsloth/llama-3.1-8b-bnb-4bit
tags:
- transformers
- unsloth
- llama
- qlora
- peft
- low-resource
- nigerian-english
- nigerian-pidgin
license: apache-2.0
language:
- en
datasets:
- saheedniyi/naijaweb
---

# NaijaLLaMA-8B (QLoRA Adapter)

## Overview

This repository contains the LoRA adapter weights for **NaijaLLaMA-8B**, a parameter-efficient fine-tuned variant of LLaMA-3.1-8B adapted for Nigerian English and Nigerian Pidgin contexts.

The base model used was:

> `unsloth/llama-3.1-8b-bnb-4bit`

Fine-tuning was performed using **QLoRA** via the [Unsloth](https://github.com/unslothai/unsloth) framework and Hugging Face TRL.

⚠️ This repository contains **LoRA adapter weights only**, not the full merged model.

---

## Motivation

Large language models are predominantly trained on Western-centric corpora and may underperform on culturally grounded Nigerian expressions and Pidgin syntax.

This work investigates whether conservative, low-rank QLoRA adaptation on a small Nigerian corpus (12,000 samples) can induce measurable contextual alignment under strict hardware constraints (single Tesla T4, 16GB VRAM).

This model accompanies the paper:

> *Low-resource Fine-Tuning of LLaMA-3.1 using QLoRA for Nigerian Language Contexts*

---

## Training Configuration

### Dataset
- Source: `saheedniyi/naijaweb`
- Subset size: 12,000 training samples
- Validation set: 500 samples
- Composition (approximate):
  - Nigerian English (45%)
  - Nigerian Pidgin (30%)
  - Code-switched text (25%)

### Hardware
- Platform: Kaggle
- GPU: Tesla T4 (16GB VRAM)

### Fine-Tuning Method
- Technique: QLoRA (4-bit NF4 quantization)
- Target modules:
  - `q_proj`
  - `k_proj`
  - `v_proj`
  - `o_proj`

### Hyperparameters
- LoRA rank (r): 64
- LoRA alpha (α): 16
- Dropout: 0
- Optimizer: AdamW (8-bit)
- Learning rate: 2e-4 (linear decay)
- Batch size: 2
- Gradient accumulation: 8
- Effective batch size: 16
- Epochs: 1
- Max sequence length: 1024
- Random seed: 42

---

## Training Behavior

Training loss decreased from approximately 2.04 to 1.98 over one epoch, indicating stable optimization under conservative scaling.

Perplexity on the held-out validation subset reduced from 8.20 (base) to 7.38 (fine-tuned).

Note: This model is an exploratory academic artifact. Improvements are modest and primarily observed in culturally grounded prompts.

---

## Usage

Load the base model and attach the adapters:

```python
from unsloth import FastLanguageModel

model, tokenizer = FastLanguageModel.from_pretrained(
    model_name="unsloth/llama-3.1-8b-bnb-4bit",
    max_seq_length=1024,
    load_in_4bit=True,
)

model.load_adapter("luminaudoabba/llama3-naijaweb-merged")
model.enable_adapter_layers()