---
language:
- sw
license: cc-by-nc-4.0
library_name: pytorch
pipeline_tag: text-generation
tags:
- swahili
- kiswahili
- gpt
- instruction-tuned
- sft
- from-scratch
- african-languages
base_model: Benjamin-png/swahili-gpt-71m
datasets:
- Benjamin-png/swahili-instruction-mix
---
# swahili-gpt-71m-instruct
An **instruction-tuned** Swahili model — [`Benjamin-png/swahili-gpt-71m`](https://huggingface.co/Benjamin-png/swahili-gpt-71m)
(71M, trained from scratch) fine-tuned (SFT) to **follow instructions and answer
questions** in Kiswahili, instead of just completing text.
[](https://colab.research.google.com/drive/1R00lE1vjbiHRXdqZqUeyKGAo8E6KZcjY?usp=sharing)
— **chat with it in your browser, no setup.**
> ✅ **Final checkpoint — SFT complete (3 epochs, 7,989 steps).** It follows the
> instruction format and answers in Swahili, but as a 71M model it has limited
> knowledge and can be repetitive or wrong. Treat it as a **proof-of-concept**,
> not a finished assistant.
## Prompt format
The model was fine-tuned on this exact template — use it for best results:
```
### Maagizo:
{your instruction or question}
### Jibu:
```
(Optionally add `### Ingizo:\n{context}\n` between them when you have extra context.)
The model generates the answer after `### Jibu:` and stops at ``.
## Usage
```bash
pip install torch sentencepiece huggingface_hub
```
```python
from huggingface_hub import hf_hub_download
import sentencepiece as spm
REPO = "Benjamin-png/swahili-gpt-71m-instruct"
hf_hub_download(REPO, "modeling_kiswahili.py", local_dir=".")
from modeling_kiswahili import load_model, generate
weights = hf_hub_download(REPO, "pytorch_model.pt")
hf_hub_download(REPO, "model_config.json")
tok = hf_hub_download(REPO, "swahili_tokenizer.model")
model, cfg = load_model(weights, device="cpu") # or "cuda"
sp = spm.SentencePieceProcessor(); sp.load(tok)
def ask(q):
prompt = f"### Maagizo:\n{q}\n### Jibu:\n"
out = generate(model, sp, prompt, device="cpu", max_new_tokens=120)
return out.split("### Jibu:")[-1].split("### Maagizo:")[0].strip()
print(ask("Eleza umuhimu wa elimu kwa jamii."))
```
Or run the included `inference.py`.
## Training
- **Base:** `Benjamin-png/swahili-gpt-71m` (decoder-only Transformer, 12 layers,
8 heads, hidden 512, 32k SentencePiece vocab).
- **Method:** supervised fine-tuning (SFT) with **loss masked to the response**
(the prompt is context only), gradient checkpointing, cosine LR (peak 2e-5).
- **Data:** [`Benjamin-png/swahili-instruction-mix`](https://huggingface.co/datasets/Benjamin-png/swahili-instruction-mix)
— ~85k examples.
### Loss trajectory
SFT loss (computed on **response tokens only**) over fine-tuning steps:
| Step | Val loss | Perplexity |
|---|---|---|
| 500 | 4.05 | ~57 |
| 1000 | 3.79 | ~44 |
| 2000 | 3.57 | ~36 |
| 3000 | 3.47 | ~32 |
| 4000 | 3.41 | ~30 |
| 5000 | 3.37 | ~29 |
| 6000 | 3.34 | ~28 |
| 7000 | 3.33 | ~28 |
| 7500 | 3.33 | ~28 |
Val loss fell steadily across the full 3-epoch run (~85k examples/epoch) and
flattened to **3.33** at this 71M model's capacity. This is the final
checkpoint; training is complete.
## Data sources & attribution
The instruction data is a blend of open Swahili datasets — full credit to:
- [CohereLabs/aya_dataset](https://huggingface.co/datasets/CohereLabs/aya_dataset) (Swahili, native human-written) — oversampled ×5.
- [MBZUAI/Bactrian-X](https://huggingface.co/datasets/MBZUAI/Bactrian-X) (`sw` split) — **CC BY-NC-4.0**.
- [NabajyotiPathak/kiswahili-ai-blended](https://huggingface.co/datasets/NabajyotiPathak/kiswahili-ai-blended) (FineTome-sw + KenSwQuAD + Code-170k-sw + Swahili-Corpus).
## License
**CC BY-NC-4.0 (non-commercial)** — inherited from Bactrian-X in the training mix.
Please keep attribution and respect the non-commercial terms.
## Limitations
71M parameters: little world knowledge, no reliable facts or reasoning, may
repeat or hallucinate. Early SFT checkpoint. Not for production or any
high-stakes use. Built as an open, from-scratch step toward Swahili-first AI.