swahili-gpt-71m-instruct

An instruction-tuned Swahili model — Benjamin-png/swahili-gpt-71m (71M, trained from scratch) fine-tuned (SFT) to follow instructions and answer questions in Kiswahili, instead of just completing text.

— chat with it in your browser, no setup.

✅ Final checkpoint — SFT complete (3 epochs, 7,989 steps). It follows the instruction format and answers in Swahili, but as a 71M model it has limited knowledge and can be repetitive or wrong. Treat it as a proof-of-concept, not a finished assistant.

Prompt format

The model was fine-tuned on this exact template — use it for best results:

### Maagizo:
{your instruction or question}
### Jibu:

(Optionally add ### Ingizo:\n{context}\n between them when you have extra context.) The model generates the answer after ### Jibu: and stops at </s>.

Usage

pip install torch sentencepiece huggingface_hub

from huggingface_hub import hf_hub_download
import sentencepiece as spm

REPO = "Benjamin-png/swahili-gpt-71m-instruct"
hf_hub_download(REPO, "modeling_kiswahili.py", local_dir=".")
from modeling_kiswahili import load_model, generate

weights = hf_hub_download(REPO, "pytorch_model.pt")
hf_hub_download(REPO, "model_config.json")
tok = hf_hub_download(REPO, "swahili_tokenizer.model")

model, cfg = load_model(weights, device="cpu")     # or "cuda"
sp = spm.SentencePieceProcessor(); sp.load(tok)

def ask(q):
    prompt = f"### Maagizo:\n{q}\n### Jibu:\n"
    out = generate(model, sp, prompt, device="cpu", max_new_tokens=120)
    return out.split("### Jibu:")[-1].split("### Maagizo:")[0].strip()

print(ask("Eleza umuhimu wa elimu kwa jamii."))

Or run the included inference.py.

Training

• Base: Benjamin-png/swahili-gpt-71m (decoder-only Transformer, 12 layers, 8 heads, hidden 512, 32k SentencePiece vocab).
• Method: supervised fine-tuning (SFT) with loss masked to the response (the prompt is context only), gradient checkpointing, cosine LR (peak 2e-5).
• Data: Benjamin-png/swahili-instruction-mix — ~85k examples.

Loss trajectory

SFT loss (computed on response tokens only) over fine-tuning steps:

Step	Val loss	Perplexity
500	4.05	~57
1000	3.79	~44
2000	3.57	~36
3000	3.47	~32
4000	3.41	~30
5000	3.37	~29
6000	3.34	~28
7000	3.33	~28
7500	3.33	~28

Val loss fell steadily across the full 3-epoch run (~85k examples/epoch) and flattened to 3.33 at this 71M model's capacity. This is the final checkpoint; training is complete.

Data sources & attribution

The instruction data is a blend of open Swahili datasets — full credit to:

• CohereLabs/aya_dataset (Swahili, native human-written) — oversampled ×5.
• MBZUAI/Bactrian-X (sw split) — CC BY-NC-4.0.
• NabajyotiPathak/kiswahili-ai-blended (FineTome-sw + KenSwQuAD + Code-170k-sw + Swahili-Corpus).

License

CC BY-NC-4.0 (non-commercial) — inherited from Bactrian-X in the training mix. Please keep attribution and respect the non-commercial terms.

Limitations

71M parameters: little world knowledge, no reliable facts or reasoning, may repeat or hallucinate. Early SFT checkpoint. Not for production or any high-stakes use. Built as an open, from-scratch step toward Swahili-first AI.