Caracal_AfroLlama_int4

This model is a fine-tuned, 4-bit quantized adapter version of Jacaranda/AfroLlama_V1 on an instruction-tuning and downstream optimization dataset. It achieves the following results on the evaluation set:

• Loss: 1.8681
• ~ 3 days on NVIDIA A100 GPU.

Authors & Acknowledgments

Author

• Theophilus Lincoln Owiti — Carnegie Mellon University Africa

Acknowledgments

We gratefully acknowledge OpenToken and their infrastructure partner Leafcloud for providing the GPU compute resources that made training and quantizing this model possible.

---

Citation

If you use this model or its outputs in your research, please cite:

@misc{owiti2026caracalafrollama,
  title        = {Caracal AfroLlama int4},
  author       = {Owiti, Theophilus Lincoln},
  year         = {2026},
  publisher    = {Hugging Face},
  journal      = {Hugging Face Repository},
  howpublished = {\url{[https://huggingface.co/theophilusowiti/Caracal_AfroLlama_int4](https://huggingface.co/theophilusowiti/Caracal_AfroLlama_int4)}},
  institution  = {Carnegie Mellon University Africa},
  note         = {GPU compute provided by OpenToken and Leafcloud}
}

Model description

Caracal_AfroLlama_int4 leverages Low-Rank Adaptation (LoRA) integrated with 4-bit precision quantization to yield a highly parameter-efficient model tailored for low-resource African languages. By building upon the robust multilingual foundations of AfroLlama_V1, this variant optimizes performance boundaries across diverse structural contexts while drastically reducing deployment memory constraints and hardware requirements.

Usage

from transformers import AutoModelForCausalLM, AutoTokenizer
from peft import PeftConfig, PeftModel
#Configuration
model_name = "theophilusowiti/Caracal_AfroLlama_int4"

# Get the adapter config to find the base model
peft_config = PeftConfig.from_pretrained(model_name)

# Load the tokenizer
tokenizer = AutoTokenizer.from_pretrained(model_name)

# Load the base model with quantization configuration
base_model = AutoModelForCausalLM.from_pretrained(
    peft_config.base_model_name_or_path,
    device_map="auto",
    trust_remote_code=True,
    torch_dtype=torch.float16,
    force_download=True,
    quantization_config = bnb_config_4bit
    # quantization_config=bnb_config_8bit, # Uncomment if bnb_config_8bit is defined
)

#Load and attach the PEFT adapter on top of the base model
model = PeftModel.from_pretrained(base_model, model_name)

if tokenizer.pad_token is None:
    tokenizer.pad_token = tokenizer.eos_token



SHOW_TAGS = True   #  True: Shows structural SFT tags | False: Clean, raw text only

# Fixed Swahili prompt
#USER_INPUT = 'Context: Rais wa Jamuhuri wa Kenya kwa sasa hivi ni William Samoei Ruto. Wanzeka katika Afrika Mashariki ambao pia ni marais ni Rais wa nchi ya kigeni Rwanda, ambaye ni Mweshimiwa Paul Kagame, upande wa Uganda ni Yoweri Kaguta Museveni na Tanzania, rais wake ni Mama Samia Suluhu.\nQuestion: Rais wa Kenya ni nani?'

#USER_INPUT = "In this task you are given a premise and two alternatives in Swahili. You must choose the alternative that is more plausibly the cause or effect of the situation described by the premise. The input format is \"premise (1)alternative_1(2)alternative_2\", the output should either be \"1\" or \"2\" based on your judgment. Naam, sikukuwa nafikiri juu ya hilo, lakini nilichanganyikiwa sana, na, hatimaye nikaendelea kuzungumza naye tena. (1)Sijaongea na yeye tena. (2)Ulitokea mambo tukawa tunawasiliana."



USER_INPUT = f"Context: Kenya na wananchi wa Kenya wamesaidika kwa ukuzaji wa Mpesa. Mpesa imefanya biashara ya wa Kenya kuwa rahisi.
\nQuestion: Niambie jinsi ambavyo Mpesa imewasadia wakenya?""

prompt = f"<Input>\n{USER_INPUT}\n</Input>\n<Answer>\n"

# Tokenize and generate
inputs = tokenizer(prompt, return_tensors="pt").to(model.device)

print("\nGenerating response, please wait...")

with torch.no_grad():
    outputs = model.generate(
        **inputs,
        max_new_tokens=120,
        do_sample=True,
        temperature=0.1,
        top_p=0.9,
        repetition_penalty=1.15,      
        no_repeat_ngram_size=3,       
        pad_token_id=tokenizer.eos_token_id,
        eos_token_id=tokenizer.eos_token_id,
    )


print("\nCaracal:")

if SHOW_TAGS:
    # Option A: Show the entire true SFT structure exactly as generated
    true_output = tokenizer.decode(outputs[0], skip_special_tokens=True)
    print(true_output.strip())
else:
    # Option B: Extract only the text between <Answer> and </Answer>
    # Slicing out the prompt first
    raw_generation = tokenizer.decode(outputs[0][inputs["input_ids"].shape[-1]:], skip_special_tokens=True)
    
    # Programmatically clean up any structural tags or artifacts from the final view
    clean_output = raw_generation.replace("<s>", "").replace("</s>", "").strip()
    print(clean_output)

Expected output

The model expects the <Input>...</Input> / <Answer>...</Answer> instruction format used during SFT, e.g.:

Generating response, please wait ...


Caracal:
<Input>
Context: Kenya na wananchi wa Kenya wamesaidika kwa ukuzaji wa Mpesa. Mpesa imefanya biashara ya wa Kenya kuwa rahisi.
Question: Niambie jinsi ambavyo Mpesa imewasadia wakenya?
</Input>
<Answer>
Mpesa inawapa wakazi wa Kenya njia salama, yenye ufanisi na rahisishaji ya kusafirisha pesa .< /Answer>

---

Intended uses & limitations

Intended Uses

Multilingual Instruction Following: Well-suited for zero-shot and few-shot task adaptation in primary African languages.

Downstream Fine-Tuning: Serves as a lightweight baseline for intent classification, sentiment parsing, and text generation.

Edge Deployment: Highly optimized for lower-tier GPU environments or edge devices requiring a minimal INT4 footprint.

Limitations

Hallucination Vectors: Like its base model, it remains susceptible to generating inaccurate facts when prompted outside its core linguistic domain.

Code-Switching Variance: Performance may degrade gracefully but noticeably when parsing extreme conversational dialect mixing or localized slang not captured during training.

Training and evaluation data

Training and Evaluation Data The model was adapted using specialized multilingual instruction sets focusing heavily on regional representation. Training subsets prioritize structural semantic integrity across high-priority regional vernaculars across East, West, Central, and Southern Africa. Specifically, for instruction fine-tuning tasks, this model utilized the MURI dataset, which resulted in approximately 60,000 instruction-tuning examples.

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

• learning_rate: 2e-05
• train_batch_size: 16
• eval_batch_size: 16
• seed: 42
• gradient_accumulation_steps: 2
• total_train_batch_size: 32
• optimizer: Use paged_adamw_8bit with betas=(0.9,0.999) and epsilon=1e-08 and optimizer_args=No additional optimizer arguments
• lr_scheduler_type: cosine
• lr_scheduler_warmup_steps: 0.03
• num_epochs: 2

Top Performing Languages

East Africa: Swahili (swa), Amharic (amh), Luganda (lug), Kinyarwanda (kin)

West Africa: Hausa (hau), Yoruba (yor), Igbo (ibo)

Central Africa: Lingala (lin)

Southern Africa: Xhosa (xho)

Framework versions

• PEFT 0.19.1
• Transformers 5.5.1
• Pytorch 2.12.0+cu126
• Datasets 4.8.5
• Tokenizers 0.22.2