--- license: llama3 base_model: GoToCompany/llama3-8b-cpt-sahabatai-v1-instruct library_name: peft pipeline_tag: text-generation language: - id - jv - su datasets: - fairleap-ai/fairleap-driver-chat-sft-43k tags: - base_model:adapter:GoToCompany/llama3-8b-cpt-sahabatai-v1-instruct - lora - qlora - sft - transformers - trl - unsloth - conversational - tool-use - indonesian - javanese - sundanese - gig-economy - ride-hailing - fairleap ---

Fairleap v1 CLM Sahabat-AI 8B Adapter

## ๐Ÿ“˜ Model Overview A QLoRA adapter that turns [`GoToCompany/llama3-8b-cpt-sahabatai-v1-instruct`](https://huggingface.co/GoToCompany/llama3-8b-cpt-sahabatai-v1-instruct) into a driver-welfare assistant for Gojek/GOTO partners in Indonesia, built for the [Fairleap AI](https://github.com/Fairleap-AI) project โ€” a platform addressing income uncertainty and wellbeing for ride-hailing drivers. It is trained for three behaviours: **answering from context it was handed** rather than asking for data, **calling a forecasting tool** when a question needs arithmetic a language model cannot do, and **declining** what falls outside the product. Training data is [`fairleap-ai/fairleap-driver-chat-sft-43k`](https://huggingface.co/datasets/fairleap-ai/fairleap-driver-chat-sft-43k). Sahabat-AI is itself a continued-pretrain of Llama-3 8B on Indonesian, Javanese and Sundanese, which is why this adapter reaches a lower loss than its [Qwen3.5-4B sibling](https://huggingface.co/fairleap-ai/fairleap-v1-clm-qwen3.5-4b-adapter) on the same 12,000 conversations. > [!IMPORTANT] > Trained entirely on **synthetic** conversations that no human reviewed. In held-out > evaluation it **invents a week-over-week earnings comparison in half of all earnings answers** > โ€” quoting the figures it was given correctly, then appending a prior-period total that was > never supplied. Read [Limitations & Biases](#%EF%B8%8F-limitations--biases) before putting it > in front of anyone. ## ๐Ÿš€ Usage This repository holds a **LoRA adapter only** โ€” no base weights. `load_model.py` handles the four things that otherwise look like broken weights: ```python from load_model import load, chat, build_system_prompt, PREDICT_EARNINGS_TOOL model, tokenizer = load() # base + adapter, 4-bit system = build_system_prompt( today="2026-08-24", city="Bekasi", vehicle="motor", risk="sedang", wellness_score=62, period="2026-08-18 s/d 2026-08-24", totals={"Total penghasilan": "Rp1.482.000", "Total order": "88", "Hari kerja": "6 dari 7 hari", "Rata-rata per hari kerja": "Rp247.000", # Supply this and it quotes it instead of inventing one. "Total 7 hari sebelumnya": "Rp1.301.000"}, ) print(chat(model, tokenizer, [ {"role": "system", "content": system}, {"role": "user", "content": "berapa penghasilan saya minggu ini?"}, ])) ``` Offer the tool only when a forecast is plausibly needed โ€” see Limitations: ```python reply = chat(model, tokenizer, messages, tools=[PREDICT_EARNINGS_TOOL]) from load_model import parse_tool_call parse_tool_call(reply) # ('predict_earnings', {'start': '2026-08-25', 'end': '2026-09-01', 'wellness_score': 62}) ``` ### Four traps, handled for you 1. **The base model's own chat template silently destroys tool calls.** Sahabat-AI ships the plain Llama-3 template, which renders every message as `content | trim` and ignores both `tool_calls` and the `tools` argument. An assistant tool-call turn carries `content: ""`, so it renders as an *empty* assistant reply โ€” no error anywhere, roughly 10% of the corpus gone. The tool-capable template is `chat_template.jinja` **in this repository**, and `from_pretrained` on this directory picks it up. Load the tokenizer from the base repo instead and tool use breaks with no warning. 2. **Adapter-only directory.** `AutoModelForCausalLM.from_pretrained(".")` fails and PEFT then retries the local path as a Hub repo id, reporting `HFValidationError: Repo id must be in the form 'repo_name'โ€ฆ`, which reads as a path bug. 3. **The base's generation config disagrees with its own template.** It names `<|end_of_text|>` (128001) as eos while every turn ends with `<|eot_id|>` (128009), so generation never stops. `chat()` passes eos explicitly. 4. **Tool calls come back as one line of JSON keyed `parameters`**, not the `arguments` an OpenAI-shaped parser looks for: `{"name": "predict_earnings", "parameters": {โ€ฆ}}`. **`fairleap-api` must parse this shape** โ€” it differs from the XML block the Qwen adapter emits. ### The system prompt carries the driver The service this was trained for is stateless and identity-blind: every fact about a driver arrives in the request. Training prompts ran ~4,400 characters โ€” persona, style, prohibitions, then city, vehicle, BPJS status, risk tolerance, a 7-day summary and up to 14 daily rows. Give it less and it has less to be correct about; give it nothing and it has nothing to read. ## ๐Ÿ—‚๏ธ Model Details | | | |---|---| | Base model | `GoToCompany/llama3-8b-cpt-sahabatai-v1-instruct` (`LlamaForCausalLM`, 8B) | | Method | QLoRA, 4-bit NF4, r=32, ฮฑ=32, dropout 0.0, bias none | | Trainable params | 83.9M across 448 tensors (1.03% of 8.11B) | | Target modules | `q_proj` `k_proj` `v_proj` `o_proj` `gate_proj` `up_proj` `down_proj` | | Adapter size | 336 MB (`adapter_model.safetensors`) | | Context | 4,096 tokens (the base has 8,192 positions) | | License | Llama 3 Community License, inherited | Every adapted module sits inside `model.layers.*` โ€” 32 layers ร— 7 projections ร— 2 matrices โ€” verified in the shipped safetensors, not just asserted at train time. ## ๐Ÿ“Š Training | | | |---|---| | Data | 12,000 stratified conversations from `fairleap-driver-chat-sft-43k` train | | Preprocessing | canned follow-up turns truncated (`training/prepare_data.py`) | | Epochs / steps | 2 / 1,500 | | Batch | 2 ร— 8 accumulation = effective 16, length-grouped | | LR | 2e-4 cosine, warmup ratio 0.03, `adamw_8bit`, weight decay 0.01 | | Masking | `train_on_responses_only` โ€” ~12% of tokens enter the loss | | Seed | 20260819 | | Hardware | 1 ร— A100-SXM4-40GB, 286 minutes, 2.81e18 FLOPs | The same 12,000 rows and the same seed as the Qwen sibling, so the two differ by base model rather than by data. | epoch | 0.33 | 0.67 | 1.00 | 1.33 | 1.67 | 2.00 | |---|---|---|---|---|---|---| | eval_loss | 0.8502 | 0.8042 | 0.7762 | 0.7646 | 0.7529 | **0.7499** | Unlike the Qwen run, this curve has largely flattened โ€” the last two passes gained 0.012 and 0.003. The remaining lever is **more data, not more epochs**. > Loss is not comparable across tokenizers. Llama-3 spends ~1.12ร— more tokens on this corpus > than Qwen does, and more tokens each carrying less information mechanically deflates > per-token loss. Qwen's 0.979 against this 0.7499 is a real margin but a smaller one than it > looks โ€” roughly 0.84 once corrected. `training/` reproduces the run end to end. ## ๐Ÿ“ˆ Evaluation **All 427 held-out test conversations, 0 generation errors.** Every reply and its per-record flags ship in `eval_results.jsonl`, so the figures below can be checked rather than trusted: ```sh python eval_model.py --backend unsloth --model . --test fairleap_test.jsonl ``` ### Tool use is the strong result | | | |---|---| | Conversations where the corpus offered the tool | 40 | | Tool calls emitted | **40** | | Malformed calls | **0 / 40** | | Calls that ran on past the call into a fabricated result | **0 / 40** | | Unsolicited calls | **0** | | Forbidden / demoted tool mentions | **0 / 427** | Perfect recall and perfect precision on the distribution it was trained for. The evaluator's `expected tool, none = 11` is not a miss: those are exactly the 11 forecast conversations where no tool was offered. ### Refusals and grounding | | | |---|---| | Out-of-scope conversations declined | **15 / 15** | | Language drift (CJK leakage) | 0 / 427 โ€” see the caveat below | | Grounding failures flagged | 4 / 120 audited | | **Replies inventing a period comparison** | **29 / 427 (6.8%)** | Auditing the 4 flagged grounding failures by hand: one is *correct* arithmetic the checker cannot do (Rp3.989.500 รท 19 = 209,974), two are small slips on derived averages (off by 105 and by 1,000), and one is a genuine fabrication. Qualitatively, it gets calendar facts right that the Qwen adapter got wrong โ€” asked for a weekly summary it rendered `2026-08-21` as *Jumat*, and that date is a Friday โ€” and it reads "minggu depan" as a sensible seven-day window rather than three days. ## โš ๏ธ Limitations & Biases **It invents week-over-week comparisons, in half of all earnings answers.** This is the defect that matters most. ``` scenario invented total rate earnings_qa 27 53 50.9% earnings_forecast 1 32 3.1% multi_intent 1 19 5.3% ALL 29 427 6.8% ``` Always the same template, with a baseline that was never in the prompt: > *"Dibanding 7 hari sebelumnya (Rp928.500), penghasilan Pak naik Rp390.000 atau sekitar 42 > persen."* The context figures around it are quoted perfectly, and the delta and percentage are computed from the invention, so the whole sentence is internally consistent. That is what makes it hard to catch. **Mitigation: put the previous period's total in the system prompt** โ€” then it quotes a real number instead of manufacturing one โ€” and treat any prior-period figure in a reply as unverified unless you supplied it. Note that `check_reply_grounding` caught only 1 of these 29. It accepts any figure within 2% of a context number times one of 26 multipliers, which lets almost anything through. `check_period_comparison` in `eval_model.py` was added for this and is what the table reports. **Small arithmetic slips on derived averages.** Monthly total รท working days came out 105 low in one audited reply and 1,000 low in another. Figures it quotes directly are exact; figures it computes are approximately right. **Javanese and Sundanese drift to Indonesian.** Asked in either, it answers in fluent casual Indonesian โ€” correct content, wrong register โ€” despite `jv` being 10.2% of the corpus. The `language drift 0/427` above measures **CJK leakage only**, a Qwen-era concern; it cannot see this and should not be read as evidence against it. **It over-calls when you offer the tool on a turn that does not need one.** On the natural distribution it never called unbidden (0 unsolicited in 427). But forced โ€” tool offered on a wellness question, or on a recall question whose answer was already in context โ€” it produced a `predict_earnings` call both times. The caller decides, not the model. **Not converged.** Trained on 12,000 of 41,461 available conversations. The loss curve has flattened for this data volume, so more epochs will not help; more data might. This is a v1. **Inherits every dataset limitation** โ€” fully synthetic, unreviewed advice, teacher-model errors about Indonesian financial products, simulated tool results. See the [dataset card](https://huggingface.co/datasets/fairleap-ai/fairleap-driver-chat-sft-43k#%EF%B8%8F-limitations--biases). **Not professional advice.** It discusses debt, insurance selection and investment for a financially vulnerable population, and none of it was expert-reviewed. Not suitable for estimating real driver income, informing platform or labour policy, or any claim about actual gig-economy conditions in Indonesia. ## ๐Ÿ› ๏ธ Tech Stacks - **peft**: The Hugging Face library implementing LoRA and other parameter-efficient finetuning methods. - **unsloth**: A finetuning library giving roughly 2x faster training and much lower VRAM use than stock peft+trl. - **trl**: The Hugging Face supervised-finetuning and RL library that ran the training loop. - **transformers**: The Hugging Face library providing the base model, tokenizer and chat template. - **bitsandbytes**: The 4-bit NF4 quantisation backend that lets an 8B model train on one 40 GB card. - **torch**: The deep learning framework everything above runs on. ## ๐Ÿ“š Citation ```bibtex @misc{fairleap_v1_clm_sahabatai_8b_adapter, title = {Fairleap v1 CLM Sahabat-AI 8B Adapter}, author = {Fairleap AI}, year = {2026}, publisher = {Hugging Face}, howpublished = {\url{https://huggingface.co/fairleap-ai/fairleap-v1-clm-sahabatai-8b-adapter}} } ``` ## ๐Ÿ“ License The base model is **Llama-3 derived**, so the [Llama 3 Community License](https://llama.meta.com/llama3/license/) governs use of this adapter too โ€” you cannot run it without loading those weights. Its Acceptable Use Policy applies, and attribution ("Built with Meta Llama 3") is required for derivative works. The code in this repository (`load_model.py`, `eval_model.py`, `training/`) is MIT.