---
license: apache-2.0
base_model: Kotichitturu/slm-125m-base
tags: [legal, finance, sft, raft, slm]
language: [en]
---

# SLM 125M — Legal/Financial RAFT (V2)

Fine-tuned for **grounded QA over retrieved context, with refusal** on US case law, SEC filings, and FineWeb-Edu
(≈40/40/20 source mix, matching the pretraining distribution).

- **Base:** [Kotichitturu/slm-125m-base](https://huggingface.co/Kotichitturu/slm-125m-base)
- **Training data:** raft_15-20K (18,614 train / 991 val), ~15% refusal
- **Method:** full supervised fine-tuning

## Evaluation — frozen `eval_v2`

Scored on a held-out benchmark built from **fresh passages** (different sampling seed), **decontaminated** against every training set, and frozen with a sha256 receipt. Judge = `claude-sonnet-5` (a different model family from both this model and the data pipeline's teacher), rubric 0–100.

| Metric | Value |
|---|---|
| **Judge score** | **16.3/100** (95% CI 14.7–17.9) |
| Correct (vs reference) | 10.9% |
| Grounded | 34.0% |
| Refusal recall (answer absent → refused) | 0.3% (n=296) |
| Over-refusal (answer present → refused) | 0.0% (n=366) |
| Numeric match | 50.0% (n=148) |
| Token F1 vs reference | 17.8% (n=1030) |
| Items scored | 1325 |

### By task type

| Task | Judge /100 | Token F1 | n |
|---|---|---|---|
| Closed-book QA (no context — recall from weights) | 5.6 | 6.6% | 381 |
| Instruction following (no context) | 5.8 | 29.2% | 283 |
| RAFT (answer IS in the retrieved context — extraction) | 42.7 | 20.8% | 366 |
| Refusal (answer is ABSENT — must decline) | 7.2 | — | 295 |

> **Read the refusal row against the RAFT row.** A high RAFT score with a low refusal score means the model answers well *when the answer is present* and **fabricates when it is not** — the two are the same behaviour measured twice.


> Gold answers are **model-authored** (`gemini-3.1-pro`) from a source passage. These measure **agreement with the reference**, not truth.

## Usage

```python
from transformers import AutoModelForCausalLM, AutoTokenizer

tok = AutoTokenizer.from_pretrained("Kotichitturu/slm-125m-sft-raft-v2")
model = AutoModelForCausalLM.from_pretrained("Kotichitturu/slm-125m-sft-raft-v2", torch_dtype="bfloat16").eval()

# This model uses CUSTOM chat tokens — the prompt must match training byte-for-byte.
SYSTEM = 'You are a legal and financial expert. Use ONLY the provided context documents to answer the question. If the answer is not in the context, reply exactly: "The context does not contain the answer."'

prompt = "<|bos|><|system|>" + SYSTEM + "<|user|>" + "<your question>" + "<|assistant|>"
ids = tok(prompt, return_tensors="pt", add_special_tokens=False)
out = model.generate(**ids, max_new_tokens=128, do_sample=False,
                     eos_token_id=tok.convert_tokens_to_ids("<|eos|>"))
print(tok.decode(out[0][ids["input_ids"].shape[1]:], skip_special_tokens=True))
```

**RAFT behaviour.** Give the model retrieved documents followed by the question. If the answer is not in the context it is trained to reply "The context does not contain the answer." rather than guess — that refusal is the point of the format, not a failure.

## How the data was built

Teacher-generated and **judge-filtered**, not scraped:

1. **Chunk** the corpus into 250–350-word passages, stratified by jurisdiction / SEC section.
2. **Generate** Q&A with `gpt-4o-mini`, then a **regex prefilter** (self-containment, anchoring)
   removes ~20% for free before any paid judging.
3. **Judge** every pair with `gemini-3.1-flash-lite` for faithfulness and correctness — a
   different model family from the teacher, so it cannot reward its own idiom.
4. **Filter chain of survivors** (not rank-top-K): MinHash near-dup removal → per-source quota
   40/40/20 → question-type cap → strata round-robin.
5. **Split by document**, so a passage's questions never straddle train and validation.

## Limitations

- **Not legal or financial advice.** Outputs can be confidently wrong.
- Trained on US case law and SEC filings; other jurisdictions and document types are
  out of distribution.
- Gold answers used for both training and evaluation are **model-authored**, so the reported
  metrics measure agreement with a reference rather than ground truth.
- A 125M-parameter model — closed-book factual recall is weak by construction. The RAFT variant with retrieval is the intended production path.