Kotichitturu/slm-125m-sft-raft-v2

SLM 125M — Legal/Financial RAFT (V2)

Fine-tuned for grounded QA over retrieved context, with refusal on US case law, SEC filings, and FineWeb-Edu (≈40/40/20 source mix, matching the pretraining distribution).

• Base: Kotichitturu/slm-125m-base
• Training data: raft_15-20K (18,614 train / 991 val), ~15% refusal
• Method: full supervised fine-tuning

Evaluation — frozen eval_v2

Scored on a held-out benchmark built from fresh passages (different sampling seed), decontaminated against every training set, and frozen with a sha256 receipt. Judge = claude-sonnet-5 (a different model family from both this model and the data pipeline's teacher), rubric 0–100.

Metric	Value
Judge score	16.3/100 (95% CI 14.7–17.9)
Correct (vs reference)	10.9%
Grounded	34.0%
Refusal recall (answer absent → refused)	0.3% (n=296)
Over-refusal (answer present → refused)	0.0% (n=366)
Numeric match	50.0% (n=148)
Token F1 vs reference	17.8% (n=1030)
Items scored	1325

By task type

Task	Judge /100	Token F1	n
Closed-book QA (no context — recall from weights)	5.6	6.6%	381
Instruction following (no context)	5.8	29.2%	283
RAFT (answer IS in the retrieved context — extraction)	42.7	20.8%	366
Refusal (answer is ABSENT — must decline)	7.2	—	295

Read the refusal row against the RAFT row. A high RAFT score with a low refusal score means the model answers well when the answer is present and fabricates when it is not — the two are the same behaviour measured twice.

Gold answers are model-authored (gemini-3.1-pro) from a source passage. These measure agreement with the reference, not truth.

Usage

from transformers import AutoModelForCausalLM, AutoTokenizer

tok = AutoTokenizer.from_pretrained("Kotichitturu/slm-125m-sft-raft-v2")
model = AutoModelForCausalLM.from_pretrained("Kotichitturu/slm-125m-sft-raft-v2", torch_dtype="bfloat16").eval()

# This model uses CUSTOM chat tokens — the prompt must match training byte-for-byte.
SYSTEM = 'You are a legal and financial expert. Use ONLY the provided context documents to answer the question. If the answer is not in the context, reply exactly: "The context does not contain the answer."'

prompt = "<|bos|><|system|>" + SYSTEM + "<|user|>" + "<your question>" + "<|assistant|>"
ids = tok(prompt, return_tensors="pt", add_special_tokens=False)
out = model.generate(**ids, max_new_tokens=128, do_sample=False,
                     eos_token_id=tok.convert_tokens_to_ids("<|eos|>"))
print(tok.decode(out[0][ids["input_ids"].shape[1]:], skip_special_tokens=True))

RAFT behaviour. Give the model retrieved documents followed by the question. If the answer is not in the context it is trained to reply "The context does not contain the answer." rather than guess — that refusal is the point of the format, not a failure.

How the data was built

Teacher-generated and judge-filtered, not scraped:

1. Chunk the corpus into 250–350-word passages, stratified by jurisdiction / SEC section.
2. Generate Q&A with gpt-4o-mini, then a regex prefilter (self-containment, anchoring) removes ~20% for free before any paid judging.
3. Judge every pair with gemini-3.1-flash-lite for faithfulness and correctness — a different model family from the teacher, so it cannot reward its own idiom.
4. Filter chain of survivors (not rank-top-K): MinHash near-dup removal → per-source quota 40/40/20 → question-type cap → strata round-robin.
5. Split by document, so a passage's questions never straddle train and validation.

Limitations

• Not legal or financial advice. Outputs can be confidently wrong.
• Trained on US case law and SEC filings; other jurisdictions and document types are out of distribution.
• Gold answers used for both training and evaluation are model-authored, so the reported metrics measure agreement with a reference rather than ground truth.
• A 125M-parameter model — closed-book factual recall is weak by construction. The RAFT variant with retrieval is the intended production path.