Kotichitturu/slm-125m-sft-qa-v2

SLM 125M — Legal/Financial Q&A SFT (V2)

Fine-tuned for closed-book Q&A on US case law, SEC filings, and FineWeb-Edu (≈40/40/20 source mix, matching the pretraining distribution).

• Base: Kotichitturu/slm-125m-base
• Training data: qa_20k (18,630 train / 1,000 val)
• Method: full supervised fine-tuning

Evaluation — frozen eval_v2

Scored on a held-out benchmark built from fresh passages (different sampling seed), decontaminated against every training set, and frozen with a sha256 receipt. Judge = claude-sonnet-5 (a different model family from both this model and the data pipeline's teacher), rubric 0–100.

Metric	Value
Judge score	7.6/100 (95% CI 6.6–8.6)
Correct (vs reference)	3.5%
Grounded	20.4%
Refusal recall (answer absent → refused)	0.0% (n=296)
Over-refusal (answer present → refused)	0.0% (n=366)
Numeric match	16.9% (n=148)
Token F1 vs reference	12.4% (n=1030)
Items scored	1325

By task type

Task	Judge /100	Token F1	n
Closed-book QA (no context — recall from weights)	7.2	7.6%	380
Instruction following (no context)	6.1	26.6%	283
RAFT (answer IS in the retrieved context — extraction)	12.7	6.4%	366
Refusal (answer is ABSENT — must decline)	3.1	—	296

Read the refusal row against the RAFT row. A high RAFT score with a low refusal score means the model answers well when the answer is present and fabricates when it is not — the two are the same behaviour measured twice.

Gold answers are model-authored (gemini-3.1-pro) from a source passage. These measure agreement with the reference, not truth.

Usage

from transformers import AutoModelForCausalLM, AutoTokenizer

tok = AutoTokenizer.from_pretrained("Kotichitturu/slm-125m-sft-qa-v2")
model = AutoModelForCausalLM.from_pretrained("Kotichitturu/slm-125m-sft-qa-v2", torch_dtype="bfloat16").eval()

# This model uses CUSTOM chat tokens — the prompt must match training byte-for-byte.
SYSTEM = 'You are a legal and financial expert. Answer accurately and concisely. If unsure, say so.'

prompt = "<|bos|><|system|>" + SYSTEM + "<|user|>" + "<your question>" + "<|assistant|>"
ids = tok(prompt, return_tensors="pt", add_special_tokens=False)
out = model.generate(**ids, max_new_tokens=128, do_sample=False,
                     eos_token_id=tok.convert_tokens_to_ids("<|eos|>"))
print(tok.decode(out[0][ids["input_ids"].shape[1]:], skip_special_tokens=True))

How the data was built

Teacher-generated and judge-filtered, not scraped:

1. Chunk the corpus into 250–350-word passages, stratified by jurisdiction / SEC section.
2. Generate Q&A with gpt-4o-mini, then a regex prefilter (self-containment, anchoring) removes ~20% for free before any paid judging.
3. Judge every pair with gemini-3.1-flash-lite for faithfulness and correctness — a different model family from the teacher, so it cannot reward its own idiom.
4. Filter chain of survivors (not rank-top-K): MinHash near-dup removal → per-source quota 40/40/20 → question-type cap → strata round-robin.
5. Split by document, so a passage's questions never straddle train and validation.

Limitations

• Not legal or financial advice. Outputs can be confidently wrong.
• Trained on US case law and SEC filings; other jurisdictions and document types are out of distribution.
• Gold answers used for both training and evaluation are model-authored, so the reported metrics measure agreement with a reference rather than ground truth.
• A 125M-parameter model — closed-book factual recall is weak by construction. The RAFT variant with retrieval is the intended production path.