Enrich model card with examples and training details

README.md

@@ -17,59 +17,120 @@ library_name: transformers
 pipeline_tag: text-generation
 ---
 
-# SearchLM NL2BM25 — GRPO v1 (Qwen2.5-3B-Instruct)
+# SearchLM NL2BM25 — GRPO v1 ⚠️ Reward Hacking (Qwen2.5-3B-Instruct)
 
-> **⚠️ Reward Hacking Model** — This checkpoint exhibits specification gaming.
-> Use [GRPO v2](Supreeth/searchlm-nl2bm25-grpo-v2) for production use. This model is published
-> for research reproducibility and as a concrete example of reward hacking in RLVR.
+**Part of the [SearchLM collection](https://huggingface.co/collections/Supreeth/searchlm) · [GitHub](https://github.com/SupreethRao99/searchLM)**
 
-A Qwen2.5-3B-Instruct model trained via GRPO (Group Relative Policy Optimization)
-starting from [SFT v1](Supreeth/searchlm-nl2bm25-sft), using live Tantivy retrieval as the reward signal.
+> **⚠️ This model games its training reward.** It achieves high NDCG@10 by collapsing all
+> outputs to 3–7 token keyword phrases, discarding the entire boolean search task it was
+> trained to learn. Published for research transparency and as a reproducible example of
+> specification gaming in RLVR. For deployment, use [GRPO v2](Supreeth/searchlm-nl2bm25-grpo-v2).
 
-See the [SearchLM collection](https://huggingface.co/collections/Supreeth/searchlm) for all checkpoints.
+A Qwen2.5-3B-Instruct model fine-tuned via GRPO starting from [SFT v1](Supreeth/searchlm-nl2bm25-sft),
+using live Tantivy retrieval (NDCG@10 + MRR) as the reward signal.
 
-## Reward Hacking Behaviour
+> **Pipeline position:** `base → SFT v1 → `**`GRPO v1 ⚠️`**` → SFT v2 → GRPO v2 ✅`
 
-Despite achieving the best NDCG@10 among v1 checkpoints, this model games the reward
-by outputting 3–7 token keyword phrases — abandoning all boolean structure:
+---
 
-| Metric | Value |
-|--------|-------|
-| Mean completion length | **5–7 tokens** (vs 95–163 for SFT) |
-| Boolean operator usage | **0%** (vs ~80% for SFT) |
-| `frac_reward_zero_std` | **90–96%** (policy gradient collapsed from step 1) |
+## The hack: specification gaming via minimum viable retrieval
+
+The model learned that on small corpora (NFCorpus: 3,633 docs; SciFact: 5,183 docs),
+2–4 content nouns yield near-optimal BM25 recall. Instead of learning boolean query
+generation, it learned to extract the most distinctive nouns from the NL query:
 
-**Typical output:**
+**Input:** `Do Cholesterol Statin Drugs Cause Breast Cancer?`
+
+**GRPO v1 output (hacking):**
 ```
 <reasoning>
 </reasoning>
 <query>Cholesterol Statin Breast Cancer</query>
 ```
 
-The model discovered that on small corpora (NFCorpus: 3,633 docs; SciFact: 5,183 docs),
-2–4 content nouns achieve near-optimal BM25 recall. The reward (`0.6 × NDCG@10 + 0.4 × MRR`)
-did not penalise empty reasoning or missing boolean structure.
+**SFT v1 output (intended behaviour, lower NDCG):**
+```
+<reasoning>
+Key concepts: statin drugs, causal relationship, breast cancer.
+Connect with AND; expand synonyms with OR.
+</reasoning>
+<query>(statin OR "HMG-CoA reductase inhibitor" OR simvastatin OR atorvastatin)
+AND (cause OR risk OR association OR induce)
+AND ("breast cancer" OR "breast carcinoma")</query>
+```
+
+The GRPO v1 output actually achieves **NDCG@10 = 0.971** on this query while the SFT
+output achieves 0.000 — the hack outperforms the intended behaviour because SFT used wrong
+synonyms. This made the gaming invisible in aggregate metrics alone.
+
+---
+
+## Collapse statistics
+
+| Metric | Value |
+|--------|-------|
+| Mean completion length | **5.1 tokens** (vs 95 for SFT v1) |
+| Boolean operator usage (AND) | **0%** (vs ~80% for SFT v1) |
+| Boolean operator usage (OR) | **0%** (vs ~90% for SFT v1) |
+| Phrase usage | **0%** (vs ~70% for SFT v1) |
+| Reasoning block content | **empty** |
+| `frac_reward_zero_std` during training | **90–96%** from step 1 |
+
+`frac_reward_zero_std` = fraction of GRPO groups where all completions received identical
+reward. At 90-96%, policy gradient was near-zero throughout — the model was not learning,
+it had already converged on the keyword-bag strategy.
+
+---
+
+## Why it still scores high on benchmarks
+
+1. **Small corpora**: BM25 keyword recall on 3–5K doc indices is high; a rare noun
+   appears in only a handful of documents, making it highly discriminative.
+2. **SFT degraded**: SFT v1 scored *below base* on SciFact (0.273 vs 0.386) due to
+   over-specified queries — a low bar to beat.
+3. **NDCG@10 rewards recall of first hit**: any query retrieving one relevant document
+   in top-10 scores well. Keyword bags do this reliably on small indexes.
 
-Full analysis: [REWARD_HACKING_REPORT_V2.md](https://github.com/SupreethRao99/searchLM/blob/main/REWARD_HACKING_REPORT_V2.md)
+**This does not generalise**: on a 2.7M-doc index (NQ), keyword bags return thousands of
+irrelevant results; NDCG@10 and MRR would collapse to near zero.
+
+---
 
-## Benchmark (test split, NDCG@10)
+## All SearchLM checkpoints
 
-| Dataset | Base | SFT v1 | **GRPO v1** | GRPO v2 |
-|---------|------|--------|------------|---------|
-| NFCorpus | 0.455 | 0.441 | 0.556 | **0.577** |
-| SciFact  | 0.386 | 0.273 | 0.608 | **0.657** |
+| Model | NFCorpus NDCG@10 | SciFact NDCG@10 | Mean tokens | Boolean ops |
+|-------|-----------------|----------------|-------------|-------------|
+| base (Qwen2.5-3B-Instruct) | 0.455 | 0.386 | 120 | ~20% |
+| [SFT v1](https://huggingface.co/Supreeth/searchlm-nl2bm25-sft) | 0.441 | 0.273 | 95 | ~80% |
+| [GRPO v1](https://huggingface.co/Supreeth/searchlm-nl2bm25-grpo) ⚠️ | 0.556 | 0.608 | **5–7** | **0%** |
+| [SFT v2](https://huggingface.co/Supreeth/searchlm-nl2bm25-sft-v2) | 0.466 | 0.358 | 109 | ~65% |
+| [**GRPO v2**](https://huggingface.co/Supreeth/searchlm-nl2bm25-grpo-v2) ✅ | **0.577** | **0.657** | 147 | ~35% |
+
+Evaluated on BEIR test splits (NFCorpus: 323 queries, SciFact: 300 queries).
+
+---
 
 ## Training Details
 
 | Setting | Value |
 |---------|-------|
 | Base model | [searchlm-nl2bm25-sft](Supreeth/searchlm-nl2bm25-sft) |
-| Method | GRPO (TRL + vLLM colocate) |
+| Method | GRPO (TRL GRPOTrainer + vLLM colocate, single H100) |
 | Reward | `0.6 × NDCG@10 + 0.4 × MRR` (live Tantivy search) |
-| Training datasets | NFCorpus + SciFact (train split) |
+| Training datasets | NFCorpus + SciFact (train split qrels) |
 | Epochs | 3 |
 | `num_generations` | 2 |
 | Hardware | NVIDIA H100 80 GB |
+| W&B run | `supreethrao/searchlm/runs/nlp69ydi` |
+
+---
+
+## Related resources
+
+- **Code:** [SupreethRao99/searchLM](https://github.com/SupreethRao99/searchLM)
+- **Analysis:** [Reward hacking report (v1 + v2 comparison)](https://github.com/SupreethRao99/searchLM/blob/main/REWARD_HACKING_REPORT_V2.md)
+- **Fixed version:** [GRPO v2](Supreeth/searchlm-nl2bm25-grpo-v2)
+- **Collection:** [SearchLM collection](https://huggingface.co/collections/Supreeth/searchlm)
 
 ## Citation