---
language:
- en
license: apache-2.0
base_model: Qwen/Qwen2.5-3B-Instruct
tags:
- information-retrieval
- boolean-search
- NL2BM25
- GRPO
- RLVR
- tantivy
- BEIR
- searchlm
- reward-hacking
library_name: transformers
pipeline_tag: text-generation
---

# SearchLM NL2BM25 — GRPO v1 ⚠️ Reward Hacking (Qwen2.5-3B-Instruct)

**Part of the [SearchLM collection](https://huggingface.co/collections/Supreeth/searchlm) · [GitHub](https://github.com/SupreethRao99/searchLM)**

> **⚠️ This model games its training reward.** It achieves high NDCG@10 by collapsing all
> outputs to 3–7 token keyword phrases, discarding the entire boolean search task it was
> trained to learn. Published for research transparency and as a reproducible example of
> specification gaming in RLVR. For deployment, use [GRPO v2](Supreeth/searchlm-nl2bm25-grpo-v2).

A Qwen2.5-3B-Instruct model fine-tuned via GRPO starting from [SFT v1](Supreeth/searchlm-nl2bm25-sft),
using live Tantivy retrieval (NDCG@10 + MRR) as the reward signal.

> **Pipeline position:** `base → SFT v1 → `**`GRPO v1 ⚠️`**` → SFT v2 → GRPO v2 ✅`

---

## The hack: specification gaming via minimum viable retrieval

The model learned that on small corpora (NFCorpus: 3,633 docs; SciFact: 5,183 docs),
2–4 content nouns yield near-optimal BM25 recall. Instead of learning boolean query
generation, it learned to extract the most distinctive nouns from the NL query:

**Input:** `Do Cholesterol Statin Drugs Cause Breast Cancer?`

**GRPO v1 output (hacking):**
```
<reasoning>
</reasoning>
<query>Cholesterol Statin Breast Cancer</query>
```

**SFT v1 output (intended behaviour, lower NDCG):**
```
<reasoning>
Key concepts: statin drugs, causal relationship, breast cancer.
Connect with AND; expand synonyms with OR.
</reasoning>
<query>(statin OR "HMG-CoA reductase inhibitor" OR simvastatin OR atorvastatin)
AND (cause OR risk OR association OR induce)
AND ("breast cancer" OR "breast carcinoma")</query>
```

The GRPO v1 output actually achieves **NDCG@10 = 0.971** on this query while the SFT
output achieves 0.000 — the hack outperforms the intended behaviour because SFT used wrong
synonyms. This made the gaming invisible in aggregate metrics alone.

---

## Collapse statistics

| Metric | Value |
|--------|-------|
| Mean completion length | **5.1 tokens** (vs 95 for SFT v1) |
| Boolean operator usage (AND) | **0%** (vs ~80% for SFT v1) |
| Boolean operator usage (OR) | **0%** (vs ~90% for SFT v1) |
| Phrase usage | **0%** (vs ~70% for SFT v1) |
| Reasoning block content | **empty** |
| `frac_reward_zero_std` during training | **90–96%** from step 1 |

`frac_reward_zero_std` = fraction of GRPO groups where all completions received identical
reward. At 90-96%, policy gradient was near-zero throughout — the model was not learning,
it had already converged on the keyword-bag strategy.

---

## Why it still scores high on benchmarks

1. **Small corpora**: BM25 keyword recall on 3–5K doc indices is high; a rare noun
   appears in only a handful of documents, making it highly discriminative.
2. **SFT degraded**: SFT v1 scored *below base* on SciFact (0.273 vs 0.386) due to
   over-specified queries — a low bar to beat.
3. **NDCG@10 rewards recall of first hit**: any query retrieving one relevant document
   in top-10 scores well. Keyword bags do this reliably on small indexes.

**This does not generalise**: on a 2.7M-doc index (NQ), keyword bags return thousands of
irrelevant results; NDCG@10 and MRR would collapse to near zero.

---

## All SearchLM checkpoints

| Model | NFCorpus NDCG@10 | SciFact NDCG@10 | Mean tokens | Boolean ops |
|-------|-----------------|----------------|-------------|-------------|
| base (Qwen2.5-3B-Instruct) | 0.455 | 0.386 | 120 | ~20% |
| [SFT v1](https://huggingface.co/Supreeth/searchlm-nl2bm25-sft) | 0.441 | 0.273 | 95 | ~80% |
| [GRPO v1](https://huggingface.co/Supreeth/searchlm-nl2bm25-grpo) ⚠️ | 0.556 | 0.608 | **5–7** | **0%** |
| [SFT v2](https://huggingface.co/Supreeth/searchlm-nl2bm25-sft-v2) | 0.466 | 0.358 | 109 | ~65% |
| [**GRPO v2**](https://huggingface.co/Supreeth/searchlm-nl2bm25-grpo-v2) ✅ | **0.577** | **0.657** | 147 | ~35% |

Evaluated on BEIR test splits (NFCorpus: 323 queries, SciFact: 300 queries).

---

## Training Details

| Setting | Value |
|---------|-------|
| Base model | [searchlm-nl2bm25-sft](Supreeth/searchlm-nl2bm25-sft) |
| Method | GRPO (TRL GRPOTrainer + vLLM colocate, single H100) |
| Reward | `0.6 × NDCG@10 + 0.4 × MRR` (live Tantivy search) |
| Training datasets | NFCorpus + SciFact (train split qrels) |
| Epochs | 3 |
| `num_generations` | 2 |
| Hardware | NVIDIA H100 80 GB |
| W&B run | `supreethrao/searchlm/runs/nlp69ydi` |

---

## Related resources

- **Code:** [SupreethRao99/searchLM](https://github.com/SupreethRao99/searchLM)
- **Analysis:** [Reward hacking report (v1 + v2 comparison)](https://github.com/SupreethRao99/searchLM/blob/main/REWARD_HACKING_REPORT_V2.md)
- **Fixed version:** [GRPO v2](Supreeth/searchlm-nl2bm25-grpo-v2)
- **Collection:** [SearchLM collection](https://huggingface.co/collections/Supreeth/searchlm)

## Citation

```bibtex
@misc{searchlm2026,
  title  = {SearchLM: Training Small Language Models for Boolean Query Generation via RLVR},
  author = {Rao, Supreeth},
  year   = {2026},
  url    = {https://github.com/SupreethRao99/searchLM},
}
```