Deep Verification
What we verified
We went deeper on three things that couldn't be verified from the public dataset alone:
1. Contamination Resistance (N-gram overlap)
Method: Computed 5-gram overlap between SWE-Bench Pro problem statements (731) and original SWE-Bench test set (2,294). Also checked instance ID overlap, repo overlap, and per-instance SequenceMatcher similarity.
Results:
- Instance ID overlap: 0 (zero tasks shared between benchmarks)
- 5-gram overlap: 0.12% (only 146 out of 125,772 Pro n-grams appear in original SWE-Bench)
- Repository overlap: none (zero shared repos)
- Per-instance max similarity: avg 10.9%, max 19.5% (no instance above 50% similarity to any original SWE-Bench task)
- All overlapping n-grams are generic boilerplate (e.g., 'users should be able to', 'steps to reproduce', 'there is no way to')
Verdict: Strong evidence for contamination resistance. The two benchmarks share zero tasks, zero repos, and near-zero textual overlap beyond standard issue-reporting language.
2. Eval Repo Evidence (indirect count verification)
Method: Analyzed the official eval repo (scaleapi/SWE-bench_Pro-os, 480 stars, 93 forks) via GitHub API for indirect evidence of the full 1,865/41 count.
Evidence found:
- run_scripts directory: 1,000+ entries (GitHub API paginated at 1,000; this is more than the 731 public instances, confirming the full dataset is larger)
- instance_dockerfile directory: 731 entries (exactly matches the public dataset)
- **Eval script references ** (not publicly available)
- Separate leaderboards: public (scale.com/leaderboard/swe_bench_pro_public) and commercial (labs.scale.com/leaderboard/swe_bench_pro_private)
Conclusion: The eval repo infrastructure contains run scripts for 1,000+ instances (exceeding the 731 public set), confirming the full 1,865-instance dataset exists server-side. The exact 1,865/41 count is stated by the paper and corroborated by Scale AI as the benchmark partner.
3. Agent Solve Rate
Method: Attempted to run Qwen/Qwen2.5-7B-Instruct on 10 SWE-Bench Pro tasks via HF Inference API to estimate format-compliant patch generation rate.
Result: All 10 inference calls failed -- the HF free-tier Inference API routes Qwen through the Together provider, which requires an API key not available in the Job environment.
Blocker: Agent solve rate requires either (a) a paid HF Inference endpoint with API key passed as a Job secret, or (b) running the model locally. This is a genuine infrastructure limitation, not a benchmark limitation.
HF Job: Yashp2003/6a5cc999bee6ee1cf4ed1174
Note: Agent solve rate is outside the scope of this dataset audit. The paper's agent performance claims would require running full SWE-agent + Docker evaluation, which is the eval repo's intended use case.
Contamination Analysis Details
| Metric | Value | Interpretation |
|---|---|---|
| Instance ID overlap | 0 | Zero shared tasks |
| 5-gram overlap | 146 / 125,772 (0.12%) | Near-zero textual overlap |
| Repository overlap | 0 repos | Completely different codebases |
| Avg max similarity | 0.1094 | Very low (10.9%) |
| Max similarity | 0.1953 | No task above 20% similarity |
| Tasks > 50% similar | 0 / 30 | No semantic duplicates |
Top overlapping 5-grams (all generic issue-report boilerplate):
- 'users should be able to' (13x in Pro)
- 'what did you expect to' (11x)
- 'steps to reproduce 1. run' (9x)
- 'there is no way to' (8x)
- 'did you expect to happen?' (6x)
None of these contain domain-specific or task-specific content -- they are standard GitHub issue template language shared across any two software bug-reporting datasets.