| # Claim 5: SWE-Bench Pro is a contamination-resistant testbed spanning business applications, B2B services, and developer tools (abstract only). |
|
|
|
|
| --- |
| <!-- trackio-cell |
| {"type": "markdown", "id": "cell_0f28b4bb7a13", "created_at": "2026-07-19T07:49:03+00:00", "title": "Claim 5 — contamination-resistant, domain-diverse"} |
| --> |
| ## Claim 5 — Contamination-resistant, domain-diverse testbed |
|
|
| **Claim (verbatim from abstract):** "SWE-BENCH PRO provides a contamination-resistant testbed that more faithfully captures the complexity and diversity of real-world software development." |
|
|
| **Verdict:** VERIFIED — zero instance ID overlap vs original SWE-Bench, 0.12% n-gram overlap (only boilerplate), containerized execution, 3 enterprise domains, 4 languages. |
|
|
| ### Contamination resistance — quantitative evidence |
| We loaded the original [princeton-nlp/SWE-bench](https://huggingface.co/datasets/princeton-nlp/SWE-bench) test split (2,294 instances) and performed cross-analysis: |
|
|
| - **Instance ID overlap: 0** — zero shared instance IDs between SWE-Bench Pro and original SWE-Bench |
| - **N-gram overlap: 0.12%** — only 146 / 125,772 5-grams overlap, all generic issue-report boilerplate ("users should be able to", "steps to reproduce", etc.) |
| - **Per-instance similarity:** average max similarity 10.9%, maximum 19.5%. No task above 50% similarity |
| - **Containerized execution:** every instance carries pinned `dockerhub_tag` + `base_commit` for isolated resolution |
|
|
| These are direct quantitative measurements confirming SWE-Bench Pro uses entirely different tasks — not relabeled or re-split versions of existing SWE-Bench problems. |
|
|
| ### Domain diversity (public set, 11 repos) |
| The 11 repos map onto the three stated categories: |
|
|
| | Domain | Repos | Instances | % | |
| |---|---|---|---| |
| | Business applications | tutanota, protonmail, openlibrary, NodeBB | 220 | 30.1% | |
| | B2B services | flipt, teleport, navidrome, vuls | 280 | 38.3% | |
| | Developer tools | ansible, element-web, qutebrowser | 231 | 31.6% | |
|
|
| ### Language diversity (public set) |
| | Language | Instances | % | |
| |---|---|---| |
| | Go | 280 | 38.3% | |
| | Python | 266 | 36.4% | |
| | JavaScript | 165 | 22.6% | |
| | TypeScript | 20 | 2.7% | |
|
|
| Four production languages vs the original SWE-Bench's Python-only scope. |
|
|
| ### Structural contamination-resistance mechanisms |
| - **Non-public splits:** held-out (12 repos) and commercial (18 repos) are deliberately not publicly accessible — models cannot have trained on their solutions |
| - **Containerized execution:** `dockerhub_tag` pins a reproducible, isolated build environment |
| - **Frozen state:** `base_commit` + `before_repo_set_cmd` + `selected_test_files_to_run` freeze the exact pre-patch state |
|
|
| ### Evidence |
| - **Dataset:** [ScaleAI/SWE-bench_Pro](https://huggingface.co/datasets/ScaleAI/SWE-bench_Pro) |
| - **Eval repo:** [scaleapi/SWE-bench_Pro-os](https://github.com/scaleapi/SWE-bench_Pro-os) |
| - **Results:** [`claim_analysis_results.json`](https://huggingface.co/Yashp2003/swebenchpro-repro-artifacts/resolve/main/claim_analysis_results.json) — zero ID overlap, 0.12% n-gram overlap |
| - **HF Job:** [Yashp2003/6a5cc999bee6ee1cf4ed1174](https://huggingface.co/jobs/Yashp2003/6a5cc999bee6ee1cf4ed1174) — deep verification (contamination analysis) |
|
|
