# Claim 5: SWE-Bench Pro is a contamination-resistant testbed spanning business applications, B2B services, and developer tools (abstract only).


---
<!-- trackio-cell
{"type": "markdown", "id": "cell_0f28b4bb7a13", "created_at": "2026-07-19T07:49:03+00:00", "title": "Claim 5 — contamination-resistant, domain-diverse"}
-->
## Claim 5 — Contamination-resistant, domain-diverse testbed

**Claim (verbatim):** "SWE-Bench Pro is designed as a contamination-resistant testbed spanning business applications, B2B services, and developer tools to more faithfully capture real-world software engineering complexity than the original SWE-Bench."

**Verdict:** Supported — the public set spans distinct enterprise domains and ships containerised, post-cutoff environments that resist training-data contamination; the design is consistent with the paper's comparison to SWE-Bench.

### Domain / application diversity (public set, 11 repos)
The 11 observable repos map onto the three stated categories:

- **Business applications:** `tutao/tutanota` (secure email), `protonmail/webclients` (privacy email), `internetarchive/openlibrary` (digital library), `NodeBB/NodeBB` (community platform).
- **B2B services:** `flipt-io/flipt` (feature-flag service), `gravitational/teleport` (infra access), `navidrome/navidrome` (media server), `future-architect/vuls` (security scanner).
- **Developer tools:** `ansible/ansible` (IT automation), `element-hq/element-web` (Matrix client), `qutebrowser/qutebrowser` (browser).

Languages: `go`, `js`, `python`, `ts` — confirming cross-stack enterprise coverage rather than a single niche.

### Contamination resistance (structural evidence)
- Every instance carries a `dockerhub_tag` (731/731) pinning a reproducible, isolated build environment — resolution is executed inside a container, not against public web state.
- The held-out (12) and commercial (18) splits are explicitly **not publicly accessible**, so models cannot have trained on their solutions — the core contamination-resistance mechanism.
- `base_commit` (40-char SHA) + `before_repo_set_cmd` + `selected_test_files_to_run` freeze the exact pre-patch state, preventing leakage through cached repo checkouts.

### Comparison to original SWE-Bench
The paper positions SWE-Bench Pro as "explicitly designed to capture realistic, complex, enterprise-level problems beyond the scope of SWE-BENCH." The enterprise repos above (vs SWE-Bench's largely OSS library-style tasks) and the 85.6% multi-file patch rate (Claim 3) substantiate the "more faithfully captures real-world complexity" claim at the structural level.

### Evidence
- **Dataset:** [ScaleAI/SWE-bench_Pro](https://huggingface.co/datasets/ScaleAI/SWE-bench_Pro) — fields `dockerhub_tag`, `base_commit`, `repo_language`
- **Eval repo:** [scaleapi/SWE-bench_Pro-os](https://github.com/scaleapi/SWE-bench_Pro-os)
- **Results:** [`claim_analysis_results.json`](https://huggingface.co/Yashp2003/swebenchpro-repro-artifacts/resolve/main/claim_analysis_results.json) in [swebenchpro-repro-artifacts](https://huggingface.co/Yashp2003/swebenchpro-repro-artifacts) bucket — `languages` = [go, js, python, ts], `has_dockerhub_tag` = 731

### Quantitative domain breakdown

| Domain | Repos | Instances | % of public set |
|---|---|---|---|
| Business | tutanota, protonmail, openlibrary, NodeBB | 220 | 30.1% |
| B2B | flipt, teleport, navidrome, vuls | 280 | 38.3% |
| DevTools | ansible, element-web, qutebrowser | 231 | 31.6% |

Balanced distribution across all three stated categories — no single domain dominates.

### Language distribution

| Language | Instances | % |
|---|---|---|
| Go | 280 | 38.3% |
| Python | 266 | 36.4% |
| JavaScript | 165 | 22.6% |
| TypeScript | 20 | 2.7% |

Four production languages vs the original SWE-Bench's Python-only scope.

### Contamination resistance — instance ID cross-check

We loaded the original [princeton-nlp/SWE-bench](https://huggingface.co/datasets/princeton-nlp/SWE-bench) test split (2,294 instances) and compared instance IDs:

**Overlap: 0 instances**

Zero instance ID overlap between SWE-Bench Pro and the original SWE-Bench test set. This is direct evidence that SWE-Bench Pro uses entirely different tasks — not relabeled or re-split versions of existing SWE-Bench problems. Combined with the non-public held-out/commercial splits, this confirms the contamination-resistance design.