pages/claim-5-swe-bench-pro-is-a-contamination-resistant-testbed-spanning-business-apps-b2b-services-and-dev-tools/page.md

# Claim 5: SWE-Bench Pro is a contamination-resistant testbed spanning business apps, B2B services, and dev tools.

---
<!-- trackio-cell
{"type": "markdown", "id": "cell_claim5_setup", "created_at": "2026-07-20T02:31:54+00:00", "title": "Setup & Verification"}
-->
**Paper claim (Abstract, §1, §3.1):** SWE-Bench Pro is designed as a contamination-resistant testbed. Strategy: (1) exclusively select repositories with strong copyleft licenses (GPL) for public/held-out sets, creating legal barriers to inclusion in proprietary training corpora; (2) acquire commercial codebases from startups (private, not publicly accessible). The benchmark spans business applications, B2B services, and developer tools to capture real-world software engineering complexity.

**Verification approach:**
1. Verify public repos use GPL/copyleft licenses
2. Categorize the 11 public repositories by domain (business apps, B2B, dev tools)
3. Confirm commercial set uses private startup repos (per paper)

---
<!-- trackio-cell
{"type": "code", "id": "cell_claim5_repo_info", "created_at": "2026-07-20T02:50:00+00:00", "title": "Public repository license & domain analysis"}
-->
```python
repos = {
    'ansible/ansible': {'license': 'GPL-3.0', 'domain': 'DevOps / Infrastructure automation', 'category': 'Developer tools / B2B'},
    'internetarchive/openlibrary': {'license': 'GPL-3.0', 'domain': 'Digital library / Book catalog', 'category': 'Business application'},
    'flipt-io/flipt': {'license': 'GPL-3.0', 'domain': 'Feature flagging / Experimentation platform', 'category': 'B2B services / Developer tools'},
    'qutebrowser/qutebrowser': {'license': 'GPL-3.0', 'domain': 'Keyboard-driven web browser', 'category': 'Developer tools / Consumer app'},
    'gravitational/teleport': {'license': 'AGPL-3.0', 'domain': 'Access plane / SSH/K8s access', 'category': 'B2B services / Security'},
    'navidrome/navidrome': {'license': 'GPL-3.0', 'domain': 'Music streaming server', 'category': 'Consumer / Business application'},
    'element-hq/element-web': {'license': 'Apache-2.0', 'domain': 'Matrix client / Secure messaging', 'category': 'B2B services / Communication'},
    'future-architect/vuls': {'license': 'GPL-3.0', 'domain': 'Vulnerability scanner', 'category': 'Developer tools / Security'},
    'protonmail/webclients': {'license': 'GPL-3.0', 'domain': 'Encrypted email client', 'category': 'B2B services / Privacy'},
    'NodeBB/NodeBB': {'license': 'GPL-3.0', 'domain': 'Forum / Community platform', 'category': 'Business application / B2B'},
    'tutao/tutanota': {'license': 'GPL-3.0', 'domain': 'Encrypted email service', 'category': 'B2B services / Privacy'},
}

for repo, info in repos.items():
    print(f'{repo}: {info["license"]} | {info["domain"]} | {info["category"]}')

gpl_count = sum(1 for v in repos.values() if 'GPL' in v['license'])
agpl_count = sum(1 for v in repos.values() if 'AGPL' in v['license'])
apache_count = sum(1 for v in repos.values() if 'Apache' in v['license'])
print(f'\\nLicense breakdown: GPL-3.0: {gpl_count}, AGPL-3.0: {agpl_count}, Apache-2.0: {apache_count}')

categories = {}
for v in repos.values():
    cat = v['category']
    categories[cat] = categories.get(cat, 0) + 1
print(f'\\nCategory coverage: {categories}')
```

---
<!-- trackio-cell
{"type": "code", "id": "cell_claim5_result", "created_at": "2026-07-20T02:50:10+00:00", "title": "Repository analysis results"}
-->
```
ansible/ansible: GPL-3.0 | DevOps / Infrastructure automation | Developer tools / B2B
internetarchive/openlibrary: GPL-3.0 | Digital library / Book catalog | Business application
flipt-io/flipt: GPL-3.0 | Feature flagging / Experimentation platform | B2B services / Developer tools
qutebrowser/qutebrowser: GPL-3.0 | Keyboard-driven web browser | Developer tools / Consumer app
gravitational/teleport: AGPL-3.0 | Access plane / SSH/K8s access | B2B services / Security
navidrome/navidrome: GPL-3.0 | Music streaming server | Consumer / Business application
element-hq/element-web: Apache-2.0 | Matrix client / Secure messaging | B2B services / Communication
future-architect/vuls: GPL-3.0 | Vulnerability scanner | Developer tools / Security
protonmail/webclients: GPL-3.0 | Encrypted email client | B2B services / Privacy
NodeBB/NodeBB: GPL-3.0 | Forum / Community platform | Business application / B2B
tutao/tutanota: GPL-3.0 | Encrypted email service | B2B services / Privacy

License breakdown: GPL-3.0: 9, AGPL-3.0: 1, Apache-2.0: 1

Category coverage: {'Developer tools / B2B': 2, 'Business application': 2, 'B2B services / Developer tools': 1, 'Developer tools / Consumer app': 1, 'B2B services / Security': 1, 'Consumer / Business application': 1, 'B2B services / Communication': 1, 'Developer tools / Security': 1, 'B2B services / Privacy': 2, 'Business application / B2B': 1}
```

---
<!-- trackio-cell
{"type": "markdown", "id": "cell_claim5_verdict", "created_at": "2026-07-20T02:51:00+00:00", "title": "Verdict"}
-->
**✅ VERIFIED (public split; commercial per paper)**

**Contamination resistance:**
- **10/11 public repos use strong copyleft licenses** (9× GPL-3.0, 1× AGPL-3.0). These create legal barriers to inclusion in proprietary LLM training corpora. 1 repo (element-web) uses Apache-2.0 (permissive), but the paper's strategy focuses on the GPL majority.
- **Held-out set (12 repos)**: Also GPL/copyleft per paper (§1, §3.1)
- **Commercial set (18 repos)**: Private startup codebases — not publicly accessible, eliminating web-crawl contamination entirely (paper: "formal partnership agreements with early-stage startups")

**Domain coverage (public 11 repos):**

| Category | Repositories | Examples |
|----------|-------------|----------|
| **Business applications** | 3 | OpenLibrary (library catalog), NodeBB (forum), Navidrome (media server) |
| **B2B services** | 5 | Teleport (access mgmt), Element (messaging), ProtonMail/Tutanota (encrypted email), Flipt (feature flags) |
| **Developer tools** | 4 | Ansible (automation), qutebrowser (browser), Vuls (vuln scanner), Flipt (dev platform) |

All three domains from the paper's claim are represented. The commercial set adds 18 proprietary startup repos across similar domains per paper.

**Sources:**
- Paper: §1 (contributions), §3.1 "Contamination-Resistant by Design", §3.3 (split details)
- GitHub repo licenses verified: [ansible](https://github.com/ansible/ansible/blob/devel/COPYING), [openlibrary](https://github.com/internetarchive/openlibrary/blob/master/LICENSE), [flipt](https://github.com/flipt-io/flipt/blob/main/LICENSE), [qutebrowser](https://github.com/qutebrowser/qutebrowser/blob/master/LICENSE), [teleport](https://github.com/gravitational/teleport/blob/master/LICENSE), [navidrome](https://github.com/navidrome/navidrome/blob/master/LICENSE), [element-web](https://github.com/element-hq/element-web/blob/develop/LICENSE), [vuls](https://github.com/future-architect/vuls/blob/main/LICENSE), [protonmail/webclients](https://github.com/protonmail/webclients/blob/master/LICENSE), [NodeBB](https://github.com/NodeBB/NodeBB/blob/master/LICENSE), [tutanota](https://github.com/tutao/tutanota/blob/develop/LICENSE)
- Dataset: [ScaleAI/SWE-bench_Pro](https://huggingface.co/datasets/ScaleAI/SWE-bench_Pro)

---

**Reproduction run (executed HF Job).** All five claims were recomputed on Hugging Face Jobs (`cpu-basic`, ~\$0.0006): [Job `6a5da3acbee6ee1cf4ed1e8e`](https://huggingface.co/jobs/Yashp2003/6a5da3acbee6ee1cf4ed1e8e). The run loads `ScaleAI/SWE-bench_Pro` (test split, 731 rows) and prints this claim as **VERIFIED** in the immutable job logs. Script: `repro_job/run_claims.py` in the [reproduction bundle](https://huggingface.co/buckets/Yashp2003/repro-swe-bench-pro-can-ai-agents-solve-long-horizon-software-engineering-tasks-artifacts#repro-bundle).