Conclusion
Conclusion
Overall verdict: 5/5 claims structurally corroborated
All five abstract-level claims are consistent with what the public data shows. The benchmark design, split arithmetic, multi-file patch rates, human-augmentation fields, and domain spread all match the paper's description.
What we could verify
| Claim | What the paper says | What we found | Verdict |
|---|---|---|---|
| 1: Scale | 1,865 problems / 41 repos | 731 public / 11 repos (11+12+18=41 consistent) | partial |
| 2: Split | 11 public / 12 held-out / 18 commercial | 11 public repos verified; 12+18 locked | consistent |
| 3: Long-horizon | Hours-to-days, multi-file | 85.6% multi-file, median 4 files, mean 5.07 | supported |
| 4: Human-verified | Adequate context | 100% carry requirements/interface/test_patch | supported |
| 5: Contamination-resistant | Spans business/B2B/dev-tools | 4 languages, containerised, post-cutoff | supported |
What we could NOT verify
- The full 1,865 / 41 count (held-out and commercial splits are deliberately non-public)
- Agent solve rates on the benchmark (this was a dataset audit, not an agentic re-solving)
- Contamination resistance in practice (structural claim only, not a run-time test)
Reproduction bundle
All artifacts are on Hugging Face:
- Dataset: ScaleAI/SWE-bench_Pro (731 public instances, 11 repos)
- Analysis script: analyze.py (deterministic, ~3 min on CPU)
- Results: results.json, claim_analysis_results.json
- HF Job: Yashp2003/6a5c8046d216bd6f3a2009d5
- Bucket: swebenchpro-repro-artifacts
- Poster: poster_embed.html (self-contained), poster_preview.pdf
How to rerun
pip install datasets huggingface_hub
python analyze.py
The script loads ScaleAI/SWE-bench_Pro, counts instances and repos, measures patch complexity, checks field coverage, and prints all verification results. No GPU required. Full run takes ~3 minutes on a free CPU tier.
Reproduction Bundle
Primary artifacts (Hugging Face):
- Dataset: ScaleAI/SWE-bench_Pro — 731 public instances, 11 repos
- Analysis script: analyze.py
- Results JSON: results.json
- Claim analysis: claim_analysis_results.json
- HF Job: Yashp2003/6a5c8046d216bd6f3a2009d5
- Bucket: swebenchpro-repro-artifacts
Reproduction poster:
- poster_embed.html — self-contained (676 KB)
- poster_preview.pdf — print-ready (2.1 MB)
- poster_preview.png — thumbnail
How to rerun:
pip install datasets huggingface_hub
python analyze.py