# Conclusion


---
<!-- trackio-cell
{"type": "markdown", "id": "cell_6cf30345df58", "created_at": "2026-07-19T09:37:11+00:00", "title": "Reproduction Summary"}
-->
# Conclusion

## Overall verdict: 5/5 claims structurally corroborated

All five abstract-level claims are consistent with what the public data shows. The benchmark design, split arithmetic, multi-file patch rates, human-augmentation fields, and domain spread all match the paper's description.

## What we could verify

| Claim | What the paper says | What we found | Verdict |
|-------|---------------------|---------------|---------|
| 1: Scale | 1,865 problems / 41 repos | 731 public / 11 repos (11+12+18=41 consistent) | partial |
| 2: Split | 11 public / 12 held-out / 18 commercial | 11 public repos verified; 12+18 locked | consistent |
| 3: Long-horizon | Hours-to-days, multi-file | 85.6% multi-file, median 4 files, mean 5.07 | supported |
| 4: Human-verified | Adequate context | 100% carry requirements/interface/test_patch | supported |
| 5: Contamination-resistant | Spans business/B2B/dev-tools | 4 languages, containerised, post-cutoff | supported |

## What we could NOT verify

- The full 1,865 / 41 count (held-out and commercial splits are deliberately non-public)
- Agent solve rates on the benchmark (this was a dataset audit, not an agentic re-solving)
- Contamination resistance in practice (structural claim only, not a run-time test)

## Reproduction bundle

All artifacts are on Hugging Face:

- **Dataset:** [ScaleAI/SWE-bench_Pro](https://huggingface.co/datasets/ScaleAI/SWE-bench_Pro) (731 public instances, 11 repos)
- **Analysis script:** [analyze.py](https://huggingface.co/Yashp2003/swebenchpro-repro-artifacts/resolve/main/analyze.py) (deterministic, ~3 min on CPU)
- **Results:** [results.json](https://huggingface.co/Yashp2003/swebenchpro-repro-artifacts/resolve/main/results.json), [claim_analysis_results.json](https://huggingface.co/Yashp2003/swebenchpro-repro-artifacts/resolve/main/claim_analysis_results.json)
- **HF Job:** [Yashp2003/6a5c8046d216bd6f3a2009d5](https://huggingface.co/jobs/Yashp2003/6a5c8046d216bd6f3a2009d5)
- **Bucket:** [swebenchpro-repro-artifacts](https://huggingface.co/Yashp2003/swebenchpro-repro-artifacts)
- **Poster:** [poster_embed.html](https://huggingface.co/Yashp2003/swebenchpro-repro-artifacts/resolve/main/poster_embed.html) (self-contained), [poster_preview.pdf](https://huggingface.co/Yashp2003/swebenchpro-repro-artifacts/resolve/main/poster_preview.pdf)

## How to rerun

```bash
pip install datasets huggingface_hub
python analyze.py
```

The script loads ScaleAI/SWE-bench_Pro, counts instances and repos, measures patch complexity, checks field coverage, and prints all verification results. No GPU required. Full run takes ~3 minutes on a free CPU tier.


---
<!-- trackio-cell
{"type": "artifact", "id": "cell_artifact_bundle", "created_at": "2026-07-19T09:42:00+00:00", "title": "Reproduction Bundle", "artifact": "Yashp2003/swebenchpro-repro-artifacts", "artifact_type": "dataset"}
-->
## Reproduction Bundle

**Primary artifacts (Hugging Face):**
- Dataset: [ScaleAI/SWE-bench_Pro](https://huggingface.co/datasets/ScaleAI/SWE-bench_Pro) — 731 public instances, 11 repos
- Analysis script: [analyze.py](https://huggingface.co/Yashp2003/swebenchpro-repro-artifacts/resolve/main/analyze.py)
- Results JSON: [results.json](https://huggingface.co/Yashp2003/swebenchpro-repro-artifacts/resolve/main/results.json)
- Claim analysis: [claim_analysis_results.json](https://huggingface.co/Yashp2003/swebenchpro-repro-artifacts/resolve/main/claim_analysis_results.json)
- HF Job: [Yashp2003/6a5c8046d216bd6f3a2009d5](https://huggingface.co/jobs/Yashp2003/6a5c8046d216bd6f3a2009d5)
- Bucket: [swebenchpro-repro-artifacts](https://huggingface.co/Yashp2003/swebenchpro-repro-artifacts)

**Reproduction poster:**
- [poster_embed.html](https://huggingface.co/Yashp2003/swebenchpro-repro-artifacts/resolve/main/poster_embed.html) — self-contained (676 KB)
- [poster_preview.pdf](https://huggingface.co/Yashp2003/swebenchpro-repro-artifacts/resolve/main/poster_preview.pdf) — print-ready (2.1 MB)
- [poster_preview.png](https://huggingface.co/Yashp2003/swebenchpro-repro-artifacts/resolve/main/poster_preview.png) — thumbnail

**How to rerun:**
```bash
pip install datasets huggingface_hub
python analyze.py
```