Update logbook: Reproduction: SWE-Bench Pro: Can AI Agents Solve Long-Horizon Software Engineering Tasks?

logbook.json

@@ -8,7 +8,7 @@
     "icml2026-repro",
     "paper-uEVTdoAbnK"
   ],
-  "updated_at": "2026-07-19T11:13:12+00:00",
+  "updated_at": "2026-07-19T12:02:49+00:00",
   "root": {
     "slug": "index",
     "title": "Reproduction: SWE-Bench Pro: Can AI Agents Solve Long-Horizon Software Engineering Tasks?",
@@ -58,6 +58,6 @@
       }
     ]
   },
-  "agent_view_tokens": 7813,
-  "revision": "1784459592408925438"
+  "agent_view_tokens": 8805,
+  "revision": "1784462569612488200"
 }

pages/conclusion/page.md

@@ -29,14 +29,30 @@ All five abstract-level claims are consistent with what the public data shows. C
 
 ## Reproduction bundle
 
-All artifacts are on Hugging Face:
+All artifacts are published to Hugging Face:
 
-- **Dataset:** [ScaleAI/SWE-bench_Pro](https://huggingface.co/datasets/ScaleAI/SWE-bench_Pro) (731 public instances, 11 repos)
-- **Analysis script:** [analyze.py](https://huggingface.co/Yashp2003/swebenchpro-repro-artifacts/resolve/main/analyze.py) (deterministic, ~3 min on CPU)
-- **Results:** [results.json](https://huggingface.co/Yashp2003/swebenchpro-repro-artifacts/resolve/main/results.json), [claim_analysis_results.json](https://huggingface.co/Yashp2003/swebenchpro-repro-artifacts/resolve/main/claim_analysis_results.json)
-- **HF Job:** [Yashp2003/6a5c8046d216bd6f3a2009d5](https://huggingface.co/jobs/Yashp2003/6a5c8046d216bd6f3a2009d5)
-- **Bucket:** [swebenchpro-repro-artifacts](https://huggingface.co/Yashp2003/swebenchpro-repro-artifacts)
-- **Poster:** [poster_embed.html](https://huggingface.co/Yashp2003/swebenchpro-repro-artifacts/resolve/main/poster_embed.html) (self-contained), [poster_preview.pdf](https://huggingface.co/Yashp2003/swebenchpro-repro-artifacts/resolve/main/poster_preview.pdf)
+### Source dataset
+- **[ScaleAI/SWE-bench_Pro](https://huggingface.co/datasets/ScaleAI/SWE-bench_Pro)** — 731 public instances, 11 repos (the official release by Scale AI, the paper's benchmark partner)
+
+### Analysis scripts & results
+- **[analyze.py](https://huggingface.co/Yashp2003/swebenchpro-repro-artifacts/resolve/main/analyze.py)** — primary analysis script (deterministic, ~3 min on CPU, no GPU required)
+- **[deep_analysis.py](https://huggingface.co/Yashp2003/swebenchpro-repro-artifacts/resolve/main/deep_analysis.py)** — cross-reference analysis vs original SWE-Bench
+- **[results.json](https://huggingface.co/Yashp2003/swebenchpro-repro-artifacts/resolve/main/results.json)** — basic analysis output (731 instances, 11 repos, field coverage)
+- **[claim_analysis_results.json](https://huggingface.co/Yashp2003/swebenchpro-repro-artifacts/resolve/main/claim_analysis_results.json)** — deep analysis with cross-reference comparisons
+
+### Compute
+- **HF Job:** [Yashp2003/6a5c8046d216bd6f3a2009d5](https://huggingface.co/jobs/Yashp2003/6a5c8046d216bd6f3a2009d5) — CPU job, ~$0.005, produced all analysis results
+- **Bucket:** [Yashp2003/swebenchpro-repro-artifacts](https://huggingface.co/Yashp2003/swebenchpro-repro-artifacts) — all artifacts stored here
+
+### Poster
+- **[poster_embed.html](https://huggingface.co/Yashp2003/swebenchpro-repro-artifacts/resolve/main/poster_embed.html)** — self-contained interactive HTML (inlined CSS, hover-to-navigate hotspots)
+- **[poster_preview.pdf](https://huggingface.co/Yashp2003/swebenchpro-repro-artifacts/resolve/main/poster_preview.pdf)** — print-ready PDF (2.2 MB)
+- **[poster_preview.png](https://huggingface.co/Yashp2003/swebenchpro-repro-artifacts/resolve/main/poster_preview.png)** — thumbnail PNG (507 KB)
+
+### Related resources
+- **Paper:** [arXiv 2509.16941](https://arxiv.org/abs/2509.16941) / [OpenReview uEVTdoAbnK](https://openreview.net/forum?id=uEVTdoAbnK)
+- **Eval repo:** [scaleapi/SWE-bench_Pro-os](https://github.com/scaleapi/SWE-bench_Pro-os) (480 stars, 93 forks)
+- **Original SWE-Bench:** [princeton-nlp/SWE-bench](https://huggingface.co/datasets/princeton-nlp/SWE-bench) (used for cross-reference comparison)
 
 ## How to rerun
 
@@ -45,7 +61,7 @@ pip install datasets huggingface_hub
 python analyze.py
 ```
 
-The script loads ScaleAI/SWE-bench_Pro, counts instances and repos, measures patch complexity, checks field coverage, and prints all verification results. No GPU required. Full run takes ~3 minutes on a free CPU tier.
+The script loads [ScaleAI/SWE-bench_Pro](https://huggingface.co/datasets/ScaleAI/SWE-bench_Pro), counts instances and repos, measures patch complexity, checks field coverage, and prints all verification results. No GPU required. Full run takes ~3 minutes on a free CPU tier.
 
 
 ---
@@ -57,15 +73,16 @@ The script loads ScaleAI/SWE-bench_Pro, counts instances and repos, measures pat
 **Primary artifacts (Hugging Face):**
 - Dataset: [ScaleAI/SWE-bench_Pro](https://huggingface.co/datasets/ScaleAI/SWE-bench_Pro) — 731 public instances, 11 repos
 - Analysis script: [analyze.py](https://huggingface.co/Yashp2003/swebenchpro-repro-artifacts/resolve/main/analyze.py)
+- Deep analysis: [deep_analysis.py](https://huggingface.co/Yashp2003/swebenchpro-repro-artifacts/resolve/main/deep_analysis.py)
 - Results JSON: [results.json](https://huggingface.co/Yashp2003/swebenchpro-repro-artifacts/resolve/main/results.json)
 - Claim analysis: [claim_analysis_results.json](https://huggingface.co/Yashp2003/swebenchpro-repro-artifacts/resolve/main/claim_analysis_results.json)
 - HF Job: [Yashp2003/6a5c8046d216bd6f3a2009d5](https://huggingface.co/jobs/Yashp2003/6a5c8046d216bd6f3a2009d5)
-- Bucket: [swebenchpro-repro-artifacts](https://huggingface.co/Yashp2003/swebenchpro-repro-artifacts)
+- Bucket: [Yashp2003/swebenchpro-repro-artifacts](https://huggingface.co/Yashp2003/swebenchpro-repro-artifacts)
 
 **Reproduction poster:**
-- [poster_embed.html](https://huggingface.co/Yashp2003/swebenchpro-repro-artifacts/resolve/main/poster_embed.html) — self-contained (676 KB)
-- [poster_preview.pdf](https://huggingface.co/Yashp2003/swebenchpro-repro-artifacts/resolve/main/poster_preview.pdf) — print-ready (2.1 MB)
-- [poster_preview.png](https://huggingface.co/Yashp2003/swebenchpro-repro-artifacts/resolve/main/poster_preview.png) — thumbnail
+- [poster_embed.html](https://huggingface.co/Yashp2003/swebenchpro-repro-artifacts/resolve/main/poster_embed.html) — self-contained interactive HTML
+- [poster_preview.pdf](https://huggingface.co/Yashp2003/swebenchpro-repro-artifacts/resolve/main/poster_preview.pdf) — print-ready (2.2 MB)
+- [poster_preview.png](https://huggingface.co/Yashp2003/swebenchpro-repro-artifacts/resolve/main/poster_preview.png) — thumbnail (507 KB)
 
 **How to rerun:**
 ```bash

pages/executive-summary/page.md

@@ -5,24 +5,34 @@
 <!-- trackio-cell
 {"type": "markdown", "id": "cell_7091adcf6ef0", "created_at": "2026-07-19T07:49:26+00:00", "title": "Executive summary", "pinned": true, "pinned_at": "2026-07-19T07:50:11+00:00"}
 -->
-We reproduced the five headline claims of **SWE-Bench Pro** (arXiv 2509.16941 / OpenReview uEVTdoAbnK) against the official public benchmark release **[ScaleAI/SWE-bench_Pro](https://huggingface.co/datasets/ScaleAI/SWE-bench_Pro)**, cross-referenced against the original [princeton-nlp/SWE-bench](https://huggingface.co/datasets/princeton-nlp/SWE-bench). All five claims are structurally corroborated:
+We reproduced the five headline claims of **SWE-Bench Pro** (arXiv [2509.16941](https://arxiv.org/abs/2509.16941) / OpenReview [uEVTdoAbnK](https://openreview.net/forum?id=uEVTdoAbnK)) against the official public benchmark release **[ScaleAI/SWE-bench_Pro](https://huggingface.co/datasets/ScaleAI/SWE-bench_Pro)** (60,512 downloads, 154 likes), cross-referenced against the original **[princeton-nlp/SWE-bench](https://huggingface.co/datasets/princeton-nlp/SWE-bench)** test split (2,294 instances). The reproduction was executed as a single Hugging Face CPU Job ([Yashp2003/6a5c8046d216bd6f3a2009d5](https://huggingface.co/jobs/Yashp2003/6a5c8046d216bd6f3a2009d5), ~$0.005 cost). All artifacts are published to the **[swebenchpro-repro-artifacts](https://huggingface.co/Yashp2003/swebenchpro-repro-artifacts)** bucket. **Verdict: 5/5 claims structurally corroborated** on the public subset.
 
-1. **Scale:** 731 public problems / 11 repos verified; eval repo confirms full 1,865 exists server-side; arithmetic 11+12+18=41 internally consistent.
-2. **Split:** 11 public repos confirmed; held-out (12) + commercial (18) non-public by design; eval repo maintains separate public/private leaderboards.
-3. **Long-horizon:** 85.6% multi-file patches (median 4 files), mean patch 13.3k chars — **4.34x larger** and **3.44x more multi-file** than original SWE-Bench. Time estimates: mean 4.4h, max 60h per task.
-4. **Human-verified:** 100% carry requirements/interface/test_patch; 32.1% include error/traceback diagnostics; mean requirements 1,483 chars.
-5. **Contamination-resistant:** **Zero instance ID overlap** with original SWE-Bench; spans 4 languages (vs Python-only original); 3 enterprise domains; non-public splits prevent training leakage.
+### What we verified
 
-Analysis ran as HF CPU Jobs; total cost < $0.01. This is a **dataset-auditing reproduction** with cross-reference validation, not an agentic re-solving of tasks.
+- **Claim 1 (Scale):** Counted **731 public problems** across exactly **11 repositories** in the official HF dataset. The paper states 1,865 total from 41 repos; arithmetic 11 + 12 + 18 = 41 is internally consistent. The official eval repo ([scaleapi/SWE-bench_Pro-os](https://github.com/scaleapi/SWE-bench_Pro-os), 480 stars) references `swe_bench_pro_full.csv` and maintains separate public/private leaderboards, confirming the full dataset exists server-side.
+
+- **Claim 2 (Split):** Verified 11 public repos match the paper's "public set of 11." The held-out (12 repos) and commercial (18 repos) splits are deliberately non-public by design — this is the contamination-resistance mechanism, not a discrepancy.
+
+- **Claim 3 (Long-horizon):** Measured patch complexity on 731 gold patches: **85.6% touch >1 file** (median 4 files, mean 5.08, max 106); mean gold-patch size ≈ 13.3k chars. Cross-reference against original SWE-Bench: SWE-Bench Pro patches are **4.34x larger** and **3.44x more multi-file**. Time-to-fix estimates: mean 4.4h, max 59.9h per task (50 chars/min slow estimate).
+
+- **Claim 4 (Human-verified):** **100%** of 731 instances carry all five human-augmentation fields: `problem_statement`, `requirements`, `interface`, `test_patch`, `dockerhub_tag`. Problem-statement quality: mean 1,297 chars (range 419–8,036); 32.1% contain error/traceback diagnostics; 12.6% contain code snippets. The `requirements` (mean 1,483 chars) and `interface` (mean 670 chars) fields are unique to SWE-Bench Pro and absent from the original.
+
+- **Claim 5 (Contamination-resistant):** **Zero instance ID overlap** with original SWE-Bench test set — these are entirely different tasks. Spans 4 production languages (Go 38.3%, Python 36.4%, JavaScript 22.6%, TypeScript 2.7%) vs Python-only original. Balanced across 3 enterprise domains: Business (220 instances, 30.1%), B2B (280, 38.3%), DevTools (231, 31.6%). Every instance carries pinned `dockerhub_tag` + `base_commit` for isolated container execution.
+
+### What we could NOT verify
+
+- The full 1,865 / 41 count (held-out and commercial splits are deliberately non-public)
+- Agent solve rates on the benchmark (this was a dataset audit, not an agentic re-solving)
+- Contamination resistance in practice (structural claim only, not a run-time test)
 
 ## Scope & cost
 
 |  | This reproduction | Full replication |
 |---|---|---|
-| Scope | Audit public benchmark release: count instances/repos, measure patch multi-file rate, context-augmentation coverage, domain/language spread | Solve 1,865 tasks with SOTA agents; report resolve rates across public/held-out/commercial splits |
-| Hardware | 1x HF CPU Job (python:3.12) | Agentic runs on many repos + containers (GPU/large CPU) |
-| Compute time | ~3 min job + dataset download | Days–weeks of agent trajectories |
-| Cost | < $0.01 | Hundreds–thousands of dollars |
+| Scope | Audit public benchmark release: count instances/repos, measure patch multi-file rate, context-augmentation coverage, domain/language spread; cross-reference vs original SWE-Bench | Solve 1,865 tasks with SOTA agents; report resolve rates across public/held-out/commercial splits |
+| Hardware | 1x HF CPU Job ([python:3.12](https://huggingface.co/jobs/Yashp2003/6a5c8046d216bd6f3a2009d5)) | Agentic runs on many repos + containers (GPU/large CPU) |
+| Compute time | ~3 min job + dataset download (~30s) | Days–weeks of agent trajectories |
+| Cost | < $0.01 (~$0.005 actual) | Hundreds–thousands of dollars |
 | Outcome | 5/5 claims structurally corroborated (public subset); full 1,865/41 + held-out/commercial not directly countable (by design) | Not attempted |
 
 
@@ -32,14 +42,12 @@ Analysis ran as HF CPU Jobs; total cost < $0.01. This is a **dataset-auditing re
 -->
 ## Reproduction Poster
 
-![Poster](https://huggingface.co/Yashp2003/swebenchpro-repro-artifacts/resolve/main/poster_embed.html)
+The poster is a 4-column ICML landscape layout (60 × 36 in) covering all 5 claims, methodology, and verdicts. Built with [Chenruishuo/posterly](https://github.com/Chenruishuo/posterly); self-contained HTML with interactive `data-logbook-target` hotspots linking to claim pages.
 
 **Download:**
-- [poster_embed.html](https://huggingface.co/Yashp2003/swebenchpro-repro-artifacts/resolve/main/poster_embed.html) — self-contained (676 KB, data-URI inlined)
-- [poster_preview.pdf](https://huggingface.co/Yashp2003/swebenchpro-repro-artifacts/resolve/main/poster_preview.pdf) — print-ready PDF (2.1 MB)
-- [poster_preview.png](https://huggingface.co/Yashp2003/swebenchpro-repro-artifacts/resolve/main/poster_preview.png) — thumbnail PNG
-
-The poster is a 4-column ICML landscape layout (60 × 36 in) covering all 5 claims, methodology, and verdicts.
+- [poster_embed.html](https://huggingface.co/Yashp2003/swebenchpro-repro-artifacts/resolve/main/poster_embed.html) — self-contained interactive HTML (inlined CSS, hover-to-navigate hotspots)
+- [poster_preview.pdf](https://huggingface.co/Yashp2003/swebenchpro-repro-artifacts/resolve/main/poster_preview.pdf) — print-ready PDF (2.2 MB)
+- [poster_preview.png](https://huggingface.co/Yashp2003/swebenchpro-repro-artifacts/resolve/main/poster_preview.png) — thumbnail PNG (507 KB)
 
 
 ---