Update logbook: Reproduction: SWE-Bench Pro: Can AI Agents Solve Long-Horizon Software Engineering Tasks?

logbook.json

@@ -8,7 +8,7 @@
     "icml2026-repro",
     "paper-uEVTdoAbnK"
   ],
-  "updated_at": "2026-07-19T10:40:23+00:00",
+  "updated_at": "2026-07-19T11:03:21+00:00",
   "root": {
     "slug": "index",
     "title": "Reproduction: SWE-Bench Pro: Can AI Agents Solve Long-Horizon Software Engineering Tasks?",
@@ -58,6 +58,6 @@
       }
     ]
   },
-  "agent_view_tokens": 7804,
-  "revision": "1784457623643127918"
+  "agent_view_tokens": 7813,
+  "revision": "1784459001597096707"
 }

pages/claim-3-tasks-are-long-horizon-potentially-requiring-hours-to-days-for-professional-engineers-and-often-spanning-multiple-files-abstract-only/page.md

@@ -14,7 +14,7 @@
 ### Measured patch complexity (public set, n=731)
 From the gold patches in **[ScaleAI/SWE-bench_Pro](https://huggingface.co/datasets/ScaleAI/SWE-bench_Pro)**:
 
-- **Files touched per patch:** min 1, median 4, mean 5.07, max 106.
+- **Files touched per patch:** min 1, median 4, mean 5.08, max 106.
 - **Multi-file patches:** **626 / 731 = 85.6%** touch more than one file.
 - **Patch size:** mean ≈ 13.3k characters (range 1.44k – 180k per the dataset README) — i.e. substantial code modifications.
 - **Language diversity:** patches span `go`, `js`, `python`, `ts` across 11 repos — realistic cross-stack scope.
@@ -25,7 +25,7 @@ A median of 4 files touched and a mean gold-patch size of ~13k characters is str
 ### Evidence
 - HF Job: https://huggingface.co/jobs/Yashp2003/6a5c8046d216bd6f3a2009d5
 - Results: `claim_analysis_results.json` in https://huggingface.co/buckets/Yashp2003/swebenchpro-repro-artifacts
-  - `instances_multi_file_pct` = 85.6, `patch_files_median` = 4, `patch_files_mean` = 5.07, `patch_files_max` = 106
+  - `instances_multi_file_pct` = 85.6, `patch_files_median` = 4, `patch_files_mean` = 5.08, `patch_files_max` = 106
 
 ### Cross-reference against original SWE-Bench
 
@@ -34,12 +34,12 @@ We loaded the original [princeton-nlp/SWE-bench](https://huggingface.co/datasets
 | Metric | Original SWE-Bench | SWE-Bench Pro | Ratio |
 |---|---|---|---|
 | Test instances | 2,294 | 731 | 0.32x |
-| Patch chars (mean) | 3,064 | 13,291 | **4.34x** |
-| Files/patch (mean) | 1.46 | 5.02 | **3.45x** |
-| Multi-file (%) | 24.8% | 85.6% | **3.45x** |
+| Patch chars (mean) | 3,061 | 13,291 | **4.34x** |
+| Files/patch (mean) | 1.66 | 5.08 | **3.07x** |
+| Multi-file (%) | 24.9% | 85.6% | **3.44x** |
 | Languages | Python only | 4 (go, js, python, ts) | — |
 
-**SWE-Bench Pro is substantially harder** than the original: patches are 4.34x larger on average, 3.45x more likely to be multi-file, and span 4 programming languages instead of just Python.
+**SWE-Bench Pro is substantially harder** than the original: patches are 4.34x larger on average, 3.44x more likely to be multi-file, and span 4 programming languages instead of just Python.
 
 ### Time-to-fix estimates
 
@@ -47,7 +47,7 @@ Using a conservative developer speed of ~50 chars/min (moderate difficulty):
 - **Mean:** 4.4 hours per task
 - **Median:** 2.6 hours per task
 - **Maximum:** 59.9 hours per task
-- **Tasks needing 4+ hours:** ~60% of the benchmark
+- **Tasks needing 4+ hours:** 10.8%–31.2% of the benchmark (fast/slow estimate)
 
 Using a faster estimate of ~100 chars/min:
 - **Mean:** 2.2 hours per task

pages/claim-4-all-swe-bench-pro-tasks-underwent-human-verification-for-adequate-resolution-context-abstract-only/page.md

@@ -38,10 +38,10 @@ The uniform presence of these augmentation fields across all 731 public instance
 **Problem statement content depth:**
 - Mean length: 1,297 chars (range 419–8,036)
 - Contains URLs (links to issues, PRs, docs): 52/731 (7.1%)
-- Contains code snippets (`def `, `class `): 28/731 (3.8%)
-- Contains error/traceback information: 228/731 (**31.2%**)
+- Contains code snippets (`def `, `class `): 92/731 (12.6%)
+- Contains error/traceback information: 235/731 (**32.1%**)
 
-The 31.2% error/traceback rate is significant — it means nearly a third of tasks include concrete failure diagnostics, not just abstract descriptions. This is exactly the kind of "adequate context for resolution" the paper claims.
+The 32.1% error/traceback rate is significant — it means nearly a third of tasks include concrete failure diagnostics, not just abstract descriptions. This is exactly the kind of "adequate context for resolution" the paper claims.
 
 **Augmentation field depth:**
 - `requirements`: mean 1,483 chars (substantial dependency/specification context)

pages/conclusion/page.md

@@ -9,7 +9,7 @@
 
 ### Overall verdict: 5/5 claims structurally corroborated (with cross-reference)
 
-All five abstract-level claims are consistent with what the public data shows. Cross-referencing against the original SWE-Bench provides additional quantitative evidence: SWE-Bench Pro patches are **4.34x larger** and **3.45x more multi-file** than the original, with **zero instance ID overlap** — confirming these are entirely different, harder tasks.
+All five abstract-level claims are consistent with what the public data shows. Cross-referencing against the original SWE-Bench provides additional quantitative evidence: SWE-Bench Pro patches are **4.34x larger** and **3.44x more multi-file** than the original, with **zero instance ID overlap** — confirming these are entirely different, harder tasks.
 
 ### Verification summary
 
@@ -17,8 +17,8 @@ All five abstract-level claims are consistent with what the public data shows. C
 |-------|-----------|---------------|--------------------------------------|---------|
 | 1: Scale | 1,865 problems / 41 repos | 731 public / 11 repos (11+12+18=41) | Eval repo confirms full dataset exists | supported |
 | 2: Split | 11 public / 12 held-out / 18 commercial | 11 public repos verified; eval repo has separate leaderboards | Non-public splits confirmed via eval repo | consistent |
-| 3: Long-horizon | Hours-to-days, multi-file | 85.6% multi-file, median 4 files | **4.34x larger patches, 3.45x more multi-file** than original | supported |
-| 4: Human-verified | Adequate context | 100% carry requirements/interface/test_patch; 31.2% include error traces | Original lacks requirements/interface fields | supported |
+| 3: Long-horizon | Hours-to-days, multi-file | 85.6% multi-file, median 4 files | **4.34x larger patches, 3.44x more multi-file** than original | supported |
+| 4: Human-verified | Adequate context | 100% carry requirements/interface/test_patch; 32.1% include error traces | Original lacks requirements/interface fields | supported |
 | 5: Contamination-resistant | Spans business/B2B/dev-tools | 4 languages, 3 domains, containerised | **Zero ID overlap** with original SWE-Bench | supported |
 
 ### What we could NOT verify

pages/executive-summary/page.md

@@ -9,8 +9,8 @@ We reproduced the five headline claims of **SWE-Bench Pro** (arXiv 2509.16941 /
 
 1. **Scale:** 731 public problems / 11 repos verified; eval repo confirms full 1,865 exists server-side; arithmetic 11+12+18=41 internally consistent.
 2. **Split:** 11 public repos confirmed; held-out (12) + commercial (18) non-public by design; eval repo maintains separate public/private leaderboards.
-3. **Long-horizon:** 85.6% multi-file patches (median 4 files), mean patch 13.3k chars — **4.34x larger** and **3.45x more multi-file** than original SWE-Bench. Time estimates: mean 4.4h, max 60h per task.
-4. **Human-verified:** 100% carry requirements/interface/test_patch; 31.2% include error/traceback diagnostics; mean requirements 1,483 chars.
+3. **Long-horizon:** 85.6% multi-file patches (median 4 files), mean patch 13.3k chars — **4.34x larger** and **3.44x more multi-file** than original SWE-Bench. Time estimates: mean 4.4h, max 60h per task.
+4. **Human-verified:** 100% carry requirements/interface/test_patch; 32.1% include error/traceback diagnostics; mean requirements 1,483 chars.
 5. **Contamination-resistant:** **Zero instance ID overlap** with original SWE-Bench; spans 4 languages (vs Python-only original); 3 enterprise domains; non-public splits prevent training leakage.
 
 Analysis ran as HF CPU Jobs; total cost < $0.01. This is a **dataset-auditing reproduction** with cross-reference validation, not an agentic re-solving of tasks.