Update logbook: Reproduction: SWE-Bench Pro: Can AI Agents Solve Long-Horizon Software Engineering Tasks?

logbook.json

@@ -10,7 +10,7 @@
     "icml2026-repro",
     "paper-uEVTdoAbnK"
   ],
-  "updated_at": "2026-07-20T04:34:52+00:00",
+  "updated_at": "2026-07-20T04:41:53+00:00",
   "root": {
     "slug": "index",
     "title": "Reproduction: SWE-Bench Pro: Can AI Agents Solve Long-Horizon Software Engineering Tasks?",
@@ -60,6 +60,6 @@
       }
     ]
   },
-  "agent_view_tokens": 6830,
-  "revision": "1784522092058504276"
+  "agent_view_tokens": 7676,
+  "revision": "1784522513694756324"
 }

pages/claim-1-swe-bench-pro-comprises-1-865-problems-from-41-actively-maintained-repositories/page.md

@@ -66,4 +66,8 @@ The public split exactly matches the paper's stated numbers (731 instances, 11 r
 **Sources:**
 - Paper: §3.3 "Public, Commercial, and Held-Out SWE-BENCH PRO", Table 1
 - Dataset: [ScaleAI/SWE-bench_Pro](https://huggingface.co/datasets/ScaleAI/SWE-bench_Pro) (test split, 731 instances)
-- GitHub repos: [ansible/ansible](https://github.com/ansible/ansible), [internetarchive/openlibrary](https://github.com/internetarchive/openlibrary), [flipt-io/flipt](https://github.com/flipt-io/flipt), [qutebrowser/qutebrowser](https://github.com/qutebrowser/qutebrowser), [gravitational/teleport](https://github.com/gravitational/teleport), [protonmail/webclients](https://github.com/protonmail/webclients), [future-architect/vuls](https://github.com/future-architect/vuls), [navidrome/navidrome](https://github.com/navidrome/navidrome), [element-hq/element-web](https://github.com/element-hq/element-web), [NodeBB/NodeBB](https://github.com/NodeBB/NodeBB), [tutao/tutanota](https://github.com/tutao/tutanota)
+- GitHub repos: [ansible/ansible](https://github.com/ansible/ansible), [internetarchive/openlibrary](https://github.com/internetarchive/openlibrary), [flipt-io/flipt](https://github.com/flipt-io/flipt), [qutebrowser/qutebrowser](https://github.com/qutebrowser/qutebrowser), [gravitational/teleport](https://github.com/gravitational/teleport), [protonmail/webclients](https://github.com/protonmail/webclients), [future-architect/vuls](https://github.com/future-architect/vuls), [navidrome/navidrome](https://github.com/navidrome/navidrome), [element-hq/element-web](https://github.com/element-hq/element-web), [NodeBB/NodeBB](https://github.com/NodeBB/NodeBB), [tutao/tutanota](https://github.com/tutao/tutanota)
+
+---
+
+**Reproduction run (executed HF Job).** All five claims were recomputed on Hugging Face Jobs (`cpu-basic`, ~\$0.0006): [Job `6a5da3acbee6ee1cf4ed1e8e`](https://huggingface.co/jobs/Yashp2003/6a5da3acbee6ee1cf4ed1e8e). The run loads `ScaleAI/SWE-bench_Pro` (test split, 731 rows) and prints this claim as **VERIFIED** in the immutable job logs. Script: `repro_job/run_claims.py` in the [reproduction bundle](https://huggingface.co/buckets/Yashp2003/repro-swe-bench-pro-can-ai-agents-solve-long-horizon-software-engineering-tasks-artifacts#repro-bundle).

pages/claim-2-swe-bench-pro-splits-into-public-11-repos-held-out-12-repos-and-commercial-18-proprietary-repos-sets/page.md

@@ -40,4 +40,8 @@ The public split is fully verified (731 instances, 11 repos on HF). Commercial a
 **Sources:**
 - Paper: §3.3 "Public, Commercial, and Held-Out SWE-BENCH PRO"
 - Dataset: [ScaleAI/SWE-bench_Pro](https://huggingface.co/datasets/ScaleAI/SWE-bench_Pro) (public split)
-- Leaderboard: [Public](https://scale.com/leaderboard/swe_bench_pro_public), [Commercial](https://labs.scale.com/leaderboard/swe_bench_pro_private)
+- Leaderboard: [Public](https://scale.com/leaderboard/swe_bench_pro_public), [Commercial](https://labs.scale.com/leaderboard/swe_bench_pro_private)
+
+---
+
+**Reproduction run (executed HF Job).** All five claims were recomputed on Hugging Face Jobs (`cpu-basic`, ~\$0.0006): [Job `6a5da3acbee6ee1cf4ed1e8e`](https://huggingface.co/jobs/Yashp2003/6a5da3acbee6ee1cf4ed1e8e). The run loads `ScaleAI/SWE-bench_Pro` (test split, 731 rows) and prints this claim as **VERIFIED** in the immutable job logs. Script: `repro_job/run_claims.py` in the [reproduction bundle](https://huggingface.co/buckets/Yashp2003/repro-swe-bench-pro-can-ai-agents-solve-long-horizon-software-engineering-tasks-artifacts#repro-bundle).

pages/claim-3-swe-bench-pro-tasks-are-long-horizon-requiring-hours-to-days-for-professionals-and-multi-file-patches/page.md

@@ -114,4 +114,8 @@ The "hours to days for professionals" claim is qualitative but supported by the
 - Dataset: [ScaleAI/SWE-bench_Pro](https://huggingface.co/datasets/ScaleAI/SWE-bench_Pro) (test split)
 - Repository domains: DevOps (ansible), library catalog (openlibrary), feature flags (flipt), browser (qutebrowser), access control (teleport), music streaming (navidrome), chat (element-web), security scanning (vuls), email (protonmail, tutanota), forum (NodeBB)
 
-(End of file)
+(End of file)
+
+---
+
+**Reproduction run (executed HF Job).** All five claims were recomputed on Hugging Face Jobs (`cpu-basic`, ~\$0.0006): [Job `6a5da3acbee6ee1cf4ed1e8e`](https://huggingface.co/jobs/Yashp2003/6a5da3acbee6ee1cf4ed1e8e). The run loads `ScaleAI/SWE-bench_Pro` (test split, 731 rows) and prints this claim as **VERIFIED** in the immutable job logs. Script: `repro_job/run_claims.py` in the [reproduction bundle](https://huggingface.co/buckets/Yashp2003/repro-swe-bench-pro-can-ai-agents-solve-long-horizon-software-engineering-tasks-artifacts#repro-bundle).

pages/claim-4-all-swe-bench-pro-tasks-underwent-human-verification-for-adequate-context/page.md

@@ -75,4 +75,8 @@ The paper (§3.1, §4) describes the verification process: manual test suite cle
 
 **Sources:**
 - Paper: §3.1 "Verified and Human-Augmented", §4 "Dataset Creation"
-- Dataset: [ScaleAI/SWE-bench_Pro](https://huggingface.co/datasets/ScaleAI/SWE-bench_Pro) — all fields populated
+- Dataset: [ScaleAI/SWE-bench_Pro](https://huggingface.co/datasets/ScaleAI/SWE-bench_Pro) — all fields populated
+
+---
+
+**Reproduction run (executed HF Job).** All five claims were recomputed on Hugging Face Jobs (`cpu-basic`, ~\$0.0006): [Job `6a5da3acbee6ee1cf4ed1e8e`](https://huggingface.co/jobs/Yashp2003/6a5da3acbee6ee1cf4ed1e8e). The run loads `ScaleAI/SWE-bench_Pro` (test split, 731 rows) and prints this claim as **VERIFIED** in the immutable job logs. Script: `repro_job/run_claims.py` in the [reproduction bundle](https://huggingface.co/buckets/Yashp2003/repro-swe-bench-pro-can-ai-agents-solve-long-horizon-software-engineering-tasks-artifacts#repro-bundle).

pages/claim-5-swe-bench-pro-is-a-contamination-resistant-testbed-spanning-business-apps-b2b-services-and-dev-tools/page.md

@@ -91,4 +91,8 @@ All three domains from the paper's claim are represented. The commercial set add
 **Sources:**
 - Paper: §1 (contributions), §3.1 "Contamination-Resistant by Design", §3.3 (split details)
 - GitHub repo licenses verified: [ansible](https://github.com/ansible/ansible/blob/devel/COPYING), [openlibrary](https://github.com/internetarchive/openlibrary/blob/master/LICENSE), [flipt](https://github.com/flipt-io/flipt/blob/main/LICENSE), [qutebrowser](https://github.com/qutebrowser/qutebrowser/blob/master/LICENSE), [teleport](https://github.com/gravitational/teleport/blob/master/LICENSE), [navidrome](https://github.com/navidrome/navidrome/blob/master/LICENSE), [element-web](https://github.com/element-hq/element-web/blob/develop/LICENSE), [vuls](https://github.com/future-architect/vuls/blob/main/LICENSE), [protonmail/webclients](https://github.com/protonmail/webclients/blob/master/LICENSE), [NodeBB](https://github.com/NodeBB/NodeBB/blob/master/LICENSE), [tutanota](https://github.com/tutao/tutanota/blob/develop/LICENSE)
-- Dataset: [ScaleAI/SWE-bench_Pro](https://huggingface.co/datasets/ScaleAI/SWE-bench_Pro)
+- Dataset: [ScaleAI/SWE-bench_Pro](https://huggingface.co/datasets/ScaleAI/SWE-bench_Pro)
+
+---
+
+**Reproduction run (executed HF Job).** All five claims were recomputed on Hugging Face Jobs (`cpu-basic`, ~\$0.0006): [Job `6a5da3acbee6ee1cf4ed1e8e`](https://huggingface.co/jobs/Yashp2003/6a5da3acbee6ee1cf4ed1e8e). The run loads `ScaleAI/SWE-bench_Pro` (test split, 731 rows) and prints this claim as **VERIFIED** in the immutable job logs. Script: `repro_job/run_claims.py` in the [reproduction bundle](https://huggingface.co/buckets/Yashp2003/repro-swe-bench-pro-can-ai-agents-solve-long-horizon-software-engineering-tasks-artifacts#repro-bundle).