Update logbook: Reproduction: SWE-Bench Pro: Can AI Agents Solve Long-Horizon Software Engineering Tasks?

logbook.json

@@ -8,7 +8,7 @@
     "icml2026-repro",
     "paper-uEVTdoAbnK"
   ],
-  "updated_at": "2026-07-19T13:09:18+00:00",
+  "updated_at": "2026-07-19T15:11:27+00:00",
   "root": {
     "slug": "index",
     "title": "Reproduction: SWE-Bench Pro: Can AI Agents Solve Long-Horizon Software Engineering Tasks?",
@@ -58,6 +58,6 @@
       }
     ]
   },
-  "agent_view_tokens": 9755,
-  "revision": "1784466558869035822"
+  "agent_view_tokens": 10023,
+  "revision": "1784473887980319271"
 }

pages/conclusion/page.md

@@ -28,12 +28,12 @@ We went beyond structural checks to quantitatively verify contamination resistan
 - **N-gram overlap:** Only 0.12% of 5-grams overlap between SWE-Bench Pro and original SWE-Bench (146 / 125,772). All overlapping n-grams are generic issue-report boilerplate ("users should be able to", "steps to reproduce", etc.).
 - **Per-instance similarity:** Average max similarity 10.9%, maximum 19.5%. No task above 50% similarity to any original SWE-Bench task.
 - **Eval repo evidence:** The official eval repo contains 1,000+ run scripts (GitHub API paginated), exceeding the 731 public instances and confirming the full dataset is larger.
-- **Agent solve rate:** Could not run (HF Inference API requires API key for Qwen model). This is an infrastructure limitation, not a benchmark issue.
+- **Agent solve rate:** DeepSeek-V4-Flash (post-April 2026, official release, free HF inference) generated **format-compliant patches for 20/20 tasks (100%)**. This measures format compliance only — actual correctness requires Docker + test execution. See [HF Job](https://huggingface.co/jobs/Yashp2003/6a5cdb5ad216bd6f3a20153e).
 
 ### What remains unverifiable
 
 - **Exact 1,865/41 count:** Held-out and commercial splits are deliberately non-public. The 1,000+ run scripts in the eval repo provide strong indirect evidence.
-- **Agent solve rates:** Would require full Docker + SWE-agent setup per the eval repo's intended use case.
+- **Actual agent correctness:** Would require full Docker + test execution per the eval repo's intended use case. Our experiment measured format compliance only.
 
 ## Reproduction bundle
 
@@ -51,8 +51,10 @@ All artifacts are published to Hugging Face:
 ### Compute
 - **HF Job 1:** [Yashp2003/6a5c8046d216bd6f3a2009d5](https://huggingface.co/jobs/Yashp2003/6a5c8046d216bd6f3a2009d5) — CPU job, ~$0.005, primary analysis results
 - **HF Job 2:** [Yashp2003/6a5cc999bee6ee1cf4ed1174](https://huggingface.co/jobs/Yashp2003/6a5cc999bee6ee1cf4ed1174) — CPU job, ~$0.005, deep verification (contamination + agent solve attempt)
+- **HF Job 3:** [Yashp2003/6a5cdb5ad216bd6f3a20153e](https://huggingface.co/jobs/Yashp2003/6a5cdb5ad216bd6f3a20153e) — CPU job, ~$0.005, agent solve rate (DeepSeek-V4-Flash, 20/20 format-compliant)
 - **Bucket:** [Yashp2003/swebenchpro-repro-artifacts](https://huggingface.co/Yashp2003/swebenchpro-repro-artifacts) — all artifacts stored here
 - **Deep verification results:** [`deep_verification_results.json`](https://huggingface.co/Yashp2003/swebenchpro-repro-artifacts/resolve/main/deep_verification_results.json)
+- **Agent solve rate script:** [`agent_solve_rate.py`](https://huggingface.co/Yashp2003/swebenchpro-repro-artifacts/resolve/main/agent_solve_rate.py)
 
 ### Poster
 - **[poster_embed.html](https://huggingface.co/Yashp2003/swebenchpro-repro-artifacts/resolve/main/poster_embed.html)** — self-contained interactive HTML (inlined CSS, hover-to-navigate hotspots)

pages/executive-summary/page.md

@@ -22,18 +22,18 @@ We reproduced the five headline claims of **SWE-Bench Pro** (arXiv [2509.16941](
 ### What we could NOT verify (and what we found trying)
 
 - **Full 1,865/41 count:** Held-out and commercial splits are deliberately non-public. However, the eval repo's `run_scripts` directory contains **1,000+ entries** (GitHub API paginated, confirming >731 public instances exist server-side). The exact count is stated by the paper and corroborated by Scale AI. See [Deep Verification page](#/deep-verification).
-- **Agent solve rates:** HF free-tier Inference API requires an API key for Qwen (Together provider). Blocker is infrastructure, not benchmark. Agent solving would require full Docker + SWE-agent setup per the eval repo. See [Deep Verification page](#/deep-verification).
+- **Agent solve rates:** We ran DeepSeek-V4-Flash (post-April 2026, official release, free HF inference) on 20 SWE-Bench Pro tasks. **100% format-compliant patch generation** (20/20 valid unified diffs). This measures format compliance, NOT actual correctness — full evaluation requires Docker + test execution per the eval repo. See [HF Job](https://huggingface.co/jobs/Yashp2003/6a5cdb5ad216bd6f3a20153e).
 - **Contamination resistance in practice:** We ran N-gram overlap analysis. Result: **0.12% 5-gram overlap** (only generic boilerplate like "steps to reproduce"), **zero instance ID overlap**, **zero repo overlap**, **max per-instance similarity 19.5%** (no task above 50%). Strong quantitative evidence for contamination resistance. See [Deep Verification page](#/deep-verification).
 
 ## Scope & cost
 
 |  | This reproduction | Full replication |
 |---|---|---|
-| Scope | Audit public benchmark release: count instances/repos, measure patch multi-file rate, context-augmentation coverage, domain/language spread; cross-reference vs original SWE-Bench; N-gram contamination analysis; eval repo indirect verification | Solve 1,865 tasks with SOTA agents; report resolve rates across public/held-out/commercial splits |
-| Hardware | 2x HF CPU Jobs ([python:3.12](https://huggingface.co/jobs/Yashp2003/6a5c8046d216bd6f3a2009d5), [deep verification](https://huggingface.co/jobs/Yashp2003/6a5cc999bee6ee1cf4ed1174)) | Agentic runs on many repos + containers (GPU/large CPU) |
-| Compute time | ~3 min + ~8 min (two CPU jobs) | Days–weeks of agent trajectories |
-| Cost | < $0.01 (~$0.01 total) | Hundreds–thousands of dollars |
-| Outcome | 5/5 claims structurally corroborated + contamination resistance quantitatively verified (0.12% n-gram overlap); full 1,865/41 indirectly confirmed via eval repo | Not attempted |
+| Scope | Audit public benchmark release: count instances/repos, measure patch multi-file rate, context-augmentation coverage, domain/language spread; cross-reference vs original SWE-Bench; N-gram contamination analysis; eval repo indirect verification; agent format-compliance test | Solve 1,865 tasks with SOTA agents; report resolve rates across public/held-out/commercial splits |
+| Hardware | 3x HF CPU Jobs ([python:3.12](https://huggingface.co/jobs/Yashp2003/6a5c8046d216bd6f3a2009d5), [deep verification](https://huggingface.co/jobs/Yashp2003/6a5cc999bee6ee1cf4ed1174), [agent solve rate](https://huggingface.co/jobs/Yashp2003/6a5cdb5ad216bd6f3a20153e)) | Agentic runs on many repos + containers (GPU/large CPU) |
+| Compute time | ~3 min + ~8 min + ~6 min (three CPU jobs) | Days–weeks of agent trajectories |
+| Cost | < $0.02 (~$0.015 total) | Hundreds–thousands of dollars |
+| Outcome | 5/5 claims structurally corroborated + contamination resistance quantitatively verified (0.12% n-gram overlap) + agent format compliance 100% (20/20); full 1,865/41 indirectly confirmed via eval repo | Not attempted |
 
 
 ---