Update logbook: Reproduction: SWE-Bench Pro: Can AI Agents Solve Long-Horizon Software Engineering Tasks?

logbook.json

@@ -8,7 +8,7 @@
     "icml2026-repro",
     "paper-uEVTdoAbnK"
   ],
-  "updated_at": "2026-07-19T13:01:37+00:00",
+  "updated_at": "2026-07-19T13:02:33+00:00",
   "root": {
     "slug": "index",
     "title": "Reproduction: SWE-Bench Pro: Can AI Agents Solve Long-Horizon Software Engineering Tasks?",
@@ -61,9 +61,15 @@
         "title": "Deep Verification: Contamination Resistance, Agent Solve Rate, and Eval Repo Evidence",
         "file": "pages/deep-verification-contamination-resistance-agent-solve-rate-and-eval-repo-evidence/page.md",
         "children": []
+      },
+      {
+        "slug": "deep-verification",
+        "title": "Deep Verification",
+        "file": "pages/deep-verification/page.md",
+        "children": []
       }
     ]
   },
-  "agent_view_tokens": 9198,
-  "revision": "1784466097528170082"
+  "agent_view_tokens": 9556,
+  "revision": "1784466153386688257"
 }

pages/deep-verification/page.md

@@ -0,0 +1,23 @@
+# Deep Verification
+
+
+---
+<!-- trackio-cell
+{"type": "markdown", "id": "cell_b2970f91d848", "created_at": "2026-07-19T13:02:27+00:00", "title": "What we verified"}
+-->
+## What we verified
+
+We went deeper on three things that couldn't be verified from the public dataset alone:
+
+### 1. Contamination Resistance (N-gram overlap)
+
+**Method:** Computed 5-gram overlap between SWE-Bench Pro problem statements (731) and original SWE-Bench test set (2,294). Also checked instance ID overlap, repo overlap, and per-instance SequenceMatcher similarity.
+
+**Results:**
+- **Instance ID overlap: 0** (zero tasks shared between benchmarks)
+- **5-gram overlap: 0.12%** (only 146 out of 125,772 Pro n-grams appear in original SWE-Bench)
+- **Repository overlap: none** (zero shared repos)
+- **Per-instance max similarity: avg 10.9%, max 19.5%** (no instance above 50% similarity to any original SWE-Bench task)
+- **All overlapping n-grams are generic boilerplate** (e.g., 'users should be able to', 'steps to reproduce', 'there is no way to')
+
+**Verdict: Strong evidence for contamination resistance.** The two benchmarks share zero tasks, zero repos, and near-zero textual overlap beyond standard issue-reporting language.

pages/index.md

@@ -14,3 +14,4 @@
 | [Claim 5: SWE-Bench Pro is a contamination-resistant testbed spanning business applications, B2B services, and developer tools (abstract only).](#/claim-5-swe-bench-pro-is-a-contamination-resistant-testbed-spanning-business-applications-b2b-services-and-developer-tools-abstract-only) |
 | [Conclusion](#/conclusion) |
 | [Deep Verification: Contamination Resistance, Agent Solve Rate, and Eval Repo Evidence](#/deep-verification-contamination-resistance-agent-solve-rate-and-eval-repo-evidence) |
+| [Deep Verification](#/deep-verification) |