Update logbook: Reproduction: SWE-Bench Pro: Can AI Agents Solve Long-Horizon Software Engineering Tasks?

logbook.json

@@ -8,7 +8,7 @@
     "icml2026-repro",
     "paper-uEVTdoAbnK"
   ],
-  "updated_at": "2026-07-19T13:03:02+00:00",
+  "updated_at": "2026-07-19T13:03:16+00:00",
   "root": {
     "slug": "index",
     "title": "Reproduction: SWE-Bench Pro: Can AI Agents Solve Long-Horizon Software Engineering Tasks?",
@@ -70,6 +70,6 @@
       }
     ]
   },
-  "agent_view_tokens": 10224,
-  "revision": "1784466182026829323"
+  "agent_view_tokens": 10499,
+  "revision": "1784466196114895058"
 }

pages/deep-verification/page.md

@@ -55,3 +55,28 @@ We went deeper on three things that couldn't be verified from the public dataset
 **HF Job:** [Yashp2003/6a5cc999bee6ee1cf4ed1174](https://huggingface.co/jobs/Yashp2003/6a5cc999bee6ee1cf4ed1174)
 
 **Note:** Agent solve rate is outside the scope of this dataset audit. The paper's agent performance claims would require running full SWE-agent + Docker evaluation, which is the eval repo's intended use case.
+
+
+---
+<!-- trackio-cell
+{"type": "markdown", "id": "cell_6509a3fa962d", "created_at": "2026-07-19T13:03:13+00:00", "title": "Contamination Analysis Details"}
+-->
+### Contamination Analysis Details
+
+| Metric | Value | Interpretation |
+|---|---|---|
+| Instance ID overlap | 0 | Zero shared tasks |
+| 5-gram overlap | 146 / 125,772 (0.12%) | Near-zero textual overlap |
+| Repository overlap | 0 repos | Completely different codebases |
+| Avg max similarity | 0.1094 | Very low (10.9%) |
+| Max similarity | 0.1953 | No task above 20% similarity |
+| Tasks > 50% similar | 0 / 30 | No semantic duplicates |
+
+**Top overlapping 5-grams** (all generic issue-report boilerplate):
+1. 'users should be able to' (13x in Pro)
+2. 'what did you expect to' (11x)
+3. 'steps to reproduce 1. run' (9x)
+4. 'there is no way to' (8x)
+5. 'did you expect to happen?' (6x)
+
+None of these contain domain-specific or task-specific content -- they are standard GitHub issue template language shared across any two software bug-reporting datasets.