pages/claim-3-swe-bench-pro-tasks-are-long-horizon-requiring-hours-to-days-for-professionals-and-multi-file-patches/page.md

Claim 3: SWE-Bench Pro tasks are long-horizon, requiring hours to days for professionals and multi-file patches.

---

Paper claim (Abstract, §3.1, Figure 3): Tasks are long-horizon, potentially requiring hours to days for a professional software engineer to complete, often involving patches across multiple files and substantial code modifications. Reference solutions span 107.4 lines across 4.1 files on average. Every problem involves at least 10 lines of change; over 100 tasks demand >100 lines.

Verification approach: Analyze the public split (731 instances) for patch complexity metrics: lines changed (added+deleted, excluding context), files modified, and problem statement detail.

---

from datasets import load_dataset
import pandas as pd

ds = load_dataset('ScaleAI/SWE-bench_Pro', split='test')
df = ds.to_pandas()

def changed_lines(patch):
    n = 0
    for l in patch.split('\n'):
        if (l.startswith('+') or l.startswith('-')) and not l.startswith('+++') and not l.startswith('---'):
            n += 1
    return n

# Changed lines (added + deleted, excluding context lines)
df['patch_lines'] = df['patch'].apply(changed_lines)
# Files modified (count of 'diff --git' headers)
df['patch_files'] = df['patch'].apply(lambda x: len([l for l in x.split('\n') if l.startswith('diff --git')]))
# Problem statement length
df['problem_len'] = df['problem_statement'].apply(len)

print('Patch changed-line stats:')
print(df['patch_lines'].describe())
print()
print('Patch files stats:')
print(df['patch_files'].describe())
print()
print(f'Min changed lines: {df["patch_lines"].min()}')
print(f'Tasks with >100 changed lines: {(df["patch_lines"] > 100).sum()} / {len(df)} ({(df["patch_lines"] > 100).mean()*100:.1f}%)')
print(f'Tasks with >=10 changed lines: {(df["patch_lines"] >= 10).sum()} / {len(df)} ({(df["patch_lines"] >= 10).mean()*100:.1f}%)')
print(f'Mean files per patch: {df["patch_files"].mean():.1f}')
print(f'Median files per patch: {df["patch_files"].median():.1f}')
print()
print('Problem statement length stats:')
print(df['problem_len'].describe())

---

Patch changed-line stats (added + deleted, excluding context):
count     731.000000
mean      169.597264
std       233.953069
min        20.000000
25%        53.000000
50%        94.000000
75%       190.000000
max      1986.000000

Patch files stats:
count    731.000000
mean       5.068399
std        6.119202
min        1.000000
25%        2.000000
50%        4.000000
75%        7.000000
max      106.000000

Min changed lines: 20
Tasks with >100 changed lines: 349 / 731 (47.7%)
Tasks with >=10 changed lines: 731 / 731 (100.0%)
Mean files per patch: 5.1
Median files per patch: 4.0

Problem statement length stats:
count     731.000000
mean     1297.465116
std       581.334987
min       419.000000
25%       976.000000
50%      1180.000000
75%       1458.000000
max      8036.000000

---

✅ VERIFIED (public split exceeds paper's stated thresholds)

Metric	Paper Claim (Full Dataset)	Public Split (Verified)
Mean lines changed	107.4	169.6
Mean files modified	4.1	5.1
Minimum lines changed	≥10	20
Tasks >100 changed lines	>100	349 (47.7%)
All tasks ≥10 changed lines	Yes	731/731 (100%)

The public split exceeds the paper's stated complexity thresholds. Every task requires ≥20 changed lines (vs. paper's ≥10), spans a median of 4 files (mean 5.1 vs. paper's 4.1), and 47.7% exceed 100 changed lines (paper states ">100 tasks" out of 1,865 total, i.e. >5.4% — the public split is far above that floor). Problem statements are substantial (mean ~1,300 chars), providing detailed context.

The "hours to days for professionals" claim is qualitative but supported by the patch complexity: multi-file edits averaging 170 changed lines across 5+ files in unfamiliar enterprise codebases aligns with long-horizon engineering tasks.

Sources:

• Paper: §3.1 "Characteristics of SWE-BENCH PRO", Figure 3 distributions
• Dataset: ScaleAI/SWE-bench_Pro (test split)
• Repository domains: DevOps (ansible), library catalog (openlibrary), feature flags (flipt), browser (qutebrowser), access control (teleport), music streaming (navidrome), chat (element-web), security scanning (vuls), email (protonmail, tutanota), forum (NodeBB)

(End of file)

---

Reproduction run (executed HF Job). All five claims were recomputed on Hugging Face Jobs (cpu-basic, ~$0.0006): Job 6a5da3acbee6ee1cf4ed1e8e. The run loads ScaleAI/SWE-bench_Pro (test split, 731 rows) and prints this claim as VERIFIED in the immutable job logs. Script: repro_job/run_claims.py in the reproduction bundle.