entity_tracking validation counts don't match the released eval data (blocks full-preds submissions)
Submitting Strict-small full predictions fails with:
Error: ambiref_0_ops has the wrong number of examples.
We traced this to a version gap specific to entity_tracking:
- This Space (
src/display/utils.py,NUM_EXPECTED_EXAMPLES) expects e.g.ambiref_0_ops = 508(entity totals: regular 2238 / ambiref 2315 / move_contents 2227). - The released eval data (
BabyLM-community/BabyLM-2026-Strict-Evals, last updated 2026-04-22) containsambiref_0_ops = 607(totals 3152 / 3224 / 3107). All rows have uniqueexample_ids, so this is not a dedup issue. - The official pipeline (
babylm-eval@ latest main) hardcodes the same 607-series incollate_preds.py(ENTITY_TRACKING_SIZES), and the 2025 leaderboard Space (babylm-leaderboard-2025-all-tasks) expects exactly the 607-series too. - Every other task (BLiMP, BLiMP-supplement, EWoK, COMPS, reading, GLUE, AoA) matches this Space's expected counts exactly — entity_tracking is the only mismatch.
So it looks like the Space validates entity_tracking against a revised dataset that was never published to the -Evals repo, meaning full-prediction submissions produced by the official pipeline on the official data are rejected.
How we worked around it for now: we resubmitted with the entity_tracking block omitted (which the validator permits — missing tasks are scored as 0), so our entry bind1-babylm2026-strict-small (strict-small) is on the leaderboard without its entity_tracking score. Once reconciled, we'll resubmit the same entry with full predictions so entity_tracking can be scored per the score-merging rules.
Could you either publish the revised entity_tracking data (we'll gladly re-run that task), or align NUM_EXPECTED_EXAMPLES with the released data? Happy to share the full per-subtask count comparison if useful. Thanks!
Update — root cause found. The expected counts changed in commit a010bbce ("Updated Entity Tracking to reflect removal of nothing answers", Jul 7, 20:24 UTC): the validator now expects entity_tracking predictions with items whose gold target contains "nothing" removed (607-series → 508-series). The released eval data (BabyLM-2026-Strict-Evals) and babylm-eval (collate_preds.py ENTITY_TRACKING_SIZES) still produce the pre-filter 607-series, so any full-predictions submission generated by the current official pipeline after that commit will hit this error until the pipeline/data are updated.
Workaround that works today: drop prediction entries at the positions where the gold target (= options[0] in the released jsonl files) contains "nothing", keeping the original order. This reproduces all 18 expected subtask counts exactly, and scoring stays index-aligned with the filtered gold. Our entry now validates fully with entity_tracking included.
Suggestion for the organizers: updating ENTITY_TRACKING_SIZES in collate_preds.py (or filtering in collate_preds itself) and noting the change in the babylm-eval README would keep the pipeline and the leaderboard in sync for other participants. Thanks!
One more point, on cross-submission consistency: the Jul 7 commit changed the entity_tracking metric (nothing-removal) and fixed a predictions-vs-gold alignment issue at the same time. That means any entity_tracking score computed before that commit is on a different, non-comparable basis than submissions scored afterwards — earlier entries were scored on the pre-fix, full-count metric, while submissions made now are scored on the corrected, nothing-removed one. On the current Strict-small board there are entries with entity_tracking in the ~39 range (the pre-removal full-set range) alongside post-change entries near ~19, which suggests the column is currently mixing the two scorings.
Would it be possible to recompute all existing entries' entity_tracking under the current scoring, so the column is consistent across all submissions? As it stands, later submitters are held to a stricter, corrected metric than earlier ones. Thanks again for looking into this!