Avifenesh/episodic-ingestion-modernbert-field-event-ranker-mixed-v2-perf-h4

episodic-ingestion-compiler / ModernBERT field-event ranker (mixed-mode v2, perf-H4)

Fine-tune of answerdotai/ModernBERT-base for the grouped multi-positive softmax ranker task with v2 semantic-reasoning labels.

What changed vs the v1 mixed-mode checkpoint

An audit of the v1 mixed-mode training data showed that 94% of eval pairs came from two fields (attempted_actions, observed_outcomes) whose labels were exact functions of event role on 97% of rows (attempted_actions=tool_call, observed_outcomes=tool_result). On these degenerate labels:

• E[MRR | random scores] ≈ 0.26 (random-permutation floor for multi-positive softmax)
• v1 perf-H4 MRR = 0.506 ≈ 1.97× random
• Architecture experiments (H3, H3a, H3b, H3b-span, etc.) could not break through because the ceiling was arithmetic, not informational.

v2 adds five semantic-reasoning fields whose labels require more than role:

field	kind	rule
initiating_command	single-pick	First tool_call in a multi-call chain
outcome_of_latest_attempt	single-pick	Tool_result whose call_id matches the LAST tool_call
recent_error	single-pick	Most-recent tool_result with success=false
attempt_outcome_pairs	array	Paired tool_results (only when window has ≥1 orphan result — otherwise empty, so it can't collapse to role==tool_result)
action_causality	array	Tool_results whose content tokens recur in a LATER assistant turn

Tautology rates on these new fields: 0–10%. Legacy fields kept for back-compat.

Training details

• Train rows: 1217 (mixed-mode v2: autonomous + prompt→action + customer-support + conversational)
• Train groups: 5882 (one per field × row pair)
• Eval rows: 220 (1900 groups, 21986 candidate pairs)
• Steps: 160 optimizer steps (--accum-steps 8)
• Optimizer: AdamW, lr 5.7e-5, warmup 30
• Precision: bf16 autocast
• Memory: gradient checkpointing (use_reentrant=False)
• Final train loss: 0.0854
• Peak VRAM: 4.08 GiB
• Hardware: RTX 5090 Laptop (Blackwell SM 12.0, 24 GiB)

Eval metrics (220-row mixed v2 eval, {'1': 0.2964509394572025, '2': 0.5173277661795407, '3': 0.6931106471816284, '5': 0.8881002087682672})

metric	value
overall MRR	0.526
top-1 recall	0.296
top-2 recall	0.517
top-3 recall	0.693
top-5 recall	0.888
mean expected rank	3.09

Per-field MRR (sorted by eval-pair count):

field	n	MRR	top-1
attempted_actions	846	0.469	0.23
observed_outcomes	845	0.486	0.24
initiating_command	199	0.667	0.50
outcome_of_latest_attempt	181	0.763	0.60
action_causality	141	0.483	0.24
attempt_outcome_pairs	53	0.563	0.30
failed_attempts	36	0.615	0.42
next_actions	32	0.565	0.34
recent_error	29	0.743	0.59
customer_identity	11	0.818	0.73
transaction_reference	5	0.467	0.20
product_name	4	1.000	1.00
discarded_options	2	0.375	0.00
invalidation_hints	2	0.750	0.50
non_promotable_context	2	0.333	0.00
payment_or_warranty_detail	2	0.350	0.00
assistant_claims_to_verify	1	0.500	0.00
explicit_decisions	1	1.000	1.00
resolved_context	1	0.250	0.00
touched_files	1	0.333	0.00
unsupported_hypotheses	1	0.333	0.00

Lineage

checkpoint	labels	overall MRR	top-1	notes
V2 (commit 87cb089)	conversational only	0.678	0.440	8-row adversarial eval, not comparable
mixed-mode stage-4	v1	0.323	0.130	pre-perf stack
mixed-mode perf-H4	v1	0.506	0.260	winning perf stack — but on role-tautological labels
mixed-mode v2 perf-H4 (this)	v2	0.526	0.296	same perf stack, semantic-reasoning labels

Intended use

Ranker head of a multi-stage episodic-ingestion compiler. Input is a JSON-serialized (requested field, candidate event) record. Output is a scalar logit that, after grouped softmax over siblings in the same trace, estimates the probability that the candidate event supports the requested field.

Limitations

• Aggregate MRR (0.526) is a blend across fields of very different difficulty. Legacy fields (attempted_actions, observed_outcomes) are still role-tautological on most rows; the model can hit ~0.48 on them without semantic reasoning. The signal of real learning is in the new fields:
  • outcome_of_latest_attempt: MRR 0.763 (2.76× random)
  • recent_error: MRR 0.743 (2.90× random)
  • initiating_command: MRR 0.667 (2.42× random)
• Not a standalone extractor. One head in a larger ingestion pipeline.

Related work

See docs/ranker-hypothesis-log-2026-05-08.md in the episodic-ingestion-compiler repo for the full experimental ladder.