README.md

---
license: apache-2.0
library_name: transformers
base_model: answerdotai/ModernBERT-base
tags:
  - modernbert
  - episodic-ingestion-compiler
  - grouped-softmax-ranker
  - field-event-ranker
  - mixed-mode-training
  - semantic-reasoning-labels
  - v2-labels
language:
  - en
---

# episodic-ingestion-compiler / ModernBERT field-event ranker (mixed-mode v2, perf-H4)

Fine-tune of `answerdotai/ModernBERT-base` for the **grouped multi-positive
softmax ranker** task with **v2 semantic-reasoning labels**.

## What changed vs the v1 mixed-mode checkpoint

An audit of the v1 mixed-mode training data showed that 94% of eval pairs came
from two fields (`attempted_actions`, `observed_outcomes`) whose labels were
**exact functions of event role** on 97% of rows (attempted_actions=tool_call,
observed_outcomes=tool_result). On these degenerate labels:

- E[MRR | random scores] ≈ 0.26 (random-permutation floor for multi-positive
  softmax)
- v1 perf-H4 MRR = 0.506 ≈ 1.97× random
- Architecture experiments (H3, H3a, H3b, H3b-span, etc.) could not break
  through because the ceiling was arithmetic, not informational.

v2 adds five semantic-reasoning fields whose labels require more than role:

| field | kind | rule |
|---|---|---|
| `initiating_command` | single-pick | First tool_call in a multi-call chain |
| `outcome_of_latest_attempt` | single-pick | Tool_result whose call_id matches the LAST tool_call |
| `recent_error` | single-pick | Most-recent tool_result with success=false |
| `attempt_outcome_pairs` | array | Paired tool_results (only when window has ≥1 orphan result — otherwise empty, so it can't collapse to role==tool_result) |
| `action_causality` | array | Tool_results whose content tokens recur in a LATER assistant turn |

Tautology rates on these new fields: 0–10%. Legacy fields kept for back-compat.

## Training details

- **Train rows**: 1217 (mixed-mode v2: autonomous + prompt→action + customer-support + conversational)
- **Train groups**: 5882 (one per field × row pair)
- **Eval rows**: 220 (1900 groups, 21986 candidate pairs)
- **Steps**: 160 optimizer steps (`--accum-steps 8`)
- **Optimizer**: AdamW, lr 5.7e-5, warmup 30
- **Precision**: bf16 autocast
- **Memory**: gradient checkpointing (use_reentrant=False)
- **Final train loss**: 0.0854
- **Peak VRAM**: 4.08 GiB
- **Hardware**: RTX 5090 Laptop (Blackwell SM 12.0, 24 GiB)

## Eval metrics (220-row mixed v2 eval, {'1': 0.2964509394572025, '2': 0.5173277661795407, '3': 0.6931106471816284, '5': 0.8881002087682672})

| metric | value |
|---|---:|
| overall MRR | 0.526 |
| top-1 recall | 0.296 |
| top-2 recall | 0.517 |
| top-3 recall | 0.693 |
| top-5 recall | 0.888 |
| mean expected rank | 3.09 |

Per-field MRR (sorted by eval-pair count):

| field | n | MRR | top-1 |
|---|---:|---:|---:|
| `attempted_actions` | 846 | 0.469 | 0.23 |
| `observed_outcomes` | 845 | 0.486 | 0.24 |
| `initiating_command` | 199 | 0.667 | 0.50 |
| `outcome_of_latest_attempt` | 181 | 0.763 | 0.60 |
| `action_causality` | 141 | 0.483 | 0.24 |
| `attempt_outcome_pairs` | 53 | 0.563 | 0.30 |
| `failed_attempts` | 36 | 0.615 | 0.42 |
| `next_actions` | 32 | 0.565 | 0.34 |
| `recent_error` | 29 | 0.743 | 0.59 |
| `customer_identity` | 11 | 0.818 | 0.73 |
| `transaction_reference` | 5 | 0.467 | 0.20 |
| `product_name` | 4 | 1.000 | 1.00 |
| `discarded_options` | 2 | 0.375 | 0.00 |
| `invalidation_hints` | 2 | 0.750 | 0.50 |
| `non_promotable_context` | 2 | 0.333 | 0.00 |
| `payment_or_warranty_detail` | 2 | 0.350 | 0.00 |
| `assistant_claims_to_verify` | 1 | 0.500 | 0.00 |
| `explicit_decisions` | 1 | 1.000 | 1.00 |
| `resolved_context` | 1 | 0.250 | 0.00 |
| `touched_files` | 1 | 0.333 | 0.00 |
| `unsupported_hypotheses` | 1 | 0.333 | 0.00 |

## Lineage

| checkpoint | labels | overall MRR | top-1 | notes |
|---|---|---:|---:|---|
| V2 (commit 87cb089) | conversational only | 0.678 | 0.440 | 8-row adversarial eval, not comparable |
| mixed-mode stage-4 | v1 | 0.323 | 0.130 | pre-perf stack |
| mixed-mode perf-H4 | v1 | 0.506 | 0.260 | winning perf stack — but on role-tautological labels |
| mixed-mode v2 perf-H4 (this) | v2 | 0.526 | 0.296 | same perf stack, semantic-reasoning labels |

## Intended use

Ranker head of a multi-stage episodic-ingestion compiler. Input is a
JSON-serialized (requested field, candidate event) record. Output is a
scalar logit that, after grouped softmax over siblings in the same trace,
estimates the probability that the candidate event supports the requested
field.

## Limitations

- Aggregate MRR (0.526) is a blend across fields of very different difficulty.
  Legacy fields (`attempted_actions`, `observed_outcomes`) are still
  role-tautological on most rows; the model can hit ~0.48 on them without
  semantic reasoning. The signal of real learning is in the new fields:
  - `outcome_of_latest_attempt`: MRR 0.763 (2.76× random)
  - `recent_error`: MRR 0.743 (2.90× random)
  - `initiating_command`: MRR 0.667 (2.42× random)
- Not a standalone extractor. One head in a larger ingestion pipeline.

## Related work

See `docs/ranker-hypothesis-log-2026-05-08.md` in the
episodic-ingestion-compiler repo for the full experimental ladder.