Upload mixed-mode v2 perf-H4 ranker (MRR 0.526, top-1 0.296)
Browse files- README.md +128 -0
- config.json +83 -0
- model.safetensors +3 -0
- tokenizer.json +0 -0
- tokenizer_config.json +17 -0
- training_metadata.json +1273 -0
README.md
ADDED
|
@@ -0,0 +1,128 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
---
|
| 2 |
+
license: apache-2.0
|
| 3 |
+
library_name: transformers
|
| 4 |
+
base_model: answerdotai/ModernBERT-base
|
| 5 |
+
tags:
|
| 6 |
+
- modernbert
|
| 7 |
+
- episodic-ingestion-compiler
|
| 8 |
+
- grouped-softmax-ranker
|
| 9 |
+
- field-event-ranker
|
| 10 |
+
- mixed-mode-training
|
| 11 |
+
- semantic-reasoning-labels
|
| 12 |
+
- v2-labels
|
| 13 |
+
language:
|
| 14 |
+
- en
|
| 15 |
+
---
|
| 16 |
+
|
| 17 |
+
# episodic-ingestion-compiler / ModernBERT field-event ranker (mixed-mode v2, perf-H4)
|
| 18 |
+
|
| 19 |
+
Fine-tune of `answerdotai/ModernBERT-base` for the **grouped multi-positive
|
| 20 |
+
softmax ranker** task with **v2 semantic-reasoning labels**.
|
| 21 |
+
|
| 22 |
+
## What changed vs the v1 mixed-mode checkpoint
|
| 23 |
+
|
| 24 |
+
An audit of the v1 mixed-mode training data showed that 94% of eval pairs came
|
| 25 |
+
from two fields (`attempted_actions`, `observed_outcomes`) whose labels were
|
| 26 |
+
**exact functions of event role** on 97% of rows (attempted_actions=tool_call,
|
| 27 |
+
observed_outcomes=tool_result). On these degenerate labels:
|
| 28 |
+
|
| 29 |
+
- E[MRR | random scores] ≈ 0.26 (random-permutation floor for multi-positive
|
| 30 |
+
softmax)
|
| 31 |
+
- v1 perf-H4 MRR = 0.506 ≈ 1.97× random
|
| 32 |
+
- Architecture experiments (H3, H3a, H3b, H3b-span, etc.) could not break
|
| 33 |
+
through because the ceiling was arithmetic, not informational.
|
| 34 |
+
|
| 35 |
+
v2 adds five semantic-reasoning fields whose labels require more than role:
|
| 36 |
+
|
| 37 |
+
| field | kind | rule |
|
| 38 |
+
|---|---|---|
|
| 39 |
+
| `initiating_command` | single-pick | First tool_call in a multi-call chain |
|
| 40 |
+
| `outcome_of_latest_attempt` | single-pick | Tool_result whose call_id matches the LAST tool_call |
|
| 41 |
+
| `recent_error` | single-pick | Most-recent tool_result with success=false |
|
| 42 |
+
| `attempt_outcome_pairs` | array | Paired tool_results (only when window has ≥1 orphan result — otherwise empty, so it can't collapse to role==tool_result) |
|
| 43 |
+
| `action_causality` | array | Tool_results whose content tokens recur in a LATER assistant turn |
|
| 44 |
+
|
| 45 |
+
Tautology rates on these new fields: 0–10%. Legacy fields kept for back-compat.
|
| 46 |
+
|
| 47 |
+
## Training details
|
| 48 |
+
|
| 49 |
+
- **Train rows**: 1217 (mixed-mode v2: autonomous + prompt→action + customer-support + conversational)
|
| 50 |
+
- **Train groups**: 5882 (one per field × row pair)
|
| 51 |
+
- **Eval rows**: 220 (1900 groups, 21986 candidate pairs)
|
| 52 |
+
- **Steps**: 160 optimizer steps (`--accum-steps 8`)
|
| 53 |
+
- **Optimizer**: AdamW, lr 5.7e-5, warmup 30
|
| 54 |
+
- **Precision**: bf16 autocast
|
| 55 |
+
- **Memory**: gradient checkpointing (use_reentrant=False)
|
| 56 |
+
- **Final train loss**: 0.0854
|
| 57 |
+
- **Peak VRAM**: 4.08 GiB
|
| 58 |
+
- **Hardware**: RTX 5090 Laptop (Blackwell SM 12.0, 24 GiB)
|
| 59 |
+
|
| 60 |
+
## Eval metrics (220-row mixed v2 eval, {'1': 0.2964509394572025, '2': 0.5173277661795407, '3': 0.6931106471816284, '5': 0.8881002087682672})
|
| 61 |
+
|
| 62 |
+
| metric | value |
|
| 63 |
+
|---|---:|
|
| 64 |
+
| overall MRR | 0.526 |
|
| 65 |
+
| top-1 recall | 0.296 |
|
| 66 |
+
| top-2 recall | 0.517 |
|
| 67 |
+
| top-3 recall | 0.693 |
|
| 68 |
+
| top-5 recall | 0.888 |
|
| 69 |
+
| mean expected rank | 3.09 |
|
| 70 |
+
|
| 71 |
+
Per-field MRR (sorted by eval-pair count):
|
| 72 |
+
|
| 73 |
+
| field | n | MRR | top-1 |
|
| 74 |
+
|---|---:|---:|---:|
|
| 75 |
+
| `attempted_actions` | 846 | 0.469 | 0.23 |
|
| 76 |
+
| `observed_outcomes` | 845 | 0.486 | 0.24 |
|
| 77 |
+
| `initiating_command` | 199 | 0.667 | 0.50 |
|
| 78 |
+
| `outcome_of_latest_attempt` | 181 | 0.763 | 0.60 |
|
| 79 |
+
| `action_causality` | 141 | 0.483 | 0.24 |
|
| 80 |
+
| `attempt_outcome_pairs` | 53 | 0.563 | 0.30 |
|
| 81 |
+
| `failed_attempts` | 36 | 0.615 | 0.42 |
|
| 82 |
+
| `next_actions` | 32 | 0.565 | 0.34 |
|
| 83 |
+
| `recent_error` | 29 | 0.743 | 0.59 |
|
| 84 |
+
| `customer_identity` | 11 | 0.818 | 0.73 |
|
| 85 |
+
| `transaction_reference` | 5 | 0.467 | 0.20 |
|
| 86 |
+
| `product_name` | 4 | 1.000 | 1.00 |
|
| 87 |
+
| `discarded_options` | 2 | 0.375 | 0.00 |
|
| 88 |
+
| `invalidation_hints` | 2 | 0.750 | 0.50 |
|
| 89 |
+
| `non_promotable_context` | 2 | 0.333 | 0.00 |
|
| 90 |
+
| `payment_or_warranty_detail` | 2 | 0.350 | 0.00 |
|
| 91 |
+
| `assistant_claims_to_verify` | 1 | 0.500 | 0.00 |
|
| 92 |
+
| `explicit_decisions` | 1 | 1.000 | 1.00 |
|
| 93 |
+
| `resolved_context` | 1 | 0.250 | 0.00 |
|
| 94 |
+
| `touched_files` | 1 | 0.333 | 0.00 |
|
| 95 |
+
| `unsupported_hypotheses` | 1 | 0.333 | 0.00 |
|
| 96 |
+
|
| 97 |
+
## Lineage
|
| 98 |
+
|
| 99 |
+
| checkpoint | labels | overall MRR | top-1 | notes |
|
| 100 |
+
|---|---|---:|---:|---|
|
| 101 |
+
| V2 (commit 87cb089) | conversational only | 0.678 | 0.440 | 8-row adversarial eval, not comparable |
|
| 102 |
+
| mixed-mode stage-4 | v1 | 0.323 | 0.130 | pre-perf stack |
|
| 103 |
+
| mixed-mode perf-H4 | v1 | 0.506 | 0.260 | winning perf stack — but on role-tautological labels |
|
| 104 |
+
| mixed-mode v2 perf-H4 (this) | v2 | 0.526 | 0.296 | same perf stack, semantic-reasoning labels |
|
| 105 |
+
|
| 106 |
+
## Intended use
|
| 107 |
+
|
| 108 |
+
Ranker head of a multi-stage episodic-ingestion compiler. Input is a
|
| 109 |
+
JSON-serialized (requested field, candidate event) record. Output is a
|
| 110 |
+
scalar logit that, after grouped softmax over siblings in the same trace,
|
| 111 |
+
estimates the probability that the candidate event supports the requested
|
| 112 |
+
field.
|
| 113 |
+
|
| 114 |
+
## Limitations
|
| 115 |
+
|
| 116 |
+
- Aggregate MRR (0.526) is a blend across fields of very different difficulty.
|
| 117 |
+
Legacy fields (`attempted_actions`, `observed_outcomes`) are still
|
| 118 |
+
role-tautological on most rows; the model can hit ~0.48 on them without
|
| 119 |
+
semantic reasoning. The signal of real learning is in the new fields:
|
| 120 |
+
- `outcome_of_latest_attempt`: MRR 0.763 (2.76× random)
|
| 121 |
+
- `recent_error`: MRR 0.743 (2.90× random)
|
| 122 |
+
- `initiating_command`: MRR 0.667 (2.42× random)
|
| 123 |
+
- Not a standalone extractor. One head in a larger ingestion pipeline.
|
| 124 |
+
|
| 125 |
+
## Related work
|
| 126 |
+
|
| 127 |
+
See `docs/ranker-hypothesis-log-2026-05-08.md` in the
|
| 128 |
+
episodic-ingestion-compiler repo for the full experimental ladder.
|
config.json
ADDED
|
@@ -0,0 +1,83 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
{
|
| 2 |
+
"architectures": [
|
| 3 |
+
"ModernBertForSequenceClassification"
|
| 4 |
+
],
|
| 5 |
+
"attention_bias": false,
|
| 6 |
+
"attention_dropout": 0.0,
|
| 7 |
+
"bos_token_id": 50281,
|
| 8 |
+
"classifier_activation": "gelu",
|
| 9 |
+
"classifier_bias": false,
|
| 10 |
+
"classifier_dropout": 0.0,
|
| 11 |
+
"classifier_pooling": "mean",
|
| 12 |
+
"cls_token_id": 50281,
|
| 13 |
+
"decoder_bias": true,
|
| 14 |
+
"deterministic_flash_attn": false,
|
| 15 |
+
"dtype": "float32",
|
| 16 |
+
"embedding_dropout": 0.0,
|
| 17 |
+
"eos_token_id": 50282,
|
| 18 |
+
"global_attn_every_n_layers": 3,
|
| 19 |
+
"gradient_checkpointing": false,
|
| 20 |
+
"hidden_activation": "gelu",
|
| 21 |
+
"hidden_size": 768,
|
| 22 |
+
"id2label": {
|
| 23 |
+
"0": "LABEL_0"
|
| 24 |
+
},
|
| 25 |
+
"initializer_cutoff_factor": 2.0,
|
| 26 |
+
"initializer_range": 0.02,
|
| 27 |
+
"intermediate_size": 1152,
|
| 28 |
+
"label2id": {
|
| 29 |
+
"LABEL_0": 0
|
| 30 |
+
},
|
| 31 |
+
"layer_norm_eps": 1e-05,
|
| 32 |
+
"layer_types": [
|
| 33 |
+
"full_attention",
|
| 34 |
+
"sliding_attention",
|
| 35 |
+
"sliding_attention",
|
| 36 |
+
"full_attention",
|
| 37 |
+
"sliding_attention",
|
| 38 |
+
"sliding_attention",
|
| 39 |
+
"full_attention",
|
| 40 |
+
"sliding_attention",
|
| 41 |
+
"sliding_attention",
|
| 42 |
+
"full_attention",
|
| 43 |
+
"sliding_attention",
|
| 44 |
+
"sliding_attention",
|
| 45 |
+
"full_attention",
|
| 46 |
+
"sliding_attention",
|
| 47 |
+
"sliding_attention",
|
| 48 |
+
"full_attention",
|
| 49 |
+
"sliding_attention",
|
| 50 |
+
"sliding_attention",
|
| 51 |
+
"full_attention",
|
| 52 |
+
"sliding_attention",
|
| 53 |
+
"sliding_attention",
|
| 54 |
+
"full_attention"
|
| 55 |
+
],
|
| 56 |
+
"local_attention": 128,
|
| 57 |
+
"max_position_embeddings": 8192,
|
| 58 |
+
"mlp_bias": false,
|
| 59 |
+
"mlp_dropout": 0.0,
|
| 60 |
+
"model_type": "modernbert",
|
| 61 |
+
"norm_bias": false,
|
| 62 |
+
"norm_eps": 1e-05,
|
| 63 |
+
"num_attention_heads": 12,
|
| 64 |
+
"num_hidden_layers": 22,
|
| 65 |
+
"pad_token_id": 50283,
|
| 66 |
+
"position_embedding_type": "absolute",
|
| 67 |
+
"rope_parameters": {
|
| 68 |
+
"full_attention": {
|
| 69 |
+
"rope_theta": 160000.0,
|
| 70 |
+
"rope_type": "default"
|
| 71 |
+
},
|
| 72 |
+
"sliding_attention": {
|
| 73 |
+
"rope_theta": 10000.0,
|
| 74 |
+
"rope_type": "default"
|
| 75 |
+
}
|
| 76 |
+
},
|
| 77 |
+
"sep_token_id": 50282,
|
| 78 |
+
"sparse_pred_ignore_index": -100,
|
| 79 |
+
"sparse_prediction": false,
|
| 80 |
+
"tie_word_embeddings": true,
|
| 81 |
+
"transformers_version": "5.8.0",
|
| 82 |
+
"vocab_size": 50368
|
| 83 |
+
}
|
model.safetensors
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:4f9218503ac695d0049a8732b23dce919fca534133a83fdfc9f7ab79d1eab811
|
| 3 |
+
size 598436708
|
tokenizer.json
ADDED
|
The diff for this file is too large to render.
See raw diff
|
|
|
tokenizer_config.json
ADDED
|
@@ -0,0 +1,17 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
{
|
| 2 |
+
"backend": "tokenizers",
|
| 3 |
+
"clean_up_tokenization_spaces": true,
|
| 4 |
+
"cls_token": "[CLS]",
|
| 5 |
+
"is_local": false,
|
| 6 |
+
"local_files_only": false,
|
| 7 |
+
"mask_token": "[MASK]",
|
| 8 |
+
"model_input_names": [
|
| 9 |
+
"input_ids",
|
| 10 |
+
"attention_mask"
|
| 11 |
+
],
|
| 12 |
+
"model_max_length": 8192,
|
| 13 |
+
"pad_token": "[PAD]",
|
| 14 |
+
"sep_token": "[SEP]",
|
| 15 |
+
"tokenizer_class": "TokenizersBackend",
|
| 16 |
+
"unk_token": "[UNK]"
|
| 17 |
+
}
|
training_metadata.json
ADDED
|
@@ -0,0 +1,1273 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
{
|
| 2 |
+
"device": "cuda",
|
| 3 |
+
"eval": "data/tmp/mixed_mode_eval.v2.field_decision.jsonl",
|
| 4 |
+
"eval_candidate_pairs": 21986,
|
| 5 |
+
"eval_groups": 1900,
|
| 6 |
+
"eval_rows": 220,
|
| 7 |
+
"last_loss": 0.08536958694458008,
|
| 8 |
+
"max_steps": 160,
|
| 9 |
+
"model": "answerdotai/ModernBERT-base",
|
| 10 |
+
"output_dir": "models/modernbert-field-event-ranker-mixed-v2-perf-H4",
|
| 11 |
+
"peak_allocated_gib": 4.076,
|
| 12 |
+
"peak_reserved_gib": 4.17,
|
| 13 |
+
"positive_train_pairs": 12854,
|
| 14 |
+
"skip_save_model": false,
|
| 15 |
+
"source_rank": {
|
| 16 |
+
"mean_expected_source_rank": 3.0918580375782883,
|
| 17 |
+
"mean_reciprocal_rank": 0.5255961949014605,
|
| 18 |
+
"median_expected_source_rank": 2,
|
| 19 |
+
"missing_prediction_pairs": 0,
|
| 20 |
+
"per_field": {
|
| 21 |
+
"action_causality": {
|
| 22 |
+
"mean_rank": 3.00709219858156,
|
| 23 |
+
"mean_reciprocal_rank": 0.48304626815265134,
|
| 24 |
+
"missing_prediction_pairs": 0,
|
| 25 |
+
"ranked_pairs": 141,
|
| 26 |
+
"source_pairs": 141,
|
| 27 |
+
"top_k_hits": {
|
| 28 |
+
"1": 34,
|
| 29 |
+
"2": 61,
|
| 30 |
+
"3": 92,
|
| 31 |
+
"5": 127
|
| 32 |
+
},
|
| 33 |
+
"top_k_recall": {
|
| 34 |
+
"1": 0.24113475177304963,
|
| 35 |
+
"2": 0.4326241134751773,
|
| 36 |
+
"3": 0.6524822695035462,
|
| 37 |
+
"5": 0.900709219858156
|
| 38 |
+
}
|
| 39 |
+
},
|
| 40 |
+
"assistant_claims_to_verify": {
|
| 41 |
+
"mean_rank": 2.0,
|
| 42 |
+
"mean_reciprocal_rank": 0.5,
|
| 43 |
+
"missing_prediction_pairs": 0,
|
| 44 |
+
"ranked_pairs": 1,
|
| 45 |
+
"source_pairs": 1,
|
| 46 |
+
"top_k_hits": {
|
| 47 |
+
"1": 0,
|
| 48 |
+
"2": 1,
|
| 49 |
+
"3": 1,
|
| 50 |
+
"5": 1
|
| 51 |
+
},
|
| 52 |
+
"top_k_recall": {
|
| 53 |
+
"1": 0.0,
|
| 54 |
+
"2": 1.0,
|
| 55 |
+
"3": 1.0,
|
| 56 |
+
"5": 1.0
|
| 57 |
+
}
|
| 58 |
+
},
|
| 59 |
+
"attempt_outcome_pairs": {
|
| 60 |
+
"mean_rank": 2.3773584905660377,
|
| 61 |
+
"mean_reciprocal_rank": 0.5625786163522012,
|
| 62 |
+
"missing_prediction_pairs": 0,
|
| 63 |
+
"ranked_pairs": 53,
|
| 64 |
+
"source_pairs": 53,
|
| 65 |
+
"top_k_hits": {
|
| 66 |
+
"1": 16,
|
| 67 |
+
"2": 31,
|
| 68 |
+
"3": 43,
|
| 69 |
+
"5": 52
|
| 70 |
+
},
|
| 71 |
+
"top_k_recall": {
|
| 72 |
+
"1": 0.3018867924528302,
|
| 73 |
+
"2": 0.5849056603773585,
|
| 74 |
+
"3": 0.8113207547169812,
|
| 75 |
+
"5": 0.9811320754716981
|
| 76 |
+
}
|
| 77 |
+
},
|
| 78 |
+
"attempted_actions": {
|
| 79 |
+
"mean_rank": 3.756501182033097,
|
| 80 |
+
"mean_reciprocal_rank": 0.4687886557994194,
|
| 81 |
+
"missing_prediction_pairs": 0,
|
| 82 |
+
"ranked_pairs": 846,
|
| 83 |
+
"source_pairs": 846,
|
| 84 |
+
"top_k_hits": {
|
| 85 |
+
"1": 196,
|
| 86 |
+
"2": 383,
|
| 87 |
+
"3": 531,
|
| 88 |
+
"5": 713
|
| 89 |
+
},
|
| 90 |
+
"top_k_recall": {
|
| 91 |
+
"1": 0.23167848699763594,
|
| 92 |
+
"2": 0.45271867612293143,
|
| 93 |
+
"3": 0.6276595744680851,
|
| 94 |
+
"5": 0.8427895981087471
|
| 95 |
+
}
|
| 96 |
+
},
|
| 97 |
+
"customer_identity": {
|
| 98 |
+
"mean_rank": 1.5454545454545454,
|
| 99 |
+
"mean_reciprocal_rank": 0.8181818181818182,
|
| 100 |
+
"missing_prediction_pairs": 0,
|
| 101 |
+
"ranked_pairs": 11,
|
| 102 |
+
"source_pairs": 11,
|
| 103 |
+
"top_k_hits": {
|
| 104 |
+
"1": 8,
|
| 105 |
+
"2": 8,
|
| 106 |
+
"3": 11,
|
| 107 |
+
"5": 11
|
| 108 |
+
},
|
| 109 |
+
"top_k_recall": {
|
| 110 |
+
"1": 0.7272727272727273,
|
| 111 |
+
"2": 0.7272727272727273,
|
| 112 |
+
"3": 1.0,
|
| 113 |
+
"5": 1.0
|
| 114 |
+
}
|
| 115 |
+
},
|
| 116 |
+
"discarded_options": {
|
| 117 |
+
"mean_rank": 3.0,
|
| 118 |
+
"mean_reciprocal_rank": 0.375,
|
| 119 |
+
"missing_prediction_pairs": 0,
|
| 120 |
+
"ranked_pairs": 2,
|
| 121 |
+
"source_pairs": 2,
|
| 122 |
+
"top_k_hits": {
|
| 123 |
+
"1": 0,
|
| 124 |
+
"2": 1,
|
| 125 |
+
"3": 1,
|
| 126 |
+
"5": 2
|
| 127 |
+
},
|
| 128 |
+
"top_k_recall": {
|
| 129 |
+
"1": 0.0,
|
| 130 |
+
"2": 0.5,
|
| 131 |
+
"3": 0.5,
|
| 132 |
+
"5": 1.0
|
| 133 |
+
}
|
| 134 |
+
},
|
| 135 |
+
"explicit_decisions": {
|
| 136 |
+
"mean_rank": 1.0,
|
| 137 |
+
"mean_reciprocal_rank": 1.0,
|
| 138 |
+
"missing_prediction_pairs": 0,
|
| 139 |
+
"ranked_pairs": 1,
|
| 140 |
+
"source_pairs": 1,
|
| 141 |
+
"top_k_hits": {
|
| 142 |
+
"1": 1,
|
| 143 |
+
"2": 1,
|
| 144 |
+
"3": 1,
|
| 145 |
+
"5": 1
|
| 146 |
+
},
|
| 147 |
+
"top_k_recall": {
|
| 148 |
+
"1": 1.0,
|
| 149 |
+
"2": 1.0,
|
| 150 |
+
"3": 1.0,
|
| 151 |
+
"5": 1.0
|
| 152 |
+
}
|
| 153 |
+
},
|
| 154 |
+
"failed_attempts": {
|
| 155 |
+
"mean_rank": 2.5,
|
| 156 |
+
"mean_reciprocal_rank": 0.6150793650793651,
|
| 157 |
+
"missing_prediction_pairs": 0,
|
| 158 |
+
"ranked_pairs": 36,
|
| 159 |
+
"source_pairs": 36,
|
| 160 |
+
"top_k_hits": {
|
| 161 |
+
"1": 15,
|
| 162 |
+
"2": 22,
|
| 163 |
+
"3": 28,
|
| 164 |
+
"5": 34
|
| 165 |
+
},
|
| 166 |
+
"top_k_recall": {
|
| 167 |
+
"1": 0.4166666666666667,
|
| 168 |
+
"2": 0.6111111111111112,
|
| 169 |
+
"3": 0.7777777777777778,
|
| 170 |
+
"5": 0.9444444444444444
|
| 171 |
+
}
|
| 172 |
+
},
|
| 173 |
+
"initiating_command": {
|
| 174 |
+
"mean_rank": 2.5728643216080402,
|
| 175 |
+
"mean_reciprocal_rank": 0.667125852929873,
|
| 176 |
+
"missing_prediction_pairs": 0,
|
| 177 |
+
"ranked_pairs": 199,
|
| 178 |
+
"source_pairs": 199,
|
| 179 |
+
"top_k_hits": {
|
| 180 |
+
"1": 99,
|
| 181 |
+
"2": 139,
|
| 182 |
+
"3": 160,
|
| 183 |
+
"5": 178
|
| 184 |
+
},
|
| 185 |
+
"top_k_recall": {
|
| 186 |
+
"1": 0.49748743718592964,
|
| 187 |
+
"2": 0.6984924623115578,
|
| 188 |
+
"3": 0.8040201005025126,
|
| 189 |
+
"5": 0.8944723618090452
|
| 190 |
+
}
|
| 191 |
+
},
|
| 192 |
+
"invalidation_hints": {
|
| 193 |
+
"mean_rank": 1.5,
|
| 194 |
+
"mean_reciprocal_rank": 0.75,
|
| 195 |
+
"missing_prediction_pairs": 0,
|
| 196 |
+
"ranked_pairs": 2,
|
| 197 |
+
"source_pairs": 2,
|
| 198 |
+
"top_k_hits": {
|
| 199 |
+
"1": 1,
|
| 200 |
+
"2": 2,
|
| 201 |
+
"3": 2,
|
| 202 |
+
"5": 2
|
| 203 |
+
},
|
| 204 |
+
"top_k_recall": {
|
| 205 |
+
"1": 0.5,
|
| 206 |
+
"2": 1.0,
|
| 207 |
+
"3": 1.0,
|
| 208 |
+
"5": 1.0
|
| 209 |
+
}
|
| 210 |
+
},
|
| 211 |
+
"next_actions": {
|
| 212 |
+
"mean_rank": 2.875,
|
| 213 |
+
"mean_reciprocal_rank": 0.5649440836940836,
|
| 214 |
+
"missing_prediction_pairs": 0,
|
| 215 |
+
"ranked_pairs": 32,
|
| 216 |
+
"source_pairs": 32,
|
| 217 |
+
"top_k_hits": {
|
| 218 |
+
"1": 11,
|
| 219 |
+
"2": 19,
|
| 220 |
+
"3": 23,
|
| 221 |
+
"5": 29
|
| 222 |
+
},
|
| 223 |
+
"top_k_recall": {
|
| 224 |
+
"1": 0.34375,
|
| 225 |
+
"2": 0.59375,
|
| 226 |
+
"3": 0.71875,
|
| 227 |
+
"5": 0.90625
|
| 228 |
+
}
|
| 229 |
+
},
|
| 230 |
+
"non_promotable_context": {
|
| 231 |
+
"mean_rank": 3.0,
|
| 232 |
+
"mean_reciprocal_rank": 0.3333333333333333,
|
| 233 |
+
"missing_prediction_pairs": 0,
|
| 234 |
+
"ranked_pairs": 2,
|
| 235 |
+
"source_pairs": 2,
|
| 236 |
+
"top_k_hits": {
|
| 237 |
+
"1": 0,
|
| 238 |
+
"2": 0,
|
| 239 |
+
"3": 2,
|
| 240 |
+
"5": 2
|
| 241 |
+
},
|
| 242 |
+
"top_k_recall": {
|
| 243 |
+
"1": 0.0,
|
| 244 |
+
"2": 0.0,
|
| 245 |
+
"3": 1.0,
|
| 246 |
+
"5": 1.0
|
| 247 |
+
}
|
| 248 |
+
},
|
| 249 |
+
"observed_outcomes": {
|
| 250 |
+
"mean_rank": 3.0272189349112426,
|
| 251 |
+
"mean_reciprocal_rank": 0.4855734009580153,
|
| 252 |
+
"missing_prediction_pairs": 0,
|
| 253 |
+
"ranked_pairs": 845,
|
| 254 |
+
"source_pairs": 845,
|
| 255 |
+
"top_k_hits": {
|
| 256 |
+
"1": 199,
|
| 257 |
+
"2": 393,
|
| 258 |
+
"3": 557,
|
| 259 |
+
"5": 754
|
| 260 |
+
},
|
| 261 |
+
"top_k_recall": {
|
| 262 |
+
"1": 0.23550295857988165,
|
| 263 |
+
"2": 0.4650887573964497,
|
| 264 |
+
"3": 0.659171597633136,
|
| 265 |
+
"5": 0.8923076923076924
|
| 266 |
+
}
|
| 267 |
+
},
|
| 268 |
+
"outcome_of_latest_attempt": {
|
| 269 |
+
"mean_rank": 1.6685082872928176,
|
| 270 |
+
"mean_reciprocal_rank": 0.7629439621152327,
|
| 271 |
+
"missing_prediction_pairs": 0,
|
| 272 |
+
"ranked_pairs": 181,
|
| 273 |
+
"source_pairs": 181,
|
| 274 |
+
"top_k_hits": {
|
| 275 |
+
"1": 108,
|
| 276 |
+
"2": 149,
|
| 277 |
+
"3": 171,
|
| 278 |
+
"5": 179
|
| 279 |
+
},
|
| 280 |
+
"top_k_recall": {
|
| 281 |
+
"1": 0.5966850828729282,
|
| 282 |
+
"2": 0.8232044198895028,
|
| 283 |
+
"3": 0.9447513812154696,
|
| 284 |
+
"5": 0.988950276243094
|
| 285 |
+
}
|
| 286 |
+
},
|
| 287 |
+
"payment_or_warranty_detail": {
|
| 288 |
+
"mean_rank": 3.5,
|
| 289 |
+
"mean_reciprocal_rank": 0.35,
|
| 290 |
+
"missing_prediction_pairs": 0,
|
| 291 |
+
"ranked_pairs": 2,
|
| 292 |
+
"source_pairs": 2,
|
| 293 |
+
"top_k_hits": {
|
| 294 |
+
"1": 0,
|
| 295 |
+
"2": 1,
|
| 296 |
+
"3": 1,
|
| 297 |
+
"5": 2
|
| 298 |
+
},
|
| 299 |
+
"top_k_recall": {
|
| 300 |
+
"1": 0.0,
|
| 301 |
+
"2": 0.5,
|
| 302 |
+
"3": 0.5,
|
| 303 |
+
"5": 1.0
|
| 304 |
+
}
|
| 305 |
+
},
|
| 306 |
+
"product_name": {
|
| 307 |
+
"mean_rank": 1.0,
|
| 308 |
+
"mean_reciprocal_rank": 1.0,
|
| 309 |
+
"missing_prediction_pairs": 0,
|
| 310 |
+
"ranked_pairs": 4,
|
| 311 |
+
"source_pairs": 4,
|
| 312 |
+
"top_k_hits": {
|
| 313 |
+
"1": 4,
|
| 314 |
+
"2": 4,
|
| 315 |
+
"3": 4,
|
| 316 |
+
"5": 4
|
| 317 |
+
},
|
| 318 |
+
"top_k_recall": {
|
| 319 |
+
"1": 1.0,
|
| 320 |
+
"2": 1.0,
|
| 321 |
+
"3": 1.0,
|
| 322 |
+
"5": 1.0
|
| 323 |
+
}
|
| 324 |
+
},
|
| 325 |
+
"recent_error": {
|
| 326 |
+
"mean_rank": 1.8620689655172413,
|
| 327 |
+
"mean_reciprocal_rank": 0.7425287356321838,
|
| 328 |
+
"missing_prediction_pairs": 0,
|
| 329 |
+
"ranked_pairs": 29,
|
| 330 |
+
"source_pairs": 29,
|
| 331 |
+
"top_k_hits": {
|
| 332 |
+
"1": 17,
|
| 333 |
+
"2": 23,
|
| 334 |
+
"3": 25,
|
| 335 |
+
"5": 28
|
| 336 |
+
},
|
| 337 |
+
"top_k_recall": {
|
| 338 |
+
"1": 0.5862068965517241,
|
| 339 |
+
"2": 0.7931034482758621,
|
| 340 |
+
"3": 0.8620689655172413,
|
| 341 |
+
"5": 0.9655172413793104
|
| 342 |
+
}
|
| 343 |
+
},
|
| 344 |
+
"resolved_context": {
|
| 345 |
+
"mean_rank": 4.0,
|
| 346 |
+
"mean_reciprocal_rank": 0.25,
|
| 347 |
+
"missing_prediction_pairs": 0,
|
| 348 |
+
"ranked_pairs": 1,
|
| 349 |
+
"source_pairs": 1,
|
| 350 |
+
"top_k_hits": {
|
| 351 |
+
"1": 0,
|
| 352 |
+
"2": 0,
|
| 353 |
+
"3": 0,
|
| 354 |
+
"5": 1
|
| 355 |
+
},
|
| 356 |
+
"top_k_recall": {
|
| 357 |
+
"1": 0.0,
|
| 358 |
+
"2": 0.0,
|
| 359 |
+
"3": 0.0,
|
| 360 |
+
"5": 1.0
|
| 361 |
+
}
|
| 362 |
+
},
|
| 363 |
+
"touched_files": {
|
| 364 |
+
"mean_rank": 3.0,
|
| 365 |
+
"mean_reciprocal_rank": 0.3333333333333333,
|
| 366 |
+
"missing_prediction_pairs": 0,
|
| 367 |
+
"ranked_pairs": 1,
|
| 368 |
+
"source_pairs": 1,
|
| 369 |
+
"top_k_hits": {
|
| 370 |
+
"1": 0,
|
| 371 |
+
"2": 0,
|
| 372 |
+
"3": 1,
|
| 373 |
+
"5": 1
|
| 374 |
+
},
|
| 375 |
+
"top_k_recall": {
|
| 376 |
+
"1": 0.0,
|
| 377 |
+
"2": 0.0,
|
| 378 |
+
"3": 1.0,
|
| 379 |
+
"5": 1.0
|
| 380 |
+
}
|
| 381 |
+
},
|
| 382 |
+
"transaction_reference": {
|
| 383 |
+
"mean_rank": 2.6,
|
| 384 |
+
"mean_reciprocal_rank": 0.4666666666666666,
|
| 385 |
+
"missing_prediction_pairs": 0,
|
| 386 |
+
"ranked_pairs": 5,
|
| 387 |
+
"source_pairs": 5,
|
| 388 |
+
"top_k_hits": {
|
| 389 |
+
"1": 1,
|
| 390 |
+
"2": 1,
|
| 391 |
+
"3": 5,
|
| 392 |
+
"5": 5
|
| 393 |
+
},
|
| 394 |
+
"top_k_recall": {
|
| 395 |
+
"1": 0.2,
|
| 396 |
+
"2": 0.2,
|
| 397 |
+
"3": 1.0,
|
| 398 |
+
"5": 1.0
|
| 399 |
+
}
|
| 400 |
+
},
|
| 401 |
+
"unsupported_hypotheses": {
|
| 402 |
+
"mean_rank": 3.0,
|
| 403 |
+
"mean_reciprocal_rank": 0.3333333333333333,
|
| 404 |
+
"missing_prediction_pairs": 0,
|
| 405 |
+
"ranked_pairs": 1,
|
| 406 |
+
"source_pairs": 1,
|
| 407 |
+
"top_k_hits": {
|
| 408 |
+
"1": 0,
|
| 409 |
+
"2": 0,
|
| 410 |
+
"3": 1,
|
| 411 |
+
"5": 1
|
| 412 |
+
},
|
| 413 |
+
"top_k_recall": {
|
| 414 |
+
"1": 0.0,
|
| 415 |
+
"2": 0.0,
|
| 416 |
+
"3": 1.0,
|
| 417 |
+
"5": 1.0
|
| 418 |
+
}
|
| 419 |
+
}
|
| 420 |
+
},
|
| 421 |
+
"rows": 220,
|
| 422 |
+
"selected_fields": 997,
|
| 423 |
+
"source_pairs": 2395,
|
| 424 |
+
"top_k_hits": {
|
| 425 |
+
"1": 710,
|
| 426 |
+
"2": 1239,
|
| 427 |
+
"3": 1660,
|
| 428 |
+
"5": 2127
|
| 429 |
+
},
|
| 430 |
+
"top_k_source_pair_recall": {
|
| 431 |
+
"1": 0.2964509394572025,
|
| 432 |
+
"2": 0.5173277661795407,
|
| 433 |
+
"3": 0.6931106471816284,
|
| 434 |
+
"5": 0.8881002087682672
|
| 435 |
+
},
|
| 436 |
+
"worst_expected_source_ranks": [
|
| 437 |
+
{
|
| 438 |
+
"expected_event": {
|
| 439 |
+
"command": "python - <<'PY'\nimport json\nfrom pathlib import Path\nfor model in ['qwen3.6-35b-a3b-nvfp4','qwen3.6-27b-text-nvfp4-mtp']:\n p=Path('/home/avifenesh/projects/aviary/models')/model\n print('MODEL', p)\n for f in ['config.json','quantization_config.json','quantize_config.json','generation_config.json']:\n q=p/f\n if q.exists():\n data=json.loads(q.read_text())\n print(f, {k:data.get(k) for k in ['model_type','architectures','quantization_config','kv_lora_rank','num_hidden_layers','num_attention_heads','num_key_value_heads','max_position_embeddings','sliding_window','use_mtp','num_nextn_predict_layers'] if k in data})\nPY",
|
| 440 |
+
"is_context_tool_echo": false,
|
| 441 |
+
"role": "tool_call",
|
| 442 |
+
"success": null,
|
| 443 |
+
"text": "python - <<'PY'\nimport json\nfrom pathlib import Path\nfor model in ['qwen3.6-35b-a3b-nvfp4','qwen3.6-27b-text-nvfp4-mtp']:\n p=Path('/home/avifenesh/projects/aviary/models')/model...",
|
| 444 |
+
"tool_name": "exec_command"
|
| 445 |
+
},
|
| 446 |
+
"expected_probability": 0.28537195920944214,
|
| 447 |
+
"expected_source_event_id": "t6",
|
| 448 |
+
"field": "attempted_actions",
|
| 449 |
+
"rank": 20,
|
| 450 |
+
"reciprocal_rank": 0.05,
|
| 451 |
+
"row_id": "pta-codex-rollout-2026-04-30T19-43-19-019ddf45-fdda-7fb2-8764-8162ba11dc24::w14",
|
| 452 |
+
"top_candidates": [
|
| 453 |
+
{
|
| 454 |
+
"event": {
|
| 455 |
+
"command": "{\"session_id\": 92479, \"chars\": \"\\u0003\", \"yield_time_ms\": 1000, \"max_output_tokens\": 20000}",
|
| 456 |
+
"is_context_tool_echo": false,
|
| 457 |
+
"role": "tool_call",
|
| 458 |
+
"success": null,
|
| 459 |
+
"text": "{\"session_id\": 92479, \"chars\": \"\\u0003\", \"yield_time_ms\": 1000, \"max_output_tokens\": 20000}",
|
| 460 |
+
"tool_name": "write_stdin"
|
| 461 |
+
},
|
| 462 |
+
"label": 1,
|
| 463 |
+
"source_event_id": "t3",
|
| 464 |
+
"support_probability": 0.9996079802513123
|
| 465 |
+
},
|
| 466 |
+
{
|
| 467 |
+
"event": {
|
| 468 |
+
"command": "{\"session_id\": 1703, \"chars\": \"\", \"yield_time_ms\": 1000, \"max_output_tokens\": 30000}",
|
| 469 |
+
"is_context_tool_echo": false,
|
| 470 |
+
"role": "tool_call",
|
| 471 |
+
"success": null,
|
| 472 |
+
"text": "{\"session_id\": 1703, \"chars\": \"\", \"yield_time_ms\": 1000, \"max_output_tokens\": 30000}",
|
| 473 |
+
"tool_name": "write_stdin"
|
| 474 |
+
},
|
| 475 |
+
"label": 1,
|
| 476 |
+
"source_event_id": "t12",
|
| 477 |
+
"support_probability": 0.9992678761482239
|
| 478 |
+
},
|
| 479 |
+
{
|
| 480 |
+
"event": {
|
| 481 |
+
"command": "{\"session_id\": 1703, \"chars\": \"\", \"yield_time_ms\": 1000, \"max_output_tokens\": 6000}",
|
| 482 |
+
"is_context_tool_echo": false,
|
| 483 |
+
"role": "tool_call",
|
| 484 |
+
"success": null,
|
| 485 |
+
"text": "{\"session_id\": 1703, \"chars\": \"\", \"yield_time_ms\": 1000, \"max_output_tokens\": 6000}",
|
| 486 |
+
"tool_name": "write_stdin"
|
| 487 |
+
},
|
| 488 |
+
"label": 1,
|
| 489 |
+
"source_event_id": "t17",
|
| 490 |
+
"support_probability": 0.9992678761482239
|
| 491 |
+
},
|
| 492 |
+
{
|
| 493 |
+
"event": {
|
| 494 |
+
"command": "nvidia-smi --query-gpu=memory.used,memory.free,utilization.gpu --format=csv,noheader,nounits && free -h && ss -ltnp sport = :8000",
|
| 495 |
+
"is_context_tool_echo": false,
|
| 496 |
+
"role": "tool_call",
|
| 497 |
+
"success": null,
|
| 498 |
+
"text": "nvidia-smi --query-gpu=memory.used,memory.free,utilization.gpu --format=csv,noheader,nounits && free -h && ss -ltnp sport = :8000",
|
| 499 |
+
"tool_name": "exec_command"
|
| 500 |
+
},
|
| 501 |
+
"label": 1,
|
| 502 |
+
"source_event_id": "t13",
|
| 503 |
+
"support_probability": 0.9985895752906799
|
| 504 |
+
},
|
| 505 |
+
{
|
| 506 |
+
"event": {
|
| 507 |
+
"command": "nvidia-smi --query-gpu=name,memory.used,memory.free,utilization.gpu --format=csv,noheader,nounits && free -h && ss -ltnp 'sport = :8000' || true",
|
| 508 |
+
"is_context_tool_echo": false,
|
| 509 |
+
"role": "tool_call",
|
| 510 |
+
"success": null,
|
| 511 |
+
"text": "nvidia-smi --query-gpu=name,memory.used,memory.free,utilization.gpu --format=csv,noheader,nounits && free -h && ss -ltnp 'sport = :8000' || true",
|
| 512 |
+
"tool_name": "exec_command"
|
| 513 |
+
},
|
| 514 |
+
"label": 1,
|
| 515 |
+
"source_event_id": "t18",
|
| 516 |
+
"support_probability": 0.998245358467102
|
| 517 |
+
}
|
| 518 |
+
]
|
| 519 |
+
},
|
| 520 |
+
{
|
| 521 |
+
"expected_event": {
|
| 522 |
+
"command": "python3 - <<'PY'\nfrom pathlib import Path\nimport yaml\nroot=Path('.')\ncanonical=set()\nfor p in Path('data/evals').glob('*/scorecards.yml'):\n d=yaml.safe_load(p.read_text()) or {}\n for c in d.get('scorecards') or []:\n if isinstance(c,dict) and c.get('result_path'):\n canonical.add(str(Path(c['result_path']).resolve()))\nprint('canonical result paths',len(canonical))\nnotcanon=[]; incanon=[]\nfor p in Path('data/runs').rglob('judge-scorecard-*.yml'):\n d=yaml.safe_load(p.read_text()) or {}\n rp=d.get('result_path')\n if not rp:\n # common default result.md in same dir\n cand=p.parent/'result.md'\n if cand.exists(): rp=str(cand)\n key=str(Path(rp).resolve()) if rp else ''\n (incanon if key in canonical else notcanon).append((p,key))\nprint('judge ",
|
| 523 |
+
"is_context_tool_echo": false,
|
| 524 |
+
"role": "tool_call",
|
| 525 |
+
"success": null,
|
| 526 |
+
"text": "python3 - <<'PY'\nfrom pathlib import Path\nimport yaml\nroot=Path('.')\ncanonical=set()\nfor p in Path('data/evals').glob('*/scorecards.yml'):\n d=yaml.safe_load(p.read_text()) or...",
|
| 527 |
+
"tool_name": "exec_command"
|
| 528 |
+
},
|
| 529 |
+
"expected_probability": 0.4863315224647522,
|
| 530 |
+
"expected_source_event_id": "t17",
|
| 531 |
+
"field": "attempted_actions",
|
| 532 |
+
"rank": 20,
|
| 533 |
+
"reciprocal_rank": 0.05,
|
| 534 |
+
"row_id": "pta-codex-rollout-2026-04-30T19-43-19-019ddf45-fdda-7fb2-8764-8162ba11dc24::w48",
|
| 535 |
+
"top_candidates": [
|
| 536 |
+
{
|
| 537 |
+
"event": {
|
| 538 |
+
"command": "find data/runs -name 'codex-scorecards-draft.yml' -print | wc -l && find data/runs -name 'codex-scorecards-draft.yml' -print",
|
| 539 |
+
"is_context_tool_echo": false,
|
| 540 |
+
"role": "tool_call",
|
| 541 |
+
"success": null,
|
| 542 |
+
"text": "find data/runs -name 'codex-scorecards-draft.yml' -print | wc -l && find data/runs -name 'codex-scorecards-draft.yml' -print",
|
| 543 |
+
"tool_name": "exec_command"
|
| 544 |
+
},
|
| 545 |
+
"label": 1,
|
| 546 |
+
"source_event_id": "t4",
|
| 547 |
+
"support_probability": 0.999783456325531
|
| 548 |
+
},
|
| 549 |
+
{
|
| 550 |
+
"event": {
|
| 551 |
+
"command": "sed -n '1,240p' data/evals/judge-review-queues/20260504-first10.txt",
|
| 552 |
+
"is_context_tool_echo": false,
|
| 553 |
+
"role": "tool_call",
|
| 554 |
+
"success": null,
|
| 555 |
+
"text": "sed -n '1,240p' data/evals/judge-review-queues/20260504-first10.txt",
|
| 556 |
+
"tool_name": "exec_command"
|
| 557 |
+
},
|
| 558 |
+
"label": 1,
|
| 559 |
+
"source_event_id": "t12",
|
| 560 |
+
"support_probability": 0.9996646642684937
|
| 561 |
+
},
|
| 562 |
+
{
|
| 563 |
+
"event": {
|
| 564 |
+
"command": "find data/evals/judge-review-queues -type f -maxdepth 4 -print 2>/dev/null | sort && printf '\\nreview markers\\n' && rg -n \"review_status|avi_corrected|pending|draft|accepted|rejected\" data/runs data/evals/judge-review-queues -g 'codex-scorecards-draft.yml' -g '*.yml'",
|
| 565 |
+
"is_context_tool_echo": false,
|
| 566 |
+
"role": "tool_call",
|
| 567 |
+
"success": null,
|
| 568 |
+
"text": "find data/evals/judge-review-queues -type f -maxdepth 4 -print 2>/dev/null | sort && printf '\\nreview markers\\n' && rg -n \"review_status|avi_corrected|pending|draft|accepted|rej...",
|
| 569 |
+
"tool_name": "exec_command"
|
| 570 |
+
},
|
| 571 |
+
"label": 1,
|
| 572 |
+
"source_event_id": "t5",
|
| 573 |
+
"support_probability": 0.998245358467102
|
| 574 |
+
},
|
| 575 |
+
{
|
| 576 |
+
"event": {
|
| 577 |
+
"command": "python3 - <<'PY'\nfrom pathlib import Path\nimport yaml\nfor p in sorted(Path('data/runs').rglob('codex-scorecards-draft.yml')):\n d=yaml.safe_load(p.read_text()) or {}\n cards=d.get('scorecards') or []\n print(f'{p}\\tfile_status={d.get(\"review_status\")}\\tcards={len(cards)}\\tcard_statuses={sorted({str(c.get(\"review_status\", d.get(\"review_status\", \"\"))) for c in cards if isinstance(c,dict)})}')\nPY",
|
| 578 |
+
"is_context_tool_echo": false,
|
| 579 |
+
"role": "tool_call",
|
| 580 |
+
"success": null,
|
| 581 |
+
"text": "python3 - <<'PY'\nfrom pathlib import Path\nimport yaml\nfor p in sorted(Path('data/runs').rglob('codex-scorecards-draft.yml')):\n d=yaml.safe_load(p.read_text()) or {}\n cards...",
|
| 582 |
+
"tool_name": "exec_command"
|
| 583 |
+
},
|
| 584 |
+
"label": 1,
|
| 585 |
+
"source_event_id": "t11",
|
| 586 |
+
"support_probability": 0.9921841025352478
|
| 587 |
+
},
|
| 588 |
+
{
|
| 589 |
+
"event": {
|
| 590 |
+
"command": null,
|
| 591 |
+
"is_context_tool_echo": false,
|
| 592 |
+
"role": "tool_result",
|
| 593 |
+
"success": true,
|
| 594 |
+
"text": "Chunk ID: c00386\nWall time: 0.0000 seconds\nProcess exited with code 0\nOriginal token count: 499\nOutput:\ncanonical result paths 46\njudge draft files already canonical by result_p...",
|
| 595 |
+
"tool_name": null
|
| 596 |
+
},
|
| 597 |
+
"label": 0,
|
| 598 |
+
"source_event_id": "r19",
|
| 599 |
+
"support_probability": 0.9759464263916016
|
| 600 |
+
}
|
| 601 |
+
]
|
| 602 |
+
},
|
| 603 |
+
{
|
| 604 |
+
"expected_event": {
|
| 605 |
+
"command": "*** Begin Patch\n*** Update File: /home/avifenesh/projects/aviary/data/evals/judge-review-status.yml\n@@\n - result_path: /home/avifenesh/projects/aviary/data/runs/session_summarize_for_avi/20260504-memory-gym-variety/qwen36-27b-autoround-64k/round-13-agent-sh-multirepo-maintenance/result.md\n- status: pending_avi_judge_review\n- note: Recent judge-draft batch; keep waiting for Avi unless verbal correction already exists.\n+ status: reviewed_with_known_owner_correction\n+ reviewed_at: '2026-05-04'\n+ note: Avi correction is score 2; the judges overranked a mistaken, failed, not-helpful session as positive.\n - result_path: /home/avifenesh/projects/aviary/data/runs/session_summarize_for_avi/20260504-memory-gym-variety/qwen36-27b-autoround-64k/round-14-agnix-next-task-workflow-policy/result.md\n- status: pending_avi_judge_review\n- note: Recent judge-draft batch; keep waiting for Avi unless verbal correction already exists.\n+ status: reviewed_with_known_owner_correction\n+ reviewed_at: '2026-05-04'\n+ note: Avi correction is score 3; not here, not there.\n@@\n - result_path: /home/avifenesh/projects/aviary/data/runs/session_summarize_for_avi/20260504-memory-gym-variety/qwen36-27b-autoround-64k/round-16-session-pairs-signal-sample/result.md\n- status: pending_avi_judge_review\n- note: Recent judge-draft batch; keep waiting for Avi unless verbal correction already exists.\n+ status: reviewed_excluded\n+ reviewed_at: '2026-05-04'\n+ note: Avi correction is exclude if leaked/contaminated; do not train on this example.\n*** End Patch\n",
|
| 606 |
+
"is_context_tool_echo": false,
|
| 607 |
+
"role": "tool_call",
|
| 608 |
+
"success": null,
|
| 609 |
+
"text": "apply_patch: *** Begin Patch\n*** Update File: /home/avifenesh/projects/aviary/data/evals/judge-review-status.yml\n@@\n - result_path: /home/avifenesh/projects/aviary/data/runs/ses...",
|
| 610 |
+
"tool_name": "apply_patch"
|
| 611 |
+
},
|
| 612 |
+
"expected_probability": 0.5248817801475525,
|
| 613 |
+
"expected_source_event_id": "t9",
|
| 614 |
+
"field": "attempted_actions",
|
| 615 |
+
"rank": 20,
|
| 616 |
+
"reciprocal_rank": 0.05,
|
| 617 |
+
"row_id": "pta-codex-rollout-2026-04-30T19-43-19-019ddf45-fdda-7fb2-8764-8162ba11dc24::w53",
|
| 618 |
+
"top_candidates": [
|
| 619 |
+
{
|
| 620 |
+
"event": {
|
| 621 |
+
"command": "python3 scripts/judge-review-queue.py --write data/evals/judge-review-queues/current-pending.txt",
|
| 622 |
+
"is_context_tool_echo": false,
|
| 623 |
+
"role": "tool_call",
|
| 624 |
+
"success": null,
|
| 625 |
+
"text": "python3 scripts/judge-review-queue.py --write data/evals/judge-review-queues/current-pending.txt",
|
| 626 |
+
"tool_name": "exec_command"
|
| 627 |
+
},
|
| 628 |
+
"label": 1,
|
| 629 |
+
"source_event_id": "t12",
|
| 630 |
+
"support_probability": 0.9999039173126221
|
| 631 |
+
},
|
| 632 |
+
{
|
| 633 |
+
"event": {
|
| 634 |
+
"command": "python3 -m py_compile scripts/judge-review-queue.py scripts/review-judge-drafts.py scripts/build-scorecard-sft-dataset.py && cargo test eval_score --quiet",
|
| 635 |
+
"is_context_tool_echo": false,
|
| 636 |
+
"role": "tool_call",
|
| 637 |
+
"success": null,
|
| 638 |
+
"text": "python3 -m py_compile scripts/judge-review-queue.py scripts/review-judge-drafts.py scripts/build-scorecard-sft-dataset.py && cargo test eval_score --quiet",
|
| 639 |
+
"tool_name": "exec_command"
|
| 640 |
+
},
|
| 641 |
+
"label": 1,
|
| 642 |
+
"source_event_id": "t14",
|
| 643 |
+
"support_probability": 0.9998910427093506
|
| 644 |
+
},
|
| 645 |
+
{
|
| 646 |
+
"event": {
|
| 647 |
+
"command": "nl -ba data/evals/session_summarize_for_avi/scorecards.yml | sed -n '1020,1235p'",
|
| 648 |
+
"is_context_tool_echo": false,
|
| 649 |
+
"role": "tool_call",
|
| 650 |
+
"success": null,
|
| 651 |
+
"text": "nl -ba data/evals/session_summarize_for_avi/scorecards.yml | sed -n '1020,1235p'",
|
| 652 |
+
"tool_name": "exec_command"
|
| 653 |
+
},
|
| 654 |
+
"label": 1,
|
| 655 |
+
"source_event_id": "t3",
|
| 656 |
+
"support_probability": 0.9996540546417236
|
| 657 |
+
},
|
| 658 |
+
{
|
| 659 |
+
"event": {
|
| 660 |
+
"command": "nl -ba data/evals/session_summarize_for_avi/scorecards.yml | sed -n '1230,1265p' && nl -ba data/evals/judge-review-status.yml | sed -n '70,140p'",
|
| 661 |
+
"is_context_tool_echo": false,
|
| 662 |
+
"role": "tool_call",
|
| 663 |
+
"success": null,
|
| 664 |
+
"text": "nl -ba data/evals/session_summarize_for_avi/scorecards.yml | sed -n '1230,1265p' && nl -ba data/evals/judge-review-status.yml | sed -n '70,140p'",
|
| 665 |
+
"tool_name": "exec_command"
|
| 666 |
+
},
|
| 667 |
+
"label": 1,
|
| 668 |
+
"source_event_id": "t5",
|
| 669 |
+
"support_probability": 0.998793363571167
|
| 670 |
+
},
|
| 671 |
+
{
|
| 672 |
+
"event": {
|
| 673 |
+
"command": "python3 scripts/build-scorecard-sft-dataset.py --lane session_summarize_for_avi --out data/training/session_summarize_for_avi/scorecard_sft_avi_verified.jsonl && python3 scripts/build-scorecard-sft-dataset.py --lane session_summarize_for_avi --out data/training/session_summarize_for_avi/scorecard_sft_with_drafts.jsonl --include-drafts",
|
| 674 |
+
"is_context_tool_echo": false,
|
| 675 |
+
"role": "tool_call",
|
| 676 |
+
"success": null,
|
| 677 |
+
"text": "python3 scripts/build-scorecard-sft-dataset.py --lane session_summarize_for_avi --out data/training/session_summarize_for_avi/scorecard_sft_avi_verified.jsonl && python3 scripts...",
|
| 678 |
+
"tool_name": "exec_command"
|
| 679 |
+
},
|
| 680 |
+
"label": 1,
|
| 681 |
+
"source_event_id": "t13",
|
| 682 |
+
"support_probability": 0.9893474578857422
|
| 683 |
+
}
|
| 684 |
+
]
|
| 685 |
+
},
|
| 686 |
+
{
|
| 687 |
+
"expected_event": {
|
| 688 |
+
"command": "*** Begin Patch\n*** Add File: /home/avifenesh/projects/episodic-ingestion-compiler/data/examples/seed.jsonl\n+{\"id\":\"seed-0001-snake-tdd-validation-gap\",\"input\":{\"source_kind\":\"agent_trace\",\"source_uri\":\"pi://local/session/snake-tdd\",\"raw\":{\"source_events\":[{\"id\":\"u1\",\"role\":\"user\",\"timestamp\":\"2026-05-02T15:40:00Z\",\"text\":\"build a snake game that run on my machine locally without using browser, write the tests upfront\"},{\"id\":\"a1\",\"role\":\"assistant\",\"timestamp\":\"2026-05-02T15:41:00Z\",\"text\":\"I will write tests first, then implement the Snake logic and run cargo test.\"},{\"id\":\"t1\",\"role\":\"tool_call\",\"timestamp\":\"2026-05-02T15:42:00Z\",\"tool_id\":\"write_file\",\"file_paths\":[\"snake_game/test_snake.py\"],\"text\":\"create test suite for movement, food, wall collision, self collision\"},{\"id\":\"t2\",\"role\":\"tool_call\",\"timestamp\":\"2026-05-02T15:43:00Z\",\"tool_id\":\"run_command\",\"command_id\":\"cmd-cargo-test-1\",\"text\":\"cargo test\"},{\"id\":\"r2\",\"role\":\"tool_result\",\"timestamp\":\"2026-05-02T15:43:20Z\",\"tool_id\":\"run_command\",\"command_id\":\"cmd-cargo-test-1\",\"text\":\"tests passed\"},{\"id\":\"a2\",\"role\":\"assistant\",\"timestamp\":\"2026-05-02T15:44:00Z\",\"text\":\"The implementation is complete and tests pass.\"},{\"id\":\"u2\",\"role\":\"user\",\"timestamp\":\"2026-05-02T15:45:00Z\",\"text\":\"did you check also that the game work or just the tests?\"},{\"id\":\"r3\",\"role\":\"tool_result\",\"timestamp\":\"2026-05-02T15:46:00Z\",\"tool_id\":\"run_command\",\"command_id\":\"cmd-run-game-1\",\"text\":\"runtime smoke failed: game window starts but input does not begin movement\"}]}},\"expected\":{\"episode_hint\":\"snake-local-game-tdd\",\"objective_hint\":\"Build a local non-browser Snake game with tests written before implementation and verify both tests and playable runtime behavior.\",\"events\":[{\"event_id\":\"e1\",\"event_type\":\"user_request\",\"summary\":\"User requested a local non-browser Snake game and explicitly required tests upfront.\",\"provenance\":{\"source_event_ids\":[\"u1\"],\"timestamps\":[\"2026-05-02T15:40:00Z\"]},\"confidence\":0.98},{\"event_id\":\"e2\",\"event_type\":\"plan\",\"summary\":\"Assistant planned to write tests first, implement logic, and run cargo test.\",\"provenance\":{\"source_event_ids\":[\"a1\"],\"timestamps\":[\"2026-05-02T15:41:00Z\"]},\"confidence\":0.9},{\"event_id\":\"e3\",\"event_type\":\"file_change\",\"summary\":\"A test suite was created for movement, food, wall collision, and self collision.\",\"provenance\":{\"source_event_ids\":[\"t1\"],\"timestamps\":[\"2026-05-02T15:42:00Z\"],\"tool_ids\":[\"write_file\"],\"file_paths\":[\"snake_game/test_snake.py\"]},\"confidence\":0.92},{\"event_id\":\"e4\",\"event_type\":\"validation\",\"summary\":\"The unit test command passed.\",\"provenance\":{\"source_event_ids\":[\"t2\",\"r2\"],\"timestamps\":[\"2026-05-02T15:43:00Z\",\"2026-05-02T15:43:20Z\"],\"tool_ids\":[\"run_command\"],\"command_ids\":[\"cmd-cargo-test-1\"]},\"confidence\":0.95},{\"event_id\":\"e5\",\"event_type\":\"blocker\",\"summary\":\"A later runtime smoke showed the game did not actually start movement correctly despite passing tests.\",\"provenance\":{\"source_event_ids\":[\"u2\",\"r3\"],\"timestamps\":[\"2026-05-02T15:45:00Z\",\"2026-05-02T15:46:00Z\"],\"tool_ids\":[\"run_command\"],\"command_ids\":[\"cmd-run-game-1\"]},\"confidence\":0.88}],\"candidate_claims\":[{\"claim_id\":\"c1\",\"kind\":\"decided\",\"text\":\"The task acceptance requires runtime/playability validation, not only unit tests.\",\"support_event_ids\":[\"e1\",\"e5\"],\"evidence\":[{\"source_event_id\":\"u1\",\"quote\":\"build a snake game that run on my machine locally without using browser\"},{\"source_event_id\":\"u2\",\"quote\":\"did you check also that the game work or just the tests?\"}],\"truth_scope\":\"session_context\",\"promotion_hint\":\"strong_candidate\",\"confidence\":0.94},{\"claim_id\":\"c2\",\"kind\":\"validation_result\",\"text\":\"Unit tests passed, but the first runtime smoke failed to confirm playable behavior.\",\"support_event_ids\":[\"e4\",\"e5\"],\"evidence\":[{\"source_event_id\":\"r2\",\"quote\":\"tests passed\"},{\"source_event_id\":\"r3\",\"quote\":\"runtime smoke failed\"}],\"truth_scope\":\"historical\",\"promotion_hint\":\"candidate\",\"confidence\":0.9},{\"claim_id\":\"c3\",\"kind\":\"failed_because\",\"text\":\"The implementation was prematurely treated as complete after tests passed without sufficient runtime smoke validation.\",\"support_event_ids\":[\"e4\",\"e5\"],\"evidence\":[{\"source_event_id\":\"a2\",\"quote\":\"The implementation is complete and tests pass.\"},{\"source_event_id\":\"u2\",\"quote\":\"did you check also that the game work or just the tests?\"}],\"truth_scope\":\"session_context\",\"promotion_hint\":\"candidate\",\"confidence\":0.82}],\"decisions\":[{\"decision_id\":\"d1\",\"text\":\"Future similar tasks should track a runtime smoke check as a separate acceptance gate from unit tests.\",\"support_event_ids\":[\"e1\",\"e5\"],\"confidence\":0.86}],\"attempts\":[{\"attempt_id\":\"a1\",\"action\":\"Create tests before implementing Snake behavior.\",\"support_event_ids\":[\"e2\",\"e3\"],\"status\":\"succeeded\"},{\"attempt_id\":\"a2\",\"action\":\"Verify implementation with unit tests.\",\"support_event_ids\":[\"e4\"],\"status\":\"succeeded\"},{\"attempt_id\":\"a3\",\"action\":\"Verify playable runtime behavior.\",\"support_event_ids\":[\"e5\"],\"status\":\"failed\"}],\"outcomes\":[{\"outcome_id\":\"o1\",\"text\":\"Unit tests passed.\",\"support_event_ids\":[\"e4\"],\"status\":\"success\"},{\"outcome_id\":\"o2\",\"text\":\"Runtime behavior was not proven by the first test pass and required further fixing.\",\"support_event_ids\":[\"e5\"],\"status\":\"failure\"}],\"open_questions\":[{\"question_id\":\"q1\",\"text\":\"Which runtime input/rendering path prevented the game from starting movement?\",\"support_event_ids\":[\"e5\"]}],\"next_actions\":[{\"action_id\":\"n1\",\"text\":\"Add or run a local runtime smoke that verifies movement starts and input is handled.\",\"support_event_ids\":[\"e5\"],\"owner_hint\":\"agent\"}],\"warnings\":[{\"code\":\"source_gap\",\"message\":\"The trace does not include the final fixed code or successful runtime smoke, so do not promote final game correctness.\",\"source_event_ids\":[\"r3\"]}]}}\n+{\"id\":\"seed-0002-agnix-tool-release-triage\",\"input\":{\"source_kind\":\"repo_tool_trace\",\"source_uri\":\"github://agent-sh/agnix/issues/868\",\"raw\":{\"source_events\":[{\"id\":\"u1\",\"role\":\"user\",\"timestamp\":\"2026-05-05T21:47:55Z\",\"text\":\"check 868\"},{\"id\":\"t1\",\"role\":\"tool_call\",\"timestamp\":\"2026-05-05T21:47:58Z\",\"tool_id\":\"github\",\"text\":\"format issue 868 in agent-sh/agnix\"},{\"id\":\"r1\",\"role\":\"tool_result\",\"timestamp\":\"2026-05-05T21:48:00Z\",\"tool_id\":\"github\",\"text\":\"Issue #868: amp (Sourcegraph) gpt-5.5. Action required: review release notes for agnix validation rules, update config if validated fields changed, update research tracking, run workflow with update_baselines.\"},{\"id\":\"a1\",\"role\":\"assistant\",\"timestamp\":\"2026-05-05T21:48:08Z\",\"text\":\"Summary: GPT-5.5 is a model update for Amp deep mode; probably no agnix schema change unless config surfaces changed.\"},{\"id\":\"u2\",\"role\":\"user\",\"timestamp\":\"2026-05-05T21:50:00Z\",\"text\":\"so you get -1 on instruction follwing and he get +1\"},{\"id\":\"t2\",\"role\":\"tool_call\",\"timestamp\":\"2026-05-05T22:01:30Z\",\"tool_id\":\"edit\",\"file_paths\":[\".github/tool-release-baselines.json\"],\"text\":\"add changes_of_interest for amp config surfaces and relevant/irrelevant changes\"}]}},\"expected\":{\"episode_hint\":\"agnix-amp-release-triage\",\"objective_hint\":\"Triage auto-opened agnix issue #868 and identify whether Amp GPT-5.5 changes affect validation rules or tracking metadata.\",\"events\":[{\"event_id\":\"e1\",\"event_type\":\"user_request\",\"summary\":\"User asked to inspect issue #868.\",\"provenance\":{\"source_event_ids\":[\"u1\"],\"timestamps\":[\"2026-05-05T21:47:55Z\"]},\"confidence\":0.98},{\"event_id\":\"e2\",\"event_type\":\"tool_call\",\"summary\":\"GitHub issue #868 was fetched for agent-sh/agnix.\",\"provenance\":{\"source_event_ids\":[\"t1\",\"r1\"],\"timestamps\":[\"2026-05-05T21:47:58Z\",\"2026-05-05T21:48:00Z\"],\"tool_ids\":[\"github\"]},\"confidence\":0.96},{\"event_id\":\"e3\",\"event_type\":\"observation\",\"summary\":\"The issue describes an Amp model update and lists required triage actions around agnix validation rules and baselines.\",\"provenance\":{\"source_event_ids\":[\"r1\"],\"timestamps\":[\"2026-05-05T21:48:00Z\"],\"tool_ids\":[\"github\"]},\"confidence\":0.95},{\"event_id\":\"e4\",\"event_type\":\"observation\",\"summary\":\"Assistant summarized the issue as likely not requiring schema changes unless config surfaces changed.\",\"provenance\":{\"source_event_ids\":[\"a1\"],\"timestamps\":[\"2026-05-05T21:48:08Z\"]},\"confidence\":0.82},{\"event_id\":\"e5\",\"event_type\":\"file_change\",\"summary\":\"The baseline config was edited to add Amp changes_of_interest metadata.\",\"provenance\":{\"source_event_ids\":[\"t2\"],\"timestamps\":[\"2026-05-05T22:01:30Z\"],\"tool_ids\":[\"edit\"],\"file_paths\":[\".github/tool-release-baselines.json\"]},\"confidence\":0.93},{\"event_id\":\"e6\",\"event_type\":\"observation\",\"summary\":\"User noted an instruction-following penalty for the orchestrating assistant, while crediting another agent for following the intended fetch scope.\",\"provenance\":{\"source_event_ids\":[\"u2\"],\"timestamps\":[\"2026-05-05T21:50:00Z\"]},\"confidence\":0.86}],\"candidate_claims\":[{\"claim_id\":\"c1\",\"kind\":\"current_state\",\"text\":\"Issue #868 concerned Amp changing its tracked model baseline to GPT-5.5, not necessarily an agnix validation schema change.\",\"support_event_ids\":[\"e3\",\"e4\"],\"evidence\":[{\"source_event_id\":\"r1\",\"quote\":\"Issue #868: amp (Sourcegraph) gpt-5.5\"},{\"source_event_id\":\"a1\",\"quote\":\"probably no agnix schema change unless config surfaces changed\"}],\"truth_scope\":\"session_context\",\"promotion_hint\":\"candidate\",\"confidence\":0.84},{\"claim_id\":\"c2\",\"kind\":\"attempted\",\"text\":\"The agent updated .github/tool-release-baselines.json with changes_of_interest metadata for Amp.\",\"support_event_ids\":[\"e5\"],\"evidence\":[{\"source_event_id\":\"t2\",\"quote\":\"add changes_of_interest for amp config surfaces\"}],\"truth_scope\":\"historical\",\"promotion_hint\":\"strong_candidate\",\"confidence\":0.93},{\"claim_id\":\"c3\",\"kind\":\"has_next_action\",\"text\":\"The deterministic system should still verify whether the release baseline workflow and issue closure happened; the trace only shows the config edit.\",\"support_event_ids\":[\"e3\",\"e5\"],\"evidence\":[{\"source_event_id\":\"r1\",\"quote\":\"run workflow with update_baselines\"},{\"source_event_id\":\"t2\",\"quote\":\"add changes_of_interest\"}],\"truth_scope\":\"session_context\",\"promotion_hint\":\"candidate\",\"confidence\":0.8}],\"decisions\":[{\"decision_id\":\"d1\",\"text\":\"Treat model-version-only Amp releases as low schema risk unless config surfaces or validation inputs changed.\",\"support_event_ids\":[\"e3\",\"e4\",\"e5\"],\"confidence\":0.82}],\"attempts\":[{\"attempt_id\":\"a1\",\"action\":\"Fetch and inspect GitHub issue #868.\",\"support_event_ids\":[\"e1\",\"e2\"],\"status\":\"succeeded\"},{\"attempt_id\":\"a2\",\"action\":\"Add triage metadata for relevant Amp change types.\",\"support_event_ids\":[\"e5\"],\"status\":\"succeeded\"}],\"outcomes\":[{\"outcome_id\":\"o1\",\"text\":\"The trace contains a config edit but not final workflow/PR status.\",\"support_event_ids\":[\"e5\"],\"status\":\"partial\"}],\"open_questions\":[{\"question_id\":\"q1\",\"text\":\"Was the update_baselines workflow run and was issue #868 closed or linked to a PR?\",\"support_event_ids\":[\"e3\"]}],\"next_actions\":[{\"action_id\":\"n1\",\"text\":\"Check the PR or workflow result before promoting the triage as complete.\",\"support_event_ids\":[\"e3\",\"e5\"],\"owner_hint\":\"agent\"}],\"warnings\":[{\"code\":\"source_gap\",\"message\":\"The trace does not include the final CI/workflow result or issue closure state.\",\"source_event_ids\":[\"r1\",\"t2\"]}]}}\n+{\"id\":\"seed-0003-qwen-training-fastpath-decision\",\"input\":{\"source_kind\":\"training_trace\",\"source_uri\":\"file://aviary/qwen35-fastpath-experiment\",\"raw\":{\"source_events\":[{\"id\":\"u1\",\"role\":\"user\",\"timestamp\":\"2026-05-06T23:00:00Z\",\"text\":\"we trained 27b on this machine, how 9b oom on 2048?\"},{\"id\":\"a1\",\"role\":\"assistant\",\"timestamp\":\"2026-05-06T23:02:00Z\",\"text\":\"The 9B run is using a different path than the 27B smoke: full modules, different activation memory, and possibly non-fast fallback.\"},{\"id\":\"t1\",\"role\":\"tool_call\",\"timestamp\":\"2026-05-06T23:10:00Z\",\"tool_id\":\"pip\",\"command_id\":\"cmd-install-fla\",\"text\":\"install flash-linear-attention, fla-core, causal-conv1d with CUDA 13 headers workaround\"},{\"id\":\"r1\",\"role\":\"tool_result\",\"timestamp\":\"2026-05-06T23:20:00Z\",\"tool_id\":\"python\",\"command_id\":\"cmd-check-fastpath\",\"text\":\"flash-linear-attention==0.5.0, fla-core==0.5.0, causal-conv1d==1.6.1, is_fast_path_available True\"},{\"id\":\"t2\",\"role\":\"tool_call\",\"timestamp\":\"2026-05-06T23:30:00Z\",\"tool_id\":\"train\",\"command_id\":\"cmd-train-v06\",\"text\":\"train qwen35-9b context ops v0.6 invariants fla full 1792 60step\"},{\"id\":\"r2\",\"role\":\"tool_result\",\"timestamp\":\"2026-05-06T23:38:00Z\",\"tool_id\":\"train\",\"command_id\":\"cmd-train-v06\",\"text\":\"train_loss 0.2719536894311508 runtime 408.7794s peak_allocated 22.1617 GiB peak_reserved 22.2383 GiB\"},{\"id\":\"r3\",\"role\":\"tool_result\",\"timestamp\":\"2026-05-06T23:45:00Z\",\"tool_id\":\"eval\",\"command_id\":\"cmd-eval-v06\",\"text\":\"challenge raw-domain-valid 10/11 repaired-valid 11/11; remaining raw failure emitted resolution ask instead of ask_user\"}]}},\"expected\":{\"episode_hint\":\"qwen35-9b-fla-training-setup\",\"objective_hint\":\"Diagnose why a Qwen3.5 9B LoRA run OOMed and establish a stable fast-path training setup for episodic context operations.\",\"events\":[{\"event_id\":\"e1\",\"event_type\":\"user_request\",\"summary\":\"User questioned why the smaller 9B model OOMed when a 27B smoke had trained locally.\",\"provenance\":{\"source_event_ids\":[\"u1\"],\"timestamps\":[\"2026-05-06T23:00:00Z\"]},\"confidence\":0.98},{\"event_id\":\"e2\",\"event_type\":\"observation\",\"summary\":\"Assistant identified that memory behavior differed because the 9B run used different module coverage, activation path, and possibly non-fast fallback.\",\"provenance\":{\"source_event_ids\":[\"a1\"],\"timestamps\":[\"2026-05-06T23:02:00Z\"]},\"confidence\":0.82},{\"event_id\":\"e3\",\"event_type\":\"attempt\",\"summary\":\"The fast-path packages were installed using a CUDA 13 headers workaround.\",\"provenance\":{\"source_event_ids\":[\"t1\"],\"timestamps\":[\"2026-05-06T23:10:00Z\"],\"tool_ids\":[\"pip\"],\"command_ids\":[\"cmd-install-fla\"]},\"confidence\":0.9},{\"event_id\":\"e4\",\"event_type\":\"validation\",\"summary\":\"The Qwen3.5 fast path was verified as available with FLA 0.5.0 and causal-conv1d 1.6.1.\",\"provenance\":{\"source_event_ids\":[\"r1\"],\"timestamps\":[\"2026-05-06T23:20:00Z\"],\"tool_ids\":[\"python\"],\"command_ids\":[\"cmd-check-fastpath\"]},\"confidence\":0.98},{\"event_id\":\"e5\",\"event_type\":\"validation\",\"summary\":\"Qwen3.5 9B v0.6 trained for 60 steps at 1792 context with peak reserved memory around 22.24 GiB.\",\"provenance\":{\"source_event_ids\":[\"t2\",\"r2\"],\"timestamps\":[\"2026-05-06T23:30:00Z\",\"2026-05-06T23:38:00Z\"],\"tool_ids\":[\"train\"],\"command_ids\":[\"cmd-train-v06\"]},\"confidence\":0.97},{\"event_id\":\"e6\",\"event_type\":\"outcome\",\"summary\":\"Challenge eval reached 10/11 raw-domain-valid and 11/11 after deterministic repair, with one enum drift failure.\",\"provenance\":{\"source_event_ids\":[\"r3\"],\"timestamps\":[\"2026-05-06T23:45:00Z\"],\"tool_ids\":[\"eval\"],\"command_ids\":[\"cmd-eval-v06\"]},\"confidence\":0.97}],\"candidate_claims\":[{\"claim_id\":\"c1\",\"kind\":\"worked_because\",\"text\":\"The Qwen3.5 9B training path became viable after the FLA/causal-conv fast path was installed and verified.\",\"support_event_ids\":[\"e3\",\"e4\",\"e5\"],\"evidence\":[{\"source_event_id\":\"r1\",\"quote\":\"is_fast_path_available True\"},{\"source_event_id\":\"r2\",\"quote\":\"peak_reserved 22.2383 GiB\"}],\"truth_scope\":\"historical\",\"promotion_hint\":\"strong_candidate\",\"confidence\":0.92},{\"claim_id\":\"c2\",\"kind\":\"validation_result\",\"text\":\"The v0.6 adapter still had one raw enum drift issue: resolution ask instead of ask_user.\",\"support_event_ids\":[\"e6\"],\"evidence\":[{\"source_event_id\":\"r3\",\"quote\":\"remaining raw failure emitted resolution ask instead of ask_user\"}],\"truth_scope\":\"historical\",\"promotion_hint\":\"strong_candidate\",\"confidence\":0.97},{\"claim_id\":\"c3\",\"kind\":\"has_next_action\",\"text\":\"The next training material should include enum/invariant correction examples rather than more generic rows.\",\"support_event_ids\":[\"e6\"],\"evidence\":[{\"source_event_id\":\"r3\",\"quote\":\"remaining raw failure emitted resolution ask instead of ask_user\"}],\"truth_scope\":\"session_context\",\"promotion_hint\":\"candidate\",\"confidence\":0.86}],\"decisions\":[{\"decision_id\":\"d1\",\"text\":\"Use the verified FLA full path as the stable Qwen3.5 9B training setup, while keeping deterministic repair in the eval loop.\",\"support_event_ids\":[\"e4\",\"e5\",\"e6\"],\"confidence\":0.91}],\"attempts\":[{\"attempt_id\":\"a1\",\"action\":\"Install and verify Qwen3.5 fast-path dependencies.\",\"support_event_ids\":[\"e3\",\"e4\"],\"status\":\"succeeded\"},{\"attempt_id\":\"a2\",\"action\":\"Train Qwen3.5 9B context-ops v0.6 for 60 steps.\",\"support_event_ids\":[\"e5\"],\"status\":\"succeeded\"},{\"attempt_id\":\"a3\",\"action\":\"Evaluate v0.6 on the challenge split.\",\"support_event_ids\":[\"e6\"],\"status\":\"succeeded\"}],\"outcomes\":[{\"outcome_id\":\"o1\",\"text\":\"Training fit within 24 GiB class VRAM with about 22.24 GiB peak reserved.\",\"support_event_ids\":[\"e5\"],\"status\":\"success\"},{\"outcome_id\":\"o2\",\"text\":\"Eval was valid after repair but not perfectly raw-valid due to enum drift.\",\"support_event_ids\":[\"e6\"],\"status\":\"partial\"}],\"open_questions\":[{\"question_id\":\"q1\",\"text\":\"Does the same fast path stay stable for larger context or broader adapter coverage?\",\"support_event_ids\":[\"e5\"]}],\"next_actions\":[{\"action_id\":\"n1\",\"text\":\"Add correction rows for enum/invariant failures and train a short v0.7 comparison from the base model.\",\"support_event_ids\":[\"e6\"],\"owner_hint\":\"agent\"}],\"warnings\":[{\"code\":\"policy_owned_by_deterministic_system\",\"message\":\"The model can propose normalized training-state claims, but the verifier should own final acceptance of adapter quality.\",\"source_event_ids\":[\"r3\"]}]}}\n*** End Patch\n",
|
| 689 |
+
"is_context_tool_echo": false,
|
| 690 |
+
"role": "tool_call",
|
| 691 |
+
"success": null,
|
| 692 |
+
"text": "apply_patch: *** Begin Patch\n*** Add File: /home/avifenesh/projects/episodic-ingestion-compiler/data/examples/seed.jsonl\n+{\"id\":\"seed-0001-snake-tdd-validation-gap\",\"input\":{\"so...",
|
| 693 |
+
"tool_name": "apply_patch"
|
| 694 |
+
},
|
| 695 |
+
"expected_probability": 0.4651445150375366,
|
| 696 |
+
"expected_source_event_id": "t19",
|
| 697 |
+
"field": "attempted_actions",
|
| 698 |
+
"rank": 20,
|
| 699 |
+
"reciprocal_rank": 0.05,
|
| 700 |
+
"row_id": "pta-codex-rollout-2026-04-30T19-43-19-019ddf45-fdda-7fb2-8764-8162ba11dc24::w88",
|
| 701 |
+
"top_candidates": [
|
| 702 |
+
{
|
| 703 |
+
"event": {
|
| 704 |
+
"command": "test -e /home/avifenesh/projects/episodic-ingestion-compiler; echo $?",
|
| 705 |
+
"is_context_tool_echo": false,
|
| 706 |
+
"role": "tool_call",
|
| 707 |
+
"success": null,
|
| 708 |
+
"text": "test -e /home/avifenesh/projects/episodic-ingestion-compiler; echo $?",
|
| 709 |
+
"tool_name": "exec_command"
|
| 710 |
+
},
|
| 711 |
+
"label": 1,
|
| 712 |
+
"source_event_id": "t6",
|
| 713 |
+
"support_probability": 0.9998511075973511
|
| 714 |
+
},
|
| 715 |
+
{
|
| 716 |
+
"event": {
|
| 717 |
+
"command": "mkdir -p /home/avifenesh/projects/episodic-ingestion-compiler/{src/episodic_ingestion_compiler,scripts,schemas,data/examples,data/sft,docs,tests}",
|
| 718 |
+
"is_context_tool_echo": false,
|
| 719 |
+
"role": "tool_call",
|
| 720 |
+
"success": null,
|
| 721 |
+
"text": "mkdir -p /home/avifenesh/projects/episodic-ingestion-compiler/{src/episodic_ingestion_compiler,scripts,schemas,data/examples,data/sft,docs,tests}",
|
| 722 |
+
"tool_name": "exec_command"
|
| 723 |
+
},
|
| 724 |
+
"label": 1,
|
| 725 |
+
"source_event_id": "t12",
|
| 726 |
+
"support_probability": 0.9995827078819275
|
| 727 |
+
},
|
| 728 |
+
{
|
| 729 |
+
"event": {
|
| 730 |
+
"command": "python3 - <<'PY'\ntry:\n import jsonschema\n print('jsonschema ok')\nexcept Exception as e:\n print(f'jsonschema missing: {e}')\nPY",
|
| 731 |
+
"is_context_tool_echo": false,
|
| 732 |
+
"role": "tool_call",
|
| 733 |
+
"success": null,
|
| 734 |
+
"text": "python3 - <<'PY'\ntry:\n import jsonschema\n print('jsonschema ok')\nexcept Exception as e:\n print(f'jsonschema missing: {e}')\nPY",
|
| 735 |
+
"tool_name": "exec_command"
|
| 736 |
+
},
|
| 737 |
+
"label": 1,
|
| 738 |
+
"source_event_id": "t13",
|
| 739 |
+
"support_probability": 0.9994966983795166
|
| 740 |
+
},
|
| 741 |
+
{
|
| 742 |
+
"event": {
|
| 743 |
+
"command": "{\"plan\": [{\"step\": \"Create clean repo scaffold\", \"status\": \"in_progress\"}, {\"step\": \"Define ingestion schema and contract\", \"status\": \"pending\"}, {\"step\": \"Add seed training examples and validators\", \"status\": \"pending\"}, {\"step\": \"Run validation and commit\", \"status\": \"pending\"}, {\"step\": \"Report next training path\", \"status\": \"pending\"}]}",
|
| 744 |
+
"is_context_tool_echo": false,
|
| 745 |
+
"role": "tool_call",
|
| 746 |
+
"success": null,
|
| 747 |
+
"text": "{\"plan\": [{\"step\": \"Create clean repo scaffold\", \"status\": \"in_progress\"}, {\"step\": \"Define ingestion schema and contract\", \"status\": \"pending\"}, {\"step\": \"Add seed training exa...",
|
| 748 |
+
"tool_name": "update_plan"
|
| 749 |
+
},
|
| 750 |
+
"label": 1,
|
| 751 |
+
"source_event_id": "t3",
|
| 752 |
+
"support_probability": 0.99935382604599
|
| 753 |
+
},
|
| 754 |
+
{
|
| 755 |
+
"event": {
|
| 756 |
+
"command": "git status --short",
|
| 757 |
+
"is_context_tool_echo": false,
|
| 758 |
+
"role": "tool_call",
|
| 759 |
+
"success": null,
|
| 760 |
+
"text": "git status --short",
|
| 761 |
+
"tool_name": "exec_command"
|
| 762 |
+
},
|
| 763 |
+
"label": 1,
|
| 764 |
+
"source_event_id": "t7",
|
| 765 |
+
"support_probability": 0.99935382604599
|
| 766 |
+
}
|
| 767 |
+
]
|
| 768 |
+
},
|
| 769 |
+
{
|
| 770 |
+
"expected_event": {
|
| 771 |
+
"command": "python3 << 'EOF'\nimport json, re, sys\n\nwith open('/tmp/valkey_prs_batch4.json') as f:\n raw = f.read()\n\n# Split by PR markers\nprs = re.split(r'=== PR #(\\d+) ===\\n', raw)[1:]\n# pairs: [num, content, num, content, ...]\n\nfor i in range(0, len(prs), 2):\n num = prs[i]\n content = prs[i+1]\n # Split content at --- inline comments ---\n parts = content.split('--- inline comments ---\\n', 1)\n pr_json_text = parts[0].strip()\n inline_text = parts[1].strip() if len(parts) > 1 else ''\n \n try:\n pr = json.loads(pr_json_text)\n except Exception as e:\n print(f\"PR {num} json err: {e}\")\n continue\n \n print(f\"\\n===== PR #{num} =====\")\n print(f\"Title: {pr.get('title')}\")\n print(f\"Base: {pr.get('baseRefName')}\")\n print(f\"Author: {pr.get('author', {}).g",
|
| 772 |
+
"is_context_tool_echo": false,
|
| 773 |
+
"role": "tool_call",
|
| 774 |
+
"success": null,
|
| 775 |
+
"text": "python3 << 'EOF'\nimport json, re, sys\n\nwith open('/tmp/valkey_prs_batch4.json') as f:\n raw = f.read()\n\n# Split by PR markers\nprs = re.split(r'=== PR #(\\d+) ===\\n', raw)[1:]\n#...",
|
| 776 |
+
"tool_name": "Bash"
|
| 777 |
+
},
|
| 778 |
+
"expected_probability": 0.3988746404647827,
|
| 779 |
+
"expected_source_event_id": "t7",
|
| 780 |
+
"field": "attempted_actions",
|
| 781 |
+
"rank": 20,
|
| 782 |
+
"reciprocal_rank": 0.05,
|
| 783 |
+
"row_id": "pta-claude_code-agent-a55fe1edceb764d9e::w0",
|
| 784 |
+
"top_candidates": [
|
| 785 |
+
{
|
| 786 |
+
"event": {
|
| 787 |
+
"command": "cd /home/avifenesh/projects/valkey && git log --all --grep=\"#3469\" --oneline | head -5 && echo \"---\" && git log --all --oneline | grep -i \"3469\" | head -5 && echo \"---forkless branch---\" && git branch -a | grep -i forkless | head",
|
| 788 |
+
"is_context_tool_echo": false,
|
| 789 |
+
"role": "tool_call",
|
| 790 |
+
"success": null,
|
| 791 |
+
"text": "cd /home/avifenesh/projects/valkey && git log --all --grep=\"#3469\" --oneline | head -5 && echo \"---\" && git log --all --oneline | grep -i \"3469\" | head -5 && echo \"---forkless b...",
|
| 792 |
+
"tool_name": "Bash"
|
| 793 |
+
},
|
| 794 |
+
"label": 1,
|
| 795 |
+
"source_event_id": "t14",
|
| 796 |
+
"support_probability": 0.999595582485199
|
| 797 |
+
},
|
| 798 |
+
{
|
| 799 |
+
"event": {
|
| 800 |
+
"command": "cd /home/avifenesh/projects/valkey && git fetch origin +refs/pull/3469/head:refs/remotes/origin/pr-3469 2>&1 | tail -3 && git log --format='%H %s%n%b' -1 origin/pr-3469 2>&1 | head -40",
|
| 801 |
+
"is_context_tool_echo": false,
|
| 802 |
+
"role": "tool_call",
|
| 803 |
+
"success": null,
|
| 804 |
+
"text": "cd /home/avifenesh/projects/valkey && git fetch origin +refs/pull/3469/head:refs/remotes/origin/pr-3469 2>&1 | tail -3 && git log --format='%H %s%n%b' -1 origin/pr-3469 2>&1 | h...",
|
| 805 |
+
"tool_name": "Bash"
|
| 806 |
+
},
|
| 807 |
+
"label": 1,
|
| 808 |
+
"source_event_id": "t16",
|
| 809 |
+
"support_probability": 0.999555766582489
|
| 810 |
+
},
|
| 811 |
+
{
|
| 812 |
+
"event": {
|
| 813 |
+
"command": "ls -la /tmp/valkey_prs_batch4.json && head -100 /tmp/valkey_prs_batch4.json",
|
| 814 |
+
"is_context_tool_echo": false,
|
| 815 |
+
"role": "tool_call",
|
| 816 |
+
"success": null,
|
| 817 |
+
"text": "ls -la /tmp/valkey_prs_batch4.json && head -100 /tmp/valkey_prs_batch4.json",
|
| 818 |
+
"tool_name": "Bash"
|
| 819 |
+
},
|
| 820 |
+
"label": 1,
|
| 821 |
+
"source_event_id": "t5",
|
| 822 |
+
"support_probability": 0.9994115829467773
|
| 823 |
+
},
|
| 824 |
+
{
|
| 825 |
+
"event": {
|
| 826 |
+
"command": "Read /home/avifenesh/.claude/projects/-home-avifenesh-projects-valkey-skills/ff8a0875-8828-483a-a804-8fd20d09aecb/tool-results/bq6pp40yy.txt",
|
| 827 |
+
"is_context_tool_echo": false,
|
| 828 |
+
"role": "tool_call",
|
| 829 |
+
"success": null,
|
| 830 |
+
"text": "Read /home/avifenesh/.claude/projects/-home-avifenesh-projects-valkey-skills/ff8a0875-8828-483a-a804-8fd20d09aecb/tool-results/bq6pp40yy.txt",
|
| 831 |
+
"tool_name": "Read"
|
| 832 |
+
},
|
| 833 |
+
"label": 1,
|
| 834 |
+
"source_event_id": "t9",
|
| 835 |
+
"support_probability": 0.9987157583236694
|
| 836 |
+
},
|
| 837 |
+
{
|
| 838 |
+
"event": {
|
| 839 |
+
"command": "for pr in 3498 3495 3472 3471 3469 3464 3463 3462 3461 3460; do\n echo \"=== PR #$pr ===\"\n gh pr view $pr --repo valkey-io/valkey --json title,body,reviews,comments,labels,baseRefName,mergeCommit,author 2>&1\n echo \"--- inline comments ---\"\n gh api repos/valkey-io/valkey/pulls/$pr/comments --paginate 2>&1\n echo\ndone > /tmp/valkey_prs_batch4.json 2>&1\nwc -l /tmp/valkey_prs_batch4.json",
|
| 840 |
+
"is_context_tool_echo": false,
|
| 841 |
+
"role": "tool_call",
|
| 842 |
+
"success": null,
|
| 843 |
+
"text": "for pr in 3498 3495 3472 3471 3469 3464 3463 3462 3461 3460; do\n echo \"=== PR #$pr ===\"\n gh pr view $pr --repo valkey-io/valkey --json title,body,reviews,comments,labels,baseR...",
|
| 844 |
+
"tool_name": "Bash"
|
| 845 |
+
},
|
| 846 |
+
"label": 1,
|
| 847 |
+
"source_event_id": "t3",
|
| 848 |
+
"support_probability": 0.9980732202529907
|
| 849 |
+
}
|
| 850 |
+
]
|
| 851 |
+
},
|
| 852 |
+
{
|
| 853 |
+
"expected_event": {
|
| 854 |
+
"command": "for sha in 5157255b82002a79fd212de9d9ba0fe08812849f 6b384e9a5b6cd4e4f8198f5df22a74feb1b70e0f 6c329dfe2c02d09b276fd5617acdb31e6169ebc1 8caa29298c51b4bb498418515801dd0720ebce62 5deebd0f131826124b3ab6231fafb5f9b188523b 6b85ca4fffff51de5f8cfcc6ce239900a48924d0 1e2ca5ec5dd6f97215e7acbcee14e89ea56a9e02 6414720504f689a3fc699338bf22565bb70c3829 9f0106755b177f8eea3d62eed6f28f20fb54d459 a3a839972ff10a68b1a7069d70d8ce1fffb8f02a; do echo \"=== $sha ===\"; git -C /home/avifenesh/projects/valkey show --format='%s%n%b%n---END---' -s $sha 2>&1 | head -30; done",
|
| 855 |
+
"is_context_tool_echo": false,
|
| 856 |
+
"role": "tool_call",
|
| 857 |
+
"success": null,
|
| 858 |
+
"text": "for sha in 5157255b82002a79fd212de9d9ba0fe08812849f 6b384e9a5b6cd4e4f8198f5df22a74feb1b70e0f 6c329dfe2c02d09b276fd5617acdb31e6169ebc1 8caa29298c51b4bb498418515801dd0720ebce62 5d...",
|
| 859 |
+
"tool_name": "Bash"
|
| 860 |
+
},
|
| 861 |
+
"expected_probability": 0.709019124507904,
|
| 862 |
+
"expected_source_event_id": "t12",
|
| 863 |
+
"field": "attempted_actions",
|
| 864 |
+
"rank": 20,
|
| 865 |
+
"reciprocal_rank": 0.05,
|
| 866 |
+
"row_id": "pta-claude_code-agent-a5a805ef49f3ca63c::w0",
|
| 867 |
+
"top_candidates": [
|
| 868 |
+
{
|
| 869 |
+
"event": {
|
| 870 |
+
"command": "cd /tmp/pr-survey && for pr in 3343 3342 3340 3338 3336 3333 3331 3329 3325 3324; do gh api repos/valkey-io/valkey/pulls/$pr/comments --paginate > inline-$pr.json 2>&1 & done; wait; ls -la inline-*",
|
| 871 |
+
"is_context_tool_echo": false,
|
| 872 |
+
"role": "tool_call",
|
| 873 |
+
"success": null,
|
| 874 |
+
"text": "cd /tmp/pr-survey && for pr in 3343 3342 3340 3338 3336 3333 3331 3329 3325 3324; do gh api repos/valkey-io/valkey/pulls/$pr/comments --paginate > inline-$pr.json 2>&1 & done; w...",
|
| 875 |
+
"tool_name": "Bash"
|
| 876 |
+
},
|
| 877 |
+
"label": 1,
|
| 878 |
+
"source_event_id": "t4",
|
| 879 |
+
"support_probability": 0.9994471669197083
|
| 880 |
+
},
|
| 881 |
+
{
|
| 882 |
+
"event": {
|
| 883 |
+
"command": "cd /tmp/pr-survey && for pr in 3343 3342 3340 3338 3336 3333 3331 3329 3325 3324; do echo \"=== PR $pr inline-comments ===\"; jq -r '.[] | \"@\\(.user.login) on \\(.path):\\(.line // .original_line): \\((.body // \"\")[:250])\"' inline-$pr.json 2>/dev/null | head -30; done 2>&1 | head -300",
|
| 884 |
+
"is_context_tool_echo": false,
|
| 885 |
+
"role": "tool_call",
|
| 886 |
+
"success": null,
|
| 887 |
+
"text": "cd /tmp/pr-survey && for pr in 3343 3342 3340 3338 3336 3333 3331 3329 3325 3324; do echo \"=== PR $pr inline-comments ===\"; jq -r '.[] | \"@\\(.user.login) on \\(.path):\\(.line //...",
|
| 888 |
+
"tool_name": "Bash"
|
| 889 |
+
},
|
| 890 |
+
"label": 1,
|
| 891 |
+
"source_event_id": "t11",
|
| 892 |
+
"support_probability": 0.9993928670883179
|
| 893 |
+
},
|
| 894 |
+
{
|
| 895 |
+
"event": {
|
| 896 |
+
"command": "cd /tmp/pr-survey && for pr in 3343 3342 3340 3338 3336 3333 3331 3329 3325 3324; do echo \"=== PR $pr reviews+comments ===\"; jq -r '.reviews[] | \"[review] [\\(.state)] @\\(.author.login): \\((.body // \"\")[:300])\"' pr-$pr.json; jq -r '.comments[] | \"[comment] @\\(.author.login): \\((.body // \"\")[:300])\"' pr-$pr.json; done 2>&1 | head -500",
|
| 897 |
+
"is_context_tool_echo": false,
|
| 898 |
+
"role": "tool_call",
|
| 899 |
+
"success": null,
|
| 900 |
+
"text": "cd /tmp/pr-survey && for pr in 3343 3342 3340 3338 3336 3333 3331 3329 3325 3324; do echo \"=== PR $pr reviews+comments ===\"; jq -r '.reviews[] | \"[review] [\\(.state)] @\\(.author...",
|
| 901 |
+
"tool_name": "Bash"
|
| 902 |
+
},
|
| 903 |
+
"label": 1,
|
| 904 |
+
"source_event_id": "t9",
|
| 905 |
+
"support_probability": 0.9992678761482239
|
| 906 |
+
},
|
| 907 |
+
{
|
| 908 |
+
"event": {
|
| 909 |
+
"command": "cd /tmp/pr-survey && for pr in 3343 3342 3340 3338 3336 3333 3331 3329 3325 3324; do echo \"=== PR $pr ===\"; jq -r '{title, author: .author.login, baseRefName, labels: [.labels[].name], mergeCommit: .mergeCommit.oid, body: (.body|.[:1500])}' pr-$pr.json; done 2>&1 | head -400",
|
| 910 |
+
"is_context_tool_echo": false,
|
| 911 |
+
"role": "tool_call",
|
| 912 |
+
"success": null,
|
| 913 |
+
"text": "cd /tmp/pr-survey && for pr in 3343 3342 3340 3338 3336 3333 3331 3329 3325 3324; do echo \"=== PR $pr ===\"; jq -r '{title, author: .author.login, baseRefName, labels: [.labels[]...",
|
| 914 |
+
"tool_name": "Bash"
|
| 915 |
+
},
|
| 916 |
+
"label": 1,
|
| 917 |
+
"source_event_id": "t7",
|
| 918 |
+
"support_probability": 0.9988664388656616
|
| 919 |
+
},
|
| 920 |
+
{
|
| 921 |
+
"event": {
|
| 922 |
+
"command": "cd /tmp && mkdir -p pr-survey && cd pr-survey && for pr in 3343 3342 3340 3338 3336 3333 3331 3329 3325 3324; do gh pr view $pr --repo valkey-io/valkey --json title,body,reviews,comments,labels,baseRefName,mergeCommit,author > pr-$pr.json 2>&1 & done; wait; ls -la",
|
| 923 |
+
"is_context_tool_echo": false,
|
| 924 |
+
"role": "tool_call",
|
| 925 |
+
"success": null,
|
| 926 |
+
"text": "cd /tmp && mkdir -p pr-survey && cd pr-survey && for pr in 3343 3342 3340 3338 3336 3333 3331 3329 3325 3324; do gh pr view $pr --repo valkey-io/valkey --json title,body,reviews...",
|
| 927 |
+
"tool_name": "Bash"
|
| 928 |
+
},
|
| 929 |
+
"label": 1,
|
| 930 |
+
"source_event_id": "t3",
|
| 931 |
+
"support_probability": 0.9987550973892212
|
| 932 |
+
}
|
| 933 |
+
]
|
| 934 |
+
},
|
| 935 |
+
{
|
| 936 |
+
"expected_event": {
|
| 937 |
+
"command": "ROOT=/home/avifenesh/projects/aviary; RUN_LOG=\"$ROOT/data/runs/qwen35-moe-8k-triton-$(date +%Y%m%d-%H%M%S).log\"; export PATH=\"$ROOT/.venv/bin:$PATH\" PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True VLLM_MEMORY_PROFILER_ESTIMATE_CUDAGRAPHS=0; echo \"log=$RUN_LOG\"; sudo -n systemd-run --scope -p MemoryMax=32G -p MemorySwapMax=0 \"$ROOT/.venv/bin/vllm\" serve \"$ROOT/models/qwen3.6-35b-a3b-nvfp4\" --host 127.0.0.1 --port 8000 --served-model-name aviary-local-qwen --trust-remote-code --language-model-only --skip-mm-profiling --chat-template \"$ROOT/configs/qwen-codex-chat-template.jinja\" --reasoning-parser qwen3 --enable-auto-tool-choice --tool-call-parser qwen3_coder --max-model-len 8192 --kv-cache-dtype fp8 --enable-prefix-caching --gpu-memory-utilization 0.92 --max-num-seqs 1 --max-num-batched-to",
|
| 938 |
+
"is_context_tool_echo": false,
|
| 939 |
+
"role": "tool_call",
|
| 940 |
+
"success": null,
|
| 941 |
+
"text": "ROOT=/home/avifenesh/projects/aviary; RUN_LOG=\"$ROOT/data/runs/qwen35-moe-8k-triton-$(date +%Y%m%d-%H%M%S).log\"; export PATH=\"$ROOT/.venv/bin:$PATH\" PYTORCH_CUDA_ALLOC_CONF=expa...",
|
| 942 |
+
"tool_name": "exec_command"
|
| 943 |
+
},
|
| 944 |
+
"expected_probability": 0.4012185037136078,
|
| 945 |
+
"expected_source_event_id": "t10",
|
| 946 |
+
"field": "attempted_actions",
|
| 947 |
+
"rank": 19,
|
| 948 |
+
"reciprocal_rank": 0.05263157894736842,
|
| 949 |
+
"row_id": "pta-codex-rollout-2026-04-30T19-43-19-019ddf45-fdda-7fb2-8764-8162ba11dc24::w14",
|
| 950 |
+
"top_candidates": [
|
| 951 |
+
{
|
| 952 |
+
"event": {
|
| 953 |
+
"command": "{\"session_id\": 92479, \"chars\": \"\\u0003\", \"yield_time_ms\": 1000, \"max_output_tokens\": 20000}",
|
| 954 |
+
"is_context_tool_echo": false,
|
| 955 |
+
"role": "tool_call",
|
| 956 |
+
"success": null,
|
| 957 |
+
"text": "{\"session_id\": 92479, \"chars\": \"\\u0003\", \"yield_time_ms\": 1000, \"max_output_tokens\": 20000}",
|
| 958 |
+
"tool_name": "write_stdin"
|
| 959 |
+
},
|
| 960 |
+
"label": 1,
|
| 961 |
+
"source_event_id": "t3",
|
| 962 |
+
"support_probability": 0.9996079802513123
|
| 963 |
+
},
|
| 964 |
+
{
|
| 965 |
+
"event": {
|
| 966 |
+
"command": "{\"session_id\": 1703, \"chars\": \"\", \"yield_time_ms\": 1000, \"max_output_tokens\": 30000}",
|
| 967 |
+
"is_context_tool_echo": false,
|
| 968 |
+
"role": "tool_call",
|
| 969 |
+
"success": null,
|
| 970 |
+
"text": "{\"session_id\": 1703, \"chars\": \"\", \"yield_time_ms\": 1000, \"max_output_tokens\": 30000}",
|
| 971 |
+
"tool_name": "write_stdin"
|
| 972 |
+
},
|
| 973 |
+
"label": 1,
|
| 974 |
+
"source_event_id": "t12",
|
| 975 |
+
"support_probability": 0.9992678761482239
|
| 976 |
+
},
|
| 977 |
+
{
|
| 978 |
+
"event": {
|
| 979 |
+
"command": "{\"session_id\": 1703, \"chars\": \"\", \"yield_time_ms\": 1000, \"max_output_tokens\": 6000}",
|
| 980 |
+
"is_context_tool_echo": false,
|
| 981 |
+
"role": "tool_call",
|
| 982 |
+
"success": null,
|
| 983 |
+
"text": "{\"session_id\": 1703, \"chars\": \"\", \"yield_time_ms\": 1000, \"max_output_tokens\": 6000}",
|
| 984 |
+
"tool_name": "write_stdin"
|
| 985 |
+
},
|
| 986 |
+
"label": 1,
|
| 987 |
+
"source_event_id": "t17",
|
| 988 |
+
"support_probability": 0.9992678761482239
|
| 989 |
+
},
|
| 990 |
+
{
|
| 991 |
+
"event": {
|
| 992 |
+
"command": "nvidia-smi --query-gpu=memory.used,memory.free,utilization.gpu --format=csv,noheader,nounits && free -h && ss -ltnp sport = :8000",
|
| 993 |
+
"is_context_tool_echo": false,
|
| 994 |
+
"role": "tool_call",
|
| 995 |
+
"success": null,
|
| 996 |
+
"text": "nvidia-smi --query-gpu=memory.used,memory.free,utilization.gpu --format=csv,noheader,nounits && free -h && ss -ltnp sport = :8000",
|
| 997 |
+
"tool_name": "exec_command"
|
| 998 |
+
},
|
| 999 |
+
"label": 1,
|
| 1000 |
+
"source_event_id": "t13",
|
| 1001 |
+
"support_probability": 0.9985895752906799
|
| 1002 |
+
},
|
| 1003 |
+
{
|
| 1004 |
+
"event": {
|
| 1005 |
+
"command": "nvidia-smi --query-gpu=name,memory.used,memory.free,utilization.gpu --format=csv,noheader,nounits && free -h && ss -ltnp 'sport = :8000' || true",
|
| 1006 |
+
"is_context_tool_echo": false,
|
| 1007 |
+
"role": "tool_call",
|
| 1008 |
+
"success": null,
|
| 1009 |
+
"text": "nvidia-smi --query-gpu=name,memory.used,memory.free,utilization.gpu --format=csv,noheader,nounits && free -h && ss -ltnp 'sport = :8000' || true",
|
| 1010 |
+
"tool_name": "exec_command"
|
| 1011 |
+
},
|
| 1012 |
+
"label": 1,
|
| 1013 |
+
"source_event_id": "t18",
|
| 1014 |
+
"support_probability": 0.998245358467102
|
| 1015 |
+
}
|
| 1016 |
+
]
|
| 1017 |
+
},
|
| 1018 |
+
{
|
| 1019 |
+
"expected_event": {
|
| 1020 |
+
"command": "*** Begin Patch\n*** Update File: /home/avifenesh/projects/aviary/scripts/build-scorecard-sft-dataset.py\n@@\n-By default this exports only canonical Avi scorecards. Use\n---include-corrected-drafts to add draft files that Avi corrected, and\n---include-drafts only for research/debug datasets.\n+By default this exports only canonical Avi scorecards. Use\n+--include-corrected-drafts to add draft files that Avi corrected, and\n+--include-drafts only for research/debug datasets. Individual cards can set\n+exclude_from_sft: true when they are useful as reviewed eval evidence but\n+unsafe or unhelpful as training targets.\n@@\n def target_yaml(card: dict[str, Any]) -> str:\n@@\n return yaml.safe_dump(target, sort_keys=False, allow_unicode=False).strip() + \"\\n\"\n+\n+\n+def is_excluded_from_sft(card: dict[str, Any]) -> bool:\n+ if bool(card.get(\"exclude_from_sft\")):\n+ return True\n+ eligibility = str(card.get(\"training_eligibility\", \"\")).strip().lower()\n+ return eligibility in {\"exclude\", \"excluded\", \"do_not_train\", \"not_for_training\"}\n@@\n records = source_records(args.lane, args.include_corrected_drafts, args.include_drafts)\n for source_file, card, source_kind in records:\n stats[f\"source:{source_kind}\"] += 1\n+ if is_excluded_from_sft(card):\n+ stats[f\"excluded_from_sft:{source_kind}\"] += 1\n+ skipped.append(\n+ {\n+ \"source_file\": str(source_file),\n+ \"run_id\": str(card.get(\"run_id\")),\n+ \"case_id\": str(card.get(\"case_id\")),\n+ \"reason\": \"card marked exclude_from_sft\",\n+ }\n+ )\n+ continue\n result_path = result_path_for_card(args.lane, card)\n*** End Patch\n",
|
| 1021 |
+
"is_context_tool_echo": false,
|
| 1022 |
+
"role": "tool_call",
|
| 1023 |
+
"success": null,
|
| 1024 |
+
"text": "apply_patch: *** Begin Patch\n*** Update File: /home/avifenesh/projects/aviary/scripts/build-scorecard-sft-dataset.py\n@@\n-By default this exports only canonical Avi scorecards. U...",
|
| 1025 |
+
"tool_name": "apply_patch"
|
| 1026 |
+
},
|
| 1027 |
+
"expected_probability": 0.6141626238822937,
|
| 1028 |
+
"expected_source_event_id": "t3",
|
| 1029 |
+
"field": "attempted_actions",
|
| 1030 |
+
"rank": 19,
|
| 1031 |
+
"reciprocal_rank": 0.05263157894736842,
|
| 1032 |
+
"row_id": "pta-codex-rollout-2026-04-30T19-43-19-019ddf45-fdda-7fb2-8764-8162ba11dc24::w32",
|
| 1033 |
+
"top_candidates": [
|
| 1034 |
+
{
|
| 1035 |
+
"event": {
|
| 1036 |
+
"command": "wc -l data/training/session_summarize_for_avi/scorecard_sft_avi_verified.jsonl data/training/session_summarize_for_avi/scorecard_sft_with_drafts.jsonl",
|
| 1037 |
+
"is_context_tool_echo": false,
|
| 1038 |
+
"role": "tool_call",
|
| 1039 |
+
"success": null,
|
| 1040 |
+
"text": "wc -l data/training/session_summarize_for_avi/scorecard_sft_avi_verified.jsonl data/training/session_summarize_for_avi/scorecard_sft_with_drafts.jsonl",
|
| 1041 |
+
"tool_name": "exec_command"
|
| 1042 |
+
},
|
| 1043 |
+
"label": 1,
|
| 1044 |
+
"source_event_id": "t13",
|
| 1045 |
+
"support_probability": 0.9992903470993042
|
| 1046 |
+
},
|
| 1047 |
+
{
|
| 1048 |
+
"event": {
|
| 1049 |
+
"command": "python3 -m py_compile scripts/build-scorecard-sft-dataset.py && scripts/build-scorecard-sft-dataset.py --out data/training/session_summarize_for_avi/scorecard_sft_avi_verified.jsonl --include-corrected-drafts",
|
| 1050 |
+
"is_context_tool_echo": false,
|
| 1051 |
+
"role": "tool_call",
|
| 1052 |
+
"success": null,
|
| 1053 |
+
"text": "python3 -m py_compile scripts/build-scorecard-sft-dataset.py && scripts/build-scorecard-sft-dataset.py --out data/training/session_summarize_for_avi/scorecard_sft_avi_verified.j...",
|
| 1054 |
+
"tool_name": "exec_command"
|
| 1055 |
+
},
|
| 1056 |
+
"label": 1,
|
| 1057 |
+
"source_event_id": "t9",
|
| 1058 |
+
"support_probability": 0.9989350438117981
|
| 1059 |
+
},
|
| 1060 |
+
{
|
| 1061 |
+
"event": {
|
| 1062 |
+
"command": "sed -n '1,120p' data/training/session_summarize_for_avi/scorecard_sft_avi_verified.jsonl.manifest.yml",
|
| 1063 |
+
"is_context_tool_echo": false,
|
| 1064 |
+
"role": "tool_call",
|
| 1065 |
+
"success": null,
|
| 1066 |
+
"text": "sed -n '1,120p' data/training/session_summarize_for_avi/scorecard_sft_avi_verified.jsonl.manifest.yml",
|
| 1067 |
+
"tool_name": "exec_command"
|
| 1068 |
+
},
|
| 1069 |
+
"label": 1,
|
| 1070 |
+
"source_event_id": "t14",
|
| 1071 |
+
"support_probability": 0.9987550973892212
|
| 1072 |
+
},
|
| 1073 |
+
{
|
| 1074 |
+
"event": {
|
| 1075 |
+
"command": "scripts/build-scorecard-sft-dataset.py --out data/training/session_summarize_for_avi/scorecard_sft_with_drafts.jsonl --include-drafts",
|
| 1076 |
+
"is_context_tool_echo": false,
|
| 1077 |
+
"role": "tool_call",
|
| 1078 |
+
"success": null,
|
| 1079 |
+
"text": "scripts/build-scorecard-sft-dataset.py --out data/training/session_summarize_for_avi/scorecard_sft_with_drafts.jsonl --include-drafts",
|
| 1080 |
+
"tool_name": "exec_command"
|
| 1081 |
+
},
|
| 1082 |
+
"label": 1,
|
| 1083 |
+
"source_event_id": "t10",
|
| 1084 |
+
"support_probability": 0.9986329674720764
|
| 1085 |
+
},
|
| 1086 |
+
{
|
| 1087 |
+
"event": {
|
| 1088 |
+
"command": "rg -n \"round-04-workbench|round-05-one-bash|round-06-qwen9\" data/training/session_summarize_for_avi/scorecard_sft_avi_verified.jsonl data/training/session_summarize_for_avi/scorecard_sft_avi_verified.jsonl.manifest.yml",
|
| 1089 |
+
"is_context_tool_echo": false,
|
| 1090 |
+
"role": "tool_call",
|
| 1091 |
+
"success": null,
|
| 1092 |
+
"text": "rg -n \"round-04-workbench|round-05-one-bash|round-06-qwen9\" data/training/session_summarize_for_avi/scorecard_sft_avi_verified.jsonl data/training/session_summarize_for_avi/scor...",
|
| 1093 |
+
"tool_name": "exec_command"
|
| 1094 |
+
},
|
| 1095 |
+
"label": 1,
|
| 1096 |
+
"source_event_id": "t15",
|
| 1097 |
+
"support_probability": 0.9969245791435242
|
| 1098 |
+
}
|
| 1099 |
+
]
|
| 1100 |
+
},
|
| 1101 |
+
{
|
| 1102 |
+
"expected_event": {
|
| 1103 |
+
"command": "*** Begin Patch\n*** Update File: /home/avifenesh/projects/aviary/data/runs/web_research_for_avi/20260503-github-web-research/codex-scorecards-draft.yml\n@@\n-review_status: needs_avi_review\n+review_status: avi_corrected_partial\n+reviewed_by: avi\n+review_note: >-\n+ Avi rated the main Aviary web_fetch result as 2-3. This is not a zero because\n+ it found the most-starred agent-sh repo, but it is still below usable because\n+ it missed the real open-work queue and over-presented incomplete research.\n@@\n - run_id: \"20260503-github-web-research/aviary-chat-once-webfetch\"\n model_label: qwen35-9b\n harness: aviary-chat-once\n case_id: round-01-github-open-issues-agent-sh-stars\n- decision: draft_negative_mixed_webfetch_result\n+ review_status: avi_corrected\n+ reviewed_by: avi\n+ decision: verified_low_mixed_webfetch_result_after_correction\n+ avi_score_range: \"2-3\"\n scores:\n- overall_usefulness: 2\n+ overall_usefulness: 2\n@@\n do_not_train_as_success: 1\n+ avi_feedback: >-\n+ I would give it 2-3.\n summary: >-\n Mixed but mostly negative web-research sample. The model used Aviary's\n web_search/web_fetch tools and correctly found `agent-sh/agentsys` as the\n most-starred repo with 777 stars. It failed the more important work-queue\n part: it did not inspect `avifenesh/*` properly, missed open\n `agent-sh/agnix` issues/PRs involving Avi, relied on shallow GitHub pages,\n- and presented incomplete findings too confidently.\n+ and presented incomplete findings too confidently. Avi corrected this to\n+ a 2-3 range; encode conservatively as 2 while preserving the range.\n*** End Patch\n",
|
| 1104 |
+
"is_context_tool_echo": false,
|
| 1105 |
+
"role": "tool_call",
|
| 1106 |
+
"success": null,
|
| 1107 |
+
"text": "apply_patch: *** Begin Patch\n*** Update File: /home/avifenesh/projects/aviary/data/runs/web_research_for_avi/20260503-github-web-research/codex-scorecards-draft.yml\n@@\n-review_s...",
|
| 1108 |
+
"tool_name": "apply_patch"
|
| 1109 |
+
},
|
| 1110 |
+
"expected_probability": 0.7799928784370422,
|
| 1111 |
+
"expected_source_event_id": "t7",
|
| 1112 |
+
"field": "attempted_actions",
|
| 1113 |
+
"rank": 19,
|
| 1114 |
+
"reciprocal_rank": 0.05263157894736842,
|
| 1115 |
+
"row_id": "pta-codex-rollout-2026-04-30T19-43-19-019ddf45-fdda-7fb2-8764-8162ba11dc24::w34",
|
| 1116 |
+
"top_candidates": [
|
| 1117 |
+
{
|
| 1118 |
+
"event": {
|
| 1119 |
+
"command": "git diff -- data/runs/web_research_for_avi/20260503-github-web-research/codex-scorecards-draft.yml",
|
| 1120 |
+
"is_context_tool_echo": false,
|
| 1121 |
+
"role": "tool_call",
|
| 1122 |
+
"success": null,
|
| 1123 |
+
"text": "git diff -- data/runs/web_research_for_avi/20260503-github-web-research/codex-scorecards-draft.yml",
|
| 1124 |
+
"tool_name": "exec_command"
|
| 1125 |
+
},
|
| 1126 |
+
"label": 1,
|
| 1127 |
+
"source_event_id": "t10",
|
| 1128 |
+
"support_probability": 0.999841570854187
|
| 1129 |
+
},
|
| 1130 |
+
{
|
| 1131 |
+
"event": {
|
| 1132 |
+
"command": "git diff --check",
|
| 1133 |
+
"is_context_tool_echo": false,
|
| 1134 |
+
"role": "tool_call",
|
| 1135 |
+
"success": null,
|
| 1136 |
+
"text": "git diff --check",
|
| 1137 |
+
"tool_name": "exec_command"
|
| 1138 |
+
},
|
| 1139 |
+
"label": 1,
|
| 1140 |
+
"source_event_id": "t11",
|
| 1141 |
+
"support_probability": 0.9997040629386902
|
| 1142 |
+
},
|
| 1143 |
+
{
|
| 1144 |
+
"event": {
|
| 1145 |
+
"command": "git add data/runs/web_research_for_avi/20260503-github-web-research/codex-scorecards-draft.yml && git commit -m \"Mark web research eval corrected\"",
|
| 1146 |
+
"is_context_tool_echo": false,
|
| 1147 |
+
"role": "tool_call",
|
| 1148 |
+
"success": null,
|
| 1149 |
+
"text": "git add data/runs/web_research_for_avi/20260503-github-web-research/codex-scorecards-draft.yml && git commit -m \"Mark web research eval corrected\"",
|
| 1150 |
+
"tool_name": "exec_command"
|
| 1151 |
+
},
|
| 1152 |
+
"label": 1,
|
| 1153 |
+
"source_event_id": "t16",
|
| 1154 |
+
"support_probability": 0.999555766582489
|
| 1155 |
+
},
|
| 1156 |
+
{
|
| 1157 |
+
"event": {
|
| 1158 |
+
"command": "rg -n \"review_status|aviary-chat-once|overall_usefulness|avi_feedback|decision|summary\" data/runs/web_research_for_avi/20260503-github-web-research/codex-scorecards-draft.yml",
|
| 1159 |
+
"is_context_tool_echo": false,
|
| 1160 |
+
"role": "tool_call",
|
| 1161 |
+
"success": null,
|
| 1162 |
+
"text": "rg -n \"review_status|aviary-chat-once|overall_usefulness|avi_feedback|decision|summary\" data/runs/web_research_for_avi/20260503-github-web-research/codex-scorecards-draft.yml",
|
| 1163 |
+
"tool_name": "exec_command"
|
| 1164 |
+
},
|
| 1165 |
+
"label": 1,
|
| 1166 |
+
"source_event_id": "t3",
|
| 1167 |
+
"support_probability": 0.999480664730072
|
| 1168 |
+
},
|
| 1169 |
+
{
|
| 1170 |
+
"event": {
|
| 1171 |
+
"command": "git status --short",
|
| 1172 |
+
"is_context_tool_echo": false,
|
| 1173 |
+
"role": "tool_call",
|
| 1174 |
+
"success": null,
|
| 1175 |
+
"text": "git status --short",
|
| 1176 |
+
"tool_name": "exec_command"
|
| 1177 |
+
},
|
| 1178 |
+
"label": 1,
|
| 1179 |
+
"source_event_id": "t18",
|
| 1180 |
+
"support_probability": 0.9994115829467773
|
| 1181 |
+
}
|
| 1182 |
+
]
|
| 1183 |
+
},
|
| 1184 |
+
{
|
| 1185 |
+
"expected_event": {
|
| 1186 |
+
"command": "*** Begin Patch\n*** Update File: /home/avifenesh/projects/aviary/data/evals/personal_agent_for_avi/scorecards.yml\n@@\n - pr_workflow_reference\n - bookkeeping_recovery_reference\n - shell_quoting_negative_reference\n - safe_merge_policy_reference\n+- id: score-20260504T144215-qwen27-agnix-claude-code-triage\n+ created_at: 2026-05-04T14:42:15+03:00\n+ rater: avi\n+ lane: personal_agent_for_avi\n+ run_id: 20260504-live-agnix-claude-code-triage\n+ model_label: aviary-local-qwen\n+ case_id: agnix-claude-code-issue-850-triage\n+ result_path: /home/avifenesh/projects/aviary/data/runs/personal_agent_for_avi/20260504-live-agnix-claude-code-triage/aviary-local-qwen/agnix-claude-code-issue-850-triage/input.jsonl\n+ input_path: /home/avifenesh/projects/aviary/data/runs/personal_agent_for_avi/20260504-live-agnix-claude-code-triage/aviary-local-qwen/agnix-claude-code-issue-850-triage/input.jsonl\n+ decision: useful_success_with_branch_and_quoting_failures\n+ scores:\n+ agentic_behavior: 4\n+ bookkeeping: 2\n+ grounding_no_hallucination: 4\n+ instruction_following: 4\n+ judgment_prioritization: 3\n+ latency_acceptability: 4\n+ long_context_handling: 4\n+ output_quality: 3\n+ overall_usefulness: 4\n+ recovery: 4\n+ safety_restraint: 3\n+ terminal_usage: 3\n+ tone_intent_understanding: 4\n+ tool_use: 3\n+ notes: >-\n+ Avi initially judged this as another useful session: helpful, fast, good\n+ terminal/tool work, and good task understanding. After review, Avi corrected\n+ that the agent did make a mistake. The material mistake was branch hygiene:\n+ it opened the Claude Code PR while still based on the previous OpenCode PR\n+ branch state, so PR #863 initially included old OpenCode commits and\n+ triggered stale Copilot/Gemini review comments. The agent also repeated the\n+ shell quoting bug from the previous session: backticks in a double-quoted\n+ `gh pr create --body` command were interpreted by the shell, leaving a\n+ mangled PR body. The final result was still useful because it recovered after\n+ Avi pointed out the wrong branch, rebased onto main, produced a clean diff,\n+ waited for all checks, and merged PR #863 with issue #850 closed.\n+ evidence:\n+ - The agent correctly found 13 open issues, separated 5 auto issues from 8 manual issues, and selected #850 as the first auto issue to process.\n+ - It correctly classified Claude Code v2.1.126 as agnix-irrelevant, updated the baseline and Last Reviewed date, ran rule bookkeeping, opened PR #863, waited for CI/reviews, and eventually merged it.\n+ - The agent created the new branch before updating local main; `git log main..HEAD` showed the previous OpenCode commits in the Claude Code PR.\n+ - Avi corrected the branch mistake with \"because you need to move to main you are on a wrong branch\"; the agent switched to main, pulled, rebased, and reduced the PR to one clean Claude Code commit.\n+ - PR #863 body still shows shell quoting damage in the Changes bullets because Markdown backticks were interpreted by the shell during `gh pr create`.\n+ - All checks passed before merge, PR #863 merged, and issue #850 closed.\n+ failure_modes:\n+ - started_pr_from_wrong_base_branch\n+ - stale_prior_pr_commits_included_initially\n+ - stale_bot_review_noise_from_dirty_base\n+ - shell_quoting_backticks_mangled_pr_body\n+ - did_not_repair_mangled_pr_body\n+ - repeated_same_pr_body_quoting_failure\n+ - gh_pr_check_typo\n+ - attempted_review_dismissal_with_wrong_api\n+ tuning_uses:\n+ - preference_dataset\n+ - eval_regression\n+ - judge_training\n+ - supervised_finetune_example\n+ - positive_agentic_terminal_reference\n+ - branch_hygiene_negative_reference\n+ - pr_workflow_recovery_reference\n+ - shell_quoting_negative_reference\n+ - owner_correction_recovery_reference\n*** End Patch\n",
|
| 1187 |
+
"is_context_tool_echo": false,
|
| 1188 |
+
"role": "tool_call",
|
| 1189 |
+
"success": null,
|
| 1190 |
+
"text": "apply_patch: *** Begin Patch\n*** Update File: /home/avifenesh/projects/aviary/data/evals/personal_agent_for_avi/scorecards.yml\n@@\n - pr_workflow_reference\n - bookkeeping_rec...",
|
| 1191 |
+
"tool_name": "apply_patch"
|
| 1192 |
+
},
|
| 1193 |
+
"expected_probability": 0.48121002316474915,
|
| 1194 |
+
"expected_source_event_id": "t10",
|
| 1195 |
+
"field": "attempted_actions",
|
| 1196 |
+
"rank": 19,
|
| 1197 |
+
"reciprocal_rank": 0.05263157894736842,
|
| 1198 |
+
"row_id": "pta-codex-rollout-2026-04-30T19-43-19-019ddf45-fdda-7fb2-8764-8162ba11dc24::w45",
|
| 1199 |
+
"top_candidates": [
|
| 1200 |
+
{
|
| 1201 |
+
"event": {
|
| 1202 |
+
"command": "cargo test eval_score --quiet",
|
| 1203 |
+
"is_context_tool_echo": false,
|
| 1204 |
+
"role": "tool_call",
|
| 1205 |
+
"success": null,
|
| 1206 |
+
"text": "cargo test eval_score --quiet",
|
| 1207 |
+
"tool_name": "exec_command"
|
| 1208 |
+
},
|
| 1209 |
+
"label": 1,
|
| 1210 |
+
"source_event_id": "t13",
|
| 1211 |
+
"support_probability": 0.9996849298477173
|
| 1212 |
+
},
|
| 1213 |
+
{
|
| 1214 |
+
"event": {
|
| 1215 |
+
"command": "tail -n 20 data/evals/personal_agent_for_avi/scorecards.yml",
|
| 1216 |
+
"is_context_tool_echo": false,
|
| 1217 |
+
"role": "tool_call",
|
| 1218 |
+
"success": null,
|
| 1219 |
+
"text": "tail -n 20 data/evals/personal_agent_for_avi/scorecards.yml",
|
| 1220 |
+
"tool_name": "exec_command"
|
| 1221 |
+
},
|
| 1222 |
+
"label": 1,
|
| 1223 |
+
"source_event_id": "t5",
|
| 1224 |
+
"support_probability": 0.9995827078819275
|
| 1225 |
+
},
|
| 1226 |
+
{
|
| 1227 |
+
"event": {
|
| 1228 |
+
"command": "date --iso-8601=seconds",
|
| 1229 |
+
"is_context_tool_echo": false,
|
| 1230 |
+
"role": "tool_call",
|
| 1231 |
+
"success": null,
|
| 1232 |
+
"text": "date --iso-8601=seconds",
|
| 1233 |
+
"tool_name": "exec_command"
|
| 1234 |
+
},
|
| 1235 |
+
"label": 1,
|
| 1236 |
+
"source_event_id": "t4",
|
| 1237 |
+
"support_probability": 0.999555766582489
|
| 1238 |
+
},
|
| 1239 |
+
{
|
| 1240 |
+
"event": {
|
| 1241 |
+
"command": "python3 - <<'PY'\nfrom pathlib import Path\nimport yaml\np=Path('data/evals/personal_agent_for_avi/scorecards.yml')\ndata=yaml.safe_load(p.read_text())\nprint(data['lane'], len(data['scorecards']))\nfor c in data['scorecards']:\n print(c['id'], c['decision'], c['scores'].get('overall_usefulness'), c['scores'].get('bookkeeping'))\nPY",
|
| 1242 |
+
"is_context_tool_echo": false,
|
| 1243 |
+
"role": "tool_call",
|
| 1244 |
+
"success": null,
|
| 1245 |
+
"text": "python3 - <<'PY'\nfrom pathlib import Path\nimport yaml\np=Path('data/evals/personal_agent_for_avi/scorecards.yml')\ndata=yaml.safe_load(p.read_text())\nprint(data['lane'], len(data[...",
|
| 1246 |
+
"tool_name": "exec_command"
|
| 1247 |
+
},
|
| 1248 |
+
"label": 1,
|
| 1249 |
+
"source_event_id": "t14",
|
| 1250 |
+
"support_probability": 0.9985449314117432
|
| 1251 |
+
},
|
| 1252 |
+
{
|
| 1253 |
+
"event": {
|
| 1254 |
+
"command": "python3 - <<'PY'\nfrom pathlib import Path\nimport yaml\nroot=Path('/home/avifenesh/projects/aviary')\ntotal=0\nfor p in sorted((root/'data/evals').glob('*/scorecards.yml')):\n data=yaml.safe_load(p.read_text()) or {}\n n=len(data.get('scorecards') or [])\n total+=n\n print(data.get('lane') or p.parent.name, n)\nprint('total', total)\nPY",
|
| 1255 |
+
"is_context_tool_echo": false,
|
| 1256 |
+
"role": "tool_call",
|
| 1257 |
+
"success": null,
|
| 1258 |
+
"text": "python3 - <<'PY'\nfrom pathlib import Path\nimport yaml\nroot=Path('/home/avifenesh/projects/aviary')\ntotal=0\nfor p in sorted((root/'data/evals').glob('*/scorecards.yml')):\n dat...",
|
| 1259 |
+
"tool_name": "exec_command"
|
| 1260 |
+
},
|
| 1261 |
+
"label": 1,
|
| 1262 |
+
"source_event_id": "t15",
|
| 1263 |
+
"support_probability": 0.9928786158561707
|
| 1264 |
+
}
|
| 1265 |
+
]
|
| 1266 |
+
}
|
| 1267 |
+
]
|
| 1268 |
+
},
|
| 1269 |
+
"train": "data/tmp/mixed_mode_train.v2.field_decision.jsonl",
|
| 1270 |
+
"train_candidate_pairs": 63752,
|
| 1271 |
+
"train_groups": 5882,
|
| 1272 |
+
"train_rows": 1217
|
| 1273 |
+
}
|