synthetic CoT reconstruction and agent trajectory design
Dear author,
Before asking detailed questions, I would like to confirm whether it is appropriate to publicly discuss aspects of the training pipeline, and whether any parts of the dataset or methodology are publicly available or fully proprietary. If some components are not intended for public disclosure, I fully understand.
I am particularly interested in the following three aspects, which seem closely related to the performance improvement from v1 to v2 (especially the tau2-bench increase from ~15% to ~55%):
Opus 4.8 CoT reconstruction for Fable5 traces
How was Opus 4.8 used to reconstruct or rewrite the original Fable5 traces?
Specifically, was it a full regeneration of reasoning from prompts/tasks, or a step-level rewriting, completion, or normalization of existing trajectories?Data cleaning and filtering pipeline
What are the main heuristics or filtering strategies used in processing synthetic data or agent trajectories?
For example (but not limited to):
invalid tool calls
incomplete read–act loops
hallucinated states
execution-based verificationDynamic context-window mechanism (training organization)
I would like to better understand the “dynamic context-window pass” used to enforce read-before-act structure in v2.
Is the ordering constraint mainly enforced at the data level (via re-segmentation / trajectory splitting), or at the training-time context construction level (dynamic window / packing policy)?
In other words, is ACT explicitly conditioned on a filtered context that guarantees prior READ/OBSERVE steps, or is this structure mainly emergent from how trajectories are segmented and packed?
Motivation
I am particularly interested in how these components interact to produce the significant improvement in agent performance (e.g., tau2-bench telecom task from ~15% to ~55%).
Any high-level clarification would be extremely valuable for academic understanding.
Thank you very much for your time and for open-sourcing this work.
@exxpre Thank you — genuinely — for asking about disclosure first rather than assuming. That's rare, and appreciated. Honest answer: I'm keeping the full data pipeline and the exact recipes private for now (I may write more up around a later release), so I'll stay at a high level here — but I'll be clear about where things are still evolving rather than paint it cleaner than it is.
At a high level, on your three points:
- Opus 4.8 CoT reconstruction. When Fable 5 was retired, only my own set still held genuinely self-authored reasoning. For the community-contributed trajectories, the reasoning was reconstructed with Opus 4.8 — closer to rebuilding reasoning to fit the existing action trajectories than free-generating tasks from scratch. I keep that reconstructed material clearly separated from the authentic-CoT pool and don't treat it as ground truth; as I note in the card, it can diverge from the original traces.
- Cleaning / filtering. A mix of structural hygiene (stripping harness/logging noise, dropping degenerate or empty turns, dedup, normalizing tool calls into the model's native format) and, where the task domain allows, execution-based verification — the coding portion is gated on passing tests. I won't overstate it: the agentic-trajectory side is more heuristic than the coding side, not a fully execution-verified agentic pipeline yet.
- Read-before-act / context window. Of your two framings, it's mainly the second: a data-level re-segmentation. The ordering comes from how trajectories are segning pairs, rather than an explicittraining-time construction that formally guarantees prior READ/OBSERVE for every ACT. And candidly — making that conditioning more robust is one of the specific things I'm tightening for v3.
On the tau2 figure: I'd gently caution against reading hard causal attribution into it. These are local, same-harness, relative measurements (all Q8_0, self-simulated user), not directly comparable to the published tau2-bench leaderboard, and local self-eval tends to run systematically lower. The improvement is real, but I treat the exact number as a relative signal, not a precise benchmark claim.
Happy to go a little deeper on any of these at the same altitude. Thanks again for engaging with the work so carefully — and for the courtesy of asking first.