AUDIT_REPORT.md

TBV NewsImpact v1 — Acımasız Denetim Raporu

Audit date: 2026-05-13 Scope: all claims in the project session brief, against local artifacts and code. Coverage: local pipeline.py (1,731 lines), all v2-v6 runners, FINAL_SESSION_REPORT.md, news_bbc_guardian.parquet (9.6M rows), 5 rescued chart parquets. Out of scope (UNKNOWN): Drive-hosted .pt checkpoints, attributed_shard_*.parquet, 1,535 chart parquets, pool.json, final_report.json, events log. Torch/transformers not installed locally, so checkpoint loading and inference smoke tests were prepared as a script (audit/audit_inference_skeleton.py) for the operator to run on Colab.

---

1. Executive Summary

Verdict: NOT TRUSTED YET.

The project's narrative — "v6 winner, encoder choice is not the bottleneck, the data/label noise is the floor" — is technically true but for the wrong reason. There are two simultaneous, mutually-reinforcing failures that fully account for the observed plateau, and neither is the data:

1. Cross-attention fusion is mathematically a no-op. With single-token query/key/value sequences, the softmax collapses to identity, so the text encoder output is bit-exactly discarded inside HybridNewsImpactModel.forward. The model is effectively a chart-only classifier wearing a 336M-parameter text encoder as decoration. (pipeline.py:1119-1124. See proof in audit/cross_attention_collapse_proof.md.)
2. The "4h" chart dir is half-daily. fetch_yfinance_daily returns DAILY bars, fetch_ccxt_4h returns 4h bars; both are written to charts_4h/. compute_impact_label's 24h pre-window catches at most 1 daily bar → pct_change().std() is NaN → label silently rejected. So labels exist only for crypto symbols. (pipeline.py:349, 625, 1029-1069.)

These two facts jointly explain every plateau observation in the iteration scoreboard:

• "v2-v6 cluster at 2.495" → it's the chart-only loss floor.
• "encoder choice doesn't matter" → encoder output is mathematically unused.
• "sig_acc = 0.557 = majority class" → chart-only signal is too weak to separate significant from non-significant news (because the chart doesn't know whether the news is bearish, bullish, or unrelated).
• "v6 winner by Δ=0.0004 over v4" → run-to-run noise of effectively the same chart-only model, not an architecture/encoder effect.

Other notable findings:

3. 36.3% of news rows have null dates. Cannot be labeled. In the first row group, every BBC entry (29,467/29,467) had a null date — strong ingestion-side data corruption.
4. News source split is BBC 73.8% / Guardian 26.2%, not the balanced "BBC + Guardian" framing.
5. "27 symbol coercions" claim is off by one — there are 26, and the table contains a no-op (^MERV→^MERV) and a circular pair (^TPX↔^TOPX).
6. merge_shards_and_test does naive weight averaging of independently trained shards without re-evaluating the merged model. The reported mean_val_loss is the mean of per-shard pre-merge val losses, not the merged-model performance. The merged model's actual test loss is unknown.
7. No seed is set anywhere in train_one_shard or runners. Reproducibility is non-existent; a Δ of 0.0004 between two runs is not distinguishable from noise without seed-controlled repeats, which the project did not perform.
8. No test files exist for the entire pipeline.
9. "reason_emb" is a 1024-dim embedding, not a human-readable explanation. The product goal "explain why a piece of news matters" requires a decoder, retrieval scheme, or LLM post-processor — none exists in the current code.
10. NaN-guard in train_one_shard has a subtle scaler bug: scaler.update() is called after a skipped step without scaler.scale().backward() first, which can corrupt the AMP scaler's internal scale factor.

The architecture and label bugs are root causes; everything else is symptomatic.

---

2. Claim Verification Matrix

Categories: A=artifact, B=size/hash, C=arch, D=params, E=frozen, F=ckpt, M=metric, V=data, L=label, S=split, T=training, I=inference, R=repro, P=perf-claim, X=prod, O=ops.

ID	Claim	Cat	Status	Evidence	Confidence	Sev-if-False
C01	pipeline.py ≈ 1670 lines	A	PARTIAL	1731 actual (≈4% off)	1.0	LOW
C02	pipeline.py syntactically valid	A	PASS	py_compile passed	1.0	HIGH
C03	All v2-v6 runner files present locally	A	PASS	All 5 files exist	1.0	LOW
C04	news_bbc_guardian.parquet ≈ 1.1 GB	B	PASS	1.07 GiB (1.12 GB)	1.0	LOW
C05	News parquet has 9.6M rows	B	PASS	9,646,340 rows	1.0	MED
C06	News source = BBC + Guardian	V	PARTIAL	Yes both, but 73.8/26.2 imbalanced	1.0	LOW
C07	_SYMBOL_COERCIONS has 27 entries	A	FAIL	26 entries, includes identity + circular	1.0	LOW
C08	14 delisted symbols skipped	A	PASS	_DELISTED_SKIP has 14 entries	1.0	LOW
C09	5 chart files locally rescued	A	PASS	5 parquets in rescued/	1.0	LOW
C10	Rescued charts have valid OHLCV	V	PASS	All invariants hold (1 minor low-close violation in URA_F)	1.0	MED
C11	All charts are "4h OHLCV"	V	FAIL	Yfinance-fetched are daily; only ccxt are 4h. charts_4h/ is heterogeneous.	1.0	CRITICAL
C12	compute_impact_label produces labels for non-crypto symbols	L	FAIL	Daily-bar pre-window yields NaN std → label rejected	1.0	CRITICAL
C13	Significance defined as	return	/pre_vol > 3.0	L	PASS	pipeline.py:1068
C14	sig threshold is dimensionally consistent	L	FAIL	pre_vol is per-4h-bar std; numerator is 48h return → off by √12	0.95	HIGH
C15	Direction is 3-class up/flat/down	L	PARTIAL	Defined as 3-class but sign(continuous return) is almost never 0 → effectively 2-class	1.0	MED
C16	HybridNewsImpactModel uses cross-attention to fuse text and chart	C	FAIL	Single-token MHA collapses softmax to 1.0 → text contribution = 0	1.0	CRITICAL
C17	head_sig dim = 2, head_dir = 3, head_mag = 1, head_reason = 1024	C	PASS (code)	pipeline.py:1114-1117. head_reason output dim = text_dim = 1024 for roberta-large	1.0	MED
C18	Fusion is 8-head 1024-dim cross-attention	C	PARTIAL	Configured as 8-head 1024-dim, but functionally degenerate (see C16)	1.0	CRITICAL
C19	Chart encoder operates on 128-step 4h OHLCV	C	PARTIAL	chart_window=128 OK; but actual feed is mixed daily/4h (see C11)	1.0	HIGH
C20	Trainable params ≈ 6.2M	D	UNKNOWN	Cannot verify w/o torch+ckpt. Skeleton script H2 will measure	n/a	MED
C21	Text encoder is FROZEN in v6	E	PASS (code)	v6_runner.py:16 + train_one_shard signature wires freeze_text	1.0	LOW
C22	Only chart_enc + fuse + heads trained in v6	E	PARTIAL	Code path supports it. But because of C16, text encoder is also effectively unused at inference	1.0	CRITICAL
C23	final_best.pt exists at Drive path	A	UNKNOWN	No Drive access	n/a	HIGH
C24	final_best.pt is ≈ 1.446 GB	B	UNKNOWN	No Drive access	n/a	LOW
C25	final_best.pt loads strict=True against HybridModel	F	UNKNOWN	Operator must run audit_inference_skeleton.py H1	n/a	HIGH
C26	Validation loss = 2.4951 reported for v6	M	PARTIAL	Cannot verify exact value; but reported value is per-shard val loss averaged across shards, NOT post-merge test loss (pipeline.py:1482)	0.95	HIGH
C27	v6 beats v4 by 0.0004 = meaningful improvement	M	FAIL	Selection logic is if v6_val < v4_best w/ no statistical test. No seed set. Within run-to-run noise.	0.95	HIGH
C28	Iteration scoreboard plateau ≈ 2.495 ± 0.005	M	PARTIAL	Cannot verify exact values, but if true it is consistent with the chart-only loss floor — not a data/label floor	0.9	HIGH
C29	"Data/label noise is the main bottleneck"	M	FAIL	The bottleneck is the architecture (C16) and the daily/4h mismatch (C11/C12). Label noise is also present but is downstream.	0.95	HIGH
C30	Train/val split is temporal (last 10%)	S	PASS	pipeline.py:1258-1261 sorts by t then slices	1.0	n/a
C31	No train/val leakage across shards	S	PARTIAL	Within shard: temporal, OK. Across shards: shard 0 train can overlap in time with shard 1's val (interleaved by iloc[shard::num_shards]), but per-shard validation is still independent	0.9	MED
C32	Cron jobs disabled at session close	O	PASS	FINAL_SESSION_REPORT.md:111	1.0	LOW
C33	Cloudflared tunnel still up	O	UNKNOWN	Local machine state — not directly verifiable	n/a	MED
C34	Vectorized attribution gives 18× speedup (80→4 min)	P	PARTIAL	The current code is vectorized (single big-regex findall). Old version not present for direct comparison. 18× ratio not independently verifiable	0.7	LOW
C35	Cross-attention seed-controlled training	R	FAIL	No torch.manual_seed, random.seed, or DataLoader worker seed anywhere in the training pipeline	1.0	HIGH
C36	"reason_emb" output is a human-readable explanation	I	FAIL	head_reason: Linear(text_dim, text_dim) → 1024-dim float vector. No decoder, no template, no retrieval mechanism in code	1.0	HIGH
C37	NaN/inf guard in training is correct	T	PARTIAL	Guard exists (lines 1363-1371) but skips scaler.scale(loss).backward() while still calling scaler.update() — can corrupt AMP scaler state	0.95	MED
C38	Test files exist	X	FAIL	No test_*.py, *_test.py, or conftest.py in project	1.0	MED
C39	News dates parse cleanly	V	FAIL	36.3% of date column is null/unparseable; appears to systematically affect BBC	1.0	HIGH
C40	"9 exchanges, 3,159 symbols" pool	V	UNKNOWN	pool.json on Drive, not accessible	n/a	LOW
C41	1,535 chart files	V	UNKNOWN	charts_4h/ on Drive, not accessible	n/a	LOW
C42	1,619 crypto fails = DEX-only	V	UNKNOWN	Cannot verify w/o chart_manifest.json	n/a	LOW
C43	Inference snippet works as written	I	PARTIAL (untested)	Constructor signature compatible; state.get('model', state) should work; but reason_emb is not an explanation as documented	0.85	HIGH

---

3. Critical Findings (sorted by severity)

CRITICAL-1: Cross-Attention Fusion is a Mathematical No-Op

Where: pipeline.py:1119-1124 inside HybridNewsImpactModel.forward.

Evidence:

fused, _ = self.fuse(t_pool.unsqueeze(1),   # (B, 1, D) query
                     c_pool.unsqueeze(1),    # (B, 1, D) key
                     c_pool.unsqueeze(1))    # (B, 1, D) value

With single-token Q/K/V, softmax over a 1-element score vector is identically 1.0, so attention weights cannot modulate the value. The output is f = W_o · W_v · c_pool + b_o, a pure linear function of the chart pool. The text encoder, the W_q projection, the W_k projection — all dead weights at the forward-pass level. The text encoder's gradients (when not frozen) train only the encoder's input distribution, never the prediction surface. Full proof in audit/cross_attention_collapse_proof.md.

Why it matters: This single bug invalidates every interpretive claim in the session report. The plateau is the chart-only loss floor; the encoder indifference is trivially true because the encoder is unused; the v6 "winner" status is a noise sample of essentially the same chart-only model evaluated twice. The entire 6.2M-trainable-parameter task layer is being trained to map a chart pool to (sig, dir, mag, reason) — without news.

Fix: Either concatenate-then-project, or have ChartEncoder.forward return a (B, T, D) sequence (don't pool with .mean(dim=1)) and use cross-attention with chart as the key/value sequence and text pool as a single query token. See audit/cross_attention_collapse_proof.md § "The fix".

Recurrence prevention: Add an automated test:

def test_text_input_affects_output():
    out_a = model(tokenize("A"), ..., chart)
    out_b = model(tokenize("B"), ..., chart)
    assert (out_a['sig_logits'] - out_b['sig_logits']).abs().max() > 1e-6

---

CRITICAL-2: charts_4h/ Contains Daily Bars; Label Function Silently Rejects Them

Where: pipeline.py:349 (fetch_yfinance_daily) writes daily bars into charts_4h/; pipeline.py:1043-1048 (compute_impact_label) computes pre['close'].pct_change().std() over a 24h pre-window.

Evidence:

• The function is named fetch_yfinance_daily and uses yf.download(...) with default interval='1d'. So one row per market day.
• All 5 locally-rescued files (URA_F, ETHUSD_X, FBHS, XAG_F, idx_IBOV) measured on disk show date diff mode = 1 day, median = 1 day, min = 1 day. Daily, not 4h.
• Only fetch_ccxt_4h explicitly requests '4h' (line 650).
• fetch_all_charts task selector (line 690): non-crypto → fetch_yfinance_daily.
• For daily data, a 24h pre-window matches at most 1-2 bars. pct_change of a 1-element series returns one NaN; .dropna().std() returns NaN. Label is rejected at line 1051 (pd.isna(pre_vol)).

Why it matters: The training corpus is implicitly filtered to crypto symbols only. Equities, FX, indices, commodities — all contribute zero labels, silently. The session's narrative of broad multi-market modeling is not supported by the data flow.

Fix: Either resample yfinance daily to 4h with a documented assumption (intraday infill is impossible from EOD data — really, use interval='1h' where supported), or shorten pre_window to e.g. 3 days when chart resolution is daily. Make the resolution mismatch loud, not silent: count rejections by reason in build_training_pairs.

Recurrence prevention: In compute_impact_label, log every rejection reason; emit a per-shard breakdown of (symbol, n_news, n_labels_produced, rejection_reasons). Today this information is lost.

---

CRITICAL-3: Reported mean_val_loss is Pre-Merge Per-Shard, Not Merged-Model Performance

Where: pipeline.py:1482 writes 'mean_val_loss': float(np.mean([m['best_val'] for m in shard_models])) to final_report.json. No re-evaluation of the merged model occurs.

Evidence: merge_shards_and_test takes the mean of per-shard best_val (line 1482), writes the soft-averaged state dict to final.pt (line 1474), and returns. There is no held-out test set used to evaluate the merged model.

Why it matters: Soft weight-averaging of independently-trained models often degrades performance — each model has a different loss basin and averaging weights places the merged point off-basin. The reported "2.4951" describes the average performance of four models before they were averaged together, not the performance of the model file that gets shipped as final_best.pt. The "winner" comparison between v4 and v6 may therefore not reflect the actual deployed-model quality.

Fix: Reserve a global held-out set up-front (e.g., a temporal hold-out of the last 5% across all shards, never seen during training). After merge_shards_and_test, evaluate the merged model on this set and report both numbers: mean-of-shards-val and merged-on-holdout.

---

HIGH-1: Significance Threshold is Dimensionally Inflated

Where: pipeline.py:1043-1068 in compute_impact_label.

Evidence:

• pre_vol = pre['close'].pct_change().std() over a 24h pre-window of 4h bars = std of 6 4h-bar returns. Unit: per-4h-bar std.
• excess = |post_return| / pre_vol where post_return = close_post/close_t - 1 measured over 48h.
• For 48h (=12 bars of 4h), the expected std-equivalent is pre_vol * √12 ≈ pre_vol * 3.46. So a 1-σ 48h move has excess ≈ 3.46, which already crosses the threshold of 3.0.

Why it matters: The "significant" rate is materially higher than the model authors intended. Many "ordinary" 48h moves are mis-labeled as significant, which is consistent with sig_acc plateauing at majority class (~55%): if ~55% of pairs are labeled significant, predicting "significant always" gets you 55% accuracy without learning anything.

Fix: Use volatility scaling: excess = |post_return| / (pre_vol * √(post_horizon/bar_size)). Or use a separate post-window-matched volatility (rolling 48h std of 4h bars). Or set threshold dynamically to the 90th percentile of excess per symbol.

---

HIGH-2: News Date Column 36.3% Null; BBC Dates Systematically Missing

Where: news_bbc_guardian.parquet date column (string-typed).

Evidence: Full scan over all 10 row groups: 3,504,560 nulls / 9,646,340 = 36.3%. In first row group, BBC had exactly 29,467 entries — identical to the date-null count in that row group — suggesting every BBC date in rg0 is null.

Why it matters: compute_impact_label requires a parseable timestamp (news_row['date'], used to select pre/post chart windows). Null-date rows are unusable. Combined with the daily-bar issue, the effective training corpus is much smaller than the 9.6M-row corpus claim suggests.

Fix: Re-parse the BBC ingestion. If the upstream source provides timestamps, something is dropping or misparsing them. Until re-parsed, drop null-date rows loudly at ingestion, not silently at label time.

---

HIGH-3: No Seed Control, Cannot Distinguish v6 from v4

Where: pipeline.py:train_one_shard (no manual_seed calls). Same for runners.

Evidence: grep -n "manual_seed\|random.seed\|np.random.seed" pipeline.py v*_runner.py returns nothing.

Why it matters: Two identical-config runs of a stochastic optimizer with random data shuffling and dropout will produce different val losses, typically Δ ≈ 0.005-0.02 for a converged model. The Δ=0.0004 between v4 and v6 is far inside this noise band. Without seed-controlled triplicate runs, no claim of "winning" is supportable.

Fix: Seed at the top of every runner:

torch.manual_seed(42); random.seed(42); np.random.seed(42)
torch.backends.cudnn.deterministic = True

Then run each config 3 times with seeds {42, 1337, 7} and report mean ± std.

---

HIGH-4: reason_emb is Not an Explanation

Where: pipeline.py:1117 self.head_reason = nn.Linear(text_dim, text_dim).

Evidence: A 1024-dim float vector is not interpretable. The product spec ("explain why a piece of news matters") needs either:

• A retrieval-based scheme (cosine-match against a labeled bank of rationales).
• A generative decoder (small LM conditioned on f).
• A feature-attribution view (e.g., Integrated Gradients of sig_logits over input tokens).

Additionally: the training loss does not appear to supervise head_reason at all. train_one_shard only computes loss over sig_logits, dir_logits, and mag. There is no target embedding being regressed against. So head_reason is trained only by gradient flowing through… no path, because nothing reads it. The weights of head_reason remain at initialization for the entire training.

Fix: Either remove the head entirely (it is dead code), or train it. To train it, supervise it against a meaningful target — e.g., a sentence-encoder embedding of a per-pair rationale ("oil-sector earnings beat" → encoded with mpnet), retrieved by manual labeling or a small LLM.

---

HIGH-5: AMP Scaler State Corruption in NaN-Guard

Where: pipeline.py:1363-1371.

Evidence:

if not torch.isfinite(loss):
    nan_skip_count += 1
    optim.zero_grad(set_to_none=True)
    scaler.update()       # <-- called without scaler.scale(loss).backward()
    ...
    continue

The recommended pattern in pytorch docs is: if you skip a step, you must NOT call scaler.update() because the scaler's growth tracker counts only valid steps. Calling update() after a skip prematurely grows the loss scale, making subsequent NaN events more likely.

Why it matters: Once you've started accumulating NaN events (which the code itself notes happen — "50 events at a time"), this guard accelerates the problem rather than fixes it. Training stability is at risk.

Fix: Remove the scaler.update() call inside the NaN branch. Pytorch's recommended pattern:

if not torch.isfinite(loss):
    optim.zero_grad(set_to_none=True)
    continue  # Do NOT touch the scaler.

---

MEDIUM-1: Direction is 3-Class but Effectively 2-Class

Where: pipeline.py:1067 'direction': int(np.sign(rret)); then pipeline.py:1313 int(r['direction']) + 1 maps to CE labels {0, 1, 2}.

Evidence: np.sign(continuous_return) is essentially never exactly 0. So the "flat" class is never observed. CrossEntropyLoss over a 3-class head with class 1 (flat) absent biases the head's logits to two-class behavior with a constant offset for the unused class.

Why it matters: Wastes capacity; dir_acc can plateau at the larger of the up/down class proportions without learning anything.

Fix: Either make direction 2-class (up/down) or define "flat" as |return| < 0.5 * pre_vol. Currently dir is conceptually 3-class but practically 2-class with a deadweight third class.

---

MEDIUM-2: Symbol Coercion Table is Off-by-One and Sloppy

Where: pipeline.py:307-338.

Evidence:

• Claim: "27-entry _SYMBOL_COERCIONS".
• Actual: 26 entries (counted programmatically).
• One identity: '^MERV': '^MERV' (line 311) — does nothing.
• One circular pair: '^TPX': '^TOPX' (line 310) and '^TOPX': '^TPX' (line 337). Apply the coercion twice and you cycle.

Why it matters: Symptomatic of unreviewed copy-paste edits to a shared mapping. The circular pair could be a latent bug if any code path applies coercion iteratively (it doesn't, but the door is open).

Fix: Pick one canonical Tokyo Topix symbol and route both ^TPX and ^TOPX to it. Drop '^MERV': '^MERV' (or change to comment that it's an explicit no-op). Update claim to 27 actual entries.

---

MEDIUM-3: No Test Files Exist

Where: Whole project.

Evidence: find for test_*.py, *_test.py, conftest.py returns nothing under merged_news/.

Why it matters: Future regressions (especially in compute_impact_label and HybridNewsImpactModel.forward) will go undetected. The two CRITICAL bugs above would have been caught by 10 lines of pytest each.

Fix: Add tests/test_pipeline.py with the recurrence-prevention tests listed in each CRITICAL finding.

---

LOW-1: Deprecated APIs

• torch.cuda.amp.autocast / torch.cuda.amp.GradScaler — deprecated since pytorch 2.0 in favor of torch.amp.autocast('cuda', ...). Currently emits DeprecationWarning at runtime.
• datetime.datetime.utcnow() (pipeline.py:46, 1484) — deprecated since python 3.12 in favor of datetime.datetime.now(timezone.utc). Currently emits DeprecationWarning.

Fixes are mechanical and one-line each.

---

4. Model Architecture Verification

Component	Declared	Actual (from code)	Compatible with final_best.pt?
Text encoder	roberta-large 336M FROZEN	✓ wired correctly when freeze_text=True	UNKNOWN (no ckpt access)
Chart encoder	small Transformer over 128-step OHLCV	ChartEncoder: in=5, hidden=128, layers=4, nhead=8, out=text_dim. Pools with mean(dim=1).	UNKNOWN
Fusion	8-head cross-attention 1024-dim	nn.MultiheadAttention(1024, 8, batch_first=True) — but collapses to identity-on-value due to T=1	UNKNOWN
head_sig	Linear(1024, 2)	✓	UNKNOWN
head_dir	Linear(1024, 3)	✓	UNKNOWN
head_mag	Linear(1024, 1)	✓	UNKNOWN
head_reason	Linear(1024, 1024)	✓ — but receives no loss signal, weights remain at init	UNKNOWN
Trainable params	6.2M	~6.5M expected (ChartEncoder ≈ 0.6M + fuse ≈ 4.2M + 4 heads ≈ 1.1M). Run script H2 to verify.	UNKNOWN

The architecture as written matches the claims at the layer-construction level. The collapse bug is a forward-pass semantics issue, not a layer-shape issue. Hence final_best.pt likely loads strict=True without errors; the silent failure is in the data flow, not the weight shapes.

---

5. Data & Label Audit

News distribution

Property	Value
Total rows	9,646,340
Columns	4 (title, summary, date, source) — all large_string
Date range	1999-01-01 → 2026-05-03
Source: The Guardian	2,525,498 (26.2%)
Source: BBC	7,120,842 (73.8%)
Date-null rows	3,504,560 (36.3%)
Date-null rows in row-group 0	29,467 = exactly the BBC count in rg0 (suspicious)
First-rg duplicate titles	152,920 / 1,048,576 = 14.6%
First-rg duplicate (title, date)	2,052 (0.2%)

Effective training corpus (after losses):

9.6M raw news
  - 3.5M null-date (-36.3%)             → 6.1M
  - news that mentions no symbol         → unknown
  - news mentioning non-crypto symbols   → labels rejected by 24h/daily issue
  ≈ "small fraction of crypto news, before deduplication"

Chart data (5 rescued, only sample available)

File	Rows	Date range	Resolution	OHLCV invariants
ETHUSD_X	3,107	2017-11 → 2026-05	daily	clean
FBHS	3,107	2014-01 → 2026-05	daily	clean
URA_F	3,107	2014-01 → 2026-05	daily	1 minor low>close violation
XAG_F	3,107	2014-01 → 2026-05	daily	219 zero-volume bars
idx_IBOV	3,064	2014-01 → 2026-05	daily	30 zero-volume bars

All five are daily, contradicting the directory name charts_4h/. Zero-volume bars for futures (XAG_F) and indices (IBOV) are expected (holidays, low-liquidity sessions) and not bugs — but pre_vol computed across these bars could include zero-return periods that artificially shrink volatility.

Label trace: what happens to a typical equity news event?

1. News at 2024-03-15 14:30 UTC: "Apple beats earnings".
2. attribute_news_to_symbols matches "Apple" → symbol AAPL.
3. build_training_pairs reads charts_4h/AAPL.parquet — daily bars.
4. compute_impact_label finds:
   • pre window (2024-03-14 14:30 → 2024-03-15 14:30): up to 1 daily bar (the 2024-03-15 EOD bar may be tomorrow's close, hence pre window may have 0-1 bars depending on UTC timing of EOD).
   • pre['close'].pct_change().std() over 0-1 values → NaN.
   • if pd.isna(pre_vol): return {} → labeled rejected.
5. Pair contributes nothing to training. AAPL effectively contributes nothing.

Label trace: a crypto news event

1. News at 2024-03-15 14:30 UTC: "Bitcoin surges past $70k".
2. Matches symbol BTC/USDT (via crypto-augmented aliases).
3. Reads ccxt-fetched 4h bars.
4. pre window 24h = 6 bars; pct_change → 5 valid returns; std OK.
5. excess = |48h return| / 4h-bar-std.
6. With BTC's typical 4h vol of ~1-2% and 48h returns often in the 2-5% range, excess ≈ 1.5-5. Many will cross threshold 3.0. Significance rate ~50% → majority class.

This trace matches the observed sig_acc plateau of 0.557 — a chart-only model on a crypto-heavy corpus with sig threshold inflated by dimensional mismatch.

---

6. Metric Audit

What does the loss decompose to?

val_loss = ce(sig) + ce(dir) + 0.5 * smoothl1(mag)

Baseline reference values:

Baseline	ce(sig)	ce(dir)	0.5·smoothl1(mag)	total
Random (uniform)	ln 2 ≈ 0.693	ln 3 ≈ 1.098	~0.5 * E[	mag
Majority class sig (55.7%)	≈ 0.685	—	—	—
Majority class dir (probably ~50%)	—	≈ 0.693	—	—
Chart-only model (observed)	?	?	?	2.495

The total of 2.495 is consistent with: majority-class sig (0.69) + slightly better than majority dir (1.0) + a partially trained mag head (~0.8 × 0.5 = 0.4), summed gives ~2.09 — under 2.495. So the model is probably worse than that decomposition would suggest, or the mag loss dominates. Without per-component val losses in the report, this can't be pinned down.

v6 vs v4: is 0.0004 meaningful?

No. Without seed-controlled triplicates:

• Two-sample t-test requires n ≥ 2 per condition with std estimate.
• The session ran each config once. Sample size = 1.
• Run-to-run std of converged validation loss on this kind of model is typically 0.01-0.05.
• Δ = 0.0004 is two orders of magnitude below the noise floor.

Verdict: "v6 winner" is a coin flip with extra steps.

Suggested baselines that were NOT run

These would distinguish "model learns from news" from "model learns prior":

• Chart-only ablation — drop text encoder entirely (or pass random text); measure val loss. If equal to current v6, the model isn't using text. (Given C16, this baseline is guaranteed equal.)
• Text-only ablation — drop chart encoder; measure val loss. If much higher, news semantics are useful; if equal, news isn't being used.
• Permuted-label baseline — shuffle labels within a shard before training. If val loss drops similarly, the model is learning train/val correlations unrelated to labels.
• Permuted-text baseline — pair each chart with a randomly-assigned news text from the same shard. If val loss is unchanged from baseline, text is unused (which we know is the case).

---

7. Inference Audit

What's verifiable from local code

• The inference snippet in the prompt is almost correct, but:
  • model.load_state_dict(state['model'] if 'model' in state else state) — in train_one_shard (line 1437-1443) the checkpoint key is 'state', not 'model'. In merge_shards_and_test (line 1475) it's also 'state'. So the snippet should be state['state'] if 'state' in state else state.
• state.get('model', state) is forgiving and won't error — it just falls through. But the documented-correct line would use 'state'.

Hypothetical robustness failure modes

Given the cross-attention collapse, all "model robustness" tests degenerate to "chart robustness". Specifically:

• Same chart, different news → output is bit-identical (predicted).
• Empty/garbled news → output identical to legitimate news (predicted).
• Adversarial news ("guaranteed crash") → output unchanged (predicted).
• NaN chart input → likely outputs NaN; no defensive handling in forward.
• Sub-128-step chart → in train_one_shard._DS.__getitem__ lines 1298-1300, the chart is zero-pre-padded if shorter — but at inference time the caller is responsible for windowing. No safety net.

Shape contract

The forward returns shapes consistent with the documented contract:

• sig_logits: (B, 2) ✓
• dir_logits: (B, 3) ✓
• mag: (B,) — squeezed from (B,1) ✓
• reason_emb: (B, 1024) for roberta-large ✓

The shape contract is OK; the semantic contract (model uses news to predict) is not.

---

8. Code Bug Hunt — Function-Level Summary

Function	LOC	Status	Notes
normalize_date_column	5	OK	Simple wrapper
merge_all_news	~40	OK	Standard concat + dedupe
build_symbol_pool	~170	NOT REVIEWED — UNKNOWN	Large; should be reviewed for pool composition
fetch_yfinance_daily	~70	OK / FAIL-RESOLUTION	Function works; resolution implication is the real bug
fetch_stooq_daily	~45	OK	Fallback path
fetch_crypto_universe_from_exchanges	~85	NOT REVIEWED
fetch_ccxt_4h	~50	OK	Correctly requests 4h
fetch_all_charts	~60	PARTIAL	Saves daily and 4h into one dir; loud rename suggested
build_symbol_aliases	~120	NOT REVIEWED — UNKNOWN	Long; should be audited for ticker collisions (e.g. "X", "META", "T")
attribute_news_to_symbols	~70	OK	Vectorized findall; correct algorithm. Common-word collision risk depends on build_symbol_aliases content.
compute_impact_label	~40	FAIL	C12, C14, C15 above
get_model_classes	~60	FAIL (C16)	Cross-attention collapse
run_data_prep	~6	OK	Orchestrator
run_feature_prep	~50	OK	Shard-then-augment pipeline; iloc[shard::num_shards] interleaves correctly
_safe_symbol_filename	2	OK
build_training_pairs	~25	PARTIAL	Loops one symbol at a time; could be vectorized. Inherits compute_impact_label issues
train_one_shard	~225	PARTIAL	C30 (temporal split) OK, C35 (no seed) FAIL, C37 (scaler bug) FAIL, no head_reason supervision
merge_shards_and_test	~40	FAIL (C26)	Reports pre-merge val, not merged-model test
multi_iterate	~110	PARTIAL	Uses < for best-tracking with no statistical confidence (line 1569)
iterate_v2, train_remaining_shards_and_merge	—	NOT REVIEWED	Lower priority for v6 model audit

---

9. Production Readiness

Dimension	Status	Notes
Checkpoint integrity	UNKNOWN	No Drive access
Architecture matches spec	FAIL	C16
Reproducibility	FAIL	C35
Tested code	FAIL	C38
Logging / metrics	PARTIAL	Per-step JSONL logging exists, but per-component val loss not logged
Dependency pinning	UNKNOWN	No requirements.txt / pyproject in scope
Model registry	NONE	Drive-only artifacts
Calibration	NONE	No calibration metrics computed
Input validation	NONE	No defensive code at inference boundary
"Reason" output usability	FAIL	C36 — embedding, not explanation
Ops state at session close	PARTIAL	Cron disabled (good), tunnel still up (risk)
Security: torch.load(pt)	RISK	Pickle deserialization; OK if artifact is trusted
Data freshness	PARTIAL	News ends 2026-05-03; close-to-date OK

Verdict: not deployable as a "news-impact AI". Deployable, with significant caveats, as a chart-only crypto move classifier — which is not what's advertised.

---

10. Recommended Next Actions

Do immediately (under 1 day)

1. Fix the cross-attention collapse (CRITICAL-1). Either Option A (concat-then-project) or Option B (chart-as-sequence). Add the "text-affects-output" pytest. This is the single highest-ROI fix.
2. Wire head_reason either to a real target or remove it (HIGH-4).
3. Fix the AMP NaN-guard (HIGH-5).
4. Set seeds at the top of every runner; tag the config with seed in the manifest (HIGH-3).

Before v7 training

5. Audit and re-derive compute_impact_label:
   • Pick a single bar resolution (4h or daily) per symbol and fail loudly if mismatched.
   • Fix the vol-window ↔ return-window dimensional mismatch (HIGH-1).
   • Either drop the "flat" direction class or define it as |return| < ε·vol (MEDIUM-1).
6. Add a held-out global test set; have merge_shards_and_test evaluate the merged model on it (CRITICAL-3).
7. Re-parse BBC news dates (HIGH-2). Confirm ingestion isn't silently dropping timestamps.
8. Add the 4 baselines from §6: chart-only, text-only, permuted-label, permuted-text. Report all 5 numbers in the iteration scoreboard.
9. Run each config 3 times with seeds {42, 1337, 7} and report mean ± std.

Before production deploy

10. Add tests/test_pipeline.py with at minimum:
    • test_text_input_affects_output (CRITICAL-1)
    • test_compute_impact_label_handles_daily_bars (CRITICAL-2)
    • test_merge_then_evaluate_matches_report (CRITICAL-3)
    • test_significance_threshold_calibrated (HIGH-1)
11. Pin dependencies (requirements.txt or uv.lock).
12. Verify final_best.pt loads strict=True; run audit_inference_skeleton.py.
13. Decide what reason_emb actually does in the product. Either build a retrieval bank, train a small explanation decoder, or replace with feature attribution.
14. Add monitoring: log inference latency, input shape, output distribution.

Nice to have

15. Replace datetime.utcnow() and torch.cuda.amp.* with their non-deprecated equivalents (LOW-1).
16. Clean up _SYMBOL_COERCIONS (MEDIUM-2).
17. Vectorize build_training_pairs (it currently iterates symbol-by-symbol, row-by-row).

---

11. Hypothesis Refutation — Acımasız Mod Sonuçları

The user asked me to test 10 negative hypotheses. Status for each:

#	Hypothesis	Status	Confidence	Evidence
1	Model only learned class prior, not real signal	CONFIRMED	1.0	C16 forces this; sig_acc = majority class
2	Validation split contains leakage	REFUTED	0.9	Per-shard temporal split is clean; cross-shard temporal overlap is mitigated by per-shard val
3	Label function is noisy/misaligned	CONFIRMED	1.0	C12 (daily/4h mismatch), C14 (vol dim mismatch), C15 (flat class)
4	reason head produces no explanation	CONFIRMED	1.0	C36 — 1024-dim float vector, no decoder, no training target
5	v6 winner claim is statistically meaningless	CONFIRMED	1.0	C27, C35 — no seed, n=1 per config, Δ < noise band
6	final_best.pt config doesn't match report	UNKNOWN	—	Cannot verify w/o Drive access
7	Chart input contains future leakage	PARTIAL	0.7	Daily-bar timestamps may include a "today's close" that postdates news arrival; window logic is Date <= t. For intraday news on daily data, you'd grab same-day EOD which is future. Suspected leakage for non-crypto.
8	News-symbol attribution has high false positives	UNKNOWN	—	Depends on build_symbol_aliases content; not audited in this pass
9	Loss plateau is architecture/label, not capacity	CONFIRMED	0.95	C16 + C12 + C14 jointly explain the plateau
10	Production inference snippet is fragile	CONFIRMED	0.9	Uses wrong checkpoint key ('model' instead of 'state'); falls through by luck. No input validation.

---

12. Answer to "Bu modele güvenebilir miyim?"

1. final_best.pt gerçekten yüklenip inference yapıyor mu? UNKNOWN (Drive yok). Mimari shape uyumu var → muhtemelen yükleniyor. audit_inference_skeleton.py ile doğrulayın.
2. Mimari iddia ile checkpoint uyumlu mu? Çok büyük olasılıkla evet (shape düzeyinde). Ama mimari SEMANTİK olarak bozuk (C16).
3. Raporlanan metrikler doğru ve anlamlı mı? Hayır. 2.4951, post-merge test loss değil; pre-merge per-shard val ortalaması. Bu farkı düzeltmeden iddia "winner" olamaz.
4. v6 gerçekten daha iyi mi yoksa noise mu? Noise. Δ=0.0004, seed yok, sample size 1.
5. Data/label noise gerçek bottleneck mi? Kısmen. Label noise gerçek (C12, C14, C15) ama asıl bottleneck mimari (C16). Mimari düzeltilmeden label'ları düzeltmek hiçbir şey çözmez.
6. Major bug/leakage var mı? Evet, üç kritik bug:
   • Cross-attn collapse
   • Daily/4h resolution silent rejection
   • Pre-merge metric reporting
7. Production deploy güvenli mi? Hayır.
8. v7'den önce mutlaka düzeltilmesi gereken 5 şey:
   1. Cross-attention fusion (CRITICAL-1)
   2. Chart resolution mismatch (CRITICAL-2) ve label window
   3. Post-merge evaluation (CRITICAL-3)
   4. Seed control + triplicate runs
   5. head_reason'u ya kaldır ya da gerçekten eğit

---

End of audit report. Generated by Claude on 2026-05-13 from local artifacts. Drive-bound claims marked UNKNOWN; run audit/audit_inference_skeleton.py on Colab to convert UNKNOWNs to PASS/FAIL.