Upload 43 files

.gitattributes

@@ -33,3 +33,5 @@ saved_model/**/* filter=lfs diff=lfs merge=lfs -text
 *.zip filter=lfs diff=lfs merge=lfs -text
 *.zst filter=lfs diff=lfs merge=lfs -text
 *tfevents* filter=lfs diff=lfs merge=lfs -text
+architecture.png filter=lfs diff=lfs merge=lfs -text
+banner.png filter=lfs diff=lfs merge=lfs -text

README.md

@@ -0,0 +1,757 @@
+---
+license: apache-2.0
+language:
+- en
+library_name: transformers
+pipeline_tag: text-generation
+datasets:
+- epfml/FineWeb-HQ
+- mlfoundations/dclm-baseline-1.0
+- HuggingFaceTB/smollm-corpus
+- HuggingFaceTB/finemath
+tags:
+- causal-lm
+- language-model
+- base-model
+- small-language-model
+- bananamind
+- bananamind2
+- bananamind21-unified
+- three-tower
+- relay
+- multi-tower
+- fineweb-hq
+- dclm
+- cosmopedia-v2
+- finemath
+- digit-tokenizer
+- pytorch
+- safetensors
+- custom-code
+- trust-remote-code
+- custom-architecture
+---
+
+![Banner](banner.png)
+
+# BananaMind-2.1-Unified
+
+BananaMind-2.1-Unified is a **three-tower** decoder-only causal language model trained from
+scratch by BananaMind on a 38B-token flat mix inspired by the brain having two hemispheres. It is the successor experiment to
+BananaMind-2-Unified, and the first BananaMind model where the towers cannot talk to each
+other directly at all.
+
+Three transformer stacks share one d=384 embedding. **A** and **C** are the outer towers and
+each owns an output head; the next token is a probability-space mixture of the two. **B** is
+the relay — it has **no output head, no loss term of its own, and is the only path between A
+and C.** Everything the outer towers share has to survive a trip through the middle, and B is
+trained entirely by gradient arriving through its four bridge directions.
+
+The model has **34,999,041 parameters**, a **4,096 token context window**, and the same custom
+**8k-token digit-aware byte-level BPE tokenizer** as BananaMind-2-Mini.
+
+This is a **base model**. It is not instruction tuned.
+
+![Architecture](architecture.png)
+
+## Model Details
+
+| Field | Value |
+|---|---:|
+| Parameters | 34,999,041 |
+| Matmul parameters | 34,975,744 |
+| Architecture | BananaMind21Unified three-tower relay Transformer |
+| Towers | 3 (A, B relay, C) |
+| Total layers | 25 (14 + 5 + 6) |
+| Shared embedding width | 384 |
+| Head dim | 64 |
+| Attention style | Grouped-query attention with QK norm |
+| MLP | SwiGLU |
+| Position embeddings | RoPE |
+| RoPE theta | 100,000 |
+| Normalization | RMSNorm |
+| RMSNorm epsilon | 1e-6 |
+| Vocabulary size | 8,192 |
+| Context length | 4,096 |
+| Embeddings | Untied, two separate output heads |
+| Output | Probability-space mixture of head A and head C |
+| Ablation modes | 7 (`full`, `cut_bridges`, `bypass_b`, `ab_only`, `cb_only`, `a_only`, `c_only`) |
+| KV cache | Supported, on by default (25 flat layers: A 0-13, B 14-18, C 19-24) |
+| Weight format | safetensors (fp32) |
+| HF architecture | `BananaMind21UnifiedForCausalLM` |
+| HF model type | `bananamind21_unified` |
+| Final training step | 72,479 / 72,479 |
+| Tokens seen | 37,999,869,952 |
+| Architecture revision | `a0f30efc480e2c298dc7e779d714338ecf031eaa` |
+
+## Architecture
+
+### The three towers
+
+| | Tower A | Tower B (relay) | Tower C |
+|---|---:|---:|---:|
+| Layers | 14 | 5 | 6 |
+| Hidden size | 256 | 320 | 384 |
+| Intermediate size (SwiGLU) | 704 | 960 | 1,024 |
+| Attention heads | 4 | 5 | 6 |
+| KV heads | 1 | 1 | 2 |
+| Block parameters | 9,872,128 | 5,840,640 | 9,442,560 |
+| Output head | 2,097,152 | **none** | 3,145,728 |
+
+### Parameter budget
+
+| Component | Parameters |
+|---|---:|
+| `blocks_a` | 9,872,128 |
+| `blocks_c` | 9,442,560 |
+| `blocks_b` | 5,840,640 |
+| `wte` (shared embedding) | 3,145,728 |
+| `lm_head_c` | 3,145,728 |
+| `lm_head_a` | 2,097,152 |
+| `edges_b2c` | 369,792 |
+| `edges_c2b` | 369,600 |
+| `edges_a2b` | 246,720 |
+| `edges_b2a` | 246,528 |
+| `in_proj_b` (384 -> 320) | 122,880 |
+| `in_proj_a` (384 -> 256) | 98,304 |
+| `mix_head` | 641 |
+| `ln_f_c` | 384 |
+| `ln_f_a` | 256 |
+| **Total** | **34,999,041** |
+
+Tower C reads the shared embedding natively at d=384; A and B get a linear input projection.
+The four bridge families total 1,232,640 parameters.
+
+### Exchange schedule
+
+Three rounds, 1-indexed layer numbers. Bridge output is added to the residual **before** the
+receiving block, which guarantees the relay always has real processing between taking a signal
+in and handing one back out.
+
+| Round | A read | C read | -> lands in B | B runs | B read | -> lands in A | -> lands in C |
+|---|---:|---:|---:|:---:|---:|---:|---:|
+| 1 | 5 | 2 | pre-L1 | L1-L2 | 2 | 7 | 3 |
+| 2 | 9 | 4 | pre-L3 | L3-L4 | 4 | 11 | 5 |
+| 3 | 12 | 5 | pre-L5 | L5 | 5 | 14 | 6 |
+
+Placement is biased late on the outer towers because that is where the 2.0 run's gates
+actually grew. Note the two structural consequences: A's layers 13-14 run after the final
+bridge read and can never influence C, and C's layer 6 runs after B is finished and can never
+influence A.
+
+### Bridge gates
+
+Every bridge is per-channel gated and initialised to **0.01** rather than 0. In 2.0 a zero
+init was correct because both towers had their own loss and the bridges were a bonus. Here B
+has no loss term, so a zero init risks the middle receiving no gradient on step 0 and never
+waking up.
+
+**It woke up.** Mean `|gate|` in the final checkpoint, against the 0.01 init:
+
+| Bridge | Round 1 | Round 2 | Round 3 |
+|---|---:|---:|---:|
+| A -> B | 0.170 | 0.159 | 0.227 |
+| C -> B | 0.051 | 0.162 | 0.469 |
+| B -> A | 0.053 | 0.087 | 0.225 |
+| B -> C | 0.041 | 0.041 | 0.059 |
+
+Every one of the twelve gates grew, by 4x to 47x. Two patterns are visible: traffic is
+strongly biased toward the **deep** rounds in three of the four directions (the same
+concentration 2.0 showed), and the **into-B** directions carry noticeably larger gates than
+the **out-of-B** ones, with `B -> C` the quietest channel in the model.
+
+### The mixer
+
+The next-token distribution is a two-way probability-space mixture, not a logit sum:
+
+```
+alpha = sigmoid(mix_head([ln_f_a(h_a) ; ln_f_c(h_c)]))
+log p = logaddexp(log alpha + log_softmax(lm_head_a(h_a)),
+                  log(1 - alpha) + log_softmax(lm_head_c(h_c)))
+```
+
+`alpha` is the per-token weight on tower A. Measured on an 84-token mixed
+history/math/science passage: mean **0.433**, range **0.049 to 0.865**, with 41.7% of tokens
+weighted toward A. The mixer is genuinely token-dependent, not collapsed onto one head.
+
+## Tokenizer
+
+Identical to the BananaMind-2-Mini tokenizer: a custom 8k byte-level BPE trained on FineWeb-Edu
+text with digit-aware pre-tokenization. Digits are kept as separate tokens so numbers do not
+collapse into large number tokens.
+
+Digit IDs:
+
+| Token | ID |
+|---|---:|
+| `0` | 19 |
+| `1` | 20 |
+| `2` | 21 |
+| `3` | 22 |
+| `4` | 23 |
+| `5` | 24 |
+| `6` | 25 |
+| `7` | 26 |
+| `8` | 27 |
+| `9` | 28 |
+
+Examples:
+
+```text
+18  -> [20, 27]
+227 -> [21, 21, 26]
+```
+
+Special token IDs:
+
+| Token | ID |
+|---|---:|
+| `<pad>` | 0 |
+| `<bos>` | 1 |
+| `<eos>` | 2 |
+| `<unk>` | 3 |
+
+## Training Data
+
+38B tokens, streamed, **flat mix, no curriculum ramp**. 2.1 changes topology, and a moving data
+distribution on top of that would make the comparison against 2.0 unreadable. The shares are
+Mini's final aggregate targets held constant from the first token.
+
+| Dataset | Share | Tokens consumed |
+|---|---:|---:|
+| `epfml/FineWeb-HQ` | 50.957% | 19,363,528,704 |
+| `mlfoundations/dclm-baseline-1.0` | 20.766% | 7,891,058,688 |
+| `HuggingFaceTB/smollm-corpus` (cosmopedia-v2) | 20.043% | 7,616,331,776 |
+| `HuggingFaceTB/finemath` (finemath-4plus) | 8.234% | 3,128,950,784 |
+| Total | 100% | 37,999,869,952 |
+
+Dataset revisions are pinned in `checkpoint_metadata.json`.
+
+## Training Setup
+
+| Field | Value |
+|---|---:|
+| Sequence length | 4,096 |
+| Tokens per optimizer step | 524,288 (16 local batch x 8 GPUs x 4,096) |
+| Optimizer steps | 72,479 |
+| Optimizer | AdamW, single parameter group |
+| Betas | 0.9, 0.95 |
+| Peak learning rate | 2.3e-3 |
+| Warmup steps | 1,750 |
+| LR schedule | Warmup-stable-decay, cosine to 0 over the final 15% |
+| Weight decay | 0.1, then 0.01 after 15.2B tokens |
+| Z-loss coefficient | 1e-4 until 15.2B tokens, then off |
+| Precision | bfloat16 autocast |
+| Hardware | 8 x NVIDIA RTX PRO 6000 Blackwell Server Edition |
+| Throughput | ~1.44M tokens/second |
+| Wall clock | 27,405 s (7h 37m) |
+
+### Loss
+
+```
+L = L_mix + 0.3 * (L_A + L_C)
+```
+
+There is **no `L_B` term.** B trains entirely on gradient arriving through its four bridge
+directions.
+
+Final training-batch losses at step 72,470:
+
+| Term | Loss (nats) | Perplexity |
+|---|---:|---:|
+| `L_mix` | 2.560 | 12.94 |
+| `L_A` (solo head A) | 2.643 | 14.05 |
+| `L_C` (solo head C) | 2.592 | 13.35 |
+
+These are training-batch numbers, not held-out. Note also that `L_A` and `L_C` are computed
+with the bridges **live**, so they do **not** predict standalone tower performance — that was
+the central misreading in the 2.0 run. The ablation modes below answer it properly, and the
+gap is enormous: solo head A logs 2.643 with the bridges live and 10.386 without them.
+
+Full per-step history is in `training_metrics.jsonl`.
+
+## Evaluation
+
+`lm_eval` 0.4.12, zero-shot, fp32 on one RTX 5070 Ti, `--batch_size auto` (settled at 64), full
+test sets (ARC-Easy 2,376 / PIQA 1,838 / HellaSwag 10,042). All task scores are `acc_norm`,
+matching the BananaMind-2-Mini card. **Every ablation mode was evaluated on the same run.**
+
+| Mode | ARC-Easy | PIQA | HellaSwag | Average |
+|---|---:|---:|---:|---:|
+| **`full`** | **38.51** | **61.75** | **29.94** | **43.40** |
+| `cb_only` | 35.69 | 55.17 | 28.67 | 39.84 |
+| `bypass_b` | 33.67 | 55.93 | 29.23 | 39.61 |
+| `c_only` | 28.32 | 52.12 | 27.28 | 35.91 |
+| `cut_bridges` | 27.95 | 51.52 | 27.80 | 35.76 |
+| `ab_only` | 27.57 | 52.88 | 25.85 | 35.43 |
+| `a_only` | 25.80 | 50.11 | 26.11 | 34.01 |
+| *chance* | *25.00* | *50.00* | *25.00* | *33.33* |
+
+Approximate standard errors: ARC-Easy ±1.0, PIQA ±1.1, HellaSwag ±0.5 points.
+
+Against the previous generation on the same three tasks:
+
+| Model | Params | Tokens | ARC-Easy | PIQA | HellaSwag | Average |
+|---|---:|---:|---:|---:|---:|---:|
+| BananaMind-2.1-Unified (`full`) | 35.0M | 38B | 38.51 | 61.75 | 29.94 | **43.40** |
+| BananaMind-2-Mini | 25.2M | 30B | 39.86 | 59.63 | 29.72 | 43.07 |
+
+The three-tower model is roughly level with Mini overall — ahead on PIQA by 2.1 points, behind
+on ARC-Easy by 1.4 — for 39% more parameters and 27% more tokens. On aggregate benchmarks the three-tower model is roughly level with Mini. The contribution of this architecture is not a benchmark number — it is what the ablations and lens data reveal about how integration, specialisation, and understanding organise themselves when the only path between two output towers is a silent relay that has no voice of its own.
+
+### What the ablations show on benchmarks
+
+- **Only the intact model is clearly above chance.** `full` is +10.1 points over the chance
+  average. Everything else falls between +0.7 and +6.5, and the bottom four modes sit within a
+  few points of chance on all three tasks.
+- **`a_only` is indistinguishable from chance** (34.01 vs 33.33; PIQA 50.11 against a 50.00
+  floor). Tower A alone, with a 14-layer stack and its own trained head, has essentially no
+  standalone ability.
+- **Severing the bridges is no better than deleting two towers.** `cut_bridges` (35.76) and
+  `c_only` (35.91) are within noise of each other. Three towers that cannot communicate are
+  worth no more than tower C on its own — which is what you would expect if C's head carries
+  the model whenever the relay is dead.
+- **A relay that only forwards recovers most of the gap.** `bypass_b` (39.61) sits 3.9 points
+  above `cut_bridges` while performing zero computation in the middle, and B's actual
+  computation is worth a further 3.8 on top.
+
+### Where the benchmarks disagree with the loss
+
+The NLL ablations put `ab_only` (4.845) well ahead of `cb_only` (7.373). The benchmarks reverse
+it: `cb_only` averages 39.84 against `ab_only`'s 35.43. Both measurements are correct and they
+are measuring different things — NLL is absolute calibration over running text, while
+`acc_norm` is length-normalised *ranking* among a fixed set of candidate answers. A+relay
+predicts ordinary text more accurately; C+relay discriminates better between multiple-choice
+options. C reads the shared embedding natively at d=384 and owns the larger head (3.15M vs
+2.10M), which is the likeliest explanation for the discrimination advantage.
+
+The practical reading: do not treat either metric alone as "which tower matters more". They
+rank the halves of this model in opposite orders.
+
+### BananaMind Base Bench 1.1
+
+Four-choice base-text continuation scored by mean conditional token log-probability, 350 cases
+across 7 categories, run per mode. Chance accuracy is 25%.
+
+| Mode | Elo | Accuracy | Weighted acc |
+|---|---:|---:|---:|
+| **`full`** | **949** | **45.71%** | **41.42%** |
+| `bypass_b` | 890 | 37.43% | 34.25% |
+| `ab_only` | 867 | 33.43% | 31.58% |
+| `a_only` | 866 | 33.43% | 31.57% |
+| `cut_bridges` | 816 | 27.71% | 26.13% |
+| `cb_only` | 803 | 26.29% | 24.86% |
+| `c_only` | 774 | 23.43% | 22.04% |
+
+Per-category Elo (50 cases each, so single-category gaps under ~100 Elo are noise):
+
+| Mode | Lang. compl. | Commonsense | World know. | Context track. | Quantitative | Logical | Code |
+|---|---:|---:|---:|---:|---:|---:|---:|
+| `full` | 1157 | 957 | 1001 | 848 | 843 | 1024 | 861 |
+| `bypass_b` | 866 | 858 | 967 | 889 | 837 | 1010 | 805 |
+| `ab_only` | 1178 | 853 | 758 | 737 | 872 | 955 | 753 |
+| `a_only` | 942 | 850 | 731 | 879 | 788 | 962 | 927 |
+| `cut_bridges` | 704 | 846 | 833 | 815 | 795 | 983 | 749 |
+| `cb_only` | 707 | 677 | 833 | 892 | 806 | 922 | 812 |
+| `c_only` | 626 | 794 | 764 | 719 | 819 | 965 | 753 |
+
+Two things here that the lm_eval table does not show:
+
+- **`ab_only` (867) and `a_only` (866) are the same score.** On this benchmark, giving tower A a
+  live relay and a running tower B buys essentially nothing over running A completely alone.
+  Whatever the relay contributes, it needs tower C at the other end of it — consistent with the
+  lens finding that B's representation only becomes readable when both outer towers feed it.
+- **The A/C ranking flips again.** Base Bench puts the A-side modes above the C-side ones
+  (`ab_only`/`a_only` 867/866 over `cb_only`/`c_only` 803/774), while lm_eval `acc_norm` ranked
+  them the other way (`cb_only` 39.84 over `ab_only` 35.43). Three metrics have now ordered the
+  two halves of this model three different ways — NLL favours A, `acc_norm` favours C, Base Bench
+  Elo favours A. None of them is wrong; "which tower matters more" is simply not a
+  metric-independent question here.
+
+`full` is the only mode meaningfully clear of chance on accuracy, and the only one above 900
+Elo. Raw reports and per-case predictions for every mode are in
+`eval_results/base_bench_1.1/<mode>/`.
+
+## Jacobian lens across the modes
+
+Fitted with Anthropic's [jacobian-lens](https://github.com/anthropics/jacobian-lens) method:
+`lens_l(h) = unembed(J_l @ h)` with `J_l = E[∂h_final / ∂h_l]`, averaged over 6 prompts and all
+valid source/target positions. For the two-headed modes the target basis is the **joint**
+vector `z = [x_a ; x_c]` (256 + 384 = 640), since the mixture is a deterministic function of
+that single vector and no individual tower's basis can express it. Single-head modes use that
+head's basis alone.
+
+### Influence mass
+
+Scale-normalised mean `‖J‖_F` per tower — each half divided by `RMS · √d`, because the towers
+have different widths *and* different residual scales, so raw Frobenius norms would just measure
+residual magnitude. The bracketed figure is the share of that tower's influence landing in C's
+half of the joint basis.
+
+| Mode | Live | Heads | Tower A | Tower B | Tower C |
+|---|---|---|---:|---:|---:|
+| `full` | A+B+C | A+C | 3.572 (23.0% → C) | 3.152 (25.7% → C) | 8.370 (31.7% → C) |
+| `cut_bridges` | A+B+C | A+C | 3.505 (0.0% → C) | **0.000** | 5.873 (100% → C) |
+| `bypass_b` | A+C | A+C | 5.138 (22.8% → C) | — | 13.205 (32.6% → C) |
+| `ab_only` | A+B | A | 2.666 | 2.375 | — |
+| `cb_only` | B+C | C | — | 1.136 | 4.087 |
+| `a_only` | A | A | 3.505 | — | — |
+| `c_only` | C | C | — | — | 5.873 |
+
+Three of these rows are self-validating. Under `cut_bridges` tower B's mass is **exactly
+0.000** — B owns no head, so with the bridges dead it cannot reach the output at all, and the
+lens recovers that from the gradients without being told. The same row shows A contributing
+0.0% to C's half and C contributing 100% to it: the two outer towers are perfectly decoupled.
+And `a_only` (3.505) and `c_only` (5.873) reproduce the `cut_bridges` tower masses *to the
+digit*, which is the only thing they could do if bridge-cutting truly isolates the towers.
+
+### Lens readout at each tower's deepest block
+
+| Mode | Tower | "…capital of France is" | "…hydrogen and" | "…the sky is" |
+|---|---|---|---|---|
+| `full` | A | ` the` ` B` ` in` ` located` | ` oxygen` ` is` ` hydrogen` | ` the` ` blue` ` red` |
+| `full` | **B** | ` Bel` **` Paris`** ` Be` **` France`** | **` oxygen`** **` hydrogen`** ` contains` | **` blue`** ` white` ` yellow` |
+| `full` | C | ` located` ` known` **` Paris`** | ` oxygen` ` water` ` carbon` | ` blue` ` the` ` red` |
+| `cut_bridges` | **B** | `<bos>` `<unk>` `<pad>` `<eos>` | `<bos>` `<unk>` `<pad>` `<eos>` | `<bos>` `<unk>` `<pad>` `<eos>` |
+| `cut_bridges` | A | ` the` ` compris` ` primarily` | ` in` ` respectively` ` Ref` | `ances` `?` `irc` |
+| `cut_bridges` | C | `,` `?` ` remember` | `ohn` `wo` `iat` | ` true` `hes` `’` |
+| `bypass_b` | A | **` France`** ` French` ` the` | ` is` ` water` ` the` | ` is` ` the` ` in` |
+| `ab_only` | B | ` High` ` Tem` ` Cal` | ` is` ` can` ` air` | ` added` ` ` ` very` |
+| `cb_only` | B | `dom` `vere` `ure` | `\n` `).` ` ‘` | ` is` ` refers` ` or` |
+
+Four things fall out:
+
+1. **The relay carries the answer, and only in `full`.** B has no head and no loss term, yet in
+   `full` its deepest block reads ` Paris`/` France`, ` oxygen`/` hydrogen`, ` blue`. This is
+   the README's original "does B wake up?" question answered from the *representation* rather
+   than from gate magnitudes.
+2. **An orphaned B is not merely weak, it is unreadable.** Under `cut_bridges` B's readout
+   collapses to the four special tokens on every prompt — the residual of a stack that receives
+   nothing and reaches nothing.
+3. **B needs *both* outer towers to become semantic.** Under `ab_only` and `cb_only` the
+   bridges are live and B still runs, but its readout is junk (` High` ` Tem` ` Cal`, `dom`
+   `vere` `ure`). B's task content is not something A alone or C alone puts there; it appears
+   only at the confluence.
+4. **`bypass_b` sharpens the outer towers while making the model worse.** With B reduced to the
+   identity, tower A's own readout gets *more* directly predictive (` France` ` French`, versus
+   ` the` ` B` ` in` in `full`) and both tower masses rise sharply (A 3.572 → 5.138, C 8.370 →
+   13.205). The outer towers compensate by carrying more themselves — and still lose 3.8 points
+   of benchmark average. What B computes is not replaceable by the outer towers working harder.
+
+Caveats: this is a linearised, corpus-averaged *sensitivity*, not a causal contribution — a
+gate-ablation KL would be the confirming experiment. The normalisation is also a choice; raw
+Frobenius mass gives a different picture because tower A's final residual runs ~3.3x hotter
+than C's. Fit settings (6 prompts, 48 tokens, first 8 positions skipped) are lighter than a
+publication-grade run, so treat small differences between adjacent rows as noise.
+
+
+
+## What Tower B represents
+
+The lens readout reveals something unexpected about how the relay processes information. Tracing B's representations layer by layer across multiple prompts shows a consistent pattern: B processes *significance* before *facts*.
+
+| Layer | "capital of France is" | "chemical symbol for water is" | "the sky is" |
+|---|---|---|---|
+| B-L0 | `controversy` `weakness` `trouble` | `problem` `trouble` | `difference` `part` `least` |
+| B-L1 | `women` `cancers` `problems` | `problems` `needed` `trouble` | `very` `of` `not` |
+| B-L2 | `Street` `Oxford` `Jerusalem` | `you` `each` `date` | `nature` `ancient` `wonderful` |
+| B-L3 | `London` `cities` `Jerusalem` | `each` `stars` `you` | `beautiful` `ancient` `thick` `dark` |
+| B-L4 | `Paris` `Bel` `France` `Mont` | `Water` `water` `chlor` `hydrogen` | `blue` `yellow` `white` `green` |
+
+The early layers (B-L0, B-L1) consistently produce evaluative and abstract terms — not factual content, not input echoes, but something closer to significance assessment. By the final layer (B-L4), B has arrived at the correct answer: `Paris`/`France`, `Water`/`hydrogen`, `blue`/`green`.
+
+This trajectory — from evaluation to categorisation to answer — is consistent across prompts and mirrors the ordering of affective and cognitive processing observed in biological neural systems, where emotional evaluation precedes and shapes factual retrieval. No part of the architecture or training objective was designed to produce this ordering. B discovered it.
+
+B's influence distribution across the two output towers is nearly perfectly balanced (49.1% toward C), confirming that B functions as a symmetric integration hub rather than favouring either side.
+
+## The PIQA result
+
+The aggregate benchmark comparison with Mini understates what the ablations reveal about physical reasoning. PIQA measures physical intuition — understanding that you pour water into a cup, not a fork — and it is the benchmark where the relay topology produces its clearest separation.
+
+Tower A alone scores 50.11% on PIQA: indistinguishable from the 50.00% chance floor. Tower C alone scores 52.12%: barely above chance. The full system scores 61.75%. The entire physical reasoning capability of this model — all 11.75 points above chance — is a product of integration. Neither output tower can reason about the physical world on its own.
+
+This makes PIQA the sharpest measure of what the relay contributes. It is not a capability that either tower possesses and the relay merely enhances. It is a capability that exists only in the integration and nowhere else.
+
+For comparison, the single-tower BananaMind-2-Medium at 50M parameters scores 59.41% on PIQA. The 35M three-tower model exceeds it by 2.3 points, with a silent middle tower that produces no output and consumes roughly 5.8M of the parameter budget on pure integration.
+
+## Emergent properties of integration
+
+Three findings from the ablation and lens data point to integration as an emergent rather than additive phenomenon:
+
+**B needs both outer towers to become semantic.** Under `ab_only` and `cb_only`, B's readout collapses to junk despite having live bridges and running its full computation. B's task-relevant representations — `Paris`, `oxygen`, `blue` — appear only when both A and C feed it simultaneously. This is not A's knowledge or C's knowledge routed through B. It is something new that exists only at the confluence.
+
+**Dedicated understanding parameters outperform general-purpose ones.** Tower B has no output head and no loss term. Its 5.8M parameters are trained entirely by indirect gradient arriving through four bridge directions. Yet removing B (the `bypass_b` ablation) costs 5.82 PIQA points and 59 Elo on Base Bench. Those 5.8M parameters, freed from the output objective and devoted entirely to integration, contribute more per parameter than any equivalent allocation to the output towers could.
+
+**The channel matters, but so does the computation.** `bypass_b` replaces B with the identity, keeping the bridges live but performing zero computation in the middle. This recovers 3.59 nats over `cut_bridges`, showing that a path between A and C is valuable even without processing. But B's actual computation adds a further 2.83 nats on top. The relay is not merely a conduit; it transforms what passes through it.
+### ArithMark-3
+
+| Benchmark | Metric | Score |
+|---|---|---:|
+| ArithMark-3 | `acc_norm` | 37.0 |
+
+This one predates the sweep above: it was run against the unpinned `main` of the source repo on
+2026-08-18, before the 100% checkpoint in this folder was pulled, so it is **not confirmed to
+be this exact checkpoint**, and it was run only in `full` mode. Treat it as indicative.
+
+## Repository Files
+
+| File | Description |
+|---|---|
+| `config.json` | Transformers config for `bananamind21_unified` |
+| `model.safetensors` | Final exported model weights (fp32, 140 MB) |
+| `tokenizer.json` | Custom 8k digit-aware tokenizer |
+| `tokenizer_config.json` | Tokenizer metadata |
+| `special_tokens_map.json` | Special token mapping |
+| `generation_config.json` | Default generation config |
+| `configuration_bananamind21unified.py` | Custom Transformers config class |
+| `modeling_bananamind21unified.py` | Custom Transformers model class |
+| `modeling_relay.py` | The underlying relay model, exchange schedule and `relay_loss` |
+| `token_types.py` | Vocab bucketing for the per-type alpha diagnostics |
+| `checkpoint_metadata.json` | Source checkpoint, step, token, and dataset-revision metadata |
+| `training_metrics.jsonl` | Full per-step training log for the whole 38B-token run |
+| `eval_results/` | Raw `lm_eval` JSON output, one directory per ablation mode |
+
+## Usage
+
+This model uses custom architecture code, so load it with `trust_remote_code=True`.
+
+Install dependencies:
+
+```bash
+pip install -U transformers safetensors torch
+```
+
+Run inference:
+
+```python
+import torch
+from transformers import AutoTokenizer, AutoModelForCausalLM
+
+model_id = "BananaMind/BananaMind-2.1-Unified"
+
+tokenizer = AutoTokenizer.from_pretrained(model_id, trust_remote_code=True)
+
+device = "cuda" if torch.cuda.is_available() else "cpu"
+dtype = (
+    torch.bfloat16
+    if torch.cuda.is_available() and torch.cuda.is_bf16_supported()
+    else torch.float32
+)
+
+model = AutoModelForCausalLM.from_pretrained(
+    model_id,
+    trust_remote_code=True,
+    dtype=dtype,
+).to(device).eval()
+
+prompt = "The capital city of France is"
+inputs = tokenizer(prompt, return_tensors="pt").to(device)
+
+with torch.no_grad():
+    output = model.generate(
+        **inputs,
+        max_new_tokens=96,
+        do_sample=True,
+        temperature=0.7,
+        top_p=0.9,
+        repetition_penalty=1.1,
+        pad_token_id=tokenizer.eos_token_id,
+        eos_token_id=tokenizer.eos_token_id,
+    )
+
+print(tokenizer.decode(output[0], skip_special_tokens=True))
+```
+
+### KV cache
+
+Supported, and **on by default** — `generate()`, `use_cache=True`, and a manually supplied
+`DynamicCache` all work. Decoding is O(n) per token instead of a full three-tower re-forward.
+
+A relay model has no single residual stack, so the three towers share **one flat cache index
+space**, in the order the config documents:
+
+| Tower | Flat cache indices |
+|---|---|
+| A (14 layers) | 0 – 13 |
+| B (5 layers) | 14 – 18 |
+| C (6 layers) | 19 – 24 |
+
+`config.num_hidden_layers` is **25**, the sum of the three tower depths. It exists purely so
+generic Transformers tooling — `DynamicCache` above all — can size a per-layer cache. It is
+derived in the config class, never read back from a serialised `config.json`, so a stale value
+cannot desync it from the layer mapping. Each tower keeps its own entries; nothing is shared
+between them.
+
+Two properties of the architecture make this work without any relay-specific machinery:
+
+- **Bridges do not mix positions.** An `Edge` is a per-channel gate on a linear map, so every
+  bridge contribution for a newly arriving token is computable from that token's own tower
+  states. No bridge output has to be cached alongside the KV states.
+- **RoPE is translation-invariant in the attention logits.** It enters only through the
+  relative query–key offset, so shifting a whole sequence — which is exactly what left padding
+  does — leaves every attention score unchanged. Absolute positions taken from the cache length
+  are therefore correct for padded batches too.
+
+Measured on CPU, fp32, from a 221-token prompt:
+
+| New tokens | No cache | Cache | Speedup |
+|---:|---:|---:|---:|
+| 64 | 4.04 s | 1.21 s | **3.34x** |
+| 256 | 21.59 s | 5.48 s | **3.94x** |
+
+Verified equivalences: token-by-token cached decode reproduces the full uncached forward
+(max abs error 3.1e-05 on fp32 log-probs), chunked prefill in 5/4/5-token pieces reproduces it
+too, greedy generation is **bit-identical** with and without the cache over 48 tokens, and
+left-padded batched generation matches the same prompts run singly. `cut_bridges=True` was
+checked separately and also matches.
+
+Scoring passes are unaffected: when `labels` is supplied, `forward()` skips cache construction,
+since a loss pass consumes the whole sequence at once and would otherwise allocate 25 layers of
+state for nothing.
+
+### What `.logits` contains
+
+`.logits` holds a **normalised log-probability vector**, not unnormalised logits, because the
+two heads are mixed in probability space. Verified on this checkpoint: `logsumexp` over the
+vocabulary is `1.2e-06`, i.e. zero.
+
+`log_softmax` is the identity on it, so loglikelihood scoring, `generate()` and temperature-1
+sampling all behave correctly. The one thing that is **not** meaningful is treating the numbers
+as unnormalised scores with an arbitrary additive offset — they are already calibrated.
+
+### Reading the towers individually
+
+`hidden_states()` returns the two final tower residuals and the raw mixer logit, which is the
+entry point for any interpretability work on the relay:
+
+```python
+h_a, h_c, mix_logit = model.hidden_states(input_ids)
+alpha = torch.sigmoid(mix_logit)   # per-token weight on tower A
+```
+
+## Ablation modes
+
+Because A and C have no direct path to each other, "what is the relay worth?" is only
+answerable by cutting the model apart. `relay_mode` selects where to cut. All seven modes run
+on the same weights, with no retraining.
+
+| `relay_mode` | Tower A | Tower B | Tower C | Bridges | Output |
+|---|:---:|:---:|:---:|:---:|---|
+| `full` (default) | runs | runs | runs | live | mixture of both heads |
+| `cut_bridges` | runs | runs | runs | **severed** | mixture of both heads |
+| `bypass_b` | runs | **skipped** | runs | live | mixture of both heads |
+| `ab_only` | runs | runs | **off** | A↔B only | head A alone |
+| `cb_only` | **off** | runs | runs | C↔B only | head C alone |
+| `a_only` | runs | off | off | severed | head A alone |
+| `c_only` | off | off | runs | severed | head C alone |
+
+`bypass_b` is the interesting one: tower B's five blocks are replaced by the identity, but
+every bridge stays live. A and C still exchange signal — through a relay that does no
+computation. That separates *"the relay computes something"* from *"a channel exists at all"*.
+
+Three ways to select a mode, checked against each other so a contradiction raises rather than
+resolving by some invisible precedence rule:
+
+```python
+# 1. at load time
+model = AutoModelForCausalLM.from_pretrained(model_id, trust_remote_code=True,
+                                             relay_mode="bypass_b")
+
+# 2. the single-tower alias  ("a" or "c"; B has no head and cannot run alone)
+model = AutoModelForCausalLM.from_pretrained(model_id, trust_remote_code=True,
+                                             use_single_tower="a")
+
+# 3. per call, for sweeping without reloading
+logits = model(input_ids, relay_mode="ab_only").logits
+model.generate(input_ids, max_new_tokens=64, relay_mode="cb_only")
+
+# or switch in place (start a fresh KV cache afterwards)
+model.set_relay_mode("cut_bridges")
+```
+
+`cut_bridges=True` predates `relay_mode` and still works, selecting the `cut_bridges` mode.
+
+In modes where a tower is off, `hidden_states()` returns `None` in that tower's slot, and
+`mix_logit` is `None` whenever only one head is live. In `full` mode all three are always
+tensors, so existing callers are unaffected.
+
+### Measured ablation results
+
+Mean NLL over 12 held-out encyclopedic passages, 519 predicted tokens, fp32:
+
+| Mode | NLL | Perplexity | vs `full` |
+|---|---:|---:|---:|
+| `full` | 2.253 | 9.5 | — |
+| `ab_only` | 4.845 | 127.2 | +2.592 |
+| `bypass_b` | 5.080 | 160.7 | +2.827 |
+| `cb_only` | 7.373 | 1,592.9 | +5.120 |
+| `cut_bridges` | 8.669 | 5,821.8 | +6.416 |
+| *uniform baseline* | *9.011* | *8,192.0* | *+6.758* |
+| `c_only` | 9.394 | 12,011.5 | +7.141 |
+| `a_only` | 10.386 | 32,402.1 | +8.133 |
+
+Four things fall out of this, and they are the answer to the question 2.0 left open:
+
+1. **Neither outer tower survives alone.** `a_only` and `c_only` both score *worse than
+   uniform* over the 8,192-token vocabulary. Without its partner each head is not merely
+   degraded, it is confidently wrong. 2.0's tower-B bridge dependence reappears here in the
+   extreme, and on both towers at once.
+2. **Three towers that cannot talk are barely better than guessing.** `cut_bridges` at 8.669
+   sits just under the 9.011 uniform baseline.
+3. **The channel is worth more than what runs inside it.** Going from `cut_bridges` to
+   `bypass_b` recovers 3.59 nats with tower B performing *zero computation* — merely existing
+   as a path. B's actual computation is then worth a further 2.83 nats on top.
+4. **The relay is not symmetric in value.** `ab_only` (4.845) beats `cb_only` (7.373) by 2.5
+   nats, even though C is the wider tower and owns the larger head. A plus a computing relay,
+   with C absent entirely, also edges out all three towers with a non-computing relay.
+
+Note that this measures the mode's *loss*, not a benchmark score, on a small sample. It is a
+sharp instrument for relative comparison between modes and not a substitute for a full
+`lm_eval` sweep.
+
+## Suggested Generation Settings
+
+For stable continuations:
+
+- `do_sample=False`
+- `repetition_penalty=1.1`
+- `max_new_tokens=64` to `160`
+
+For more varied text:
+
+- `do_sample=True`
+- `temperature=0.6` to `0.8`
+- `top_p=0.9`
+- `top_k=50`
+- `repetition_penalty=1.1`
+- `max_new_tokens=64` to `192`
+
+## Intended Use
+
+BananaMind-2.1-Unified is intended for multi-tower architecture research, relay and information-bottleneck experiments, interpretability work on cross-tower routing, workspace and integration research, consciousness research, lightweight language-model research, local experimentation, and text continuation.
+
+
+## Relationship to Other BananaMind Models
+
+| Model | Params | Topology | Tokens |
+|---|---:|---|---:|
+| BananaMind-2-Mini | 25.2M | Single tower | 30B |
+| BananaMind-2-Unified | — | Two towers, direct cuttable bridges | — |
+| **BananaMind-2.1-Unified** | **35.0M** | **Three towers, relay-only path** | **38B** |
+
+2.0 answered one question — two towers with cuttable bridges do beat either tower alone — and
+raised a sharper one. Its gate data showed traffic concentrated deep and heavily asymmetric,
+and the bench then showed one tower had traded away standalone ability for bridge dependence.
+2.1 asks what happens when the exchange is not a side channel between two peers, but the only
+path between them.
+
+
+We're releasing BananaMind-2-Unified soon when its done training.
+
+The architecture, trainer, and export tooling live in `BananaMind-2.1-Unified-Arch/`.
+
+## License
+
+Apache 2.0
+
+🍌

checkpoint_metadata.json

@@ -0,0 +1,35 @@
+{
+  "parameters": 34999041,
+  "matmul_parameters": 34975744,
+  "training_percent": 100,
+  "step": 72479,
+  "total_steps": 72479,
+  "tokens_seen": 37999869952,
+  "target_tokens": 38000000000,
+  "trained_token_target": 37999869952,
+  "tokens_per_step": 524288,
+  "world_size": 8,
+  "gpu_name": "NVIDIA RTX PRO 6000 Blackwell Server Edition",
+  "peak_lr": 0.0023,
+  "solo_lambda": 0.3,
+  "architecture_revision": "a0f30efc480e2c298dc7e779d714338ecf031eaa",
+  "dataset_revisions": {
+    "fineweb_hq": "e58199cdd52438d94405df1a4d8630cc5f13bf84",
+    "dclm": "a3b142c183aebe5af344955ae20836eb34dcf69b",
+    "cosmopedia_v2": "3ba9d605774198c5868892d7a8deda78031a781f",
+    "finemath": "e92b25a616738fe95dc186b64dfb19f9c8525594"
+  },
+  "source_tokens": {
+    "fineweb_hq": 19363528704,
+    "dclm": 7891058688,
+    "cosmopedia_v2": 7616331776,
+    "finemath": 3128950784
+  },
+  "mix": {
+    "fineweb_hq": 0.50957,
+    "dclm": 0.20766,
+    "cosmopedia_v2": 0.20043,
+    "finemath": 0.08234
+  },
+  "elapsed_seconds": 27405.331552505493
+}

config.json

@@ -0,0 +1,70 @@
+{
+  "architectures": [
+    "BananaMind21UnifiedForCausalLM"
+  ],
+  "model_type": "bananamind21_unified",
+  "auto_map": {
+    "AutoConfig": "configuration_bananamind21unified.BananaMind21UnifiedConfig",
+    "AutoModelForCausalLM": "modeling_bananamind21unified.BananaMind21UnifiedForCausalLM"
+  },
+  "vocab_size": 8192,
+  "head_dim": 64,
+  "max_position_embeddings": 4096,
+  "rope_theta": 100000.0,
+  "rms_norm_eps": 1e-06,
+  "embed_width": 384,
+  "hidden_size_a": 256,
+  "num_hidden_layers_a": 14,
+  "num_attention_heads_a": 4,
+  "num_key_value_heads_a": 1,
+  "intermediate_size_a": 704,
+  "hidden_size_b": 320,
+  "num_hidden_layers_b": 5,
+  "num_attention_heads_b": 5,
+  "num_key_value_heads_b": 1,
+  "intermediate_size_b": 960,
+  "hidden_size_c": 384,
+  "num_hidden_layers_c": 6,
+  "num_attention_heads_c": 6,
+  "num_key_value_heads_c": 2,
+  "intermediate_size_c": 1024,
+  "a_read": [
+    5,
+    9,
+    12
+  ],
+  "a_land": [
+    7,
+    11,
+    14
+  ],
+  "c_read": [
+    2,
+    4,
+    5
+  ],
+  "c_land": [
+    3,
+    5,
+    6
+  ],
+  "b_land": [
+    1,
+    3,
+    5
+  ],
+  "b_read": [
+    2,
+    4,
+    5
+  ],
+  "gate_init": 0.01,
+  "cut_bridges": false,
+  "hidden_size": 384,
+  "tie_word_embeddings": false,
+  "bos_token_id": 1,
+  "eos_token_id": 2,
+  "pad_token_id": 0,
+  "torch_dtype": "float32",
+  "_name_or_path": "Banaxi-Tech/unified-2.1-test"
+}

configuration_bananamind21unified.py

@@ -0,0 +1,120 @@
+from transformers import PretrainedConfig
+
+
+class BananaMind21UnifiedConfig(PretrainedConfig):
+    """BananaMind 2.1 Unified: three towers, a relay middle, one token out.
+
+    A and C are the outer towers and each owns an output head.  B is the relay:
+    no head, no solo loss, and the only path between A and C.
+    """
+
+    model_type = "bananamind21_unified"
+
+    def __init__(
+        self,
+        vocab_size=8192,
+        head_dim=64,
+        max_position_embeddings=4096,
+        rope_theta=100000.0,
+        rms_norm_eps=1e-6,
+        embed_width=384,
+        hidden_size_a=256,
+        num_hidden_layers_a=14,
+        num_attention_heads_a=4,
+        num_key_value_heads_a=1,
+        intermediate_size_a=704,
+        hidden_size_b=320,
+        num_hidden_layers_b=5,
+        num_attention_heads_b=5,
+        num_key_value_heads_b=1,
+        intermediate_size_b=960,
+        hidden_size_c=384,
+        num_hidden_layers_c=6,
+        num_attention_heads_c=6,
+        num_key_value_heads_c=2,
+        intermediate_size_c=1024,
+        a_read=(5, 9, 12),
+        a_land=(7, 11, 14),
+        c_read=(2, 4, 5),
+        c_land=(3, 5, 6),
+        b_land=(1, 3, 5),
+        b_read=(2, 4, 5),
+        gate_init=0.01,
+        cut_bridges=False,
+        relay_mode=None,
+        use_single_tower=None,
+        use_cache=True,
+        tie_word_embeddings=False,
+        **kwargs,
+    ):
+        self.vocab_size = vocab_size
+        self.head_dim = head_dim
+        self.max_position_embeddings = max_position_embeddings
+        self.rope_theta = rope_theta
+        self.rms_norm_eps = rms_norm_eps
+        self.embed_width = embed_width
+
+        self.hidden_size_a = hidden_size_a
+        self.num_hidden_layers_a = num_hidden_layers_a
+        self.num_attention_heads_a = num_attention_heads_a
+        self.num_key_value_heads_a = num_key_value_heads_a
+        self.intermediate_size_a = intermediate_size_a
+
+        self.hidden_size_b = hidden_size_b
+        self.num_hidden_layers_b = num_hidden_layers_b
+        self.num_attention_heads_b = num_attention_heads_b
+        self.num_key_value_heads_b = num_key_value_heads_b
+        self.intermediate_size_b = intermediate_size_b
+
+        self.hidden_size_c = hidden_size_c
+        self.num_hidden_layers_c = num_hidden_layers_c
+        self.num_attention_heads_c = num_attention_heads_c
+        self.num_key_value_heads_c = num_key_value_heads_c
+        self.intermediate_size_c = intermediate_size_c
+
+        self.a_read = list(a_read)
+        self.a_land = list(a_land)
+        self.c_read = list(c_read)
+        self.c_land = list(c_land)
+        self.b_land = list(b_land)
+        self.b_read = list(b_read)
+        self.gate_init = gate_init
+
+        # Ablation controls.  `relay_mode` is the general switch - see
+        # `RELAY_MODES` in the modeling file for the full list:
+        #
+        #   full         three towers, all bridges, both heads mixed (default)
+        #   cut_bridges  every bridge severed; A and C run standalone
+        #   bypass_b     B's blocks skipped, bridges still live
+        #   ab_only      tower C off; A and B run; head A alone
+        #   cb_only      tower A off; C and B run; head C alone
+        #   a_only       tower A alone, no bridges
+        #   c_only       tower C alone, no bridges
+        #
+        # `use_single_tower="a"` / `"c"` is an alias for `a_only` / `c_only`, and
+        # `cut_bridges=True` predates both and still selects `cut_bridges`.
+        # Setting two of them to different modes is an error rather than a
+        # silent precedence rule.
+        self.cut_bridges = cut_bridges
+        self.relay_mode = relay_mode
+        self.use_single_tower = use_single_tower
+        self.use_cache = use_cache
+
+        # `hidden_size` is what generic HF tooling looks for
+        self.hidden_size = hidden_size_c
+
+        # Total attention layers over all three towers.  There is no single
+        # "number of layers" in a relay model, but the KV cache needs one flat
+        # index space, and generic HF tooling - `DynamicCache` above all - sizes
+        # itself from `num_hidden_layers`.  The flat order is A, then B, then C:
+        #
+        #     A -> 0 .. n_a-1        B -> n_a .. n_a+n_b-1        C -> n_a+n_b ..
+        #
+        # `BananaMind21UnifiedForCausalLM` assigns the matching `layer_idx` to
+        # every block, so the two must stay consistent.  Derived, never read
+        # from a serialised config, so a stale value cannot desync the mapping.
+        kwargs.pop("num_hidden_layers", None)
+        self.num_hidden_layers = (
+            num_hidden_layers_a + num_hidden_layers_b + num_hidden_layers_c
+        )
+        super().__init__(tie_word_embeddings=tie_word_embeddings, **kwargs)

eval_results/a_only/__home__banaxi__Desktop__BananaMind__BananaMind-2__BananaMind-2.1-Unified/results_2026-08-19T13-10-53.260045.json

@@ -0,0 +1,287 @@
+{
+  "results": {
+    "arc_easy": {
+      "name": "arc_easy",
+      "alias": "arc_easy",
+      "sample_len": 2376,
+      "acc,none": 0.2521043771043771,
+      "acc_stderr,none": 0.00891002416321825,
+      "acc_norm,none": 0.257996632996633,
+      "acc_norm_stderr,none": 0.00897797000520329
+    },
+    "piqa": {
+      "name": "piqa",
+      "alias": "piqa",
+      "sample_len": 1838,
+      "acc,none": 0.5331882480957563,
+      "acc_stderr,none": 0.011640096923563308,
+      "acc_norm,none": 0.501088139281828,
+      "acc_norm_stderr,none": 0.011665796539540774
+    },
+    "hellaswag": {
+      "name": "hellaswag",
+      "alias": "hellaswag",
+      "sample_len": 10042,
+      "acc,none": 0.2577175861382195,
+      "acc_stderr,none": 0.004364838000336011,
+      "acc_norm,none": 0.26110336586337385,
+      "acc_norm_stderr,none": 0.0043833847840387945
+    }
+  },
+  "group_subtasks": {},
+  "configs": {
+    "arc_easy": {
+      "task": "arc_easy",
+      "dataset_path": "allenai/ai2_arc",
+      "dataset_name": "ARC-Easy",
+      "training_split": "train",
+      "validation_split": "validation",
+      "test_split": "test",
+      "doc_to_text": "Question: {{question}}\nAnswer:",
+      "doc_to_target": "{{choices.label.index(answerKey)}}",
+      "unsafe_code": false,
+      "doc_to_choice": "{{choices.text}}",
+      "description": "",
+      "target_delimiter": " ",
+      "fewshot_delimiter": "\n\n",
+      "fewshot_config": {
+        "sampler": "default",
+        "split": null,
+        "process_docs": null,
+        "fewshot_indices": null,
+        "samples": null,
+        "doc_to_text": "Question: {{question}}\nAnswer:",
+        "doc_to_choice": "{{choices.text}}",
+        "doc_to_target": "{{choices.label.index(answerKey)}}",
+        "gen_prefix": null,
+        "fewshot_delimiter": "\n\n",
+        "target_delimiter": " "
+      },
+      "num_fewshot": 0,
+      "metric_list": [
+        {
+          "metric": "acc",
+          "aggregation": "mean",
+          "higher_is_better": true
+        },
+        {
+          "metric": "acc_norm",
+          "aggregation": "mean",
+          "higher_is_better": true
+        }
+      ],
+      "output_type": "multiple_choice",
+      "repeats": 1,
+      "should_decontaminate": true,
+      "doc_to_decontamination_query": "Question: {{question}}\nAnswer:",
+      "metadata": {
+        "version": 1.0,
+        "pretrained": "/home/banaxi/Desktop/BananaMind/BananaMind-2/BananaMind-2.1-Unified",
+        "trust_remote_code": true,
+        "dtype": "float32",
+        "use_cache": false,
+        "relay_mode": "a_only",
+        "config_source": "/home/banaxi/.local/lib/python3.14/site-packages/lm_eval/tasks/arc/arc_easy.yaml"
+      }
+    },
+    "hellaswag": {
+      "task": "hellaswag",
+      "dataset_path": "Rowan/hellaswag",
+      "training_split": "train",
+      "validation_split": "validation",
+      "process_docs": "def process_docs(dataset: datasets.Dataset) -> datasets.Dataset:\n    def _process_doc(doc):\n        ctx = doc[\"ctx_a\"] + \" \" + doc[\"ctx_b\"].capitalize()\n        out_doc = {\n            \"query\": preprocess(doc[\"activity_label\"] + \": \" + ctx),\n            \"choices\": [preprocess(ending) for ending in doc[\"endings\"]],\n            \"gold\": int(doc[\"label\"]),\n        }\n        return out_doc\n\n    return dataset.map(_process_doc)\n",
+      "doc_to_text": "{{query}}",
+      "doc_to_target": "{{label}}",
+      "unsafe_code": false,
+      "doc_to_choice": "choices",
+      "description": "",
+      "target_delimiter": " ",
+      "fewshot_delimiter": "\n\n",
+      "fewshot_config": {
+        "sampler": "default",
+        "split": null,
+        "process_docs": "<function process_docs at 0x7f75196e7950>",
+        "fewshot_indices": null,
+        "samples": null,
+        "doc_to_text": "{{query}}",
+        "doc_to_choice": "choices",
+        "doc_to_target": "{{label}}",
+        "gen_prefix": null,
+        "fewshot_delimiter": "\n\n",
+        "target_delimiter": " "
+      },
+      "num_fewshot": 0,
+      "metric_list": [
+        {
+          "metric": "acc",
+          "aggregation": "mean",
+          "higher_is_better": true
+        },
+        {
+          "metric": "acc_norm",
+          "aggregation": "mean",
+          "higher_is_better": true
+        }
+      ],
+      "output_type": "multiple_choice",
+      "repeats": 1,
+      "should_decontaminate": false,
+      "metadata": {
+        "version": 1.0,
+        "pretrained": "/home/banaxi/Desktop/BananaMind/BananaMind-2/BananaMind-2.1-Unified",
+        "trust_remote_code": true,
+        "dtype": "float32",
+        "use_cache": false,
+        "relay_mode": "a_only",
+        "config_source": "/home/banaxi/.local/lib/python3.14/site-packages/lm_eval/tasks/hellaswag/hellaswag.yaml"
+      }
+    },
+    "piqa": {
+      "task": "piqa",
+      "dataset_path": "baber/piqa",
+      "training_split": "train",
+      "validation_split": "validation",
+      "doc_to_text": "Question: {{goal}}\nAnswer:",
+      "doc_to_target": "label",
+      "unsafe_code": false,
+      "doc_to_choice": "{{[sol1, sol2]}}",
+      "description": "",
+      "target_delimiter": " ",
+      "fewshot_delimiter": "\n\n",
+      "fewshot_config": {
+        "sampler": "default",
+        "split": null,
+        "process_docs": null,
+        "fewshot_indices": null,
+        "samples": null,
+        "doc_to_text": "Question: {{goal}}\nAnswer:",
+        "doc_to_choice": "{{[sol1, sol2]}}",
+        "doc_to_target": "label",
+        "gen_prefix": null,
+        "fewshot_delimiter": "\n\n",
+        "target_delimiter": " "
+      },
+      "num_fewshot": 0,
+      "metric_list": [
+        {
+          "metric": "acc",
+          "aggregation": "mean",
+          "higher_is_better": true
+        },
+        {
+          "metric": "acc_norm",
+          "aggregation": "mean",
+          "higher_is_better": true
+        }
+      ],
+      "output_type": "multiple_choice",
+      "repeats": 1,
+      "should_decontaminate": true,
+      "doc_to_decontamination_query": "goal",
+      "metadata": {
+        "version": 1.0,
+        "pretrained": "/home/banaxi/Desktop/BananaMind/BananaMind-2/BananaMind-2.1-Unified",
+        "trust_remote_code": true,
+        "dtype": "float32",
+        "use_cache": false,
+        "relay_mode": "a_only",
+        "config_source": "/home/banaxi/.local/lib/python3.14/site-packages/lm_eval/tasks/piqa/piqa.yaml"
+      }
+    }
+  },
+  "versions": {
+    "arc_easy": 1.0,
+    "hellaswag": 1.0,
+    "piqa": 1.0
+  },
+  "n-shot": {
+    "arc_easy": 0,
+    "hellaswag": 0,
+    "piqa": 0
+  },
+  "higher_is_better": {
+    "arc_easy": {
+      "acc": true,
+      "acc_norm": true
+    },
+    "hellaswag": {
+      "acc": true,
+      "acc_norm": true
+    },
+    "piqa": {
+      "acc": true,
+      "acc_norm": true
+    }
+  },
+  "n-samples": {
+    "arc_easy": {
+      "original": 2376,
+      "effective": 2376
+    },
+    "piqa": {
+      "original": 1838,
+      "effective": 1838
+    },
+    "hellaswag": {
+      "original": 10042,
+      "effective": 10042
+    }
+  },
+  "config": {
+    "model": "hf",
+    "model_args": {
+      "pretrained": "/home/banaxi/Desktop/BananaMind/BananaMind-2/BananaMind-2.1-Unified",
+      "trust_remote_code": true,
+      "dtype": "float32",
+      "use_cache": false,
+      "relay_mode": "a_only"
+    },
+    "model_num_parameters": 34999041,
+    "model_dtype": "torch.float32",
+    "model_revision": "main",
+    "model_sha": "",
+    "batch_size": "auto",
+    "batch_sizes": [
+      64
+    ],
+    "device": "cuda:0",
+    "use_cache": null,
+    "limit": null,
+    "bootstrap_iters": 100000,
+    "gen_kwargs": {},
+    "random_seed": 0,
+    "numpy_seed": 1234,
+    "torch_seed": 1234,
+    "fewshot_seed": 1234
+  },
+  "git_hash": null,
+  "date": 1787137810.9721441,
+  "pretty_env_info": "PyTorch version: 2.11.0+cu130\nIs debug build: False\nCUDA used to build PyTorch: 13.0\nROCM used to build PyTorch: N/A\n\nOS: Arch Linux (x86_64)\nGCC version: (GCC) 16.2.1 20260810\nClang version: 22.1.8\nCMake version: version 4.4.2\nLibc version: glibc-2.44\n\nPython version: 3.14.7 (main, Aug 10 2026, 07:46:56) [GCC 16.1.1 20260728] (64-bit runtime)\nPython platform: Linux-7.1.8-arch1-3-x86_64-with-glibc2.44\nIs CUDA available: True\nCUDA runtime version: 13.3.73\nCUDA_MODULE_LOADING set to: \nGPU models and configuration: GPU 0: NVIDIA GeForce RTX 5070 Ti\nNvidia driver version: Could not collect\ncuDNN version: Probably one of the following:\n/usr/lib/libcudnn.so.9.25.0\n/usr/lib/libcudnn_adv.so.9.25.0\n/usr/lib/libcudnn_cnn.so.9.25.0\n/usr/lib/libcudnn_engines_precompiled.so.9.25.0\n/usr/lib/libcudnn_engines_runtime_compiled.so.9.25.0\n/usr/lib/libcudnn_engines_tensor_ir.so.9.25.0\n/usr/lib/libcudnn_ext.so.9.25.0\n/usr/lib/libcudnn_graph.so.9.25.0\n/usr/lib/libcudnn_heuristic.so.9.25.0\n/usr/lib/libcudnn_ops.so.9.25.0\nIs XPU available: False\nHIP runtime version: N/A\nMIOpen runtime version: N/A\nIs XNNPACK available: True\nCaching allocator config: N/A\n\nCPU:\nArchitecture:                            x86_64\nCPU op-mode(s):                          32-bit, 64-bit\nAddress sizes:                           39 bits physical, 48 bits virtual\nByte Order:                              Little Endian\nCPU(s):                                  24\nOn-line CPU(s) list:                     0-23\nVendor ID:                               GenuineIntel\nModel name:                              12th Gen Intel(R) Core(TM) i9-12900KF\nCPU family:                              6\nModel:                                   151\nThread(s) per core:                      2\nCore(s) per socket:                      16\nSocket(s):                               1\nStepping:                                2\nMicrocode version:                       0x3e\nCPU(s) scaling MHz:                      18%\nCPU max MHz:                             5200.0000\nCPU min MHz:                             800.0000\nBogoMIPS:                                6374.40\nFlags:                                   fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc art arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc cpuid aperfmperf tsc_known_freq pni pclmulqdq dtes64 monitor ds_cpl vmx est tm2 ssse3 sdbg fma cx16 xtpr pdcm pcid sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm abm 3dnowprefetch cpuid_fault ssbd ibrs ibpb stibp ibrs_enhanced tpr_shadow flexpriority ept vpid ept_ad fsgsbase tsc_adjust bmi1 avx2 smep bmi2 erms invpcid rdseed adx smap clflushopt clwb intel_pt sha_ni xsaveopt xsavec xgetbv1 xsaves split_lock_detect user_shstk avx_vnni dtherm ida arat pln pts hwp hwp_notify hwp_act_window hwp_epp hwp_pkg_req hfi vnmi umip pku ospke waitpkg gfni vaes vpclmulqdq rdpid movdiri movdir64b fsrm md_clear serialize arch_lbr ibt flush_l1d arch_capabilities\nVirtualization:                          VT-x\nL1d cache:                               640 KiB (16 instances)\nL1i cache:                               768 KiB (16 instances)\nL2 cache:                                14 MiB (10 instances)\nL3 cache:                                30 MiB (1 instance)\nNUMA node(s):                            1\nNUMA node0 CPU(s):                       0-23\nVulnerability Gather data sampling:      Not affected\nVulnerability Ghostwrite:                Not affected\nVulnerability Indirect target selection: Not affected\nVulnerability Itlb multihit:             Not affected\nVulnerability L1tf:                      Not affected\nVulnerability Mds:                       Not affected\nVulnerability Meltdown:                  Not affected\nVulnerability Mmio stale data:           Not affected\nVulnerability Old microcode:             Not affected\nVulnerability Reg file data sampling:    Mitigation; Clear Register File\nVulnerability Retbleed:                  Not affected\nVulnerability Spec rstack overflow:      Not affected\nVulnerability Spec store bypass:         Mitigation; Speculative Store Bypass disabled via prctl\nVulnerability Spectre v1:                Mitigation; usercopy/swapgs barriers and __user pointer sanitization\nVulnerability Spectre v2:                Mitigation; Enhanced / Automatic IBRS; IBPB conditional; PBRSB-eIBRS SW sequence; BHI BHI_DIS_S\nVulnerability Srbds:                     Not affected\nVulnerability Tsa:                       Not affected\nVulnerability Tsx async abort:           Not affected\nVulnerability Vmscape:                   Mitigation; IBPB before exit to userspace\n\nVersions of relevant libraries:\n[pip3] numpy==2.5.2\n[pip3] nvidia-cublas==13.1.0.3\n[pip3] nvidia-cuda-cupti==13.0.85\n[pip3] nvidia-cuda-nvrtc==13.0.88\n[pip3] nvidia-cuda-runtime==13.0.96\n[pip3] nvidia-cudnn-cu13==9.19.0.56\n[pip3] nvidia-cufft==12.0.0.61\n[pip3] nvidia-curand==10.4.0.35\n[pip3] nvidia-cusolver==12.0.4.66\n[pip3] nvidia-cusparse==12.6.3.3\n[pip3] nvidia-cusparselt-cu13==0.8.0\n[pip3] nvidia-nccl-cu13==2.28.9\n[pip3] nvidia-nvjitlink==13.0.88\n[pip3] nvidia-nvtx==13.0.85\n[pip3] segmentation_models_pytorch==0.5.0\n[pip3] torch==2.11.0\n[pip3] torchao==0.17.0\n[pip3] torchaudio==2.11.0\n[pip3] torchcodec==0.16.0\n[pip3] torchvision==0.26.0\n[pip3] triton==3.6.0\n[conda] Could not collect",
+  "transformers_version": "5.7.0",
+  "lm_eval_version": "0.4.12",
+  "upper_git_hash": null,
+  "tokenizer_pad_token": [
+    "<|pad|>",
+    "0"
+  ],
+  "tokenizer_eos_token": [
+    "<|eos|>",
+    "2"
+  ],
+  "tokenizer_bos_token": [
+    "<|bos|>",
+    "1"
+  ],
+  "eot_token_id": 2,
+  "max_length": 4096,
+  "task_hashes": {},
+  "model_source": "hf",
+  "model_name": "/home/banaxi/Desktop/BananaMind/BananaMind-2/BananaMind-2.1-Unified",
+  "model_name_sanitized": "__home__banaxi__Desktop__BananaMind__BananaMind-2__BananaMind-2.1-Unified",
+  "system_instruction": null,
+  "system_instruction_sha": null,
+  "fewshot_as_multiturn": null,
+  "chat_template": null,
+  "chat_template_sha": null,
+  "total_evaluation_time_seconds": "44.84569375999854"
+}

eval_results/ab_only/__home__banaxi__Desktop__BananaMind__BananaMind-2__BananaMind-2.1-Unified/results_2026-08-19T13-08-19.026128.json

@@ -0,0 +1,287 @@
+{
+  "results": {
+    "arc_easy": {
+      "name": "arc_easy",
+      "alias": "arc_easy",
+      "sample_len": 2376,
+      "acc,none": 0.26304713804713803,
+      "acc_stderr,none": 0.009034514898865944,
+      "acc_norm,none": 0.2756734006734007,
+      "acc_norm_stderr,none": 0.009169229476542745
+    },
+    "piqa": {
+      "name": "piqa",
+      "alias": "piqa",
+      "sample_len": 1838,
+      "acc,none": 0.5397170837867247,
+      "acc_stderr,none": 0.011628961491718543,
+      "acc_norm,none": 0.528835690968444,
+      "acc_norm_stderr,none": 0.011646407809944788
+    },
+    "hellaswag": {
+      "name": "hellaswag",
+      "alias": "hellaswag",
+      "sample_len": 10042,
+      "acc,none": 0.25721967735510853,
+      "acc_stderr,none": 0.004362081806560604,
+      "acc_norm,none": 0.25851424019119695,
+      "acc_norm_stderr,none": 0.004369232540126326
+    }
+  },
+  "group_subtasks": {},
+  "configs": {
+    "arc_easy": {
+      "task": "arc_easy",
+      "dataset_path": "allenai/ai2_arc",
+      "dataset_name": "ARC-Easy",
+      "training_split": "train",
+      "validation_split": "validation",
+      "test_split": "test",
+      "doc_to_text": "Question: {{question}}\nAnswer:",
+      "doc_to_target": "{{choices.label.index(answerKey)}}",
+      "unsafe_code": false,
+      "doc_to_choice": "{{choices.text}}",
+      "description": "",
+      "target_delimiter": " ",
+      "fewshot_delimiter": "\n\n",
+      "fewshot_config": {
+        "sampler": "default",
+        "split": null,
+        "process_docs": null,
+        "fewshot_indices": null,
+        "samples": null,
+        "doc_to_text": "Question: {{question}}\nAnswer:",
+        "doc_to_choice": "{{choices.text}}",
+        "doc_to_target": "{{choices.label.index(answerKey)}}",
+        "gen_prefix": null,
+        "fewshot_delimiter": "\n\n",
+        "target_delimiter": " "
+      },
+      "num_fewshot": 0,
+      "metric_list": [
+        {
+          "metric": "acc",
+          "aggregation": "mean",
+          "higher_is_better": true
+        },
+        {
+          "metric": "acc_norm",
+          "aggregation": "mean",
+          "higher_is_better": true
+        }
+      ],
+      "output_type": "multiple_choice",
+      "repeats": 1,
+      "should_decontaminate": true,
+      "doc_to_decontamination_query": "Question: {{question}}\nAnswer:",
+      "metadata": {
+        "version": 1.0,
+        "pretrained": "/home/banaxi/Desktop/BananaMind/BananaMind-2/BananaMind-2.1-Unified",
+        "trust_remote_code": true,
+        "dtype": "float32",
+        "use_cache": false,
+        "relay_mode": "ab_only",
+        "config_source": "/home/banaxi/.local/lib/python3.14/site-packages/lm_eval/tasks/arc/arc_easy.yaml"
+      }
+    },
+    "hellaswag": {
+      "task": "hellaswag",
+      "dataset_path": "Rowan/hellaswag",
+      "training_split": "train",
+      "validation_split": "validation",
+      "process_docs": "def process_docs(dataset: datasets.Dataset) -> datasets.Dataset:\n    def _process_doc(doc):\n        ctx = doc[\"ctx_a\"] + \" \" + doc[\"ctx_b\"].capitalize()\n        out_doc = {\n            \"query\": preprocess(doc[\"activity_label\"] + \": \" + ctx),\n            \"choices\": [preprocess(ending) for ending in doc[\"endings\"]],\n            \"gold\": int(doc[\"label\"]),\n        }\n        return out_doc\n\n    return dataset.map(_process_doc)\n",
+      "doc_to_text": "{{query}}",
+      "doc_to_target": "{{label}}",
+      "unsafe_code": false,
+      "doc_to_choice": "choices",
+      "description": "",
+      "target_delimiter": " ",
+      "fewshot_delimiter": "\n\n",
+      "fewshot_config": {
+        "sampler": "default",
+        "split": null,
+        "process_docs": "<function process_docs at 0x7f4831777950>",
+        "fewshot_indices": null,
+        "samples": null,
+        "doc_to_text": "{{query}}",
+        "doc_to_choice": "choices",
+        "doc_to_target": "{{label}}",
+        "gen_prefix": null,
+        "fewshot_delimiter": "\n\n",
+        "target_delimiter": " "
+      },
+      "num_fewshot": 0,
+      "metric_list": [
+        {
+          "metric": "acc",
+          "aggregation": "mean",
+          "higher_is_better": true
+        },
+        {
+          "metric": "acc_norm",
+          "aggregation": "mean",
+          "higher_is_better": true
+        }
+      ],
+      "output_type": "multiple_choice",
+      "repeats": 1,
+      "should_decontaminate": false,
+      "metadata": {
+        "version": 1.0,
+        "pretrained": "/home/banaxi/Desktop/BananaMind/BananaMind-2/BananaMind-2.1-Unified",
+        "trust_remote_code": true,
+        "dtype": "float32",
+        "use_cache": false,
+        "relay_mode": "ab_only",
+        "config_source": "/home/banaxi/.local/lib/python3.14/site-packages/lm_eval/tasks/hellaswag/hellaswag.yaml"
+      }
+    },
+    "piqa": {
+      "task": "piqa",
+      "dataset_path": "baber/piqa",
+      "training_split": "train",
+      "validation_split": "validation",
+      "doc_to_text": "Question: {{goal}}\nAnswer:",
+      "doc_to_target": "label",
+      "unsafe_code": false,
+      "doc_to_choice": "{{[sol1, sol2]}}",
+      "description": "",
+      "target_delimiter": " ",
+      "fewshot_delimiter": "\n\n",
+      "fewshot_config": {
+        "sampler": "default",
+        "split": null,
+        "process_docs": null,
+        "fewshot_indices": null,
+        "samples": null,
+        "doc_to_text": "Question: {{goal}}\nAnswer:",
+        "doc_to_choice": "{{[sol1, sol2]}}",
+        "doc_to_target": "label",
+        "gen_prefix": null,
+        "fewshot_delimiter": "\n\n",
+        "target_delimiter": " "
+      },
+      "num_fewshot": 0,
+      "metric_list": [
+        {
+          "metric": "acc",
+          "aggregation": "mean",
+          "higher_is_better": true
+        },
+        {
+          "metric": "acc_norm",
+          "aggregation": "mean",
+          "higher_is_better": true
+        }
+      ],
+      "output_type": "multiple_choice",
+      "repeats": 1,
+      "should_decontaminate": true,
+      "doc_to_decontamination_query": "goal",
+      "metadata": {
+        "version": 1.0,
+        "pretrained": "/home/banaxi/Desktop/BananaMind/BananaMind-2/BananaMind-2.1-Unified",
+        "trust_remote_code": true,
+        "dtype": "float32",
+        "use_cache": false,
+        "relay_mode": "ab_only",
+        "config_source": "/home/banaxi/.local/lib/python3.14/site-packages/lm_eval/tasks/piqa/piqa.yaml"
+      }
+    }
+  },
+  "versions": {
+    "arc_easy": 1.0,
+    "hellaswag": 1.0,
+    "piqa": 1.0
+  },
+  "n-shot": {
+    "arc_easy": 0,
+    "hellaswag": 0,
+    "piqa": 0
+  },
+  "higher_is_better": {
+    "arc_easy": {
+      "acc": true,
+      "acc_norm": true
+    },
+    "hellaswag": {
+      "acc": true,
+      "acc_norm": true
+    },
+    "piqa": {
+      "acc": true,
+      "acc_norm": true
+    }
+  },
+  "n-samples": {
+    "arc_easy": {
+      "original": 2376,
+      "effective": 2376
+    },
+    "piqa": {
+      "original": 1838,
+      "effective": 1838
+    },
+    "hellaswag": {
+      "original": 10042,
+      "effective": 10042
+    }
+  },
+  "config": {
+    "model": "hf",
+    "model_args": {
+      "pretrained": "/home/banaxi/Desktop/BananaMind/BananaMind-2/BananaMind-2.1-Unified",
+      "trust_remote_code": true,
+      "dtype": "float32",
+      "use_cache": false,
+      "relay_mode": "ab_only"
+    },
+    "model_num_parameters": 34999041,
+    "model_dtype": "torch.float32",
+    "model_revision": "main",
+    "model_sha": "",
+    "batch_size": "auto",
+    "batch_sizes": [
+      64
+    ],
+    "device": "cuda:0",
+    "use_cache": null,
+    "limit": null,
+    "bootstrap_iters": 100000,
+    "gen_kwargs": {},
+    "random_seed": 0,
+    "numpy_seed": 1234,
+    "torch_seed": 1234,
+    "fewshot_seed": 1234
+  },
+  "git_hash": null,
+  "date": 1787137649.6364806,
+  "pretty_env_info": "PyTorch version: 2.11.0+cu130\nIs debug build: False\nCUDA used to build PyTorch: 13.0\nROCM used to build PyTorch: N/A\n\nOS: Arch Linux (x86_64)\nGCC version: (GCC) 16.2.1 20260810\nClang version: 22.1.8\nCMake version: version 4.4.2\nLibc version: glibc-2.44\n\nPython version: 3.14.7 (main, Aug 10 2026, 07:46:56) [GCC 16.1.1 20260728] (64-bit runtime)\nPython platform: Linux-7.1.8-arch1-3-x86_64-with-glibc2.44\nIs CUDA available: True\nCUDA runtime version: 13.3.73\nCUDA_MODULE_LOADING set to: \nGPU models and configuration: GPU 0: NVIDIA GeForce RTX 5070 Ti\nNvidia driver version: Could not collect\ncuDNN version: Probably one of the following:\n/usr/lib/libcudnn.so.9.25.0\n/usr/lib/libcudnn_adv.so.9.25.0\n/usr/lib/libcudnn_cnn.so.9.25.0\n/usr/lib/libcudnn_engines_precompiled.so.9.25.0\n/usr/lib/libcudnn_engines_runtime_compiled.so.9.25.0\n/usr/lib/libcudnn_engines_tensor_ir.so.9.25.0\n/usr/lib/libcudnn_ext.so.9.25.0\n/usr/lib/libcudnn_graph.so.9.25.0\n/usr/lib/libcudnn_heuristic.so.9.25.0\n/usr/lib/libcudnn_ops.so.9.25.0\nIs XPU available: False\nHIP runtime version: N/A\nMIOpen runtime version: N/A\nIs XNNPACK available: True\nCaching allocator config: N/A\n\nCPU:\nArchitecture:                            x86_64\nCPU op-mode(s):                          32-bit, 64-bit\nAddress sizes:                           39 bits physical, 48 bits virtual\nByte Order:                              Little Endian\nCPU(s):                                  24\nOn-line CPU(s) list:                     0-23\nVendor ID:                               GenuineIntel\nModel name:                              12th Gen Intel(R) Core(TM) i9-12900KF\nCPU family:                              6\nModel:                                   151\nThread(s) per core:                      2\nCore(s) per socket:                      16\nSocket(s):                               1\nStepping:                                2\nMicrocode version:                       0x3e\nCPU(s) scaling MHz:                      22%\nCPU max MHz:                             5200.0000\nCPU min MHz:                             800.0000\nBogoMIPS:                                6374.40\nFlags:                                   fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc art arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc cpuid aperfmperf tsc_known_freq pni pclmulqdq dtes64 monitor ds_cpl vmx est tm2 ssse3 sdbg fma cx16 xtpr pdcm pcid sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm abm 3dnowprefetch cpuid_fault ssbd ibrs ibpb stibp ibrs_enhanced tpr_shadow flexpriority ept vpid ept_ad fsgsbase tsc_adjust bmi1 avx2 smep bmi2 erms invpcid rdseed adx smap clflushopt clwb intel_pt sha_ni xsaveopt xsavec xgetbv1 xsaves split_lock_detect user_shstk avx_vnni dtherm ida arat pln pts hwp hwp_notify hwp_act_window hwp_epp hwp_pkg_req hfi vnmi umip pku ospke waitpkg gfni vaes vpclmulqdq rdpid movdiri movdir64b fsrm md_clear serialize arch_lbr ibt flush_l1d arch_capabilities\nVirtualization:                          VT-x\nL1d cache:                               640 KiB (16 instances)\nL1i cache:                               768 KiB (16 instances)\nL2 cache:                                14 MiB (10 instances)\nL3 cache:                                30 MiB (1 instance)\nNUMA node(s):                            1\nNUMA node0 CPU(s):                       0-23\nVulnerability Gather data sampling:      Not affected\nVulnerability Ghostwrite:                Not affected\nVulnerability Indirect target selection: Not affected\nVulnerability Itlb multihit:             Not affected\nVulnerability L1tf:                      Not affected\nVulnerability Mds:                       Not affected\nVulnerability Meltdown:                  Not affected\nVulnerability Mmio stale data:           Not affected\nVulnerability Old microcode:             Not affected\nVulnerability Reg file data sampling:    Mitigation; Clear Register File\nVulnerability Retbleed:                  Not affected\nVulnerability Spec rstack overflow:      Not affected\nVulnerability Spec store bypass:         Mitigation; Speculative Store Bypass disabled via prctl\nVulnerability Spectre v1:                Mitigation; usercopy/swapgs barriers and __user pointer sanitization\nVulnerability Spectre v2:                Mitigation; Enhanced / Automatic IBRS; IBPB conditional; PBRSB-eIBRS SW sequence; BHI BHI_DIS_S\nVulnerability Srbds:                     Not affected\nVulnerability Tsa:                       Not affected\nVulnerability Tsx async abort:           Not affected\nVulnerability Vmscape:                   Mitigation; IBPB before exit to userspace\n\nVersions of relevant libraries:\n[pip3] numpy==2.5.2\n[pip3] nvidia-cublas==13.1.0.3\n[pip3] nvidia-cuda-cupti==13.0.85\n[pip3] nvidia-cuda-nvrtc==13.0.88\n[pip3] nvidia-cuda-runtime==13.0.96\n[pip3] nvidia-cudnn-cu13==9.19.0.56\n[pip3] nvidia-cufft==12.0.0.61\n[pip3] nvidia-curand==10.4.0.35\n[pip3] nvidia-cusolver==12.0.4.66\n[pip3] nvidia-cusparse==12.6.3.3\n[pip3] nvidia-cusparselt-cu13==0.8.0\n[pip3] nvidia-nccl-cu13==2.28.9\n[pip3] nvidia-nvjitlink==13.0.88\n[pip3] nvidia-nvtx==13.0.85\n[pip3] segmentation_models_pytorch==0.5.0\n[pip3] torch==2.11.0\n[pip3] torchao==0.17.0\n[pip3] torchaudio==2.11.0\n[pip3] torchcodec==0.16.0\n[pip3] torchvision==0.26.0\n[pip3] triton==3.6.0\n[conda] Could not collect",
+  "transformers_version": "5.7.0",
+  "lm_eval_version": "0.4.12",
+  "upper_git_hash": null,
+  "tokenizer_pad_token": [
+    "<|pad|>",
+    "0"
+  ],
+  "tokenizer_eos_token": [
+    "<|eos|>",
+    "2"
+  ],
+  "tokenizer_bos_token": [
+    "<|bos|>",
+    "1"
+  ],
+  "eot_token_id": 2,
+  "max_length": 4096,
+  "task_hashes": {},
+  "model_source": "hf",
+  "model_name": "/home/banaxi/Desktop/BananaMind/BananaMind-2/BananaMind-2.1-Unified",
+  "model_name_sanitized": "__home__banaxi__Desktop__BananaMind__BananaMind-2__BananaMind-2.1-Unified",
+  "system_instruction": null,
+  "system_instruction_sha": null,
+  "fewshot_as_multiturn": null,
+  "chat_template": null,
+  "chat_template_sha": null,
+  "total_evaluation_time_seconds": "51.955665409972426"
+}

eval_results/base_bench_1.1/a_only/run_config.json

@@ -0,0 +1,12 @@
+{
+  "benchmark": "BananaMind Base Bench 1.1",
+  "model": "/home/banaxi/Desktop/BananaMind/BananaMind-2/BananaMind-2.1-Unified",
+  "tokenizer": "/home/banaxi/Desktop/BananaMind/BananaMind-2/BananaMind-2.1-Unified",
+  "model_revision": "main",
+  "dataset_id": "BananaMind/BananaMind-Base-Bench-1.1",
+  "dataset_revision": "main",
+  "dataset_sha256": "2f563bb46df778ca494fa20f994a8d3045d4c51fbbffeee433764e2813abea21",
+  "score_mode": "mean_logprob",
+  "add_bos": false,
+  "max_context": null
+}

eval_results/base_bench_1.1/ab_only/run_config.json

@@ -0,0 +1,12 @@
+{
+  "benchmark": "BananaMind Base Bench 1.1",
+  "model": "/home/banaxi/Desktop/BananaMind/BananaMind-2/BananaMind-2.1-Unified",
+  "tokenizer": "/home/banaxi/Desktop/BananaMind/BananaMind-2/BananaMind-2.1-Unified",
+  "model_revision": "main",
+  "dataset_id": "BananaMind/BananaMind-Base-Bench-1.1",
+  "dataset_revision": "main",
+  "dataset_sha256": "2f563bb46df778ca494fa20f994a8d3045d4c51fbbffeee433764e2813abea21",
+  "score_mode": "mean_logprob",
+  "add_bos": false,
+  "max_context": null
+}

eval_results/base_bench_1.1/bypass_b/run_config.json

@@ -0,0 +1,12 @@
+{
+  "benchmark": "BananaMind Base Bench 1.1",
+  "model": "/home/banaxi/Desktop/BananaMind/BananaMind-2/BananaMind-2.1-Unified",
+  "tokenizer": "/home/banaxi/Desktop/BananaMind/BananaMind-2/BananaMind-2.1-Unified",
+  "model_revision": "main",
+  "dataset_id": "BananaMind/BananaMind-Base-Bench-1.1",
+  "dataset_revision": "main",
+  "dataset_sha256": "2f563bb46df778ca494fa20f994a8d3045d4c51fbbffeee433764e2813abea21",
+  "score_mode": "mean_logprob",
+  "add_bos": false,
+  "max_context": null
+}

eval_results/base_bench_1.1/c_only/run_config.json

@@ -0,0 +1,12 @@
+{
+  "benchmark": "BananaMind Base Bench 1.1",
+  "model": "/home/banaxi/Desktop/BananaMind/BananaMind-2/BananaMind-2.1-Unified",
+  "tokenizer": "/home/banaxi/Desktop/BananaMind/BananaMind-2/BananaMind-2.1-Unified",
+  "model_revision": "main",
+  "dataset_id": "BananaMind/BananaMind-Base-Bench-1.1",
+  "dataset_revision": "main",
+  "dataset_sha256": "2f563bb46df778ca494fa20f994a8d3045d4c51fbbffeee433764e2813abea21",
+  "score_mode": "mean_logprob",
+  "add_bos": false,
+  "max_context": null
+}

eval_results/base_bench_1.1/cb_only/run_config.json

@@ -0,0 +1,12 @@
+{
+  "benchmark": "BananaMind Base Bench 1.1",
+  "model": "/home/banaxi/Desktop/BananaMind/BananaMind-2/BananaMind-2.1-Unified",
+  "tokenizer": "/home/banaxi/Desktop/BananaMind/BananaMind-2/BananaMind-2.1-Unified",
+  "model_revision": "main",
+  "dataset_id": "BananaMind/BananaMind-Base-Bench-1.1",
+  "dataset_revision": "main",
+  "dataset_sha256": "2f563bb46df778ca494fa20f994a8d3045d4c51fbbffeee433764e2813abea21",
+  "score_mode": "mean_logprob",
+  "add_bos": false,
+  "max_context": null
+}

eval_results/base_bench_1.1/cut_bridges/run_config.json

@@ -0,0 +1,12 @@
+{
+  "benchmark": "BananaMind Base Bench 1.1",
+  "model": "/home/banaxi/Desktop/BananaMind/BananaMind-2/BananaMind-2.1-Unified",
+  "tokenizer": "/home/banaxi/Desktop/BananaMind/BananaMind-2/BananaMind-2.1-Unified",
+  "model_revision": "main",
+  "dataset_id": "BananaMind/BananaMind-Base-Bench-1.1",
+  "dataset_revision": "main",
+  "dataset_sha256": "2f563bb46df778ca494fa20f994a8d3045d4c51fbbffeee433764e2813abea21",
+  "score_mode": "mean_logprob",
+  "add_bos": false,
+  "max_context": null
+}

eval_results/base_bench_1.1/full/run_config.json

@@ -0,0 +1,12 @@
+{
+  "benchmark": "BananaMind Base Bench 1.1",
+  "model": "/home/banaxi/Desktop/BananaMind/BananaMind-2/BananaMind-2.1-Unified",
+  "tokenizer": "/home/banaxi/Desktop/BananaMind/BananaMind-2/BananaMind-2.1-Unified",
+  "model_revision": "main",
+  "dataset_id": "BananaMind/BananaMind-Base-Bench-1.1",
+  "dataset_revision": "main",
+  "dataset_sha256": "2f563bb46df778ca494fa20f994a8d3045d4c51fbbffeee433764e2813abea21",
+  "score_mode": "mean_logprob",
+  "add_bos": false,
+  "max_context": null
+}

eval_results/bypass_b/__home__banaxi__Desktop__BananaMind__BananaMind-2__BananaMind-2.1-Unified/results_2026-08-19T13-07-25.864618.json

@@ -0,0 +1,287 @@
+{
+  "results": {
+    "arc_easy": {
+      "name": "arc_easy",
+      "alias": "arc_easy",
+      "sample_len": 2376,
+      "acc,none": 0.3345959595959596,
+      "acc_stderr,none": 0.009682137724327718,
+      "acc_norm,none": 0.3367003367003367,
+      "acc_norm_stderr,none": 0.009697166595752359
+    },
+    "piqa": {
+      "name": "piqa",
+      "alias": "piqa",
+      "sample_len": 1838,
+      "acc,none": 0.5707290533188248,
+      "acc_stderr,none": 0.01154851547125808,
+      "acc_norm,none": 0.55930359085963,
+      "acc_norm_stderr,none": 0.011583478090656981
+    },
+    "hellaswag": {
+      "name": "hellaswag",
+      "alias": "hellaswag",
+      "sample_len": 10042,
+      "acc,none": 0.276638119896435,
+      "acc_stderr,none": 0.004464217420693715,
+      "acc_norm,none": 0.2922724556861183,
+      "acc_norm_stderr,none": 0.004538773493746909
+    }
+  },
+  "group_subtasks": {},
+  "configs": {
+    "arc_easy": {
+      "task": "arc_easy",
+      "dataset_path": "allenai/ai2_arc",
+      "dataset_name": "ARC-Easy",
+      "training_split": "train",
+      "validation_split": "validation",
+      "test_split": "test",
+      "doc_to_text": "Question: {{question}}\nAnswer:",
+      "doc_to_target": "{{choices.label.index(answerKey)}}",
+      "unsafe_code": false,
+      "doc_to_choice": "{{choices.text}}",
+      "description": "",
+      "target_delimiter": " ",
+      "fewshot_delimiter": "\n\n",
+      "fewshot_config": {
+        "sampler": "default",
+        "split": null,
+        "process_docs": null,
+        "fewshot_indices": null,
+        "samples": null,
+        "doc_to_text": "Question: {{question}}\nAnswer:",
+        "doc_to_choice": "{{choices.text}}",
+        "doc_to_target": "{{choices.label.index(answerKey)}}",
+        "gen_prefix": null,
+        "fewshot_delimiter": "\n\n",
+        "target_delimiter": " "
+      },
+      "num_fewshot": 0,
+      "metric_list": [
+        {
+          "metric": "acc",
+          "aggregation": "mean",
+          "higher_is_better": true
+        },
+        {
+          "metric": "acc_norm",
+          "aggregation": "mean",
+          "higher_is_better": true
+        }
+      ],
+      "output_type": "multiple_choice",
+      "repeats": 1,
+      "should_decontaminate": true,
+      "doc_to_decontamination_query": "Question: {{question}}\nAnswer:",
+      "metadata": {
+        "version": 1.0,
+        "pretrained": "/home/banaxi/Desktop/BananaMind/BananaMind-2/BananaMind-2.1-Unified",
+        "trust_remote_code": true,
+        "dtype": "float32",
+        "use_cache": false,
+        "relay_mode": "bypass_b",
+        "config_source": "/home/banaxi/.local/lib/python3.14/site-packages/lm_eval/tasks/arc/arc_easy.yaml"
+      }
+    },
+    "hellaswag": {
+      "task": "hellaswag",
+      "dataset_path": "Rowan/hellaswag",
+      "training_split": "train",
+      "validation_split": "validation",
+      "process_docs": "def process_docs(dataset: datasets.Dataset) -> datasets.Dataset:\n    def _process_doc(doc):\n        ctx = doc[\"ctx_a\"] + \" \" + doc[\"ctx_b\"].capitalize()\n        out_doc = {\n            \"query\": preprocess(doc[\"activity_label\"] + \": \" + ctx),\n            \"choices\": [preprocess(ending) for ending in doc[\"endings\"]],\n            \"gold\": int(doc[\"label\"]),\n        }\n        return out_doc\n\n    return dataset.map(_process_doc)\n",
+      "doc_to_text": "{{query}}",
+      "doc_to_target": "{{label}}",
+      "unsafe_code": false,
+      "doc_to_choice": "choices",
+      "description": "",
+      "target_delimiter": " ",
+      "fewshot_delimiter": "\n\n",
+      "fewshot_config": {
+        "sampler": "default",
+        "split": null,
+        "process_docs": "<function process_docs at 0x7fefeb1f7950>",
+        "fewshot_indices": null,
+        "samples": null,
+        "doc_to_text": "{{query}}",
+        "doc_to_choice": "choices",
+        "doc_to_target": "{{label}}",
+        "gen_prefix": null,
+        "fewshot_delimiter": "\n\n",
+        "target_delimiter": " "
+      },
+      "num_fewshot": 0,
+      "metric_list": [
+        {
+          "metric": "acc",
+          "aggregation": "mean",
+          "higher_is_better": true
+        },
+        {
+          "metric": "acc_norm",
+          "aggregation": "mean",
+          "higher_is_better": true
+        }
+      ],
+      "output_type": "multiple_choice",
+      "repeats": 1,
+      "should_decontaminate": false,
+      "metadata": {
+        "version": 1.0,
+        "pretrained": "/home/banaxi/Desktop/BananaMind/BananaMind-2/BananaMind-2.1-Unified",
+        "trust_remote_code": true,
+        "dtype": "float32",
+        "use_cache": false,
+        "relay_mode": "bypass_b",
+        "config_source": "/home/banaxi/.local/lib/python3.14/site-packages/lm_eval/tasks/hellaswag/hellaswag.yaml"
+      }
+    },
+    "piqa": {
+      "task": "piqa",
+      "dataset_path": "baber/piqa",
+      "training_split": "train",
+      "validation_split": "validation",
+      "doc_to_text": "Question: {{goal}}\nAnswer:",
+      "doc_to_target": "label",
+      "unsafe_code": false,
+      "doc_to_choice": "{{[sol1, sol2]}}",
+      "description": "",
+      "target_delimiter": " ",
+      "fewshot_delimiter": "\n\n",
+      "fewshot_config": {
+        "sampler": "default",
+        "split": null,
+        "process_docs": null,
+        "fewshot_indices": null,
+        "samples": null,
+        "doc_to_text": "Question: {{goal}}\nAnswer:",
+        "doc_to_choice": "{{[sol1, sol2]}}",
+        "doc_to_target": "label",
+        "gen_prefix": null,
+        "fewshot_delimiter": "\n\n",
+        "target_delimiter": " "
+      },
+      "num_fewshot": 0,
+      "metric_list": [
+        {
+          "metric": "acc",
+          "aggregation": "mean",
+          "higher_is_better": true
+        },
+        {
+          "metric": "acc_norm",
+          "aggregation": "mean",
+          "higher_is_better": true
+        }
+      ],
+      "output_type": "multiple_choice",
+      "repeats": 1,
+      "should_decontaminate": true,
+      "doc_to_decontamination_query": "goal",
+      "metadata": {
+        "version": 1.0,
+        "pretrained": "/home/banaxi/Desktop/BananaMind/BananaMind-2/BananaMind-2.1-Unified",
+        "trust_remote_code": true,
+        "dtype": "float32",
+        "use_cache": false,
+        "relay_mode": "bypass_b",
+        "config_source": "/home/banaxi/.local/lib/python3.14/site-packages/lm_eval/tasks/piqa/piqa.yaml"
+      }
+    }
+  },
+  "versions": {
+    "arc_easy": 1.0,
+    "hellaswag": 1.0,
+    "piqa": 1.0
+  },
+  "n-shot": {
+    "arc_easy": 0,
+    "hellaswag": 0,
+    "piqa": 0
+  },
+  "higher_is_better": {
+    "arc_easy": {
+      "acc": true,
+      "acc_norm": true
+    },
+    "hellaswag": {
+      "acc": true,
+      "acc_norm": true
+    },
+    "piqa": {
+      "acc": true,
+      "acc_norm": true
+    }
+  },
+  "n-samples": {
+    "arc_easy": {
+      "original": 2376,
+      "effective": 2376
+    },
+    "piqa": {
+      "original": 1838,
+      "effective": 1838
+    },
+    "hellaswag": {
+      "original": 10042,
+      "effective": 10042
+    }
+  },
+  "config": {
+    "model": "hf",
+    "model_args": {
+      "pretrained": "/home/banaxi/Desktop/BananaMind/BananaMind-2/BananaMind-2.1-Unified",
+      "trust_remote_code": true,
+      "dtype": "float32",
+      "use_cache": false,
+      "relay_mode": "bypass_b"
+    },
+    "model_num_parameters": 34999041,
+    "model_dtype": "torch.float32",
+    "model_revision": "main",
+    "model_sha": "",
+    "batch_size": "auto",
+    "batch_sizes": [
+      64
+    ],
+    "device": "cuda:0",
+    "use_cache": null,
+    "limit": null,
+    "bootstrap_iters": 100000,
+    "gen_kwargs": {},
+    "random_seed": 0,
+    "numpy_seed": 1234,
+    "torch_seed": 1234,
+    "fewshot_seed": 1234
+  },
+  "git_hash": null,
+  "date": 1787137593.3229036,
+  "pretty_env_info": "PyTorch version: 2.11.0+cu130\nIs debug build: False\nCUDA used to build PyTorch: 13.0\nROCM used to build PyTorch: N/A\n\nOS: Arch Linux (x86_64)\nGCC version: (GCC) 16.2.1 20260810\nClang version: 22.1.8\nCMake version: version 4.4.2\nLibc version: glibc-2.44\n\nPython version: 3.14.7 (main, Aug 10 2026, 07:46:56) [GCC 16.1.1 20260728] (64-bit runtime)\nPython platform: Linux-7.1.8-arch1-3-x86_64-with-glibc2.44\nIs CUDA available: True\nCUDA runtime version: 13.3.73\nCUDA_MODULE_LOADING set to: \nGPU models and configuration: GPU 0: NVIDIA GeForce RTX 5070 Ti\nNvidia driver version: Could not collect\ncuDNN version: Probably one of the following:\n/usr/lib/libcudnn.so.9.25.0\n/usr/lib/libcudnn_adv.so.9.25.0\n/usr/lib/libcudnn_cnn.so.9.25.0\n/usr/lib/libcudnn_engines_precompiled.so.9.25.0\n/usr/lib/libcudnn_engines_runtime_compiled.so.9.25.0\n/usr/lib/libcudnn_engines_tensor_ir.so.9.25.0\n/usr/lib/libcudnn_ext.so.9.25.0\n/usr/lib/libcudnn_graph.so.9.25.0\n/usr/lib/libcudnn_heuristic.so.9.25.0\n/usr/lib/libcudnn_ops.so.9.25.0\nIs XPU available: False\nHIP runtime version: N/A\nMIOpen runtime version: N/A\nIs XNNPACK available: True\nCaching allocator config: N/A\n\nCPU:\nArchitecture:                            x86_64\nCPU op-mode(s):                          32-bit, 64-bit\nAddress sizes:                           39 bits physical, 48 bits virtual\nByte Order:                              Little Endian\nCPU(s):                                  24\nOn-line CPU(s) list:                     0-23\nVendor ID:                               GenuineIntel\nModel name:                              12th Gen Intel(R) Core(TM) i9-12900KF\nCPU family:                              6\nModel:                                   151\nThread(s) per core:                      2\nCore(s) per socket:                      16\nSocket(s):                               1\nStepping:                                2\nMicrocode version:                       0x3e\nCPU(s) scaling MHz:                      18%\nCPU max MHz:                             5200.0000\nCPU min MHz:                             800.0000\nBogoMIPS:                                6374.40\nFlags:                                   fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc art arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc cpuid aperfmperf tsc_known_freq pni pclmulqdq dtes64 monitor ds_cpl vmx est tm2 ssse3 sdbg fma cx16 xtpr pdcm pcid sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm abm 3dnowprefetch cpuid_fault ssbd ibrs ibpb stibp ibrs_enhanced tpr_shadow flexpriority ept vpid ept_ad fsgsbase tsc_adjust bmi1 avx2 smep bmi2 erms invpcid rdseed adx smap clflushopt clwb intel_pt sha_ni xsaveopt xsavec xgetbv1 xsaves split_lock_detect user_shstk avx_vnni dtherm ida arat pln pts hwp hwp_notify hwp_act_window hwp_epp hwp_pkg_req hfi vnmi umip pku ospke waitpkg gfni vaes vpclmulqdq rdpid movdiri movdir64b fsrm md_clear serialize arch_lbr ibt flush_l1d arch_capabilities\nVirtualization:                          VT-x\nL1d cache:                               640 KiB (16 instances)\nL1i cache:                               768 KiB (16 instances)\nL2 cache:                                14 MiB (10 instances)\nL3 cache:                                30 MiB (1 instance)\nNUMA node(s):                            1\nNUMA node0 CPU(s):                       0-23\nVulnerability Gather data sampling:      Not affected\nVulnerability Ghostwrite:                Not affected\nVulnerability Indirect target selection: Not affected\nVulnerability Itlb multihit:             Not affected\nVulnerability L1tf:                      Not affected\nVulnerability Mds:                       Not affected\nVulnerability Meltdown:                  Not affected\nVulnerability Mmio stale data:           Not affected\nVulnerability Old microcode:             Not affected\nVulnerability Reg file data sampling:    Mitigation; Clear Register File\nVulnerability Retbleed:                  Not affected\nVulnerability Spec rstack overflow:      Not affected\nVulnerability Spec store bypass:         Mitigation; Speculative Store Bypass disabled via prctl\nVulnerability Spectre v1:                Mitigation; usercopy/swapgs barriers and __user pointer sanitization\nVulnerability Spectre v2:                Mitigation; Enhanced / Automatic IBRS; IBPB conditional; PBRSB-eIBRS SW sequence; BHI BHI_DIS_S\nVulnerability Srbds:                     Not affected\nVulnerability Tsa:                       Not affected\nVulnerability Tsx async abort:           Not affected\nVulnerability Vmscape:                   Mitigation; IBPB before exit to userspace\n\nVersions of relevant libraries:\n[pip3] numpy==2.5.2\n[pip3] nvidia-cublas==13.1.0.3\n[pip3] nvidia-cuda-cupti==13.0.85\n[pip3] nvidia-cuda-nvrtc==13.0.88\n[pip3] nvidia-cuda-runtime==13.0.96\n[pip3] nvidia-cudnn-cu13==9.19.0.56\n[pip3] nvidia-cufft==12.0.0.61\n[pip3] nvidia-curand==10.4.0.35\n[pip3] nvidia-cusolver==12.0.4.66\n[pip3] nvidia-cusparse==12.6.3.3\n[pip3] nvidia-cusparselt-cu13==0.8.0\n[pip3] nvidia-nccl-cu13==2.28.9\n[pip3] nvidia-nvjitlink==13.0.88\n[pip3] nvidia-nvtx==13.0.85\n[pip3] segmentation_models_pytorch==0.5.0\n[pip3] torch==2.11.0\n[pip3] torchao==0.17.0\n[pip3] torchaudio==2.11.0\n[pip3] torchcodec==0.16.0\n[pip3] torchvision==0.26.0\n[pip3] triton==3.6.0\n[conda] Could not collect",
+  "transformers_version": "5.7.0",
+  "lm_eval_version": "0.4.12",
+  "upper_git_hash": null,
+  "tokenizer_pad_token": [
+    "<|pad|>",
+    "0"
+  ],
+  "tokenizer_eos_token": [
+    "<|eos|>",
+    "2"
+  ],
+  "tokenizer_bos_token": [
+    "<|bos|>",
+    "1"
+  ],
+  "eot_token_id": 2,
+  "max_length": 4096,
+  "task_hashes": {},
+  "model_source": "hf",
+  "model_name": "/home/banaxi/Desktop/BananaMind/BananaMind-2/BananaMind-2.1-Unified",
+  "model_name_sanitized": "__home__banaxi__Desktop__BananaMind__BananaMind-2__BananaMind-2.1-Unified",
+  "system_instruction": null,
+  "system_instruction_sha": null,
+  "fewshot_as_multiturn": null,
+  "chat_template": null,
+  "chat_template_sha": null,
+  "total_evaluation_time_seconds": "55.09879929400631"
+}

eval_results/c_only/__home__banaxi__Desktop__BananaMind__BananaMind-2__BananaMind-2.1-Unified/results_2026-08-19T13-11-40.562917.json

@@ -0,0 +1,287 @@
+{
+  "results": {
+    "arc_easy": {
+      "name": "arc_easy",
+      "alias": "arc_easy",
+      "sample_len": 2376,
+      "acc,none": 0.27104377104377103,
+      "acc_stderr,none": 0.009120919741760646,
+      "acc_norm,none": 0.28324915824915825,
+      "acc_norm_stderr,none": 0.009245632200075576
+    },
+    "piqa": {
+      "name": "piqa",
+      "alias": "piqa",
+      "sample_len": 1838,
+      "acc,none": 0.544613710554951,
+      "acc_stderr,none": 0.011619292444156876,
+      "acc_norm,none": 0.5212187159956474,
+      "acc_norm_stderr,none": 0.011655314732288771
+    },
+    "hellaswag": {
+      "name": "hellaswag",
+      "alias": "hellaswag",
+      "sample_len": 10042,
+      "acc,none": 0.2640908185620394,
+      "acc_stderr,none": 0.0043994691682725275,
+      "acc_norm,none": 0.2727544313881697,
+      "acc_norm_stderr,none": 0.004444654076551038
+    }
+  },
+  "group_subtasks": {},
+  "configs": {
+    "arc_easy": {
+      "task": "arc_easy",
+      "dataset_path": "allenai/ai2_arc",
+      "dataset_name": "ARC-Easy",
+      "training_split": "train",
+      "validation_split": "validation",
+      "test_split": "test",
+      "doc_to_text": "Question: {{question}}\nAnswer:",
+      "doc_to_target": "{{choices.label.index(answerKey)}}",
+      "unsafe_code": false,
+      "doc_to_choice": "{{choices.text}}",
+      "description": "",
+      "target_delimiter": " ",
+      "fewshot_delimiter": "\n\n",
+      "fewshot_config": {
+        "sampler": "default",
+        "split": null,
+        "process_docs": null,
+        "fewshot_indices": null,
+        "samples": null,
+        "doc_to_text": "Question: {{question}}\nAnswer:",
+        "doc_to_choice": "{{choices.text}}",
+        "doc_to_target": "{{choices.label.index(answerKey)}}",
+        "gen_prefix": null,
+        "fewshot_delimiter": "\n\n",
+        "target_delimiter": " "
+      },
+      "num_fewshot": 0,
+      "metric_list": [
+        {
+          "metric": "acc",
+          "aggregation": "mean",
+          "higher_is_better": true
+        },
+        {
+          "metric": "acc_norm",
+          "aggregation": "mean",
+          "higher_is_better": true
+        }
+      ],
+      "output_type": "multiple_choice",
+      "repeats": 1,
+      "should_decontaminate": true,
+      "doc_to_decontamination_query": "Question: {{question}}\nAnswer:",
+      "metadata": {
+        "version": 1.0,
+        "pretrained": "/home/banaxi/Desktop/BananaMind/BananaMind-2/BananaMind-2.1-Unified",
+        "trust_remote_code": true,
+        "dtype": "float32",
+        "use_cache": false,
+        "relay_mode": "c_only",
+        "config_source": "/home/banaxi/.local/lib/python3.14/site-packages/lm_eval/tasks/arc/arc_easy.yaml"
+      }
+    },
+    "hellaswag": {
+      "task": "hellaswag",
+      "dataset_path": "Rowan/hellaswag",
+      "training_split": "train",
+      "validation_split": "validation",
+      "process_docs": "def process_docs(dataset: datasets.Dataset) -> datasets.Dataset:\n    def _process_doc(doc):\n        ctx = doc[\"ctx_a\"] + \" \" + doc[\"ctx_b\"].capitalize()\n        out_doc = {\n            \"query\": preprocess(doc[\"activity_label\"] + \": \" + ctx),\n            \"choices\": [preprocess(ending) for ending in doc[\"endings\"]],\n            \"gold\": int(doc[\"label\"]),\n        }\n        return out_doc\n\n    return dataset.map(_process_doc)\n",
+      "doc_to_text": "{{query}}",
+      "doc_to_target": "{{label}}",
+      "unsafe_code": false,
+      "doc_to_choice": "choices",
+      "description": "",
+      "target_delimiter": " ",
+      "fewshot_delimiter": "\n\n",
+      "fewshot_config": {
+        "sampler": "default",
+        "split": null,
+        "process_docs": "<function process_docs at 0x7f91773f3950>",
+        "fewshot_indices": null,
+        "samples": null,
+        "doc_to_text": "{{query}}",
+        "doc_to_choice": "choices",
+        "doc_to_target": "{{label}}",
+        "gen_prefix": null,
+        "fewshot_delimiter": "\n\n",
+        "target_delimiter": " "
+      },
+      "num_fewshot": 0,
+      "metric_list": [
+        {
+          "metric": "acc",
+          "aggregation": "mean",
+          "higher_is_better": true
+        },
+        {
+          "metric": "acc_norm",
+          "aggregation": "mean",
+          "higher_is_better": true
+        }
+      ],
+      "output_type": "multiple_choice",
+      "repeats": 1,
+      "should_decontaminate": false,
+      "metadata": {
+        "version": 1.0,
+        "pretrained": "/home/banaxi/Desktop/BananaMind/BananaMind-2/BananaMind-2.1-Unified",
+        "trust_remote_code": true,
+        "dtype": "float32",
+        "use_cache": false,
+        "relay_mode": "c_only",
+        "config_source": "/home/banaxi/.local/lib/python3.14/site-packages/lm_eval/tasks/hellaswag/hellaswag.yaml"
+      }
+    },
+    "piqa": {
+      "task": "piqa",
+      "dataset_path": "baber/piqa",
+      "training_split": "train",
+      "validation_split": "validation",
+      "doc_to_text": "Question: {{goal}}\nAnswer:",
+      "doc_to_target": "label",
+      "unsafe_code": false,
+      "doc_to_choice": "{{[sol1, sol2]}}",
+      "description": "",
+      "target_delimiter": " ",
+      "fewshot_delimiter": "\n\n",
+      "fewshot_config": {
+        "sampler": "default",
+        "split": null,
+        "process_docs": null,
+        "fewshot_indices": null,
+        "samples": null,
+        "doc_to_text": "Question: {{goal}}\nAnswer:",
+        "doc_to_choice": "{{[sol1, sol2]}}",
+        "doc_to_target": "label",
+        "gen_prefix": null,
+        "fewshot_delimiter": "\n\n",
+        "target_delimiter": " "
+      },
+      "num_fewshot": 0,
+      "metric_list": [
+        {
+          "metric": "acc",
+          "aggregation": "mean",
+          "higher_is_better": true
+        },
+        {
+          "metric": "acc_norm",
+          "aggregation": "mean",
+          "higher_is_better": true
+        }
+      ],
+      "output_type": "multiple_choice",
+      "repeats": 1,
+      "should_decontaminate": true,
+      "doc_to_decontamination_query": "goal",
+      "metadata": {
+        "version": 1.0,
+        "pretrained": "/home/banaxi/Desktop/BananaMind/BananaMind-2/BananaMind-2.1-Unified",
+        "trust_remote_code": true,
+        "dtype": "float32",
+        "use_cache": false,
+        "relay_mode": "c_only",
+        "config_source": "/home/banaxi/.local/lib/python3.14/site-packages/lm_eval/tasks/piqa/piqa.yaml"
+      }
+    }
+  },
+  "versions": {
+    "arc_easy": 1.0,
+    "hellaswag": 1.0,
+    "piqa": 1.0
+  },
+  "n-shot": {
+    "arc_easy": 0,
+    "hellaswag": 0,
+    "piqa": 0
+  },
+  "higher_is_better": {
+    "arc_easy": {
+      "acc": true,
+      "acc_norm": true
+    },
+    "hellaswag": {
+      "acc": true,
+      "acc_norm": true
+    },
+    "piqa": {
+      "acc": true,
+      "acc_norm": true
+    }
+  },
+  "n-samples": {
+    "arc_easy": {
+      "original": 2376,
+      "effective": 2376
+    },
+    "piqa": {
+      "original": 1838,
+      "effective": 1838
+    },
+    "hellaswag": {
+      "original": 10042,
+      "effective": 10042
+    }
+  },
+  "config": {
+    "model": "hf",
+    "model_args": {
+      "pretrained": "/home/banaxi/Desktop/BananaMind/BananaMind-2/BananaMind-2.1-Unified",
+      "trust_remote_code": true,
+      "dtype": "float32",
+      "use_cache": false,
+      "relay_mode": "c_only"
+    },
+    "model_num_parameters": 34999041,
+    "model_dtype": "torch.float32",
+    "model_revision": "main",
+    "model_sha": "",
+    "batch_size": "auto",
+    "batch_sizes": [
+      64
+    ],
+    "device": "cuda:0",
+    "use_cache": null,
+    "limit": null,
+    "bootstrap_iters": 100000,
+    "gen_kwargs": {},
+    "random_seed": 0,
+    "numpy_seed": 1234,
+    "torch_seed": 1234,
+    "fewshot_seed": 1234
+  },
+  "git_hash": null,
+  "date": 1787137857.061881,
+  "pretty_env_info": "PyTorch version: 2.11.0+cu130\nIs debug build: False\nCUDA used to build PyTorch: 13.0\nROCM used to build PyTorch: N/A\n\nOS: Arch Linux (x86_64)\nGCC version: (GCC) 16.2.1 20260810\nClang version: 22.1.8\nCMake version: version 4.4.2\nLibc version: glibc-2.44\n\nPython version: 3.14.7 (main, Aug 10 2026, 07:46:56) [GCC 16.1.1 20260728] (64-bit runtime)\nPython platform: Linux-7.1.8-arch1-3-x86_64-with-glibc2.44\nIs CUDA available: True\nCUDA runtime version: 13.3.73\nCUDA_MODULE_LOADING set to: \nGPU models and configuration: GPU 0: NVIDIA GeForce RTX 5070 Ti\nNvidia driver version: Could not collect\ncuDNN version: Probably one of the following:\n/usr/lib/libcudnn.so.9.25.0\n/usr/lib/libcudnn_adv.so.9.25.0\n/usr/lib/libcudnn_cnn.so.9.25.0\n/usr/lib/libcudnn_engines_precompiled.so.9.25.0\n/usr/lib/libcudnn_engines_runtime_compiled.so.9.25.0\n/usr/lib/libcudnn_engines_tensor_ir.so.9.25.0\n/usr/lib/libcudnn_ext.so.9.25.0\n/usr/lib/libcudnn_graph.so.9.25.0\n/usr/lib/libcudnn_heuristic.so.9.25.0\n/usr/lib/libcudnn_ops.so.9.25.0\nIs XPU available: False\nHIP runtime version: N/A\nMIOpen runtime version: N/A\nIs XNNPACK available: True\nCaching allocator config: N/A\n\nCPU:\nArchitecture:                            x86_64\nCPU op-mode(s):                          32-bit, 64-bit\nAddress sizes:                           39 bits physical, 48 bits virtual\nByte Order:                              Little Endian\nCPU(s):                                  24\nOn-line CPU(s) list:                     0-23\nVendor ID:                               GenuineIntel\nModel name:                              12th Gen Intel(R) Core(TM) i9-12900KF\nCPU family:                              6\nModel:                                   151\nThread(s) per core:                      2\nCore(s) per socket:                      16\nSocket(s):                               1\nStepping:                                2\nMicrocode version:                       0x3e\nCPU(s) scaling MHz:                      33%\nCPU max MHz:                             5200.0000\nCPU min MHz:                             800.0000\nBogoMIPS:                                6374.40\nFlags:                                   fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc art arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc cpuid aperfmperf tsc_known_freq pni pclmulqdq dtes64 monitor ds_cpl vmx est tm2 ssse3 sdbg fma cx16 xtpr pdcm pcid sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm abm 3dnowprefetch cpuid_fault ssbd ibrs ibpb stibp ibrs_enhanced tpr_shadow flexpriority ept vpid ept_ad fsgsbase tsc_adjust bmi1 avx2 smep bmi2 erms invpcid rdseed adx smap clflushopt clwb intel_pt sha_ni xsaveopt xsavec xgetbv1 xsaves split_lock_detect user_shstk avx_vnni dtherm ida arat pln pts hwp hwp_notify hwp_act_window hwp_epp hwp_pkg_req hfi vnmi umip pku ospke waitpkg gfni vaes vpclmulqdq rdpid movdiri movdir64b fsrm md_clear serialize arch_lbr ibt flush_l1d arch_capabilities\nVirtualization:                          VT-x\nL1d cache:                               640 KiB (16 instances)\nL1i cache:                               768 KiB (16 instances)\nL2 cache:                                14 MiB (10 instances)\nL3 cache:                                30 MiB (1 instance)\nNUMA node(s):                            1\nNUMA node0 CPU(s):                       0-23\nVulnerability Gather data sampling:      Not affected\nVulnerability Ghostwrite:                Not affected\nVulnerability Indirect target selection: Not affected\nVulnerability Itlb multihit:             Not affected\nVulnerability L1tf:                      Not affected\nVulnerability Mds:                       Not affected\nVulnerability Meltdown:                  Not affected\nVulnerability Mmio stale data:           Not affected\nVulnerability Old microcode:             Not affected\nVulnerability Reg file data sampling:    Mitigation; Clear Register File\nVulnerability Retbleed:                  Not affected\nVulnerability Spec rstack overflow:      Not affected\nVulnerability Spec store bypass:         Mitigation; Speculative Store Bypass disabled via prctl\nVulnerability Spectre v1:                Mitigation; usercopy/swapgs barriers and __user pointer sanitization\nVulnerability Spectre v2:                Mitigation; Enhanced / Automatic IBRS; IBPB conditional; PBRSB-eIBRS SW sequence; BHI BHI_DIS_S\nVulnerability Srbds:                     Not affected\nVulnerability Tsa:                       Not affected\nVulnerability Tsx async abort:           Not affected\nVulnerability Vmscape:                   Mitigation; IBPB before exit to userspace\n\nVersions of relevant libraries:\n[pip3] numpy==2.5.2\n[pip3] nvidia-cublas==13.1.0.3\n[pip3] nvidia-cuda-cupti==13.0.85\n[pip3] nvidia-cuda-nvrtc==13.0.88\n[pip3] nvidia-cuda-runtime==13.0.96\n[pip3] nvidia-cudnn-cu13==9.19.0.56\n[pip3] nvidia-cufft==12.0.0.61\n[pip3] nvidia-curand==10.4.0.35\n[pip3] nvidia-cusolver==12.0.4.66\n[pip3] nvidia-cusparse==12.6.3.3\n[pip3] nvidia-cusparselt-cu13==0.8.0\n[pip3] nvidia-nccl-cu13==2.28.9\n[pip3] nvidia-nvjitlink==13.0.88\n[pip3] nvidia-nvtx==13.0.85\n[pip3] segmentation_models_pytorch==0.5.0\n[pip3] torch==2.11.0\n[pip3] torchao==0.17.0\n[pip3] torchaudio==2.11.0\n[pip3] torchcodec==0.16.0\n[pip3] torchvision==0.26.0\n[pip3] triton==3.6.0\n[conda] Could not collect",
+  "transformers_version": "5.7.0",
+  "lm_eval_version": "0.4.12",
+  "upper_git_hash": null,
+  "tokenizer_pad_token": [
+    "<|pad|>",
+    "0"
+  ],
+  "tokenizer_eos_token": [
+    "<|eos|>",
+    "2"
+  ],
+  "tokenizer_bos_token": [
+    "<|bos|>",
+    "1"
+  ],
+  "eot_token_id": 2,
+  "max_length": 4096,
+  "task_hashes": {},
+  "model_source": "hf",
+  "model_name": "/home/banaxi/Desktop/BananaMind/BananaMind-2/BananaMind-2.1-Unified",
+  "model_name_sanitized": "__home__banaxi__Desktop__BananaMind__BananaMind-2__BananaMind-2.1-Unified",
+  "system_instruction": null,
+  "system_instruction_sha": null,
+  "fewshot_as_multiturn": null,
+  "chat_template": null,
+  "chat_template_sha": null,
+  "total_evaluation_time_seconds": "46.08944170398172"
+}

eval_results/cb_only/__home__banaxi__Desktop__BananaMind__BananaMind-2__BananaMind-2.1-Unified/results_2026-08-19T13-09-08.771260.json

@@ -0,0 +1,287 @@
+{
+  "results": {
+    "arc_easy": {
+      "name": "arc_easy",
+      "alias": "arc_easy",
+      "sample_len": 2376,
+      "acc,none": 0.35395622895622897,
+      "acc_stderr,none": 0.009812370644174563,
+      "acc_norm,none": 0.3569023569023569,
+      "acc_norm_stderr,none": 0.009830630210347113
+    },
+    "piqa": {
+      "name": "piqa",
+      "alias": "piqa",
+      "sample_len": 1838,
+      "acc,none": 0.5658324265505985,
+      "acc_stderr,none": 0.011564264866016225,
+      "acc_norm,none": 0.5516866158868335,
+      "acc_norm_stderr,none": 0.011603326108334476
+    },
+    "hellaswag": {
+      "name": "hellaswag",
+      "alias": "hellaswag",
+      "sample_len": 10042,
+      "acc,none": 0.26946823341963755,
+      "acc_stderr,none": 0.0044277679963018075,
+      "acc_norm,none": 0.28669587731527585,
+      "acc_norm_stderr,none": 0.004512940497463054
+    }
+  },
+  "group_subtasks": {},
+  "configs": {
+    "arc_easy": {
+      "task": "arc_easy",
+      "dataset_path": "allenai/ai2_arc",
+      "dataset_name": "ARC-Easy",
+      "training_split": "train",
+      "validation_split": "validation",
+      "test_split": "test",
+      "doc_to_text": "Question: {{question}}\nAnswer:",
+      "doc_to_target": "{{choices.label.index(answerKey)}}",
+      "unsafe_code": false,
+      "doc_to_choice": "{{choices.text}}",
+      "description": "",
+      "target_delimiter": " ",
+      "fewshot_delimiter": "\n\n",
+      "fewshot_config": {
+        "sampler": "default",
+        "split": null,
+        "process_docs": null,
+        "fewshot_indices": null,
+        "samples": null,
+        "doc_to_text": "Question: {{question}}\nAnswer:",
+        "doc_to_choice": "{{choices.text}}",
+        "doc_to_target": "{{choices.label.index(answerKey)}}",
+        "gen_prefix": null,
+        "fewshot_delimiter": "\n\n",
+        "target_delimiter": " "
+      },
+      "num_fewshot": 0,
+      "metric_list": [
+        {
+          "metric": "acc",
+          "aggregation": "mean",
+          "higher_is_better": true
+        },
+        {
+          "metric": "acc_norm",
+          "aggregation": "mean",
+          "higher_is_better": true
+        }
+      ],
+      "output_type": "multiple_choice",
+      "repeats": 1,
+      "should_decontaminate": true,
+      "doc_to_decontamination_query": "Question: {{question}}\nAnswer:",
+      "metadata": {
+        "version": 1.0,
+        "pretrained": "/home/banaxi/Desktop/BananaMind/BananaMind-2/BananaMind-2.1-Unified",
+        "trust_remote_code": true,
+        "dtype": "float32",
+        "use_cache": false,
+        "relay_mode": "cb_only",
+        "config_source": "/home/banaxi/.local/lib/python3.14/site-packages/lm_eval/tasks/arc/arc_easy.yaml"
+      }
+    },
+    "hellaswag": {
+      "task": "hellaswag",
+      "dataset_path": "Rowan/hellaswag",
+      "training_split": "train",
+      "validation_split": "validation",
+      "process_docs": "def process_docs(dataset: datasets.Dataset) -> datasets.Dataset:\n    def _process_doc(doc):\n        ctx = doc[\"ctx_a\"] + \" \" + doc[\"ctx_b\"].capitalize()\n        out_doc = {\n            \"query\": preprocess(doc[\"activity_label\"] + \": \" + ctx),\n            \"choices\": [preprocess(ending) for ending in doc[\"endings\"]],\n            \"gold\": int(doc[\"label\"]),\n        }\n        return out_doc\n\n    return dataset.map(_process_doc)\n",
+      "doc_to_text": "{{query}}",
+      "doc_to_target": "{{label}}",
+      "unsafe_code": false,
+      "doc_to_choice": "choices",
+      "description": "",
+      "target_delimiter": " ",
+      "fewshot_delimiter": "\n\n",
+      "fewshot_config": {
+        "sampler": "default",
+        "split": null,
+        "process_docs": "<function process_docs at 0x7fcab37ff950>",
+        "fewshot_indices": null,
+        "samples": null,
+        "doc_to_text": "{{query}}",
+        "doc_to_choice": "choices",
+        "doc_to_target": "{{label}}",
+        "gen_prefix": null,
+        "fewshot_delimiter": "\n\n",
+        "target_delimiter": " "
+      },
+      "num_fewshot": 0,
+      "metric_list": [
+        {
+          "metric": "acc",
+          "aggregation": "mean",
+          "higher_is_better": true
+        },
+        {
+          "metric": "acc_norm",
+          "aggregation": "mean",
+          "higher_is_better": true
+        }
+      ],
+      "output_type": "multiple_choice",
+      "repeats": 1,
+      "should_decontaminate": false,
+      "metadata": {
+        "version": 1.0,
+        "pretrained": "/home/banaxi/Desktop/BananaMind/BananaMind-2/BananaMind-2.1-Unified",
+        "trust_remote_code": true,
+        "dtype": "float32",
+        "use_cache": false,
+        "relay_mode": "cb_only",
+        "config_source": "/home/banaxi/.local/lib/python3.14/site-packages/lm_eval/tasks/hellaswag/hellaswag.yaml"
+      }
+    },
+    "piqa": {
+      "task": "piqa",
+      "dataset_path": "baber/piqa",
+      "training_split": "train",
+      "validation_split": "validation",
+      "doc_to_text": "Question: {{goal}}\nAnswer:",
+      "doc_to_target": "label",
+      "unsafe_code": false,
+      "doc_to_choice": "{{[sol1, sol2]}}",
+      "description": "",
+      "target_delimiter": " ",
+      "fewshot_delimiter": "\n\n",
+      "fewshot_config": {
+        "sampler": "default",
+        "split": null,
+        "process_docs": null,
+        "fewshot_indices": null,
+        "samples": null,
+        "doc_to_text": "Question: {{goal}}\nAnswer:",
+        "doc_to_choice": "{{[sol1, sol2]}}",
+        "doc_to_target": "label",
+        "gen_prefix": null,
+        "fewshot_delimiter": "\n\n",
+        "target_delimiter": " "
+      },
+      "num_fewshot": 0,
+      "metric_list": [
+        {
+          "metric": "acc",
+          "aggregation": "mean",
+          "higher_is_better": true
+        },
+        {
+          "metric": "acc_norm",
+          "aggregation": "mean",
+          "higher_is_better": true
+        }
+      ],
+      "output_type": "multiple_choice",
+      "repeats": 1,
+      "should_decontaminate": true,
+      "doc_to_decontamination_query": "goal",
+      "metadata": {
+        "version": 1.0,
+        "pretrained": "/home/banaxi/Desktop/BananaMind/BananaMind-2/BananaMind-2.1-Unified",
+        "trust_remote_code": true,
+        "dtype": "float32",
+        "use_cache": false,
+        "relay_mode": "cb_only",
+        "config_source": "/home/banaxi/.local/lib/python3.14/site-packages/lm_eval/tasks/piqa/piqa.yaml"
+      }
+    }
+  },
+  "versions": {
+    "arc_easy": 1.0,
+    "hellaswag": 1.0,
+    "piqa": 1.0
+  },
+  "n-shot": {
+    "arc_easy": 0,
+    "hellaswag": 0,
+    "piqa": 0
+  },
+  "higher_is_better": {
+    "arc_easy": {
+      "acc": true,
+      "acc_norm": true
+    },
+    "hellaswag": {
+      "acc": true,
+      "acc_norm": true
+    },
+    "piqa": {
+      "acc": true,
+      "acc_norm": true
+    }
+  },
+  "n-samples": {
+    "arc_easy": {
+      "original": 2376,
+      "effective": 2376
+    },
+    "piqa": {
+      "original": 1838,
+      "effective": 1838
+    },
+    "hellaswag": {
+      "original": 10042,
+      "effective": 10042
+    }
+  },
+  "config": {
+    "model": "hf",
+    "model_args": {
+      "pretrained": "/home/banaxi/Desktop/BananaMind/BananaMind-2/BananaMind-2.1-Unified",
+      "trust_remote_code": true,
+      "dtype": "float32",
+      "use_cache": false,
+      "relay_mode": "cb_only"
+    },
+    "model_num_parameters": 34999041,
+    "model_dtype": "torch.float32",
+    "model_revision": "main",
+    "model_sha": "",
+    "batch_size": "auto",
+    "batch_sizes": [
+      64
+    ],
+    "device": "cuda:0",
+    "use_cache": null,
+    "limit": null,
+    "bootstrap_iters": 100000,
+    "gen_kwargs": {},
+    "random_seed": 0,
+    "numpy_seed": 1234,
+    "torch_seed": 1234,
+    "fewshot_seed": 1234
+  },
+  "git_hash": null,
+  "date": 1787137702.9870973,
+  "pretty_env_info": "PyTorch version: 2.11.0+cu130\nIs debug build: False\nCUDA used to build PyTorch: 13.0\nROCM used to build PyTorch: N/A\n\nOS: Arch Linux (x86_64)\nGCC version: (GCC) 16.2.1 20260810\nClang version: 22.1.8\nCMake version: version 4.4.2\nLibc version: glibc-2.44\n\nPython version: 3.14.7 (main, Aug 10 2026, 07:46:56) [GCC 16.1.1 20260728] (64-bit runtime)\nPython platform: Linux-7.1.8-arch1-3-x86_64-with-glibc2.44\nIs CUDA available: True\nCUDA runtime version: 13.3.73\nCUDA_MODULE_LOADING set to: \nGPU models and configuration: GPU 0: NVIDIA GeForce RTX 5070 Ti\nNvidia driver version: Could not collect\ncuDNN version: Probably one of the following:\n/usr/lib/libcudnn.so.9.25.0\n/usr/lib/libcudnn_adv.so.9.25.0\n/usr/lib/libcudnn_cnn.so.9.25.0\n/usr/lib/libcudnn_engines_precompiled.so.9.25.0\n/usr/lib/libcudnn_engines_runtime_compiled.so.9.25.0\n/usr/lib/libcudnn_engines_tensor_ir.so.9.25.0\n/usr/lib/libcudnn_ext.so.9.25.0\n/usr/lib/libcudnn_graph.so.9.25.0\n/usr/lib/libcudnn_heuristic.so.9.25.0\n/usr/lib/libcudnn_ops.so.9.25.0\nIs XPU available: False\nHIP runtime version: N/A\nMIOpen runtime version: N/A\nIs XNNPACK available: True\nCaching allocator config: N/A\n\nCPU:\nArchitecture:                            x86_64\nCPU op-mode(s):                          32-bit, 64-bit\nAddress sizes:                           39 bits physical, 48 bits virtual\nByte Order:                              Little Endian\nCPU(s):                                  24\nOn-line CPU(s) list:                     0-23\nVendor ID:                               GenuineIntel\nModel name:                              12th Gen Intel(R) Core(TM) i9-12900KF\nCPU family:                              6\nModel:                                   151\nThread(s) per core:                      2\nCore(s) per socket:                      16\nSocket(s):                               1\nStepping:                                2\nMicrocode version:                       0x3e\nCPU(s) scaling MHz:                      19%\nCPU max MHz:                             5200.0000\nCPU min MHz:                             800.0000\nBogoMIPS:                                6374.40\nFlags:                                   fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc art arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc cpuid aperfmperf tsc_known_freq pni pclmulqdq dtes64 monitor ds_cpl vmx est tm2 ssse3 sdbg fma cx16 xtpr pdcm pcid sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm abm 3dnowprefetch cpuid_fault ssbd ibrs ibpb stibp ibrs_enhanced tpr_shadow flexpriority ept vpid ept_ad fsgsbase tsc_adjust bmi1 avx2 smep bmi2 erms invpcid rdseed adx smap clflushopt clwb intel_pt sha_ni xsaveopt xsavec xgetbv1 xsaves split_lock_detect user_shstk avx_vnni dtherm ida arat pln pts hwp hwp_notify hwp_act_window hwp_epp hwp_pkg_req hfi vnmi umip pku ospke waitpkg gfni vaes vpclmulqdq rdpid movdiri movdir64b fsrm md_clear serialize arch_lbr ibt flush_l1d arch_capabilities\nVirtualization:                          VT-x\nL1d cache:                               640 KiB (16 instances)\nL1i cache:                               768 KiB (16 instances)\nL2 cache:                                14 MiB (10 instances)\nL3 cache:                                30 MiB (1 instance)\nNUMA node(s):                            1\nNUMA node0 CPU(s):                       0-23\nVulnerability Gather data sampling:      Not affected\nVulnerability Ghostwrite:                Not affected\nVulnerability Indirect target selection: Not affected\nVulnerability Itlb multihit:             Not affected\nVulnerability L1tf:                      Not affected\nVulnerability Mds:                       Not affected\nVulnerability Meltdown:                  Not affected\nVulnerability Mmio stale data:           Not affected\nVulnerability Old microcode:             Not affected\nVulnerability Reg file data sampling:    Mitigation; Clear Register File\nVulnerability Retbleed:                  Not affected\nVulnerability Spec rstack overflow:      Not affected\nVulnerability Spec store bypass:         Mitigation; Speculative Store Bypass disabled via prctl\nVulnerability Spectre v1:                Mitigation; usercopy/swapgs barriers and __user pointer sanitization\nVulnerability Spectre v2:                Mitigation; Enhanced / Automatic IBRS; IBPB conditional; PBRSB-eIBRS SW sequence; BHI BHI_DIS_S\nVulnerability Srbds:                     Not affected\nVulnerability Tsa:                       Not affected\nVulnerability Tsx async abort:           Not affected\nVulnerability Vmscape:                   Mitigation; IBPB before exit to userspace\n\nVersions of relevant libraries:\n[pip3] numpy==2.5.2\n[pip3] nvidia-cublas==13.1.0.3\n[pip3] nvidia-cuda-cupti==13.0.85\n[pip3] nvidia-cuda-nvrtc==13.0.88\n[pip3] nvidia-cuda-runtime==13.0.96\n[pip3] nvidia-cudnn-cu13==9.19.0.56\n[pip3] nvidia-cufft==12.0.0.61\n[pip3] nvidia-curand==10.4.0.35\n[pip3] nvidia-cusolver==12.0.4.66\n[pip3] nvidia-cusparse==12.6.3.3\n[pip3] nvidia-cusparselt-cu13==0.8.0\n[pip3] nvidia-nccl-cu13==2.28.9\n[pip3] nvidia-nvjitlink==13.0.88\n[pip3] nvidia-nvtx==13.0.85\n[pip3] segmentation_models_pytorch==0.5.0\n[pip3] torch==2.11.0\n[pip3] torchao==0.17.0\n[pip3] torchaudio==2.11.0\n[pip3] torchcodec==0.16.0\n[pip3] torchvision==0.26.0\n[pip3] triton==3.6.0\n[conda] Could not collect",
+  "transformers_version": "5.7.0",
+  "lm_eval_version": "0.4.12",
+  "upper_git_hash": null,
+  "tokenizer_pad_token": [
+    "<|pad|>",
+    "0"
+  ],
+  "tokenizer_eos_token": [
+    "<|eos|>",
+    "2"
+  ],
+  "tokenizer_bos_token": [
+    "<|bos|>",
+    "1"
+  ],
+  "eot_token_id": 2,
+  "max_length": 4096,
+  "task_hashes": {},
+  "model_source": "hf",
+  "model_name": "/home/banaxi/Desktop/BananaMind/BananaMind-2/BananaMind-2.1-Unified",
+  "model_name_sanitized": "__home__banaxi__Desktop__BananaMind__BananaMind-2__BananaMind-2.1-Unified",
+  "system_instruction": null,
+  "system_instruction_sha": null,
+  "fewshot_as_multiturn": null,
+  "chat_template": null,
+  "chat_template_sha": null,
+  "total_evaluation_time_seconds": "48.45971779001411"
+}

eval_results/cut_bridges/__home__banaxi__Desktop__BananaMind__BananaMind-2__BananaMind-2.1-Unified/results_2026-08-19T13-10-07.184445.json

@@ -0,0 +1,287 @@
+{
+  "results": {
+    "arc_easy": {
+      "name": "arc_easy",
+      "alias": "arc_easy",
+      "sample_len": 2376,
+      "acc,none": 0.26515151515151514,
+      "acc_stderr,none": 0.009057621139172679,
+      "acc_norm,none": 0.27946127946127947,
+      "acc_norm_stderr,none": 0.009207838142597383
+    },
+    "piqa": {
+      "name": "piqa",
+      "alias": "piqa",
+      "sample_len": 1838,
+      "acc,none": 0.543525571273123,
+      "acc_stderr,none": 0.011621538875661558,
+      "acc_norm,none": 0.515233949945593,
+      "acc_norm_stderr,none": 0.011660408257153443
+    },
+    "hellaswag": {
+      "name": "hellaswag",
+      "alias": "hellaswag",
+      "sample_len": 10042,
+      "acc,none": 0.26289583748257317,
+      "acc_stderr,none": 0.004393066760916671,
+      "acc_norm,none": 0.2780322644891456,
+      "acc_norm_stderr,none": 0.004471137333619756
+    }
+  },
+  "group_subtasks": {},
+  "configs": {
+    "arc_easy": {
+      "task": "arc_easy",
+      "dataset_path": "allenai/ai2_arc",
+      "dataset_name": "ARC-Easy",
+      "training_split": "train",
+      "validation_split": "validation",
+      "test_split": "test",
+      "doc_to_text": "Question: {{question}}\nAnswer:",
+      "doc_to_target": "{{choices.label.index(answerKey)}}",
+      "unsafe_code": false,
+      "doc_to_choice": "{{choices.text}}",
+      "description": "",
+      "target_delimiter": " ",
+      "fewshot_delimiter": "\n\n",
+      "fewshot_config": {
+        "sampler": "default",
+        "split": null,
+        "process_docs": null,
+        "fewshot_indices": null,
+        "samples": null,
+        "doc_to_text": "Question: {{question}}\nAnswer:",
+        "doc_to_choice": "{{choices.text}}",
+        "doc_to_target": "{{choices.label.index(answerKey)}}",
+        "gen_prefix": null,
+        "fewshot_delimiter": "\n\n",
+        "target_delimiter": " "
+      },
+      "num_fewshot": 0,
+      "metric_list": [
+        {
+          "metric": "acc",
+          "aggregation": "mean",
+          "higher_is_better": true
+        },
+        {
+          "metric": "acc_norm",
+          "aggregation": "mean",
+          "higher_is_better": true
+        }
+      ],
+      "output_type": "multiple_choice",
+      "repeats": 1,
+      "should_decontaminate": true,
+      "doc_to_decontamination_query": "Question: {{question}}\nAnswer:",
+      "metadata": {
+        "version": 1.0,
+        "pretrained": "/home/banaxi/Desktop/BananaMind/BananaMind-2/BananaMind-2.1-Unified",
+        "trust_remote_code": true,
+        "dtype": "float32",
+        "use_cache": false,
+        "relay_mode": "cut_bridges",
+        "config_source": "/home/banaxi/.local/lib/python3.14/site-packages/lm_eval/tasks/arc/arc_easy.yaml"
+      }
+    },
+    "hellaswag": {
+      "task": "hellaswag",
+      "dataset_path": "Rowan/hellaswag",
+      "training_split": "train",
+      "validation_split": "validation",
+      "process_docs": "def process_docs(dataset: datasets.Dataset) -> datasets.Dataset:\n    def _process_doc(doc):\n        ctx = doc[\"ctx_a\"] + \" \" + doc[\"ctx_b\"].capitalize()\n        out_doc = {\n            \"query\": preprocess(doc[\"activity_label\"] + \": \" + ctx),\n            \"choices\": [preprocess(ending) for ending in doc[\"endings\"]],\n            \"gold\": int(doc[\"label\"]),\n        }\n        return out_doc\n\n    return dataset.map(_process_doc)\n",
+      "doc_to_text": "{{query}}",
+      "doc_to_target": "{{label}}",
+      "unsafe_code": false,
+      "doc_to_choice": "choices",
+      "description": "",
+      "target_delimiter": " ",
+      "fewshot_delimiter": "\n\n",
+      "fewshot_config": {
+        "sampler": "default",
+        "split": null,
+        "process_docs": "<function process_docs at 0x7fa7efa2b950>",
+        "fewshot_indices": null,
+        "samples": null,
+        "doc_to_text": "{{query}}",
+        "doc_to_choice": "choices",
+        "doc_to_target": "{{label}}",
+        "gen_prefix": null,
+        "fewshot_delimiter": "\n\n",
+        "target_delimiter": " "
+      },
+      "num_fewshot": 0,
+      "metric_list": [
+        {
+          "metric": "acc",
+          "aggregation": "mean",
+          "higher_is_better": true
+        },
+        {
+          "metric": "acc_norm",
+          "aggregation": "mean",
+          "higher_is_better": true
+        }
+      ],
+      "output_type": "multiple_choice",
+      "repeats": 1,
+      "should_decontaminate": false,
+      "metadata": {
+        "version": 1.0,
+        "pretrained": "/home/banaxi/Desktop/BananaMind/BananaMind-2/BananaMind-2.1-Unified",
+        "trust_remote_code": true,
+        "dtype": "float32",
+        "use_cache": false,
+        "relay_mode": "cut_bridges",
+        "config_source": "/home/banaxi/.local/lib/python3.14/site-packages/lm_eval/tasks/hellaswag/hellaswag.yaml"
+      }
+    },
+    "piqa": {
+      "task": "piqa",
+      "dataset_path": "baber/piqa",
+      "training_split": "train",
+      "validation_split": "validation",
+      "doc_to_text": "Question: {{goal}}\nAnswer:",
+      "doc_to_target": "label",
+      "unsafe_code": false,
+      "doc_to_choice": "{{[sol1, sol2]}}",
+      "description": "",
+      "target_delimiter": " ",
+      "fewshot_delimiter": "\n\n",
+      "fewshot_config": {
+        "sampler": "default",
+        "split": null,
+        "process_docs": null,
+        "fewshot_indices": null,
+        "samples": null,
+        "doc_to_text": "Question: {{goal}}\nAnswer:",
+        "doc_to_choice": "{{[sol1, sol2]}}",
+        "doc_to_target": "label",
+        "gen_prefix": null,
+        "fewshot_delimiter": "\n\n",
+        "target_delimiter": " "
+      },
+      "num_fewshot": 0,
+      "metric_list": [
+        {
+          "metric": "acc",
+          "aggregation": "mean",
+          "higher_is_better": true
+        },
+        {
+          "metric": "acc_norm",
+          "aggregation": "mean",
+          "higher_is_better": true
+        }
+      ],
+      "output_type": "multiple_choice",
+      "repeats": 1,
+      "should_decontaminate": true,
+      "doc_to_decontamination_query": "goal",
+      "metadata": {
+        "version": 1.0,
+        "pretrained": "/home/banaxi/Desktop/BananaMind/BananaMind-2/BananaMind-2.1-Unified",
+        "trust_remote_code": true,
+        "dtype": "float32",
+        "use_cache": false,
+        "relay_mode": "cut_bridges",
+        "config_source": "/home/banaxi/.local/lib/python3.14/site-packages/lm_eval/tasks/piqa/piqa.yaml"
+      }
+    }
+  },
+  "versions": {
+    "arc_easy": 1.0,
+    "hellaswag": 1.0,
+    "piqa": 1.0
+  },
+  "n-shot": {
+    "arc_easy": 0,
+    "hellaswag": 0,
+    "piqa": 0
+  },
+  "higher_is_better": {
+    "arc_easy": {
+      "acc": true,
+      "acc_norm": true
+    },
+    "hellaswag": {
+      "acc": true,
+      "acc_norm": true
+    },
+    "piqa": {
+      "acc": true,
+      "acc_norm": true
+    }
+  },
+  "n-samples": {
+    "arc_easy": {
+      "original": 2376,
+      "effective": 2376
+    },
+    "piqa": {
+      "original": 1838,
+      "effective": 1838
+    },
+    "hellaswag": {
+      "original": 10042,
+      "effective": 10042
+    }
+  },
+  "config": {
+    "model": "hf",
+    "model_args": {
+      "pretrained": "/home/banaxi/Desktop/BananaMind/BananaMind-2/BananaMind-2.1-Unified",
+      "trust_remote_code": true,
+      "dtype": "float32",
+      "use_cache": false,
+      "relay_mode": "cut_bridges"
+    },
+    "model_num_parameters": 34999041,
+    "model_dtype": "torch.float32",
+    "model_revision": "main",
+    "model_sha": "",
+    "batch_size": "auto",
+    "batch_sizes": [
+      64
+    ],
+    "device": "cuda:0",
+    "use_cache": null,
+    "limit": null,
+    "bootstrap_iters": 100000,
+    "gen_kwargs": {},
+    "random_seed": 0,
+    "numpy_seed": 1234,
+    "torch_seed": 1234,
+    "fewshot_seed": 1234
+  },
+  "git_hash": null,
+  "date": 1787137752.5487585,
+  "pretty_env_info": "PyTorch version: 2.11.0+cu130\nIs debug build: False\nCUDA used to build PyTorch: 13.0\nROCM used to build PyTorch: N/A\n\nOS: Arch Linux (x86_64)\nGCC version: (GCC) 16.2.1 20260810\nClang version: 22.1.8\nCMake version: version 4.4.2\nLibc version: glibc-2.44\n\nPython version: 3.14.7 (main, Aug 10 2026, 07:46:56) [GCC 16.1.1 20260728] (64-bit runtime)\nPython platform: Linux-7.1.8-arch1-3-x86_64-with-glibc2.44\nIs CUDA available: True\nCUDA runtime version: 13.3.73\nCUDA_MODULE_LOADING set to: \nGPU models and configuration: GPU 0: NVIDIA GeForce RTX 5070 Ti\nNvidia driver version: Could not collect\ncuDNN version: Probably one of the following:\n/usr/lib/libcudnn.so.9.25.0\n/usr/lib/libcudnn_adv.so.9.25.0\n/usr/lib/libcudnn_cnn.so.9.25.0\n/usr/lib/libcudnn_engines_precompiled.so.9.25.0\n/usr/lib/libcudnn_engines_runtime_compiled.so.9.25.0\n/usr/lib/libcudnn_engines_tensor_ir.so.9.25.0\n/usr/lib/libcudnn_ext.so.9.25.0\n/usr/lib/libcudnn_graph.so.9.25.0\n/usr/lib/libcudnn_heuristic.so.9.25.0\n/usr/lib/libcudnn_ops.so.9.25.0\nIs XPU available: False\nHIP runtime version: N/A\nMIOpen runtime version: N/A\nIs XNNPACK available: True\nCaching allocator config: N/A\n\nCPU:\nArchitecture:                            x86_64\nCPU op-mode(s):                          32-bit, 64-bit\nAddress sizes:                           39 bits physical, 48 bits virtual\nByte Order:                              Little Endian\nCPU(s):                                  24\nOn-line CPU(s) list:                     0-23\nVendor ID:                               GenuineIntel\nModel name:                              12th Gen Intel(R) Core(TM) i9-12900KF\nCPU family:                              6\nModel:                                   151\nThread(s) per core:                      2\nCore(s) per socket:                      16\nSocket(s):                               1\nStepping:                                2\nMicrocode version:                       0x3e\nCPU(s) scaling MHz:                      23%\nCPU max MHz:                             5200.0000\nCPU min MHz:                             800.0000\nBogoMIPS:                                6374.40\nFlags:                                   fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc art arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc cpuid aperfmperf tsc_known_freq pni pclmulqdq dtes64 monitor ds_cpl vmx est tm2 ssse3 sdbg fma cx16 xtpr pdcm pcid sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm abm 3dnowprefetch cpuid_fault ssbd ibrs ibpb stibp ibrs_enhanced tpr_shadow flexpriority ept vpid ept_ad fsgsbase tsc_adjust bmi1 avx2 smep bmi2 erms invpcid rdseed adx smap clflushopt clwb intel_pt sha_ni xsaveopt xsavec xgetbv1 xsaves split_lock_detect user_shstk avx_vnni dtherm ida arat pln pts hwp hwp_notify hwp_act_window hwp_epp hwp_pkg_req hfi vnmi umip pku ospke waitpkg gfni vaes vpclmulqdq rdpid movdiri movdir64b fsrm md_clear serialize arch_lbr ibt flush_l1d arch_capabilities\nVirtualization:                          VT-x\nL1d cache:                               640 KiB (16 instances)\nL1i cache:                               768 KiB (16 instances)\nL2 cache:                                14 MiB (10 instances)\nL3 cache:                                30 MiB (1 instance)\nNUMA node(s):                            1\nNUMA node0 CPU(s):                       0-23\nVulnerability Gather data sampling:      Not affected\nVulnerability Ghostwrite:                Not affected\nVulnerability Indirect target selection: Not affected\nVulnerability Itlb multihit:             Not affected\nVulnerability L1tf:                      Not affected\nVulnerability Mds:                       Not affected\nVulnerability Meltdown:                  Not affected\nVulnerability Mmio stale data:           Not affected\nVulnerability Old microcode:             Not affected\nVulnerability Reg file data sampling:    Mitigation; Clear Register File\nVulnerability Retbleed:                  Not affected\nVulnerability Spec rstack overflow:      Not affected\nVulnerability Spec store bypass:         Mitigation; Speculative Store Bypass disabled via prctl\nVulnerability Spectre v1:                Mitigation; usercopy/swapgs barriers and __user pointer sanitization\nVulnerability Spectre v2:                Mitigation; Enhanced / Automatic IBRS; IBPB conditional; PBRSB-eIBRS SW sequence; BHI BHI_DIS_S\nVulnerability Srbds:                     Not affected\nVulnerability Tsa:                       Not affected\nVulnerability Tsx async abort:           Not affected\nVulnerability Vmscape:                   Mitigation; IBPB before exit to userspace\n\nVersions of relevant libraries:\n[pip3] numpy==2.5.2\n[pip3] nvidia-cublas==13.1.0.3\n[pip3] nvidia-cuda-cupti==13.0.85\n[pip3] nvidia-cuda-nvrtc==13.0.88\n[pip3] nvidia-cuda-runtime==13.0.96\n[pip3] nvidia-cudnn-cu13==9.19.0.56\n[pip3] nvidia-cufft==12.0.0.61\n[pip3] nvidia-curand==10.4.0.35\n[pip3] nvidia-cusolver==12.0.4.66\n[pip3] nvidia-cusparse==12.6.3.3\n[pip3] nvidia-cusparselt-cu13==0.8.0\n[pip3] nvidia-nccl-cu13==2.28.9\n[pip3] nvidia-nvjitlink==13.0.88\n[pip3] nvidia-nvtx==13.0.85\n[pip3] segmentation_models_pytorch==0.5.0\n[pip3] torch==2.11.0\n[pip3] torchao==0.17.0\n[pip3] torchaudio==2.11.0\n[pip3] torchcodec==0.16.0\n[pip3] torchvision==0.26.0\n[pip3] triton==3.6.0\n[conda] Could not collect",
+  "transformers_version": "5.7.0",
+  "lm_eval_version": "0.4.12",
+  "upper_git_hash": null,
+  "tokenizer_pad_token": [
+    "<|pad|>",
+    "0"
+  ],
+  "tokenizer_eos_token": [
+    "<|eos|>",
+    "2"
+  ],
+  "tokenizer_bos_token": [
+    "<|bos|>",
+    "1"
+  ],
+  "eot_token_id": 2,
+  "max_length": 4096,
+  "task_hashes": {},
+  "model_source": "hf",
+  "model_name": "/home/banaxi/Desktop/BananaMind/BananaMind-2/BananaMind-2.1-Unified",
+  "model_name_sanitized": "__home__banaxi__Desktop__BananaMind__BananaMind-2__BananaMind-2.1-Unified",
+  "system_instruction": null,
+  "system_instruction_sha": null,
+  "fewshot_as_multiturn": null,
+  "chat_template": null,
+  "chat_template_sha": null,
+  "total_evaluation_time_seconds": "57.18551829599892"
+}

eval_results/full/__home__banaxi__Desktop__BananaMind__BananaMind-2__BananaMind-2.1-Unified/results_2026-08-19T13-06-29.559607.json

@@ -0,0 +1,287 @@
+{
+  "results": {
+    "arc_easy": {
+      "name": "arc_easy",
+      "alias": "arc_easy",
+      "sample_len": 2376,
+      "acc,none": 0.43223905723905726,
+      "acc_stderr,none": 0.010165130379698673,
+      "acc_norm,none": 0.3851010101010101,
+      "acc_norm_stderr,none": 0.009985214798737322
+    },
+    "piqa": {
+      "name": "piqa",
+      "alias": "piqa",
+      "sample_len": 1838,
+      "acc,none": 0.6251360174102285,
+      "acc_stderr,none": 0.01129456580561892,
+      "acc_norm,none": 0.6175190424374319,
+      "acc_norm_stderr,none": 0.011339019654272245
+    },
+    "hellaswag": {
+      "name": "hellaswag",
+      "alias": "hellaswag",
+      "sample_len": 10042,
+      "acc,none": 0.28649671380203146,
+      "acc_stderr,none": 0.00451200245975772,
+      "acc_norm,none": 0.29944234216291576,
+      "acc_norm_stderr,none": 0.004570777326263939
+    }
+  },
+  "group_subtasks": {},
+  "configs": {
+    "arc_easy": {
+      "task": "arc_easy",
+      "dataset_path": "allenai/ai2_arc",
+      "dataset_name": "ARC-Easy",
+      "training_split": "train",
+      "validation_split": "validation",
+      "test_split": "test",
+      "doc_to_text": "Question: {{question}}\nAnswer:",
+      "doc_to_target": "{{choices.label.index(answerKey)}}",
+      "unsafe_code": false,
+      "doc_to_choice": "{{choices.text}}",
+      "description": "",
+      "target_delimiter": " ",
+      "fewshot_delimiter": "\n\n",
+      "fewshot_config": {
+        "sampler": "default",
+        "split": null,
+        "process_docs": null,
+        "fewshot_indices": null,
+        "samples": null,
+        "doc_to_text": "Question: {{question}}\nAnswer:",
+        "doc_to_choice": "{{choices.text}}",
+        "doc_to_target": "{{choices.label.index(answerKey)}}",
+        "gen_prefix": null,
+        "fewshot_delimiter": "\n\n",
+        "target_delimiter": " "
+      },
+      "num_fewshot": 0,
+      "metric_list": [
+        {
+          "metric": "acc",
+          "aggregation": "mean",
+          "higher_is_better": true
+        },
+        {
+          "metric": "acc_norm",
+          "aggregation": "mean",
+          "higher_is_better": true
+        }
+      ],
+      "output_type": "multiple_choice",
+      "repeats": 1,
+      "should_decontaminate": true,
+      "doc_to_decontamination_query": "Question: {{question}}\nAnswer:",
+      "metadata": {
+        "version": 1.0,
+        "pretrained": "/home/banaxi/Desktop/BananaMind/BananaMind-2/BananaMind-2.1-Unified",
+        "trust_remote_code": true,
+        "dtype": "float32",
+        "use_cache": false,
+        "relay_mode": "full",
+        "config_source": "/home/banaxi/.local/lib/python3.14/site-packages/lm_eval/tasks/arc/arc_easy.yaml"
+      }
+    },
+    "hellaswag": {
+      "task": "hellaswag",
+      "dataset_path": "Rowan/hellaswag",
+      "training_split": "train",
+      "validation_split": "validation",
+      "process_docs": "def process_docs(dataset: datasets.Dataset) -> datasets.Dataset:\n    def _process_doc(doc):\n        ctx = doc[\"ctx_a\"] + \" \" + doc[\"ctx_b\"].capitalize()\n        out_doc = {\n            \"query\": preprocess(doc[\"activity_label\"] + \": \" + ctx),\n            \"choices\": [preprocess(ending) for ending in doc[\"endings\"]],\n            \"gold\": int(doc[\"label\"]),\n        }\n        return out_doc\n\n    return dataset.map(_process_doc)\n",
+      "doc_to_text": "{{query}}",
+      "doc_to_target": "{{label}}",
+      "unsafe_code": false,
+      "doc_to_choice": "choices",
+      "description": "",
+      "target_delimiter": " ",
+      "fewshot_delimiter": "\n\n",
+      "fewshot_config": {
+        "sampler": "default",
+        "split": null,
+        "process_docs": "<function process_docs at 0x7fae1b617950>",
+        "fewshot_indices": null,
+        "samples": null,
+        "doc_to_text": "{{query}}",
+        "doc_to_choice": "choices",
+        "doc_to_target": "{{label}}",
+        "gen_prefix": null,
+        "fewshot_delimiter": "\n\n",
+        "target_delimiter": " "
+      },
+      "num_fewshot": 0,
+      "metric_list": [
+        {
+          "metric": "acc",
+          "aggregation": "mean",
+          "higher_is_better": true
+        },
+        {
+          "metric": "acc_norm",
+          "aggregation": "mean",
+          "higher_is_better": true
+        }
+      ],
+      "output_type": "multiple_choice",
+      "repeats": 1,
+      "should_decontaminate": false,
+      "metadata": {
+        "version": 1.0,
+        "pretrained": "/home/banaxi/Desktop/BananaMind/BananaMind-2/BananaMind-2.1-Unified",
+        "trust_remote_code": true,
+        "dtype": "float32",
+        "use_cache": false,
+        "relay_mode": "full",
+        "config_source": "/home/banaxi/.local/lib/python3.14/site-packages/lm_eval/tasks/hellaswag/hellaswag.yaml"
+      }
+    },
+    "piqa": {
+      "task": "piqa",
+      "dataset_path": "baber/piqa",
+      "training_split": "train",
+      "validation_split": "validation",
+      "doc_to_text": "Question: {{goal}}\nAnswer:",
+      "doc_to_target": "label",
+      "unsafe_code": false,
+      "doc_to_choice": "{{[sol1, sol2]}}",
+      "description": "",
+      "target_delimiter": " ",
+      "fewshot_delimiter": "\n\n",
+      "fewshot_config": {
+        "sampler": "default",
+        "split": null,
+        "process_docs": null,
+        "fewshot_indices": null,
+        "samples": null,
+        "doc_to_text": "Question: {{goal}}\nAnswer:",
+        "doc_to_choice": "{{[sol1, sol2]}}",
+        "doc_to_target": "label",
+        "gen_prefix": null,
+        "fewshot_delimiter": "\n\n",
+        "target_delimiter": " "
+      },
+      "num_fewshot": 0,
+      "metric_list": [
+        {
+          "metric": "acc",
+          "aggregation": "mean",
+          "higher_is_better": true
+        },
+        {
+          "metric": "acc_norm",
+          "aggregation": "mean",
+          "higher_is_better": true
+        }
+      ],
+      "output_type": "multiple_choice",
+      "repeats": 1,
+      "should_decontaminate": true,
+      "doc_to_decontamination_query": "goal",
+      "metadata": {
+        "version": 1.0,
+        "pretrained": "/home/banaxi/Desktop/BananaMind/BananaMind-2/BananaMind-2.1-Unified",
+        "trust_remote_code": true,
+        "dtype": "float32",
+        "use_cache": false,
+        "relay_mode": "full",
+        "config_source": "/home/banaxi/.local/lib/python3.14/site-packages/lm_eval/tasks/piqa/piqa.yaml"
+      }
+    }
+  },
+  "versions": {
+    "arc_easy": 1.0,
+    "hellaswag": 1.0,
+    "piqa": 1.0
+  },
+  "n-shot": {
+    "arc_easy": 0,
+    "hellaswag": 0,
+    "piqa": 0
+  },
+  "higher_is_better": {
+    "arc_easy": {
+      "acc": true,
+      "acc_norm": true
+    },
+    "hellaswag": {
+      "acc": true,
+      "acc_norm": true
+    },
+    "piqa": {
+      "acc": true,
+      "acc_norm": true
+    }
+  },
+  "n-samples": {
+    "arc_easy": {
+      "original": 2376,
+      "effective": 2376
+    },
+    "piqa": {
+      "original": 1838,
+      "effective": 1838
+    },
+    "hellaswag": {
+      "original": 10042,
+      "effective": 10042
+    }
+  },
+  "config": {
+    "model": "hf",
+    "model_args": {
+      "pretrained": "/home/banaxi/Desktop/BananaMind/BananaMind-2/BananaMind-2.1-Unified",
+      "trust_remote_code": true,
+      "dtype": "float32",
+      "use_cache": false,
+      "relay_mode": "full"
+    },
+    "model_num_parameters": 34999041,
+    "model_dtype": "torch.float32",
+    "model_revision": "main",
+    "model_sha": "",
+    "batch_size": "auto",
+    "batch_sizes": [
+      64
+    ],
+    "device": "cuda:0",
+    "use_cache": null,
+    "limit": null,
+    "bootstrap_iters": 100000,
+    "gen_kwargs": {},
+    "random_seed": 0,
+    "numpy_seed": 1234,
+    "torch_seed": 1234,
+    "fewshot_seed": 1234
+  },
+  "git_hash": null,
+  "date": 1787137532.7548308,
+  "pretty_env_info": "PyTorch version: 2.11.0+cu130\nIs debug build: False\nCUDA used to build PyTorch: 13.0\nROCM used to build PyTorch: N/A\n\nOS: Arch Linux (x86_64)\nGCC version: (GCC) 16.2.1 20260810\nClang version: 22.1.8\nCMake version: version 4.4.2\nLibc version: glibc-2.44\n\nPython version: 3.14.7 (main, Aug 10 2026, 07:46:56) [GCC 16.1.1 20260728] (64-bit runtime)\nPython platform: Linux-7.1.8-arch1-3-x86_64-with-glibc2.44\nIs CUDA available: True\nCUDA runtime version: 13.3.73\nCUDA_MODULE_LOADING set to: \nGPU models and configuration: GPU 0: NVIDIA GeForce RTX 5070 Ti\nNvidia driver version: Could not collect\ncuDNN version: Probably one of the following:\n/usr/lib/libcudnn.so.9.25.0\n/usr/lib/libcudnn_adv.so.9.25.0\n/usr/lib/libcudnn_cnn.so.9.25.0\n/usr/lib/libcudnn_engines_precompiled.so.9.25.0\n/usr/lib/libcudnn_engines_runtime_compiled.so.9.25.0\n/usr/lib/libcudnn_engines_tensor_ir.so.9.25.0\n/usr/lib/libcudnn_ext.so.9.25.0\n/usr/lib/libcudnn_graph.so.9.25.0\n/usr/lib/libcudnn_heuristic.so.9.25.0\n/usr/lib/libcudnn_ops.so.9.25.0\nIs XPU available: False\nHIP runtime version: N/A\nMIOpen runtime version: N/A\nIs XNNPACK available: True\nCaching allocator config: N/A\n\nCPU:\nArchitecture:                            x86_64\nCPU op-mode(s):                          32-bit, 64-bit\nAddress sizes:                           39 bits physical, 48 bits virtual\nByte Order:                              Little Endian\nCPU(s):                                  24\nOn-line CPU(s) list:                     0-23\nVendor ID:                               GenuineIntel\nModel name:                              12th Gen Intel(R) Core(TM) i9-12900KF\nCPU family:                              6\nModel:                                   151\nThread(s) per core:                      2\nCore(s) per socket:                      16\nSocket(s):                               1\nStepping:                                2\nMicrocode version:                       0x3e\nCPU(s) scaling MHz:                      32%\nCPU max MHz:                             5200.0000\nCPU min MHz:                             800.0000\nBogoMIPS:                                6374.40\nFlags:                                   fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc art arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc cpuid aperfmperf tsc_known_freq pni pclmulqdq dtes64 monitor ds_cpl vmx est tm2 ssse3 sdbg fma cx16 xtpr pdcm pcid sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm abm 3dnowprefetch cpuid_fault ssbd ibrs ibpb stibp ibrs_enhanced tpr_shadow flexpriority ept vpid ept_ad fsgsbase tsc_adjust bmi1 avx2 smep bmi2 erms invpcid rdseed adx smap clflushopt clwb intel_pt sha_ni xsaveopt xsavec xgetbv1 xsaves split_lock_detect user_shstk avx_vnni dtherm ida arat pln pts hwp hwp_notify hwp_act_window hwp_epp hwp_pkg_req hfi vnmi umip pku ospke waitpkg gfni vaes vpclmulqdq rdpid movdiri movdir64b fsrm md_clear serialize arch_lbr ibt flush_l1d arch_capabilities\nVirtualization:                          VT-x\nL1d cache:                               640 KiB (16 instances)\nL1i cache:                               768 KiB (16 instances)\nL2 cache:                                14 MiB (10 instances)\nL3 cache:                                30 MiB (1 instance)\nNUMA node(s):                            1\nNUMA node0 CPU(s):                       0-23\nVulnerability Gather data sampling:      Not affected\nVulnerability Ghostwrite:                Not affected\nVulnerability Indirect target selection: Not affected\nVulnerability Itlb multihit:             Not affected\nVulnerability L1tf:                      Not affected\nVulnerability Mds:                       Not affected\nVulnerability Meltdown:                  Not affected\nVulnerability Mmio stale data:           Not affected\nVulnerability Old microcode:             Not affected\nVulnerability Reg file data sampling:    Mitigation; Clear Register File\nVulnerability Retbleed:                  Not affected\nVulnerability Spec rstack overflow:      Not affected\nVulnerability Spec store bypass:         Mitigation; Speculative Store Bypass disabled via prctl\nVulnerability Spectre v1:                Mitigation; usercopy/swapgs barriers and __user pointer sanitization\nVulnerability Spectre v2:                Mitigation; Enhanced / Automatic IBRS; IBPB conditional; PBRSB-eIBRS SW sequence; BHI BHI_DIS_S\nVulnerability Srbds:                     Not affected\nVulnerability Tsa:                       Not affected\nVulnerability Tsx async abort:           Not affected\nVulnerability Vmscape:                   Mitigation; IBPB before exit to userspace\n\nVersions of relevant libraries:\n[pip3] numpy==2.5.2\n[pip3] nvidia-cublas==13.1.0.3\n[pip3] nvidia-cuda-cupti==13.0.85\n[pip3] nvidia-cuda-nvrtc==13.0.88\n[pip3] nvidia-cuda-runtime==13.0.96\n[pip3] nvidia-cudnn-cu13==9.19.0.56\n[pip3] nvidia-cufft==12.0.0.61\n[pip3] nvidia-curand==10.4.0.35\n[pip3] nvidia-cusolver==12.0.4.66\n[pip3] nvidia-cusparse==12.6.3.3\n[pip3] nvidia-cusparselt-cu13==0.8.0\n[pip3] nvidia-nccl-cu13==2.28.9\n[pip3] nvidia-nvjitlink==13.0.88\n[pip3] nvidia-nvtx==13.0.85\n[pip3] segmentation_models_pytorch==0.5.0\n[pip3] torch==2.11.0\n[pip3] torchao==0.17.0\n[pip3] torchaudio==2.11.0\n[pip3] torchcodec==0.16.0\n[pip3] torchvision==0.26.0\n[pip3] triton==3.6.0\n[conda] Could not collect",
+  "transformers_version": "5.7.0",
+  "lm_eval_version": "0.4.12",
+  "upper_git_hash": null,
+  "tokenizer_pad_token": [
+    "<|pad|>",
+    "0"
+  ],
+  "tokenizer_eos_token": [
+    "<|eos|>",
+    "2"
+  ],
+  "tokenizer_bos_token": [
+    "<|bos|>",
+    "1"
+  ],
+  "eot_token_id": 2,
+  "max_length": 4096,
+  "task_hashes": {},
+  "model_source": "hf",
+  "model_name": "/home/banaxi/Desktop/BananaMind/BananaMind-2/BananaMind-2.1-Unified",
+  "model_name_sanitized": "__home__banaxi__Desktop__BananaMind__BananaMind-2__BananaMind-2.1-Unified",
+  "system_instruction": null,
+  "system_instruction_sha": null,
+  "fewshot_as_multiturn": null,
+  "chat_template": null,
+  "chat_template_sha": null,
+  "total_evaluation_time_seconds": "59.47798355802661"
+}

generation_config.json

@@ -0,0 +1,7 @@
+{
+  "_from_model_config": true,
+  "bos_token_id": 1,
+  "eos_token_id": 2,
+  "pad_token_id": 0,
+  "transformers_version": "5"
+}

model.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:3b23f2e1f59a5631fea477026f7b9b14f0b6fddeb42a1f448e7eff2eb365887a
+size 140025876

modeling_bananamind21unified.py

@@ -0,0 +1,610 @@
+"""BananaMind 2.1 Unified as an HF causal LM.
+
+IMPORTANT - what `.logits` contains
+-----------------------------------
+Towers A and C are mixed in *probability* space, so the natural output of this
+model is a normalised log-probability vector, not a logit vector:
+
+    log p = logaddexp(log a + log_softmax(logits_A),
+                      log(1-a) + log_softmax(logits_C))
+
+`.logits` carries that log-probability vector directly.  This is safe for every
+standard consumer, because `log_softmax` is the identity on an already
+normalised log-probability vector (its logsumexp is 0), and `softmax(log p) = p`.
+So loglikelihood scoring, `generate()`, and temperature-1 sampling all behave
+correctly.  What is *not* meaningful is treating these numbers as unnormalised
+scores with an arbitrary additive offset - they are already calibrated.
+
+Tower B is the relay.  It has no output head and never appears in the mixture;
+it exists only to carry signal between A and C, which have no other path to
+each other.  Set `config.cut_bridges = True` to sever every bridge, which turns
+A and C into two ordinary standalone transformers.
+
+KV cache
+--------
+Supported, and on by default.  A relay model has no single residual stack, so
+the three towers share one flat cache index space - tower A first, then B, then
+C - sized by `config.num_hidden_layers`, which is the sum of the three tower
+depths.  Each tower keeps its own entries; nothing is shared between them.
+
+Two properties of this architecture make that work without any extra machinery:
+
+* A bridge (`Edge`) is a per-channel gate on a linear map, so it mixes channels
+  but never positions.  Every bridge contribution for a newly arriving token is
+  computable from that token's own tower states, so no bridge output has to be
+  cached alongside the KV states.
+* RoPE enters the attention logits only through the relative offset between
+  query and key, so shifting a whole sequence - which is what left padding does
+  - leaves every attention score unchanged.  Absolute positions taken from the
+  cache length are therefore correct for padded batches too.
+
+Parameter names match the training module exactly, so a checkpoint transfers
+without any key rewriting.
+"""
+import math
+from typing import Optional
+
+import torch
+import torch.nn as nn
+import torch.nn.functional as F
+from transformers import PreTrainedModel
+from transformers.cache_utils import Cache, DynamicCache
+from transformers.generation import GenerationMixin
+from transformers.modeling_outputs import CausalLMOutputWithPast
+
+from .configuration_bananamind21unified import BananaMind21UnifiedConfig
+
+
+class RMSNorm(nn.Module):
+    def __init__(self, dim, eps=1e-6):
+        super().__init__()
+        self.eps = eps
+        self.weight = nn.Parameter(torch.ones(dim))
+
+    def forward(self, x):
+        x_float = x.float()
+        rms = torch.rsqrt(x_float.pow(2).mean(-1, keepdim=True) + self.eps)
+        return (x_float * rms * self.weight.float()).type_as(x)
+
+
+def precompute_freqs_cis(head_dim, seq_len, theta=100000.0):
+    inv = 1.0 / (theta ** (torch.arange(0, head_dim, 2, dtype=torch.float32) / head_dim))
+    t = torch.arange(seq_len, dtype=torch.float32)
+    return torch.polar(torch.ones_like(torch.outer(t, inv)), torch.outer(t, inv))
+
+
+def apply_rotary_emb(q, k, freqs_cis):
+    q_complex = torch.view_as_complex(q.float().reshape(*q.shape[:-1], -1, 2))
+    k_complex = torch.view_as_complex(k.float().reshape(*k.shape[:-1], -1, 2))
+    freqs_cis = freqs_cis.unsqueeze(0).unsqueeze(0)
+    q_out = torch.view_as_real(q_complex * freqs_cis).flatten(-2)
+    k_out = torch.view_as_real(k_complex * freqs_cis).flatten(-2)
+    return q_out.type_as(q), k_out.type_as(k)
+
+
+class _Spec:
+    def __init__(self, hidden_size, num_hidden_layers, num_attention_heads, num_key_value_heads, intermediate_size):
+        self.hidden_size = hidden_size
+        self.num_hidden_layers = num_hidden_layers
+        self.num_attention_heads = num_attention_heads
+        self.num_key_value_heads = num_key_value_heads
+        self.intermediate_size = intermediate_size
+
+
+class TowerAttention(nn.Module):
+    def __init__(self, spec, head_dim, rms_norm_eps, layer_idx=None):
+        super().__init__()
+        self.n_head = spec.num_attention_heads
+        self.n_kv_heads = spec.num_key_value_heads
+        self.head_dim = head_dim
+        self.n_rep = self.n_head // self.n_kv_heads
+        # flat index into the shared KV cache; see `num_hidden_layers` in the config
+        self.layer_idx = layer_idx
+        self.q_proj = nn.Linear(spec.hidden_size, self.n_head * head_dim, bias=False)
+        self.k_proj = nn.Linear(spec.hidden_size, self.n_kv_heads * head_dim, bias=False)
+        self.v_proj = nn.Linear(spec.hidden_size, self.n_kv_heads * head_dim, bias=False)
+        self.o_proj = nn.Linear(self.n_head * head_dim, spec.hidden_size, bias=False)
+        self.q_norm = RMSNorm(head_dim, eps=rms_norm_eps)
+        self.k_norm = RMSNorm(head_dim, eps=rms_norm_eps)
+
+    def _attn_mask(self, attention_mask, q_len, kv_len, past_len, device):
+        """Bottom-right aligned causal mask, intersected with the padding mask.
+
+        `F.scaled_dot_product_attention(is_causal=True)` aligns its mask to the
+        *top left*, which is only correct when `q_len == kv_len`.  With a cache
+        the queries sit at positions `past_len .. past_len+q_len-1` while the
+        keys start at 0, so the mask has to be built explicitly.  Returns
+        `(attn_mask, is_causal)` and prefers the cheap fast paths.
+        """
+        if attention_mask is None:
+            if kv_len == q_len:
+                return None, True       # prefill: top-left alignment is correct
+            if q_len == 1:
+                return None, False      # one new token attends to the whole cache
+        q_pos = torch.arange(q_len, device=device) + past_len
+        k_pos = torch.arange(kv_len, device=device)
+        mask = k_pos[None, :] <= q_pos[:, None]
+        mask = mask[None, None, :, :]
+        if attention_mask is not None:
+            mask = mask & attention_mask.to(torch.bool)[:, None, None, :kv_len]
+            # A fully-padded query row would be all-False, and SDPA turns an
+            # all-masked row into NaN, which then leaks through the residual
+            # stream into every later position.  A token may always attend to
+            # itself, which is causally legal and keeps every row non-empty.
+            mask = mask | (k_pos[None, :] == q_pos[:, None])[None, None, :, :]
+        return mask, False
+
+    def forward(self, x, freqs_cis, attention_mask=None, past_key_values=None, past_len=0):
+        bsz, q_len, _ = x.size()
+        q = self.q_proj(x).view(bsz, q_len, self.n_head, self.head_dim).transpose(1, 2)
+        k = self.k_proj(x).view(bsz, q_len, self.n_kv_heads, self.head_dim).transpose(1, 2)
+        v = self.v_proj(x).view(bsz, q_len, self.n_kv_heads, self.head_dim).transpose(1, 2)
+        q = self.q_norm(q)
+        k = self.k_norm(k)
+        # `freqs_cis` is already sliced to this step's absolute positions
+        q, k = apply_rotary_emb(q, k, freqs_cis)
+
+        # Cache the pre-GQA-expansion states: n_kv_heads is 1 or 2 here, so
+        # storing the expanded copies would cost up to 4x the memory for nothing.
+        if past_key_values is not None:
+            k, v = past_key_values.update(k, v, self.layer_idx)
+        kv_len = k.size(-2)
+
+        k = k.unsqueeze(2).expand(bsz, self.n_kv_heads, self.n_rep, kv_len, self.head_dim)
+        k = k.reshape(bsz, self.n_head, kv_len, self.head_dim).contiguous()
+        v = v.unsqueeze(2).expand(bsz, self.n_kv_heads, self.n_rep, kv_len, self.head_dim)
+        v = v.reshape(bsz, self.n_head, kv_len, self.head_dim).contiguous()
+
+        attn_mask, is_causal = self._attn_mask(
+            attention_mask, q_len, kv_len, past_len, x.device
+        )
+        y = F.scaled_dot_product_attention(q, k, v, attn_mask=attn_mask, is_causal=is_causal)
+        y = y.transpose(1, 2).contiguous().view(bsz, q_len, self.n_head * self.head_dim)
+        return self.o_proj(y)
+
+
+class TowerSwiGLUMLP(nn.Module):
+    def __init__(self, spec):
+        super().__init__()
+        self.w_gate = nn.Linear(spec.hidden_size, spec.intermediate_size, bias=False)
+        self.w_up = nn.Linear(spec.hidden_size, spec.intermediate_size, bias=False)
+        self.w_down = nn.Linear(spec.intermediate_size, spec.hidden_size, bias=False)
+
+    def forward(self, x):
+        return self.w_down(F.silu(self.w_gate(x)) * self.w_up(x))
+
+
+class TowerBlock(nn.Module):
+    def __init__(self, spec, head_dim, rms_norm_eps, layer_idx=None):
+        super().__init__()
+        self.ln_1 = RMSNorm(spec.hidden_size, eps=rms_norm_eps)
+        self.attn = TowerAttention(spec, head_dim, rms_norm_eps, layer_idx=layer_idx)
+        self.ln_2 = RMSNorm(spec.hidden_size, eps=rms_norm_eps)
+        self.mlp = TowerSwiGLUMLP(spec)
+
+    def forward(self, x, freqs_cis, attention_mask=None, past_key_values=None, past_len=0):
+        x = x + self.attn(
+            self.ln_1(x),
+            freqs_cis,
+            attention_mask=attention_mask,
+            past_key_values=past_key_values,
+            past_len=past_len,
+        )
+        x = x + self.mlp(self.ln_2(x))
+        return x
+
+
+class Edge(nn.Module):
+    def __init__(self, dim_in, dim_out, gate_init=0.01):
+        super().__init__()
+        self.w = nn.Linear(dim_in, dim_out, bias=False)
+        self.g = nn.Parameter(torch.full((dim_out,), float(gate_init)))
+
+    def forward(self, x):
+        return self.g * self.w(x)
+
+
+class RelayMode:
+    """Which towers run, whether the bridges carry, and which heads vote.
+
+    The point of the 2.1 topology is that A and C have no direct path to each
+    other, so "what is the relay worth?" is only answerable by ablation.  Each
+    mode below cuts the model somewhere different; every one of them runs on the
+    same weights, with no retraining and no reloading.
+
+    ``run_b=False`` with ``bridges=True`` is the interesting one: tower B's
+    blocks are replaced by the identity, but the bridge wiring stays live, so A
+    and C still exchange signal - through a channel that does no computation.
+    That separates "the relay computes something" from "a channel exists at all".
+    """
+
+    __slots__ = ("name", "run_a", "run_b", "run_c", "bridges", "head_a", "head_c", "doc")
+
+    def __init__(self, name, run_a, run_b, run_c, bridges, head_a, head_c, doc):
+        self.name = name
+        self.run_a, self.run_b, self.run_c = run_a, run_b, run_c
+        self.bridges = bridges
+        self.head_a, self.head_c = head_a, head_c
+        self.doc = doc
+
+    @property
+    def two_headed(self):
+        return self.head_a and self.head_c
+
+
+RELAY_MODES = {
+    m.name: m
+    for m in (
+        RelayMode("full", True, True, True, True, True, True,
+                  "the model as trained: three towers, all bridges, both heads mixed"),
+        RelayMode("cut_bridges", True, True, True, False, True, True,
+                  "all 12 bridges severed; A and C become standalone transformers, "
+                  "B is orphaned; the two heads are still mixed"),
+        RelayMode("bypass_b", True, False, True, True, True, True,
+                  "B's 5 blocks are skipped but every bridge stays live: A and C "
+                  "still exchange, through a relay that does no computation"),
+        RelayMode("ab_only", True, True, False, True, True, False,
+                  "tower C is switched off; A and B run with the A<->B bridges "
+                  "live; head A alone produces the distribution"),
+        RelayMode("cb_only", False, True, True, True, False, True,
+                  "tower A is switched off; C and B run with the C<->B bridges "
+                  "live; head C alone produces the distribution"),
+        RelayMode("a_only", True, False, False, False, True, False,
+                  "tower A alone, no bridges, head A alone"),
+        RelayMode("c_only", False, False, True, False, False, True,
+                  "tower C alone, no bridges, head C alone"),
+    )
+}
+
+_SINGLE_TOWER_MODES = {"a": "a_only", "c": "c_only"}
+
+
+def resolve_relay_mode(relay_mode=None, use_single_tower=None, cut_bridges=False):
+    """Fold the three user-facing switches into exactly one `RelayMode`.
+
+    They overlap on purpose - `cut_bridges` predates the others and stays
+    supported - so anything contradictory is rejected rather than silently
+    resolved in an order nobody can guess.
+    """
+    chosen = []
+    if relay_mode is not None:
+        name = str(relay_mode).lower()
+        if name not in RELAY_MODES:
+            raise ValueError(
+                f"unknown relay_mode {relay_mode!r}; expected one of "
+                f"{sorted(RELAY_MODES)}"
+            )
+        chosen.append(("relay_mode", name))
+    if use_single_tower is not None:
+        tower = str(use_single_tower).lower()
+        if tower not in _SINGLE_TOWER_MODES:
+            raise ValueError(
+                f"use_single_tower must be 'a' or 'c' (tower B has no output "
+                f"head and cannot run alone), got {use_single_tower!r}"
+            )
+        chosen.append(("use_single_tower", _SINGLE_TOWER_MODES[tower]))
+    if cut_bridges:
+        chosen.append(("cut_bridges", "cut_bridges"))
+
+    names = {name for _, name in chosen}
+    if len(names) > 1:
+        detail = ", ".join(f"{src}={name!r}" for src, name in chosen)
+        raise ValueError(f"conflicting relay settings: {detail}")
+    return RELAY_MODES[names.pop()] if names else RELAY_MODES["full"]
+
+
+class BananaMind21UnifiedPreTrainedModel(PreTrainedModel):
+    config_class = BananaMind21UnifiedConfig
+    base_model_prefix = "model"
+    supports_gradient_checkpointing = False
+
+    def _init_weights(self, module):
+        if isinstance(module, nn.Linear):
+            torch.nn.init.normal_(module.weight, mean=0.0, std=0.02)
+            if module.bias is not None:
+                torch.nn.init.zeros_(module.bias)
+        elif isinstance(module, nn.Embedding):
+            torch.nn.init.normal_(module.weight, mean=0.0, std=0.02)
+
+
+class BananaMind21UnifiedForCausalLM(BananaMind21UnifiedPreTrainedModel, GenerationMixin):
+    def __init__(self, config):
+        super().__init__(config)
+        self.config = config
+        a = _Spec(config.hidden_size_a, config.num_hidden_layers_a, config.num_attention_heads_a,
+                  config.num_key_value_heads_a, config.intermediate_size_a)
+        b = _Spec(config.hidden_size_b, config.num_hidden_layers_b, config.num_attention_heads_b,
+                  config.num_key_value_heads_b, config.intermediate_size_b)
+        c = _Spec(config.hidden_size_c, config.num_hidden_layers_c, config.num_attention_heads_c,
+                  config.num_key_value_heads_c, config.intermediate_size_c)
+
+        self.wte = nn.Embedding(config.vocab_size, config.embed_width)
+        self.in_proj_a = nn.Linear(config.embed_width, a.hidden_size, bias=False)
+        self.in_proj_b = nn.Linear(config.embed_width, b.hidden_size, bias=False)
+
+        # Flat KV-cache index space over the three towers, in the order the
+        # config documents: A first, then B, then C.  Each tower keeps its own
+        # cache entries; nothing is shared between them.
+        self._cache_offset_a = 0
+        self._cache_offset_b = a.num_hidden_layers
+        self._cache_offset_c = a.num_hidden_layers + b.num_hidden_layers
+
+        self.blocks_a = nn.ModuleList([TowerBlock(a, config.head_dim, config.rms_norm_eps,
+                                                  layer_idx=self._cache_offset_a + i)
+                                       for i in range(a.num_hidden_layers)])
+        self.blocks_b = nn.ModuleList([TowerBlock(b, config.head_dim, config.rms_norm_eps,
+                                                  layer_idx=self._cache_offset_b + i)
+                                       for i in range(b.num_hidden_layers)])
+        self.blocks_c = nn.ModuleList([TowerBlock(c, config.head_dim, config.rms_norm_eps,
+                                                  layer_idx=self._cache_offset_c + i)
+                                       for i in range(c.num_hidden_layers)])
+
+        n = len(config.a_read)
+        gi = config.gate_init
+        self.edges_a2b = nn.ModuleList([Edge(a.hidden_size, b.hidden_size, gi) for _ in range(n)])
+        self.edges_c2b = nn.ModuleList([Edge(c.hidden_size, b.hidden_size, gi) for _ in range(n)])
+        self.edges_b2a = nn.ModuleList([Edge(b.hidden_size, a.hidden_size, gi) for _ in range(n)])
+        self.edges_b2c = nn.ModuleList([Edge(b.hidden_size, c.hidden_size, gi) for _ in range(n)])
+
+        self.ln_f_a = RMSNorm(a.hidden_size, eps=config.rms_norm_eps)
+        self.ln_f_c = RMSNorm(c.hidden_size, eps=config.rms_norm_eps)
+        self.lm_head_a = nn.Linear(a.hidden_size, config.vocab_size, bias=False)
+        self.lm_head_c = nn.Linear(c.hidden_size, config.vocab_size, bias=False)
+        self.mix_head = nn.Linear(a.hidden_size + c.hidden_size, 1, bias=True)
+
+        self._embd_scale = math.sqrt(config.embed_width)
+        self._freqs_cis_cache = None
+        # Validated once at construction so a bad mode fails on load, not on the
+        # first forward pass halfway through an evaluation sweep.
+        self._config_mode = resolve_relay_mode(
+            getattr(config, "relay_mode", None),
+            getattr(config, "use_single_tower", None),
+            getattr(config, "cut_bridges", False),
+        )
+        self.post_init()
+
+    @property
+    def relay_mode(self):
+        """The `RelayMode` this model runs in unless a call overrides it."""
+        return self._config_mode
+
+    def set_relay_mode(self, relay_mode=None, use_single_tower=None):
+        """Switch ablation mode in place, for sweeping without reloading.
+
+        Returns the resolved `RelayMode`.  Any KV cache built under the previous
+        mode is invalid afterwards - the towers it holds state for may no longer
+        be the towers that run - so start a fresh cache after calling this.
+        """
+        mode = resolve_relay_mode(relay_mode, use_single_tower)
+        self._config_mode = mode
+        self.config.relay_mode = mode.name
+        self.config.use_single_tower = None
+        self.config.cut_bridges = mode.name == "cut_bridges"
+        return mode
+
+    def get_input_embeddings(self):
+        return self.wte
+
+    def set_input_embeddings(self, value):
+        self.wte = value
+
+    def get_output_embeddings(self):
+        return self.lm_head_c
+
+    def resolve_mode(self, relay_mode=None, use_single_tower=None):
+        """The `RelayMode` in force, per-call arguments overriding the config."""
+        if relay_mode is None and use_single_tower is None:
+            return self._config_mode
+        return resolve_relay_mode(relay_mode, use_single_tower)
+
+    def prepare_inputs_for_generation(
+        self, input_ids, past_key_values=None, attention_mask=None, use_cache=True,
+        relay_mode=None, use_single_tower=None, **kwargs
+    ):
+        # With a warm cache only the tokens the towers have not seen yet need to
+        # be forwarded.  Every live tower advances over the same positions, so
+        # one probed cache length describes all of them.
+        mode = self.resolve_mode(relay_mode, use_single_tower)
+        past_len = self._cache_len(past_key_values, mode)
+        if past_len > 0:
+            input_ids = input_ids[:, past_len:]
+        out = {
+            "input_ids": input_ids,
+            "attention_mask": attention_mask,
+            "past_key_values": past_key_values,
+            "use_cache": use_cache,
+        }
+        # Only forward an explicit override; otherwise the config mode applies.
+        if relay_mode is not None:
+            out["relay_mode"] = relay_mode
+        if use_single_tower is not None:
+            out["use_single_tower"] = use_single_tower
+        return out
+
+    def _get_freqs_cis(self, seq_len, device):
+        cache = self._freqs_cis_cache
+        if cache is None or cache.device != device or cache.size(0) < seq_len:
+            cache = precompute_freqs_cis(
+                self.config.head_dim, seq_len, self.config.rope_theta
+            ).to(device)
+            self._freqs_cis_cache = cache
+        return cache[:seq_len]
+
+    @staticmethod
+    def _advance(blocks, x, cursor, target, freqs_cis, pending, attention_mask,
+                 past_key_values=None, past_len=0, run_blocks=True):
+        """Run `blocks[cursor:target]`, folding in bridge arrivals as they land.
+
+        With `run_blocks=False` the blocks are replaced by the identity but the
+        arrivals are still added and the cursor still advances, which is what
+        turns tower B into a pass-through channel in `bypass_b`.
+        """
+        while cursor < target:
+            arriving = pending.pop(cursor + 1, None)
+            if arriving is not None:
+                x = x + arriving
+            if run_blocks:
+                x = blocks[cursor](
+                    x,
+                    freqs_cis,
+                    attention_mask=attention_mask,
+                    past_key_values=past_key_values,
+                    past_len=past_len,
+                )
+            cursor += 1
+        return x, cursor
+
+    def _cache_probe_idx(self, mode):
+        """Flat cache index of the first block that actually runs in `mode`.
+
+        `Cache.get_seq_length()` defaults to layer 0, which is tower A's first
+        block - but A does not run in every mode, and an untouched layer reports
+        length 0 forever.  Probing a layer that really runs keeps the cache
+        length honest under every ablation.
+        """
+        if mode.run_a:
+            return self._cache_offset_a
+        if mode.run_b:
+            return self._cache_offset_b
+        return self._cache_offset_c
+
+    def _cache_len(self, past_key_values, mode):
+        if past_key_values is None:
+            return 0
+        return past_key_values.get_seq_length(self._cache_probe_idx(mode))
+
+    def hidden_states(self, input_ids, attention_mask=None, past_key_values=None,
+                      relay_mode=None, use_single_tower=None):
+        """Run the towers and return `(h_a, h_c, mix_logit)`.
+
+        A tower switched off by the active mode returns `None` in its slot, and
+        `mix_logit` is `None` whenever only one head is live - there is nothing
+        for the mixer to weigh.  In the default `full` mode all three are always
+        tensors, so existing callers are unaffected.
+        """
+        cfg = self.config
+        mode = self.resolve_mode(relay_mode, use_single_tower)
+        _, seq_len = input_ids.size()
+        # Read the cache length once, before any layer writes to it.
+        past_len = self._cache_len(past_key_values, mode)
+        # RoPE is applied at absolute positions, so a cached run has to skip the
+        # `past_len` entries the earlier tokens already used.
+        freqs_cis = self._get_freqs_cis(past_len + seq_len, input_ids.device)[past_len:]
+
+        embedded = self.wte(input_ids) * self._embd_scale
+        x_a = self.in_proj_a(embedded) if mode.run_a else None
+        x_b = self.in_proj_b(embedded) if (mode.run_b or mode.bridges) else None
+        x_c = embedded if mode.run_c else None
+
+        pend_a, pend_b, pend_c = {}, {}, {}
+        cur_a = cur_b = cur_c = 0
+        kv = dict(past_key_values=past_key_values, past_len=past_len)
+        step_a = dict(kv, run_blocks=mode.run_a)
+        step_b = dict(kv, run_blocks=mode.run_b)
+        step_c = dict(kv, run_blocks=mode.run_c)
+
+        # A bridge is only carried when both of its endpoints are live: in
+        # `ab_only` there is no tower C to read from or write back to, so the
+        # C-side edges are skipped rather than fed zeros.
+        carry_a = mode.bridges and mode.run_a
+        carry_c = mode.bridges and mode.run_c
+        relay_live = mode.bridges and x_b is not None
+
+        # An `Edge` is a per-channel gate on a linear map, so a bridge mixes
+        # channels but never positions.  Every bridge contribution for the new
+        # tokens is therefore computable from the new tokens' own tower states,
+        # and no bridge output needs to be cached alongside the KV states.
+        for k in range(len(cfg.a_read)):
+            if x_a is not None:
+                x_a, cur_a = self._advance(self.blocks_a, x_a, cur_a, cfg.a_read[k],
+                                           freqs_cis, pend_a, attention_mask, **step_a)
+            if x_c is not None:
+                x_c, cur_c = self._advance(self.blocks_c, x_c, cur_c, cfg.c_read[k],
+                                           freqs_cis, pend_c, attention_mask, **step_c)
+            if relay_live:
+                into_b = None
+                if carry_a:
+                    into_b = self.edges_a2b[k](x_a)
+                if carry_c:
+                    contrib = self.edges_c2b[k](x_c)
+                    into_b = contrib if into_b is None else into_b + contrib
+                if into_b is not None:
+                    pend_b[cfg.b_land[k]] = into_b
+            if x_b is not None:
+                x_b, cur_b = self._advance(self.blocks_b, x_b, cur_b, cfg.b_read[k],
+                                           freqs_cis, pend_b, attention_mask, **step_b)
+            if relay_live:
+                if carry_a:
+                    pend_a[cfg.a_land[k]] = self.edges_b2a[k](x_b)
+                if carry_c:
+                    pend_c[cfg.c_land[k]] = self.edges_b2c[k](x_b)
+
+        if x_a is not None:
+            x_a, _ = self._advance(self.blocks_a, x_a, cur_a, cfg.num_hidden_layers_a,
+                                   freqs_cis, pend_a, attention_mask, **step_a)
+        if x_c is not None:
+            x_c, _ = self._advance(self.blocks_c, x_c, cur_c, cfg.num_hidden_layers_c,
+                                   freqs_cis, pend_c, attention_mask, **step_c)
+        if x_b is not None:
+            x_b, _ = self._advance(self.blocks_b, x_b, cur_b, cfg.num_hidden_layers_b,
+                                   freqs_cis, pend_b, attention_mask, **step_b)
+
+        h_a = self.ln_f_a(x_a) if x_a is not None else None
+        h_c = self.ln_f_c(x_c) if x_c is not None else None
+        mix_logit = None
+        if mode.two_headed:
+            mix_logit = self.mix_head(torch.cat([h_a, h_c], dim=-1)).squeeze(-1)
+        return h_a, h_c, mix_logit
+
+    def forward(
+        self,
+        input_ids,
+        attention_mask=None,
+        labels=None,
+        past_key_values: Optional[Cache] = None,
+        use_cache: Optional[bool] = None,
+        relay_mode=None,
+        use_single_tower=None,
+        **kwargs,
+    ):
+        mode = self.resolve_mode(relay_mode, use_single_tower)
+
+        if use_cache is None:
+            use_cache = getattr(self.config, "use_cache", True)
+        if labels is not None:
+            # A scoring or training pass consumes the whole sequence in one go
+            # and throws the states away, so building 25 layers of cache is pure
+            # overhead.  An explicitly supplied cache is still honoured.
+            use_cache = use_cache and past_key_values is not None
+        if use_cache and past_key_values is None:
+            past_key_values = DynamicCache(config=self.config)
+        if not use_cache:
+            past_key_values = None
+
+        h_a, h_c, mix_logit = self.hidden_states(
+            input_ids,
+            attention_mask=attention_mask,
+            past_key_values=past_key_values,
+            relay_mode=mode.name,
+        )
+
+        # With one head live there is nothing to mix, and `log_softmax` alone
+        # already gives the normalised log-probability vector `.logits` promises.
+        if mode.two_headed:
+            log_p_a = F.log_softmax(self.lm_head_a(h_a).float(), dim=-1)
+            log_p_c = F.log_softmax(self.lm_head_c(h_c).float(), dim=-1)
+            log_alpha = F.logsigmoid(mix_logit).unsqueeze(-1)
+            log_one_minus = F.logsigmoid(-mix_logit).unsqueeze(-1)
+            log_p = torch.logaddexp(log_alpha + log_p_a, log_one_minus + log_p_c)
+        elif mode.head_a:
+            log_p = F.log_softmax(self.lm_head_a(h_a).float(), dim=-1)
+        else:
+            log_p = F.log_softmax(self.lm_head_c(h_c).float(), dim=-1)
+
+        loss = None
+        if labels is not None:
+            loss = F.nll_loss(
+                log_p[..., :-1, :].reshape(-1, log_p.size(-1)),
+                labels[..., 1:].reshape(-1),
+            )
+        return CausalLMOutputWithPast(loss=loss, logits=log_p, past_key_values=past_key_values)

modeling_relay.py

@@ -0,0 +1,680 @@
+#!/usr/bin/env python3
+"""BananaMind 2.1 Unified - three towers, relay middle, 35.0M params.
+
+Three independent transformer stacks share one input embedding.  A and C are
+the outer towers; each owns an output head and contributes to the mixture.  B is
+the relay: it has no head, no solo loss term, and no gradient of its own except
+what arrives through its four bridge directions.  There is no direct A<->C path,
+so anything the two outer towers share about a token has to survive a trip
+through B.
+
+Exchange schedule
+-----------------
+Three rounds.  Each round reads the outer towers, lands the sum in B, lets B
+process, then reads B and lands the result back in both outer towers.
+
+    round | A read | C read | -> B  | B runs | B read | -> A  | -> C
+      1   |   5    |   2    | pre-1 |  1-2   |   2    |   7   |   3
+      2   |   9    |   4    | pre-3 |  3-4   |   4    |  11   |   5
+      3   |  12    |   5    | pre-5 |   5    |   5    |  14   |   6
+
+Bridge output is added to the residual *before* the receiving block rather than
+after it, which is what guarantees B always has at least one full block between
+taking a signal in and handing one back out.
+
+Gates are per-channel and initialise to 0.01, not 0.  B is the only path between
+A and C and it carries no solo loss, so a zero-init would leave the middle with
+no gradient signal at all on step 0 and a real chance of never waking up.  0.01
+is small enough that the towers still start as three near-independent models.
+"""
+from __future__ import annotations
+
+import math
+from dataclasses import dataclass, field
+
+import torch
+import torch.nn as nn
+import torch.nn.functional as F
+
+
+# --------------------------------------------------------------------------- #
+# config
+# --------------------------------------------------------------------------- #
+
+
+@dataclass
+class TowerSpec:
+    hidden_size: int
+    num_hidden_layers: int
+    num_attention_heads: int
+    num_key_value_heads: int
+    intermediate_size: int
+
+
+@dataclass
+class RelayConfig:
+    # inherited from Mini / 2.0 Unified, unchanged
+    vocab_size: int = 8192
+    head_dim: int = 64
+    max_position_embeddings: int = 4096
+    rope_theta: float = 100000.0
+    rms_norm_eps: float = 1e-6
+
+    # the shared embedding lives at tower C's width, so C reads it natively and
+    # A and B each get one projection down.
+    embed_width: int = 384
+
+    tower_a: TowerSpec = field(
+        default_factory=lambda: TowerSpec(
+            hidden_size=256,
+            num_hidden_layers=14,
+            num_attention_heads=4,
+            num_key_value_heads=1,
+            intermediate_size=704,
+        )
+    )
+    tower_b: TowerSpec = field(
+        default_factory=lambda: TowerSpec(
+            hidden_size=320,
+            num_hidden_layers=5,
+            num_attention_heads=5,
+            num_key_value_heads=1,
+            intermediate_size=960,
+        )
+    )
+    tower_c: TowerSpec = field(
+        default_factory=lambda: TowerSpec(
+            hidden_size=384,
+            num_hidden_layers=6,
+            num_attention_heads=6,
+            num_key_value_heads=2,
+            intermediate_size=1024,
+        )
+    )
+
+    # 1-indexed exchange schedule.  `*_read` is where a bridge takes a tower's
+    # state; `*_land` is where the returning signal is added.  Biased deep on
+    # the outer towers, because in the 2.0 run the deep bridges carried 4-25x
+    # the gate magnitude of the shallow ones.
+    a_read: tuple[int, ...] = (5, 9, 12)
+    a_land: tuple[int, ...] = (7, 11, 14)
+    c_read: tuple[int, ...] = (2, 4, 5)
+    c_land: tuple[int, ...] = (3, 5, 6)
+    b_land: tuple[int, ...] = (1, 3, 5)
+    b_read: tuple[int, ...] = (2, 4, 5)
+
+    gate_init: float = 0.01
+
+    # untied on purpose: A and C each own an output space, so either one still
+    # decodes by itself once the bridges come off.
+    tie_word_embeddings: bool = False
+
+    @property
+    def n_rounds(self) -> int:
+        return len(self.a_read)
+
+    def validate(self) -> None:
+        if self.embed_width != self.tower_c.hidden_size:
+            raise ValueError("shared embedding width must match tower C width")
+        n = self.n_rounds
+        for name in ("a_read", "a_land", "c_read", "c_land", "b_land", "b_read"):
+            if len(getattr(self, name)) != n:
+                raise ValueError(f"{name} must have {n} entries")
+
+        # Every tower is visited strictly left to right, and every bridge reads
+        # a state that already exists.  If this passes, the forward pass below
+        # is a valid topological order.
+        for read, land, spec, tag in (
+            (self.a_read, self.a_land, self.tower_a, "A"),
+            (self.c_read, self.c_land, self.tower_c, "C"),
+        ):
+            cursor = 0
+            for k in range(n):
+                # `==` is legal: the previous round's signal is added before that
+                # layer runs, so reading right after it is still causal.
+                if not cursor <= read[k]:
+                    raise ValueError(f"tower {tag} round {k}: read layer {read[k]} is behind cursor {cursor}")
+                if not read[k] < land[k]:
+                    raise ValueError(f"tower {tag} round {k}: land {land[k]} must follow read {read[k]}")
+                cursor = land[k]
+            if max(land) > spec.num_hidden_layers or min(read) < 1:
+                raise ValueError(f"tower {tag} schedule leaves the stack")
+
+        cursor = 0
+        for k in range(n):
+            if not cursor < self.b_land[k]:
+                raise ValueError(f"tower B round {k}: land {self.b_land[k]} is not ahead of {cursor}")
+            if not self.b_land[k] <= self.b_read[k]:
+                raise ValueError(f"tower B round {k}: read {self.b_read[k]} precedes land {self.b_land[k]}")
+            cursor = self.b_read[k]
+        if max(self.b_read) > self.tower_b.num_hidden_layers or min(self.b_land) < 1:
+            raise ValueError("tower B schedule leaves the stack")
+
+
+# --------------------------------------------------------------------------- #
+# primitives (identical to train_mini.py / 2.0 Unified)
+# --------------------------------------------------------------------------- #
+
+
+class RMSNorm(nn.Module):
+    def __init__(self, dim: int, eps: float = 1e-6):
+        super().__init__()
+        self.eps = eps
+        self.weight = nn.Parameter(torch.ones(dim))
+
+    def forward(self, x: torch.Tensor) -> torch.Tensor:
+        x_float = x.float()
+        rms = torch.rsqrt(x_float.pow(2).mean(-1, keepdim=True) + self.eps)
+        return (x_float * rms * self.weight.float()).type_as(x)
+
+
+def build_rope_inv_freq(head_dim: int, theta: float = 100000.0) -> torch.Tensor:
+    return 1.0 / (theta ** (torch.arange(0, head_dim, 2, dtype=torch.float32) / head_dim))
+
+
+def precompute_freqs_cis(head_dim: int, seq_len: int, theta: float = 100000.0) -> torch.Tensor:
+    freqs = torch.outer(torch.arange(seq_len, dtype=torch.float32), build_rope_inv_freq(head_dim, theta))
+    return torch.polar(torch.ones_like(freqs), freqs)
+
+
+def apply_rotary_emb(
+    q: torch.Tensor, k: torch.Tensor, freqs_cis: torch.Tensor
+) -> tuple[torch.Tensor, torch.Tensor]:
+    q_complex = torch.view_as_complex(q.float().reshape(*q.shape[:-1], -1, 2))
+    k_complex = torch.view_as_complex(k.float().reshape(*k.shape[:-1], -1, 2))
+    freqs_cis = freqs_cis.unsqueeze(0).unsqueeze(0)
+    q_out = torch.view_as_real(q_complex * freqs_cis).flatten(-2)
+    k_out = torch.view_as_real(k_complex * freqs_cis).flatten(-2)
+    return q_out.type_as(q), k_out.type_as(k)
+
+
+class TowerAttention(nn.Module):
+    def __init__(self, spec: TowerSpec, head_dim: int, rms_norm_eps: float):
+        super().__init__()
+        self.n_head = spec.num_attention_heads
+        self.n_kv_heads = spec.num_key_value_heads
+        self.head_dim = head_dim
+        self.n_rep = self.n_head // self.n_kv_heads
+
+        self.q_proj = nn.Linear(spec.hidden_size, self.n_head * head_dim, bias=False)
+        self.k_proj = nn.Linear(spec.hidden_size, self.n_kv_heads * head_dim, bias=False)
+        self.v_proj = nn.Linear(spec.hidden_size, self.n_kv_heads * head_dim, bias=False)
+        self.o_proj = nn.Linear(self.n_head * head_dim, spec.hidden_size, bias=False)
+        self.o_proj.NANOGPT_SCALE_INIT = 1
+
+        self.q_norm = RMSNorm(head_dim, eps=rms_norm_eps)
+        self.k_norm = RMSNorm(head_dim, eps=rms_norm_eps)
+
+    def forward(self, x: torch.Tensor, freqs_cis: torch.Tensor) -> torch.Tensor:
+        bsz, seq_len, _ = x.size()
+
+        q = self.q_proj(x).view(bsz, seq_len, self.n_head, self.head_dim).transpose(1, 2)
+        k = self.k_proj(x).view(bsz, seq_len, self.n_kv_heads, self.head_dim).transpose(1, 2)
+        v = self.v_proj(x).view(bsz, seq_len, self.n_kv_heads, self.head_dim).transpose(1, 2)
+
+        q = self.q_norm(q)
+        k = self.k_norm(k)
+        q, k = apply_rotary_emb(q, k, freqs_cis)
+
+        # .contiguous() is required for torch.compile: without it the expanded
+        # view reaches the flash-attention backward with strides Inductor
+        # rejects.  Pure layout change, no effect on the math.
+        k = (
+            k.unsqueeze(2)
+            .expand(bsz, self.n_kv_heads, self.n_rep, seq_len, self.head_dim)
+            .reshape(bsz, self.n_head, seq_len, self.head_dim)
+            .contiguous()
+        )
+        v = (
+            v.unsqueeze(2)
+            .expand(bsz, self.n_kv_heads, self.n_rep, seq_len, self.head_dim)
+            .reshape(bsz, self.n_head, seq_len, self.head_dim)
+            .contiguous()
+        )
+
+        y = F.scaled_dot_product_attention(q, k, v, is_causal=True)
+        y = y.transpose(1, 2).contiguous().view(bsz, seq_len, self.n_head * self.head_dim)
+        return self.o_proj(y)
+
+
+class TowerSwiGLUMLP(nn.Module):
+    def __init__(self, spec: TowerSpec):
+        super().__init__()
+        self.w_gate = nn.Linear(spec.hidden_size, spec.intermediate_size, bias=False)
+        self.w_up = nn.Linear(spec.hidden_size, spec.intermediate_size, bias=False)
+        self.w_down = nn.Linear(spec.intermediate_size, spec.hidden_size, bias=False)
+        self.w_down.NANOGPT_SCALE_INIT = 1
+
+    def forward(self, x: torch.Tensor) -> torch.Tensor:
+        return self.w_down(F.silu(self.w_gate(x)) * self.w_up(x))
+
+
+class TowerBlock(nn.Module):
+    """A plain Mini block.  Nothing in here knows about bridges."""
+
+    def __init__(self, spec: TowerSpec, head_dim: int, rms_norm_eps: float):
+        super().__init__()
+        self.ln_1 = RMSNorm(spec.hidden_size, eps=rms_norm_eps)
+        self.attn = TowerAttention(spec, head_dim, rms_norm_eps)
+        self.ln_2 = RMSNorm(spec.hidden_size, eps=rms_norm_eps)
+        self.mlp = TowerSwiGLUMLP(spec)
+
+    def forward(self, x: torch.Tensor, freqs_cis: torch.Tensor) -> torch.Tensor:
+        x = x + self.attn(self.ln_1(x), freqs_cis)
+        x = x + self.mlp(self.ln_2(x))
+        return x
+
+
+# --------------------------------------------------------------------------- #
+# bridge
+# --------------------------------------------------------------------------- #
+
+
+class Edge(nn.Module):
+    """One direction of one exchange point.
+
+    W is a plain linear, g a per-channel gate.  The gate starts small but not
+    zero so the relay gets traffic on step 0.
+    """
+
+    def __init__(self, dim_in: int, dim_out: int, gate_init: float):
+        super().__init__()
+        self.w = nn.Linear(dim_in, dim_out, bias=False)
+        self.g = nn.Parameter(torch.full((dim_out,), float(gate_init)))
+
+    def forward(self, x: torch.Tensor) -> torch.Tensor:
+        return self.g * self.w(x)
+
+
+# --------------------------------------------------------------------------- #
+# model
+# --------------------------------------------------------------------------- #
+
+
+class BananaMindRelay(nn.Module):
+    def __init__(self, config: RelayConfig):
+        super().__init__()
+        config.validate()
+        self.config = config
+        a, b, c = config.tower_a, config.tower_b, config.tower_c
+
+        self.wte = nn.Embedding(config.vocab_size, config.embed_width)
+        self.in_proj_a = nn.Linear(config.embed_width, a.hidden_size, bias=False)
+        self.in_proj_b = nn.Linear(config.embed_width, b.hidden_size, bias=False)
+        # tower C reads the shared embedding natively; no projection
+
+        self.blocks_a = nn.ModuleList(
+            [TowerBlock(a, config.head_dim, config.rms_norm_eps) for _ in range(a.num_hidden_layers)]
+        )
+        self.blocks_b = nn.ModuleList(
+            [TowerBlock(b, config.head_dim, config.rms_norm_eps) for _ in range(b.num_hidden_layers)]
+        )
+        self.blocks_c = nn.ModuleList(
+            [TowerBlock(c, config.head_dim, config.rms_norm_eps) for _ in range(c.num_hidden_layers)]
+        )
+
+        n = config.n_rounds
+        gi = config.gate_init
+        self.edges_a2b = nn.ModuleList([Edge(a.hidden_size, b.hidden_size, gi) for _ in range(n)])
+        self.edges_c2b = nn.ModuleList([Edge(c.hidden_size, b.hidden_size, gi) for _ in range(n)])
+        self.edges_b2a = nn.ModuleList([Edge(b.hidden_size, a.hidden_size, gi) for _ in range(n)])
+        self.edges_b2c = nn.ModuleList([Edge(b.hidden_size, c.hidden_size, gi) for _ in range(n)])
+
+        # B has no final norm because it has no head: its only outputs are the
+        # bridge reads, which are normalised by the receiving tower's blocks.
+        self.ln_f_a = RMSNorm(a.hidden_size, eps=config.rms_norm_eps)
+        self.ln_f_c = RMSNorm(c.hidden_size, eps=config.rms_norm_eps)
+
+        self.lm_head_a = nn.Linear(a.hidden_size, config.vocab_size, bias=False)
+        self.lm_head_c = nn.Linear(c.hidden_size, config.vocab_size, bias=False)
+        if config.tie_word_embeddings:
+            self.lm_head_c.weight = self.wte.weight
+
+        # still a 2-way mixer: it reads the two towers that have heads
+        self.mix_head = nn.Linear(a.hidden_size + c.hidden_size, 1, bias=True)
+
+        self._embd_scale = math.sqrt(config.embed_width)
+        self._freqs_cis_cache: torch.Tensor | None = None
+
+        self._init_all()
+
+    # ---- init -------------------------------------------------------------- #
+
+    def _init_module(self, module: nn.Module, depth: int) -> None:
+        std = 0.02
+        if hasattr(module, "NANOGPT_SCALE_INIT"):
+            std *= 2 * depth**-0.5
+        if isinstance(module, nn.Linear):
+            torch.nn.init.normal_(module.weight, mean=0.0, std=std)
+            if module.bias is not None:
+                torch.nn.init.zeros_(module.bias)
+        elif isinstance(module, nn.Embedding):
+            torch.nn.init.normal_(module.weight, mean=0.0, std=0.02)
+
+    def _init_all(self) -> None:
+        cfg = self.config
+        depth_a = cfg.tower_a.num_hidden_layers
+        depth_b = cfg.tower_b.num_hidden_layers
+        depth_c = cfg.tower_c.num_hidden_layers
+
+        for blocks, depth in (
+            (self.blocks_a, depth_a),
+            (self.blocks_b, depth_b),
+            (self.blocks_c, depth_c),
+        ):
+            for m in blocks.modules():
+                self._init_module(m, depth)
+
+        for m in (
+            self.wte,
+            self.in_proj_a,
+            self.in_proj_b,
+            self.lm_head_a,
+            self.lm_head_c,
+            self.mix_head,
+        ):
+            self._init_module(m, depth_a)
+
+        for edges, depth in (
+            (self.edges_a2b, depth_b),
+            (self.edges_c2b, depth_b),
+            (self.edges_b2a, depth_a),
+            (self.edges_b2c, depth_c),
+        ):
+            for edge in edges:
+                self._init_module(edge.w, depth)
+                torch.nn.init.constant_(edge.g, cfg.gate_init)
+
+        # start the mixer at alpha = 0.5 so neither outer tower is favoured
+        torch.nn.init.zeros_(self.mix_head.weight)
+        torch.nn.init.zeros_(self.mix_head.bias)
+
+    # ---- rope cache -------------------------------------------------------- #
+
+    def _get_freqs_cis(self, seq_len: int, device: torch.device) -> torch.Tensor:
+        cache = self._freqs_cis_cache
+        if cache is None or cache.device != device or cache.size(0) < seq_len:
+            cache = precompute_freqs_cis(
+                self.config.head_dim, seq_len, self.config.rope_theta
+            ).to(device)
+            self._freqs_cis_cache = cache
+        return cache[:seq_len]
+
+    # ---- forward ----------------------------------------------------------- #
+
+    @staticmethod
+    def _advance(
+        blocks: nn.ModuleList,
+        x: torch.Tensor,
+        cursor: int,
+        target: int,
+        freqs_cis: torch.Tensor,
+        pending: dict[int, torch.Tensor],
+    ) -> tuple[torch.Tensor, int]:
+        """Run `blocks` from `cursor` up to and including 1-indexed `target`.
+
+        A pending bridge value keyed by a 1-indexed layer is added to the
+        residual immediately before that layer runs.
+        """
+        while cursor < target:
+            layer = cursor + 1
+            arriving = pending.pop(layer, None)
+            if arriving is not None:
+                x = x + arriving
+            x = blocks[cursor](x, freqs_cis)
+            cursor += 1
+        return x, cursor
+
+    def forward(
+        self,
+        input_ids: torch.Tensor,
+        cut_bridges: bool = False,
+        collect_diagnostics: bool = False,
+    ) -> tuple[torch.Tensor, torch.Tensor, torch.Tensor, dict]:
+        """Returns (h_a, h_c, mix_logit, diagnostics).
+
+        With `cut_bridges=True` the three stacks run completely independently:
+        A and C become ordinary transformers and B becomes dead weight that
+        influences nothing.
+        """
+        cfg = self.config
+        _, seq_len = input_ids.size()
+        freqs_cis = self._get_freqs_cis(seq_len, input_ids.device)
+
+        embedded = self.wte(input_ids) * self._embd_scale
+        x_a = self.in_proj_a(embedded)
+        x_b = self.in_proj_b(embedded)
+        x_c = embedded
+
+        pend_a: dict[int, torch.Tensor] = {}
+        pend_b: dict[int, torch.Tensor] = {}
+        pend_c: dict[int, torch.Tensor] = {}
+        cur_a = cur_b = cur_c = 0
+        diagnostics: dict = {}
+
+        for k in range(cfg.n_rounds):
+            # 1. outer towers walk to their read points, applying whatever the
+            #    previous round handed them on the way.
+            x_a, cur_a = self._advance(self.blocks_a, x_a, cur_a, cfg.a_read[k], freqs_cis, pend_a)
+            x_c, cur_c = self._advance(self.blocks_c, x_c, cur_c, cfg.c_read[k], freqs_cis, pend_c)
+
+            # 2. both outer signals land in the relay at the same layer.
+            if not cut_bridges:
+                pend_b[cfg.b_land[k]] = self.edges_a2b[k](x_a) + self.edges_c2b[k](x_c)
+
+            # 3. the relay processes, then hands one signal back to each side.
+            x_b, cur_b = self._advance(self.blocks_b, x_b, cur_b, cfg.b_read[k], freqs_cis, pend_b)
+
+            if not cut_bridges:
+                pend_a[cfg.a_land[k]] = self.edges_b2a[k](x_b)
+                pend_c[cfg.c_land[k]] = self.edges_b2c[k](x_b)
+
+            if collect_diagnostics:
+                with torch.no_grad():
+                    fa, fb, fc = x_a.float(), x_b.float(), x_c.float()
+                    diagnostics[f"cos_a2b/round{k}"] = float(
+                        F.cosine_similarity(self.edges_a2b[k].w(fa), fb, dim=-1).mean()
+                    )
+                    diagnostics[f"cos_c2b/round{k}"] = float(
+                        F.cosine_similarity(self.edges_c2b[k].w(fc), fb, dim=-1).mean()
+                    )
+                    diagnostics[f"cos_b2a/round{k}"] = float(
+                        F.cosine_similarity(self.edges_b2a[k].w(fb), fa, dim=-1).mean()
+                    )
+                    diagnostics[f"cos_b2c/round{k}"] = float(
+                        F.cosine_similarity(self.edges_b2c[k].w(fb), fc, dim=-1).mean()
+                    )
+                    # how much of A survives the trip to C and back through B
+                    diagnostics[f"rms_a/round{k}"] = float(fa.pow(2).mean().sqrt())
+                    diagnostics[f"rms_b/round{k}"] = float(fb.pow(2).mean().sqrt())
+                    diagnostics[f"rms_c/round{k}"] = float(fc.pow(2).mean().sqrt())
+
+        # 4. finish every stack
+        x_a, cur_a = self._advance(
+            self.blocks_a, x_a, cur_a, cfg.tower_a.num_hidden_layers, freqs_cis, pend_a
+        )
+        x_c, cur_c = self._advance(
+            self.blocks_c, x_c, cur_c, cfg.tower_c.num_hidden_layers, freqs_cis, pend_c
+        )
+        # B's tail runs only so its parameters see gradient in later rounds; with
+        # the default schedule B is already finished here.
+        x_b, cur_b = self._advance(
+            self.blocks_b, x_b, cur_b, cfg.tower_b.num_hidden_layers, freqs_cis, pend_b
+        )
+
+        if pend_a or pend_b or pend_c:
+            raise RuntimeError(f"bridge values were never consumed: {sorted(pend_a) + sorted(pend_b) + sorted(pend_c)}")
+
+        h_a = self.ln_f_a(x_a)
+        h_c = self.ln_f_c(x_c)
+        mix_logit = self.mix_head(torch.cat([h_a, h_c], dim=-1)).squeeze(-1)
+        return h_a, h_c, mix_logit, diagnostics
+
+    # ---- diagnostics ------------------------------------------------------- #
+
+    @torch.no_grad()
+    def gate_stats(self) -> dict[str, float]:
+        """Per-round, per-direction gate magnitudes."""
+        cfg = self.config
+        stats: dict[str, float] = {}
+        groups = (
+            ("a2b", self.edges_a2b, cfg.b_land),
+            ("c2b", self.edges_c2b, cfg.b_land),
+            ("b2a", self.edges_b2a, cfg.a_land),
+            ("b2c", self.edges_b2c, cfg.c_land),
+        )
+        for name, edges, layers in groups:
+            for i, edge in enumerate(edges):
+                g = edge.g.detach().float()
+                stats[f"gate_{name}/L{layers[i]}_absmean"] = float(g.abs().mean())
+                stats[f"gate_{name}/L{layers[i]}_absmax"] = float(g.abs().max())
+                stats[f"gate_{name}/L{layers[i]}_rms"] = float(g.pow(2).mean().sqrt())
+        return stats
+
+    # ---- parameter accounting ---------------------------------------------- #
+
+    def param_breakdown(self) -> dict[str, int]:
+        def n(module: nn.Module) -> int:
+            return sum(p.numel() for p in module.parameters())
+
+        bridges = sum(n(e) for e in (self.edges_a2b, self.edges_c2b, self.edges_b2a, self.edges_b2c))
+        return {
+            "shared_embedding": n(self.wte),
+            "in_proj_384_to_256": n(self.in_proj_a),
+            "in_proj_384_to_320": n(self.in_proj_b),
+            "tower_a_layers": n(self.blocks_a),
+            "tower_b_layers": n(self.blocks_b),
+            "tower_c_layers": n(self.blocks_c),
+            "bridges": bridges,
+            "ln_f_a": n(self.ln_f_a),
+            "ln_f_c": n(self.ln_f_c),
+            "lm_head_a": n(self.lm_head_a),
+            "lm_head_c": n(self.lm_head_c),
+            "mix_head": n(self.mix_head),
+            "total": sum(p.numel() for p in self.parameters()),
+        }
+
+    def matmul_parameters(self) -> int:
+        """Total excluding norms, gates and the mixer.
+
+        This is the number the architecture is specified against: everything
+        that is a weight matrix, and nothing that is a per-channel scalar.
+        """
+        skip = set()
+        for module in self.modules():
+            if isinstance(module, RMSNorm):
+                skip.add(id(module.weight))
+        for edges in (self.edges_a2b, self.edges_c2b, self.edges_b2a, self.edges_b2c):
+            for edge in edges:
+                skip.add(id(edge.g))
+        for p in self.mix_head.parameters():
+            skip.add(id(p))
+        return sum(p.numel() for p in self.parameters() if id(p) not in skip)
+
+
+# --------------------------------------------------------------------------- #
+# loss
+# --------------------------------------------------------------------------- #
+
+
+def _loss_chunk(
+    h_a: torch.Tensor,
+    h_c: torch.Tensor,
+    mix_logit: torch.Tensor,
+    targets: torch.Tensor,
+    w_a: torch.Tensor,
+    w_c: torch.Tensor,
+) -> tuple[torch.Tensor, torch.Tensor, torch.Tensor, torch.Tensor]:
+    """One chunk of positions.  Returns summed NLLs and summed squared logits."""
+    logits_a = F.linear(h_a, w_a).float()
+    logits_c = F.linear(h_c, w_c).float()
+
+    log_p_a = F.log_softmax(logits_a, dim=-1)
+    log_p_c = F.log_softmax(logits_c, dim=-1)
+
+    # Mixing happens in probability space, computed in log space:
+    #   log(a*p_A + (1-a)*p_C) = logaddexp(log a + log p_A, log(1-a) + log p_C)
+    log_alpha = F.logsigmoid(mix_logit).unsqueeze(-1)
+    log_one_minus_alpha = F.logsigmoid(-mix_logit).unsqueeze(-1)
+    log_p_mix = torch.logaddexp(log_alpha + log_p_a, log_one_minus_alpha + log_p_c)
+
+    idx = targets.unsqueeze(-1)
+    nll_mix = -log_p_mix.gather(-1, idx).squeeze(-1).sum()
+    nll_a = -log_p_a.gather(-1, idx).squeeze(-1).sum()
+    nll_c = -log_p_c.gather(-1, idx).squeeze(-1).sum()
+    sq_logits = logits_a.pow(2).sum() + logits_c.pow(2).sum()
+    return nll_mix, nll_a, nll_c, sq_logits
+
+
+def relay_loss(
+    model: BananaMindRelay,
+    h_a: torch.Tensor,
+    h_c: torch.Tensor,
+    mix_logit: torch.Tensor,
+    targets: torch.Tensor,
+    solo_lambda: float,
+    z_loss_coeff: float = 0.0,
+    chunk_size: int = 16384,
+) -> dict[str, torch.Tensor]:
+    """L = L_mix + lambda * (L_A + L_C).
+
+    There is no L_B term.  B has no head to compute one with, and giving it one
+    would turn the relay into a third predictor instead of a channel.  B trains
+    entirely on gradient arriving through its four bridge directions.
+
+    Positions are processed in chunks under gradient checkpointing: three
+    vocab-sized float tensors per position do not fit at 4096 context otherwise.
+    """
+    w_a = model.lm_head_a.weight
+    w_c = model.lm_head_c.weight
+    vocab = w_a.size(0)
+
+    flat_a = h_a.reshape(-1, h_a.size(-1))
+    flat_c = h_c.reshape(-1, h_c.size(-1))
+    flat_z = mix_logit.reshape(-1)
+    flat_y = targets.reshape(-1)
+    n_pos = flat_y.numel()
+
+    if chunk_size <= 0 or chunk_size >= n_pos:
+        bounds = [(0, n_pos)]
+    else:
+        bounds = [(i, min(i + chunk_size, n_pos)) for i in range(0, n_pos, chunk_size)]
+
+    mix_sum = flat_a.new_zeros((), dtype=torch.float32)
+    a_sum = flat_a.new_zeros((), dtype=torch.float32)
+    c_sum = flat_a.new_zeros((), dtype=torch.float32)
+    sq_sum = flat_a.new_zeros((), dtype=torch.float32)
+
+    use_ckpt = torch.is_grad_enabled() and len(bounds) > 1
+    for lo, hi in bounds:
+        args = (flat_a[lo:hi], flat_c[lo:hi], flat_z[lo:hi], flat_y[lo:hi], w_a, w_c)
+        if use_ckpt:
+            part = torch.utils.checkpoint.checkpoint(_loss_chunk, *args, use_reentrant=False)
+        else:
+            part = _loss_chunk(*args)
+        mix_sum = mix_sum + part[0]
+        a_sum = a_sum + part[1]
+        c_sum = c_sum + part[2]
+        sq_sum = sq_sum + part[3]
+
+    l_mix = mix_sum / n_pos
+    l_a = a_sum / n_pos
+    l_c = c_sum / n_pos
+    z_loss = sq_sum / (2.0 * n_pos * vocab)
+
+    loss = l_mix + solo_lambda * (l_a + l_c) + z_loss_coeff * z_loss
+    return {"loss": loss, "l_mix": l_mix, "l_a": l_a, "l_c": l_c, "z_loss": z_loss}
+
+
+def count_parameters(model: nn.Module) -> int:
+    return sum(p.numel() for p in model.parameters())
+
+
+def estimate_training_flops_per_token(config: RelayConfig, param_count: int, seq_len: int) -> int:
+    dense_flops = 6 * param_count
+    attention_flops = 0
+    for spec in (config.tower_a, config.tower_b, config.tower_c):
+        attention_flops += (
+            12 * spec.num_hidden_layers * spec.num_attention_heads * config.head_dim * seq_len
+        )
+    return dense_flops + attention_flops

special_tokens_map.json

@@ -0,0 +1,6 @@
+{
+  "pad_token": "<|pad|>",
+  "bos_token": "<|bos|>",
+  "eos_token": "<|eos|>",
+  "unk_token": "<|unk|>"
+}

token_types.py

@@ -0,0 +1,108 @@
+#!/usr/bin/env python3
+"""Bucket the 8192-token vocab into coarse types.
+
+Used to log the mixing scalar's distribution over token types: knowing that
+alpha averages 0.5 overall says nothing, knowing that it sits at 0.8 on digits
+and 0.4 on word continuations says a lot about what each tower is doing.
+
+The Mini tokenizer is byte-level BPE (with digits split out by a pre-tokenizer),
+so token strings are byte-level encoded and have to be mapped back through the
+GPT-2 byte<->unicode table before they can be classified.
+"""
+from __future__ import annotations
+
+import json
+from pathlib import Path
+
+import numpy as np
+
+
+TYPE_NAMES = (
+    "special",
+    "whitespace",
+    "digit",
+    "word_start",
+    "word_cont",
+    "punct",
+    "other",
+)
+TYPE_INDEX = {name: i for i, name in enumerate(TYPE_NAMES)}
+
+
+def _byte_decoder() -> dict[str, int]:
+    bs = (
+        list(range(ord("!"), ord("~") + 1))
+        + list(range(ord("¡"), ord("¬") + 1))
+        + list(range(ord("®"), ord("ÿ") + 1))
+    )
+    cs = bs[:]
+    n = 0
+    for b in range(256):
+        if b not in bs:
+            bs.append(b)
+            cs.append(256 + n)
+            n += 1
+    return {chr(c): b for b, c in zip(bs, cs)}
+
+
+def _decode(token: str, byte_decoder: dict[str, int]) -> str:
+    try:
+        raw = bytes(byte_decoder[ch] for ch in token)
+    except KeyError:
+        return token
+    return raw.decode("utf-8", errors="replace")
+
+
+def classify(text: str) -> int:
+    if text == "":
+        return TYPE_INDEX["other"]
+    core = text.lstrip(" \t")
+    leading_space = core != text
+    if core.strip() == "":
+        return TYPE_INDEX["whitespace"]
+    if any(ch.isdigit() for ch in core):
+        return TYPE_INDEX["digit"]
+    letters = core.replace("'", "").replace("’", "")
+    if letters and all(ch.isalpha() for ch in letters):
+        return TYPE_INDEX["word_start"] if leading_space else TYPE_INDEX["word_cont"]
+    if all(not ch.isalnum() for ch in core):
+        return TYPE_INDEX["punct"]
+    return TYPE_INDEX["other"]
+
+
+def build_token_type_table(tokenizer_path: Path, vocab_size: int) -> np.ndarray:
+    """Returns an int64 array of length vocab_size mapping token id -> type index."""
+    path = Path(tokenizer_path)
+    if path.is_dir():
+        path = path / "tokenizer.json"
+    with path.open("r", encoding="utf-8") as f:
+        data = json.load(f)
+
+    vocab: dict[str, int] = data["model"]["vocab"]
+    specials = {entry["content"] for entry in data.get("added_tokens", [])}
+    byte_decoder = _byte_decoder()
+
+    table = np.full(vocab_size, TYPE_INDEX["other"], dtype=np.int64)
+    for token, idx in vocab.items():
+        if idx >= vocab_size:
+            continue
+        if token in specials:
+            table[idx] = TYPE_INDEX["special"]
+            continue
+        table[idx] = classify(_decode(token, byte_decoder))
+    return table
+
+
+if __name__ == "__main__":
+    import sys
+    from collections import Counter
+
+    tok = Path(
+        sys.argv[1]
+        if len(sys.argv) > 1
+        else "/home/banaxi/Desktop/BananaMind/BananaMind-2/tokenizers/fineweb_edu_first_50gib_8k_digits/tokenizer.json"
+    )
+    table = build_token_type_table(tok, 8192)
+    counts = Counter(table.tolist())
+    for i, name in enumerate(TYPE_NAMES):
+        print(f"{name:12s} {counts.get(i, 0):5d}  {counts.get(i, 0) / len(table):6.2%}")

tokenizer_config.json

@@ -0,0 +1,8 @@
+{
+  "tokenizer_class": "PreTrainedTokenizerFast",
+  "model_max_length": 4096,
+  "pad_token": "<|pad|>",
+  "bos_token": "<|bos|>",
+  "eos_token": "<|eos|>",
+  "unk_token": "<|unk|>"
+}