---
license: apache-2.0
library_name: transformers
pipeline_tag: text-generation
tags:
  - causal-lm
  - moe
  - sequence-level-routing
  - custom-code
  - research
---

# BananaMind-2-SLMoE Effective 8M / 25M Total

![BananaMind-2-SLMoE](banner.png)

**E8M means 8 million effective parameters per message.**

BananaMind-2-SLMoE is an experimental sequence-level mixture-of-experts
language model. The checkpoint theoretically contains **25.45M total
parameters**, but one fixed set of 13 experts is selected for an entire
message, leaving **7.90M parameters active per message**.

This is not a production model. It is a research checkpoint built to test
whether sequence-level MoE routing works at this scale at all. A V2 is planned
with routing, expert-balance, and implementation fixes.

## Model summary

| Property | Value |
|---|---:|
| Effective size | **E8M** |
| Active parameters per message | 7,902,208 |
| Theoretical total parameters | 25,449,472 |
| Layers | 8 |
| Hidden size | 256 |
| Attention heads / KV heads | 8 / 2 |
| Experts | 64 |
| Active experts per message | 13 |
| Expert intermediate size | 56 |
| Router prefix | First 32 valid tokens |
| Context length | 4,096 |
| Vocabulary | 8,192 |
| Expert MLP | SwiGLU |
| Position encoding | RoPE, theta 100,000 |

The router reads a causal prefix and selects one top-13 route. That same route
is then reused for every token in the message and generated response. This is
different from token-level MoE models, which may select a different route for
each token.

## Why sequence-level MoE?

This model is a small-scale test of how this idea could work; it is research,
not a production-ready model. The longer-term idea is that sequence-level
routing could make very large sparse models usable on smaller machines. For
example, a hypothetical **744B-total, E30B** model
could keep inactive experts on disk and load only its selected 30B active set
into RAM for a message. With sufficient quantization, and provided the shared
weights and KV cache also fit, that could potentially bring such a model within
reach of a consumer PC.

The same basic offloading idea works with normal token-level MoE. The problem
is that its selected experts can change at every token, so experts not already
in RAM may need to be loaded from disk repeatedly during generation. Disk I/O
would make that extremely slow. Sequence-level MoE chooses one expert set for
the message, loads it from disk once, and reuses it for the entire response.

## Expert utilization

The final checkpoint does **not** show full expert collapse, but expert usage
is not fully balanced.

A 48-prompt routing probe found:

- 38 of 64 experts selected at least once
- 23.43 effective experts across the probe
- 47 unique top-13 routes across 48 prompts
- normalized routing entropy of 0.758
- five experts present in every tested route

The checkpoint therefore has meaningful route diversity, while still showing
a persistent group of dominant experts. It should not be described as having
perfect expert specialization.

## Benchmarks

![BananaMind-2-SLMoE benchmark comparison](benchmarks.png)

| Model | Parameters | ARC Easy | HellaSwag | PIQA | ARC Challenge | ArithMark 3 | ArithMark 2 |
|---|---:|---:|---:|---:|---:|---:|---:|
| **BananaMind-2-SLMoE E8M** | **7.90M active / 25.45M total** | **33.92** | **27.16** | **55.44** | **23.12** | **33.60** | **26.32** |
| [BananaMind-2-Nano](https://huggingface.co/BananaMind/BananaMind-2-Nano) | 9.97M | 36.20 | 27.50 | 55.98 | 23.38 | 33.70 | 27.68 |
| [BananaMind-2-MoE](https://huggingface.co/BananaMind/BananaMind-2-MoE) | 25.1M total | 34.64 | 27.45 | 56.37 | 21.16 | 33.80 | 28.44 |

ARC Easy, HellaSwag, PIQA, ARC Challenge, and ArithMark 3 use normalized
continuation accuracy. ArithMark 2 uses raw continuation accuracy.


We're not claiming SLMoE beats token-level MoE here. This run used ~2× the training tokens and ~4× the active compute (top-13 vs top-1) compared to BananaMind-2-MoE, so the comparison isn't controlled. The point of this checkpoint is to show that sequence-level routing trains stably at this scale — not that it's the better architecture.

### BananaMind Base Bench 1.1

| Metric | Result |
|---|---:|
| Overall Elo | **887** |
| Accuracy | **36.29% (127/350)** |
| Weighted accuracy | **33.90%** |
| Language completion | 64.00% |
| Commonsense | 32.00% |
| World knowledge | 48.00% |
| Context tracking | 34.00% |
| Quantitative | 24.00% |
| Logical reasoning | 36.00% |
| Code completion | 16.00% |

These are research results, not guarantees of downstream quality. Scores can
vary with evaluator version, precision, and batching configuration.

## Usage

This repository uses custom Transformers architecture code, so
`trust_remote_code=True` is required.

### Standard RAM mode

RAM mode loads all 64 experts as regular model parameters. Computation remains
sparse: only the selected 13 experts are evaluated for each message.

```python
import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

model_id = "BananaMind/BananaMind-2-SLMoE"
device = "cuda" if torch.cuda.is_available() else "cpu"
dtype = torch.bfloat16 if device == "cuda" else torch.float32

tokenizer = AutoTokenizer.from_pretrained(model_id, trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained(
    model_id,
    trust_remote_code=True,
    dtype=dtype,
    expert_storage="ram",
).to(device)

inputs = tokenizer("The color of the sky is", return_tensors="pt").to(device)

with torch.no_grad():
    output = model.generate(
        **inputs,
        max_new_tokens=96,
        do_sample=True,
        temperature=0.7,
        top_p=0.9,
        repetition_penalty=1.1,
        pad_token_id=tokenizer.eos_token_id,
        eos_token_id=tokenizer.eos_token_id,
        use_cache=True,
        only_use_active_experts=False,
    )

print(tokenizer.decode(output[0], skip_special_tokens=True))
```

### Disk-backed active-expert mode

Disk mode does not register the complete expert bank as in-memory model
parameters. The router runs first, only the selected expert slices are read
from `model.safetensors`, and those slices are cached for the response.

```python
model = AutoModelForCausalLM.from_pretrained(
    model_id,
    trust_remote_code=True,
    dtype=dtype,
    expert_storage="disk",
).to(device)

with torch.no_grad():
    output = model.generate(
        **inputs,
        max_new_tokens=96,
        do_sample=True,
        temperature=0.7,
        top_p=0.9,
        repetition_penalty=1.1,
        pad_token_id=tokenizer.eos_token_id,
        eos_token_id=tokenizer.eos_token_id,
        use_cache=True,
        only_use_active_experts=True,
    )
```

In disk mode, the full 64-expert bank stays in SafeTensors storage. The active
expert slices temporarily occupy system memory and the target device while the
request runs. The operating system may retain recently accessed file pages in
its disk cache. Disk mode is inference-only and is most efficient with
`use_cache=True`.

For a batch containing multiple messages, each message receives its own top-13
route. The total number of distinct experts materialized across the batch can
therefore exceed 13.

## Training

The checkpoint was pretrained for approximately 60B tokens with AdamW using a
curriculum containing FineWeb-HQ, FineWeb-Edu, DCLM, Cosmopedia v2, FineMath,
and NPSet2 data.

## Limitations

- This is an experimental base model, not an instruction-tuned assistant.
- It is not intended for production or high-stakes use.
- Expert usage remains concentrated even though full collapse was not observed.
- A single route is fixed for the full response and cannot adapt token by token.
- Disk-backed inference trades memory residency for per-request I/O latency.
- Generated text may be incorrect, repetitive, biased, or unsafe.

## License

Apache-2.0.