---
license: apache-2.0
language:
- en
tags:
- symbiogenesis
- cross-species
- weight-projection
- pca
- vocabfusion
- causal-lm
base_model:
- EleutherAI/pythia-160m
datasets:
- wikitext
pipeline_tag: text-generation
---

# VocabFusion-Pythia160m-Fused

Full pretrained weight projection from **Pythia-160m** (768d, 12 layers) into
**VocabFusion-SymbioGPT-10M** (320d, 8 layers) with **blend alpha = 0.3**.

This is the first cross-species experiment where source and target share both
**vocabulary** (50K GPTNeoX) and **head dimension** (64), eliminating the two
largest information-loss channels from previous projection experiments.

## Architecture Mapping

| Component | Pythia-160m (Source) | VocabFusion (Target) | Method |
|-----------|---------------------|---------------------|--------|
| d_model | 768 | 320 | PCA (92.1% variance) |
| Layers | 12 | 8 | Proportional grouping |
| Heads | 12 MHA | 5 MHA | Top-5 by L2 norm |
| Head dim | 64 | 64 | **Exact match** |
| FFN | 3072 (GELU) | 832 (SwiGLU) | Neuron selection + 2→3 matrix map |
| Vocab | 50,304 | 50,304 (frozen Pythia) | **Exact match** |

## Projection Details

- **56 weights** projected (8 layers × 7 weights: wq, wk, wv, wo, w1, v, w2)
- **PCA avg variance**: 92.1% (range: 82.6% — 97.3%)
- **Blend formula**: `new_W = 0.7 * vocabfusion_W + 0.3 * projected_pythia_W`
- **Zero fine-tuning**: No training after projection
- **What's NOT projected**: Embeddings (shared), junction layer, organelle weights
  (CausalConv, Monarch, LongConv), OrganelleGate, norms, skip gates

## Key Advantages Over Gemma→SymbioGPT Experiment

| Aspect | Gemma → SymbioGPT | Pythia → VocabFusion |
|--------|-------------------|---------------------|
| Vocabulary | Mismatched (256K→2K) | **Shared** (50K=50K) |
| Head dim | 64=64 | 64=64 |
| Attention type | GQA→MHA (complex) | **MHA→MHA** (simple) |
| Transfer type | LoRA deltas only | Full pretrained weights |
| PCA ratio | 2:1 (640→320) | 2.4:1 (768→320) |

## Usage

```python
import torch
checkpoint = torch.load("vocabfusion_pythia160m_fused.pt", map_location="cpu")
```

Load into a VocabFusionModel instance from the
[symbiogenesis-experiments](https://github.com/MonumentalSystems/symbiogenesis-experiments)
repo (`vocab_fusion_experiment/model.py`).

## Links

- **VocabFusion original**: [LisaMegaWatts/VocabFusion-SymbioGPT-10M](https://huggingface.co/LisaMegaWatts/VocabFusion-SymbioGPT-10M)
- **Source Pythia**: [EleutherAI/pythia-160m](https://huggingface.co/EleutherAI/pythia-160m)
- **Gemma LoRA experiment**: [LisaMegaWatts/SymbioGPT-Gemma-Fused](https://huggingface.co/LisaMegaWatts/SymbioGPT-Gemma-Fused)
- **Framework**: [symbiogenesis](https://github.com/DavinciDreams/symbiogenesis)
- **Projection script**: `cross_species_lora/project_pythia_to_vocabfusion.py`