---
tags:
  - model-garage
  - decomposed
  - gpt2
  - interpretability
  - model-surgery
license: apache-2.0
library_name: model-garage
---

# GPT-2 Decomposed — Model Garage

Full component-level decomposition of GPT-2 (124M parameters) using [Model Garage](https://github.com/Lumi-node/model-garage).

## What's Here

64 individually extracted `nn.Module` components:

| Component Type | Count | Dimensions |
|---------------|-------|-----------|
| Attention (self_attn) | 12 | 768d, 12 heads, 64 head_dim |
| Feed-Forward (mlp) | 12 | 768→3072→768 |
| Layer Norm (ln_1, ln_2) | 24 | 768d |
| Full Layers | 12 | 768d |
| Embeddings | 2 | token (50257→768), position (1024→768) |
| Output Head | 1 | 768→50257 |
| Final Norm | 1 | 768d |

## Usage

```python
from model_garage.extract.pytorch import PyTorchExtractor

extractor = PyTorchExtractor("gpt2")
extractor.load_model()

# Extract any component
attn = extractor.extract_component("self_attention", layer_idx=6)
ffn = extractor.extract_component("feed_forward", layer_idx=6)

# Test in isolation
from model_garage.extract.pytorch import ComponentTester
tester = ComponentTester()
print(tester.test_attention(attn))
```

## Install

```bash
pip install model-garage
```

## Links

- [Model Garage GitHub](https://github.com/Lumi-node/model-garage)
- [PyPI](https://pypi.org/project/model-garage/)
- [Research Papers](https://github.com/Lumi-node/model-garage/tree/main/research/papers)