---
language:
- en
- zh
license: apache-2.0
library_name: transformers
pipeline_tag: text-generation
tags:
- qwen2.5
- experiential-plasticity
- forged
- head-pruning
- neural-plasticity
- sentinel-ai
- continuum
- safetensors
- general-purpose
- local-inference
- efficient
- optimized
- pruned
- 0.5b
base_model:
- Qwen/Qwen2.5-0.5B
datasets:
- Salesforce/wikitext
---

> **Methodology note.** This artifact predates the [§4.1.3.4 calibration-aware importance metric](https://github.com/CambrianTech/continuum/blob/main/docs/papers/PLASTICITY-COMPACTION.md) (April 2026). The current best forge for the Qwen3-Coder family is [`continuum-ai/qwen3-coder-30b-a3b-compacted-19b-256k`](https://huggingface.co/continuum-ai/qwen3-coder-30b-a3b-compacted-19b-256k), which lands within −3.7 HumanEval points of the unmodified base via calibration-aware MoE expert pruning. This older artifact is preserved as part of the methodology arc and remains usable for its original tier; for new deployments, prefer artifacts forged under the §4.1.3.4 metric.

# qwen2.5-0.5b-general-forged

**-3.2% better than baseline.** Forged from [Qwen/Qwen2.5-0.5B](https://huggingface.co/Qwen/Qwen2.5-0.5B) for **general** tasks.

**Not quantized. Not distilled. Structurally reshaped.**

The architecture co-evolves with training: heads that contribute to the domain specialize, heads that don't are removed. The result is a model architecturally optimized for its task — like biological synaptic pruning during brain development.

## Results

| Metric | Value |
|--------|-------|
| Base Model | [Qwen/Qwen2.5-0.5B](https://huggingface.co/Qwen/Qwen2.5-0.5B) |
| Baseline Perplexity | 2.83 |
| **Forged Perplexity** | **2.92** |
| **Improvement** | **-3.2%** |
| Domain | general |
| Training Data | wikitext-2 |
| Strategy | combined |
| Pruning Level | 30% |
| Cycles | 3 |
| Steps/Cycle | 1000 |

## Runs On

| Device | Format | Verified |
|--------|--------|----------|
| Phone / Raspberry Pi | Q4_K_M | Expected |
| MacBook Air 8GB | Q4_K_M | Expected |

These models are designed for **consumer hardware**. No A100s required. Your MacBook, your gaming PC, your home server.

## Quick Start

```python
from transformers import AutoModelForCausalLM, AutoTokenizer

model = AutoModelForCausalLM.from_pretrained("continuum-ai/qwen2.5-0.5b-general-forged",
    torch_dtype="auto", device_map="auto")
tokenizer = AutoTokenizer.from_pretrained("continuum-ai/qwen2.5-0.5b-general-forged")

inputs = tokenizer("Write a Python decorator that caches results:", return_tensors="pt").to(model.device)
output = model.generate(**inputs, max_new_tokens=200, do_sample=True, temperature=0.7)
print(tokenizer.decode(output[0], skip_special_tokens=True))
```

## Forge Your Own

Three commands. Any NVIDIA GPU with 8GB+ VRAM.

```bash
git clone https://github.com/CambrianTech/sentinel-ai && cd sentinel-ai && ./setup.sh
source .venv/bin/activate
python scripts/forge_model.py Qwen/Qwen2.5-0.5B --domain general
```

The forge script auto-detects your GPU, picks the right memory tier (fp16 / 4-bit NF4), trains with LoRA + AMP, prunes attention heads, defrags, and saves. Progress observable via `status.json`.

## The Science: Experiential Plasticity

Traditional model compression (quantization, distillation) makes models **smaller but worse**. Experiential Plasticity makes them **smaller AND better**.

### How It Works

1. **Train** on domain-specific data (LoRA + AMP mixed precision)
2. **Measure** each attention head's information contribution (entropy-based importance)
3. **Prune** the lowest-contributing heads
4. **Retrain** on the same domain data — surviving heads specialize and compensate
5. **Defrag** — structurally remove dead heads, free VRAM
6. **Repeat** — each cycle the model improves on its domain

### Scaling Law

Larger models harbor more architectural redundancy. Plasticity exploits this — bigger models benefit more:

| Model | Params | Domain | Improvement |
|-------|--------|--------|------------|
| Qwen2.5-0.5B | 0.5B | General | -3.2% (too small to prune) |
| Qwen2.5-1.5B | 1.5B | General | +3.0% |
| Qwen2.5-7B | 7.6B | General | +11.8% |
| **Qwen3.5-4B** | **3.4B** | **Code** | **+24.0%** |
| **Qwen3.5-27B** | **23.6B** | **Code** | **+3.5%** (4-bit, runs in 17GB) |

Domain-specific training amplifies the effect. Qwen3.5-4B on code (+24%) exceeds Qwen2.5-7B on generic text (+11.8%) despite being a smaller model.

### Transfer Function

Recovery from iterative pruning follows a measurable exponential decay:

```
recovery = 1.45 * exp(-0.18 * cycle) - 0.03
```

This connects transformer optimization to classical control theory — the same mathematics used in electrical engineering and robotics for decades. A PID controller can manage the entire forging process with zero human hyperparameters.

### Continuous Defrag

Traditional pruning masks heads but doesn't free memory. Continuous defrag structurally removes dead heads between cycles:

```
Cycle 1: train (batch=1, 27B, 17.9GB) -> prune -> defrag -> freed 1.7GB
Cycle 2: train (batch=2, 24.5B, 16.2GB) -> prune -> defrag -> freed 1.7GB  (2x faster)
Cycle 3: train (batch=3, 22B, 14.5GB)  -> prune -> defrag                  (2.8x faster)
```

40% faster total training and a 33% smaller final model.

### Head Mitosis

Pruning frees slots. Mitosis fills them. When a head is overutilized, it gets cloned into a pruned slot — each copy at 50% gate value to maintain output continuity. After continued training, the clones **diverge and specialize**, like cell differentiation after biological mitosis. The model grows new specialized capacity exactly where it's needed.

**Read the full paper**: [Experiential Plasticity: Transformers That Grow Their Own Architecture From Experience](https://github.com/CambrianTech/continuum/blob/main/docs/papers/EXPERIENTIAL-PLASTICITY.md)

## Output Samples

Generated by the forged model immediately after forging — **no cherry-picking, no post-processing**.

*No generation samples available for this model.*

## Forging Metadata

```json
{
  "model": "Qwen/Qwen2.5-0.5B",
  "strategy": "combined",
  "pruning_level": 0.3,
  "cycles": 3,
  "training_steps": 1000,
  "baseline_ppl": 2.83,
  "final_ppl": 2.92,
  "forged_at": "2026-03-27T08:22:03-05:00",
  "device": "cuda",
  "domain": "general",
  "training_data": "wikitext-2",
  "hardware_targets": [
    {
      "device": "Phone / Raspberry Pi",
      "format": "Q4_K_M",
      "verified": false
    },
    {
      "device": "MacBook Air 8GB",
      "format": "Q4_K_M",
      "verified": false
    }
  ],
  "improvement_pct": -3.18
}
```

## Research

- **[Experiential Plasticity](https://github.com/CambrianTech/continuum/blob/main/docs/papers/EXPERIENTIAL-PLASTICITY.md)** — Scaling law, transfer function, self-directed controller, domain forging, continuous defrag
- **[Neural Plasticity in Transformers](https://github.com/CambrianTech/continuum/blob/main/docs/papers/SENTINEL-AI-NEURAL-PLASTICITY.md)** — Foundation paper with cross-architecture results
- **[Plasticity Compaction](https://github.com/CambrianTech/continuum/blob/main/docs/papers/PLASTICITY-COMPACTION-MOE.md)** — MoE expert pruning (67GB to 14GB)

## Links

- [All published models](https://huggingface.co/continuum-ai)
- [sentinel-ai](https://github.com/CambrianTech/sentinel-ai) — Open source forge framework
- [continuum](https://github.com/CambrianTech/continuum) — Distributed AI on consumer hardware

---

## More from continuum-ai

`continuum-ai` ships **structurally compacted models for hardware tiers nobody else targets**. Every artifact is calibration-aware, hardware-anchored, and shipped with [ForgeAlloy](https://github.com/CambrianTech/forge-alloy) cryptographic provenance — the per-problem benchmark JSONLs are uploaded with sha256 hashes recorded in the alloy so anyone can re-score against the same anchor without trusting the producer's claim.

### Currently shipped

| Model | Base | HumanEval (vs base) | Tier | What's new |
|---|---|---|---|---|
| [**qwen3-coder-30b-a3b-compacted-19b-256k**](https://huggingface.co/continuum-ai/qwen3-coder-30b-a3b-compacted-19b-256k) | Qwen3-Coder-30B-A3B-Instruct | **88.4** (base 92.1, Δ −3.7) | **12 GB Q4_K_M** | First 30B-class coder that fits a 12 GB consumer GPU. Calibration-aware MoE expert pruning (§4.1.3.4). 256K context. |
| [**qwen2.5-coder-7b-compacted**](https://huggingface.co/continuum-ai/qwen2.5-coder-7b-compacted) | Qwen2.5-Coder-7B | 61.0 (base 62.2, Δ −1.2) | 16 GB fp16 | Methodology validation artifact for §4.1.3.3 — compensation LoRA closes the dense-head pruning gap to within ±3pt of base. |
| [**olmoe-1b-7b-compacted-5b**](https://huggingface.co/continuum-ai/olmoe-1b-7b-compacted-5b) | OLMoE-1B-7B-0924-Instruct (Allen AI, fully open) | **36.0** (base 40.9, Δ −4.9) | **4 GB Q5_K_M / phone tier** | Cross-architecture validation of §4.1.3.4 — same forge scripts ported `Qwen3MoeForCausalLM` → `OlmoeForCausalLM` without modification. The +8.0 within-model swing between broad-corpus and code-corpus calibration is the second empirical anchor for the discipline gate. |

### Forge methodology in one paragraph

A prunable unit's importance MUST be derived from **task-conditioned activation profiling on a held-out corpus** that reflects the artifact's intended workload. Architectural-only metrics (router gate norms, weight norms, magnitudes) are first-pass shortcuts that systematically underperform task-specific activation metrics — empirically validated at two structurally distinct units (dense heads in §4.1.3.1, MoE experts in §4.1.3.4) with a +9.7 HumanEval swing on the same prune budget. **Get the metric right AND the calibration corpus right; the artifact follows.** Two discipline gates now derived from empirical failures, not asserted from first principles: **§4.1.4.1 anchor-reproduction gate** (the base anchor must reproduce within ±3pt on the publishing pipeline before any calibrated delta is reported), and **§4.1.3.4.1 calibration-corpus discipline gate** (the calibration corpus used for importance profiling must be hash-pinned in the alloy AND must be a representative sample of the eval workload distribution — wrong-corpus and wrong-metric saturate at the same ~13 HumanEval damage ceiling, demonstrated empirically across two architectures). Full methodology in [PLASTICITY-COMPACTION.md](https://github.com/CambrianTech/continuum/blob/main/docs/papers/PLASTICITY-COMPACTION.md).

### The empty-quadrant frontier

A live HuggingFace audit (April 2026) confirmed that **the entire structurally-pruned-MoE quadrant is empty for every frontier model except Llama 3.3 70B**. Quantization is everywhere; structural pruning is nowhere. The forge methodology validated on `qwen3-coder-30b-a3b` ports directly to every other MoE family. The forge queue below is the comprehensive map of empty quadrants we are claiming, one architecture at a time.

### Forge queue — comprehensive new-architecture coverage

| # | Target | Arch | License | Total/Active | Tier post-prune | Status |
|---|---|---|---|---|---|---|
| 1 | OLMoE-1B-7B (`OlmoeForCausalLM`) | `OlmoeForCausalLM` | Apache-2.0 | 7B/1.3B → 5B/1.0B | **Phone / 4 GB Q5** | ✅ **SHIPPED** as `olmoe-1b-7b-compacted-5b`. Second cross-arch validation of §4.1.3.4. |
| 2 | [ibm-granite/granite-3.1-3b-a800m-instruct](https://huggingface.co/ibm-granite/granite-3.1-3b-a800m-instruct) | `GraniteMoeForCausalLM` | Apache-2.0 | 3.3B/800M (40e/top-8) | Edge tier | **Downloading now.** IBM enterprise brand, ultra-rare tiny-MoE niche, zero pruned variants. |
| 3 | [deepseek-ai/DeepSeek-V2-Lite-Chat](https://huggingface.co/deepseek-ai/DeepSeek-V2-Lite-Chat) | `DeepseekV2ForCausalLM` | DeepSeek (commercial OK) | 15.7B/2.4B | Single GPU | **Downloading now.** The forgotten DeepSeek sibling — DeepSeek brand without 670 GB of VRAM. |
| 4 | [microsoft/Phi-3.5-MoE-instruct](https://huggingface.co/microsoft/Phi-3.5-MoE-instruct) | `PhiMoEForCausalLM` | **MIT** | 42B/6.6B (16e/top-2) | Single 5090 Q4 | Queued. MIT-licensed Microsoft MoE that nobody runs because 42B is the awkward middle tier — until you prune to 12 experts. |
| 5 | [mistralai/Mixtral-8x22B-Instruct-v0.1](https://huggingface.co/mistralai/Mixtral-8x22B-Instruct-v0.1) | `MixtralForCausalLM` | Apache-2.0 | 141B/39B (8e/top-2) | Single 5090 Q4 | Queued. Two-year overdue Pareto win — the textbook MoE that nobody has ever calibration-pruned. |
| 6 | [Qwen/Qwen3-235B-A22B-Instruct-2507](https://huggingface.co/Qwen/Qwen3-235B-A22B-Instruct-2507) | `Qwen3MoeForCausalLM` | Apache-2.0 | 235B/22B (128e/top-8) | Single 5090 Q4 | Queued. Same family as our shipped 30B-A3B → methodology ports trivially. |
| 7 | [Qwen/Qwen3-Coder-480B-A35B-Instruct](https://huggingface.co/Qwen/Qwen3-Coder-480B-A35B-Instruct) | `Qwen3MoeForCausalLM` | Apache-2.0 | 480B/35B (160e/top-8) | **Grid moonshot** (4×24GB) | Queued. First consumer-accessible 480B coder. |
| 8 | [deepseek-ai/DeepSeek-Coder-V2-Instruct](https://huggingface.co/deepseek-ai/DeepSeek-Coder-V2-Instruct) | `DeepseekV2ForCausalLM` | DeepSeek | 236B/21B | Grid | Queued. Direct methodology replay at higher tier. |
| 9 | [Snowflake/snowflake-arctic-instruct](https://huggingface.co/Snowflake/snowflake-arctic-instruct) | `ArcticForCausalLM` | Apache-2.0 | 480B/17B (128e/top-2) | Grid | Queued. The forgotten Apache frontier MoE — dense+sparse hybrid arch is a novel research contribution by itself. |
| 10 | [deepseek-ai/DeepSeek-R1](https://huggingface.co/deepseek-ai/DeepSeek-R1) | `DeepseekV3ForCausalLM` | **MIT** | 671B/37B | **Grid moonshot** | Queued. The viral king. First non-distill R1 compaction. |

**8 distinct architecture classes** covered across **5 hardware tiers** (edge → phone → single GPU → 5090 → grid). When the queue completes, the calibration-aware-importance metric has been validated on `Qwen3MoeForCausalLM`, `OlmoeForCausalLM`, `GraniteMoeForCausalLM`, `DeepseekV2ForCausalLM`, `PhiMoEForCausalLM`, `MixtralForCausalLM`, `ArcticForCausalLM`, and `DeepseekV3ForCausalLM` — the cross-family invariance claim becomes empirical, not theoretical.

### Hard prerequisites being built in parallel

- **LiveCodeBench v6 anchor extension** for `eval_with_calibration.py` — HumanEval is no longer reported on frontier model cards (Qwen3-Coder, DeepSeek-V3.1, Mixtral 8x22B all use SWE-bench / LiveCodeBench / Aider-Polyglot). Without LCB v6 wired up, frontier targets are blocked at the §4.1.4.1 calibration discipline gate. ~1-2 days of mechanical pipeline work.
- **Offline teacher-logit precomputation** for `compensation_lora.py` — at 30B+ class, transformers' `caching_allocator_warmup` pre-allocates an fp16 buffer equal to full model size before bnb 4-bit takes effect, exceeding total VRAM on a single 32 GB GPU. The architecturally correct fix is phase-1-load-teacher / phase-2-unload / phase-3-load-student-and-train-against-on-disk-logits. Prerequisite for compensation v2 of every artifact ≥30B.
- **Grid expert sharding** for the 480B+ moonshots — `cpu_expert_prune_v2.py`'s streaming pruner already handles shards bigger than any single GPU, but distributed inference + cross-machine activation profiling for the calibration-aware metric needs the grid layer. This is the §4.1.3.5 distributed forge methodology paper section.

### Sensory bridge stack (separate from the LLM forge queue)

For Continuum's own sensory architecture (vision/audio/embedding bridges), the right targets are not forge candidates — they're curated bridge components used as-is:

| Component | Model | Use |
|---|---|---|
| Vision encoder | [`google/siglip-so400m-patch14-384`](https://huggingface.co/google/siglip-so400m-patch14-384) | Image embeddings for the vision bridge |
| Vision describer | [`microsoft/Phi-3.5-vision-instruct`](https://huggingface.co/microsoft/Phi-3.5-vision-instruct) | Small VLM that generates text descriptions consumed by text-only LLMs |
| STT | [`openai/whisper-large-v3`](https://huggingface.co/openai/whisper-large-v3) | Speech transcription for audio bridge |
| Multilingual embedding | [`BAAI/bge-m3`](https://huggingface.co/BAAI/bge-m3) | Sensory cache embeddings |
| Avatar diffusion | [`black-forest-labs/FLUX.1-schnell`](https://huggingface.co/black-forest-labs/FLUX.1-schnell) | Apache-licensed avatar generation for Continuum universes |

### What we DON'T target

The Llama 3.3 70B slot is saturated (six publishers, every quant level). We're not shipping a third compacted MoE in the middle tier. The lab's brand pitch is **models that no individual hardware tier can run, made runnable by structural compaction + grid distribution** — empty-quadrant headlines, not catalog filler. That's the intersection only continuum has, and the forge queue above is the map.