SCLM v2 - Stateful Coherent Language Model with EARCP

README.md

@@ -0,0 +1,118 @@
+---
+license: mit
+language:
+- en
+library_name: transformers
+tags:
+- sclm
+- stateful
+- memory
+- earcp
+- text-generation
+- conversational
+pipeline_tag: text-generation
+base_model: mistralai/Mistral-7B-v0.1
+widget:
+- text: "The wizard Elara lived in Silverwood forest. One day, she discovered"
+  example_title: "Fantasy Story"
+- text: "In the year 2050, humanity had finally achieved"
+  example_title: "Science Fiction"
+- text: "The detective examined the crime scene carefully. The clues pointed to"
+  example_title: "Mystery"
+inference:
+  parameters:
+    max_new_tokens: 100
+    temperature: 0.7
+    top_p: 0.9
+    repetition_penalty: 1.1
+---
+
+# 🧠 SCLM: Stateful Coherent Language Model
+
+**SCLM** adds **persistent latent memory** to transformer language models, enabling better coherence across long conversations and multi-turn generation.
+
+## 🎯 Key Features
+
+- **Persistent State**: Memory that evolves across conversation turns
+- **Entity Coherence**: Maintains context about characters, places, and objects
+- **Edit Mode**: Make local changes without affecting global memory
+- **Lightweight**: Only 91.7M additional parameters (2.44% overhead)
+
+## 📊 Architecture: EARCP
+
+```
+EARCP = Encapsulation + Alignment + Revision + Coherence + Propagation
+```
+
+| Component | Function |
+|-----------|----------|
+| **Encapsulation** | GRU-style state update from hidden states |
+| **Alignment** | Cross-attention between state and hidden layers |
+| **Revision** | Drift detection and correction |
+| **Coherence** | Mixture-of-Experts for consistency |
+| **Propagation** | State injection into transformer layers |
+
+## 🔧 Model Details
+
+| Parameter | Value |
+|-----------|-------|
+| Base Model | mistralai/Mistral-7B-v0.1 |
+| EARCP Parameters | 91.7M |
+| Latent State Dim | 256 |
+| Injection Layers | [8, 16] |
+| Alpha (injection strength) | 0.02 |
+| Experts | 2 |
+
+## 🚀 Quick Start
+
+```python
+# Note: Full SCLM requires custom loading (see below)
+# The inference widget uses the base model only
+
+from transformers import AutoTokenizer
+import torch
+
+# Load tokenizer
+tokenizer = AutoTokenizer.from_pretrained("amewebstudio/ananke-sclm")
+
+# For full SCLM functionality, load weights separately:
+# 1. Load base Mistral-7B
+# 2. Load EARCP weights from earcp_weights.pt
+# 3. Apply SCLM wrapper
+```
+
+## 📈 Validation Results
+
+| Test | Result |
+|------|--------|
+| Forward Pass | ✅ |
+| State Evolution | ✅ (norm: 0 → 4.6 → 7.5) |
+| Coherent Generation | ✅ |
+| Edit Mode | ✅ |
+| Entity Memory | ✅ (Elara, Nimbus retained) |
+
+## 💡 Use Cases
+
+- **Interactive Fiction**: Characters and plot points remain consistent
+- **Long Conversations**: Context persists without growing prompts
+- **Creative Writing**: Maintain story coherence across chapters
+- **Role-Playing**: NPCs remember past interactions
+
+## 📝 Citation
+
+```bibtex
+@article{amega2025sclm,
+  title={SCLM: Stateful Coherent Language Models with EARCP Architecture},
+  author={Amega, Mike},
+  year={2025},
+  note={Ame Web Studio}
+}
+```
+
+## 👤 Author
+
+**Mike Amega** - [Ame Web Studio](https://github.com/Volgat)
+
+---
+
+*SCLM is an experimental architecture exploring persistent memory in language models.*

earcp_weights.pt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:76d6e5369d11288e4329e75b59b83ada490fadc367fdf6c9dcb0ecf170e37ec8
+size 366837710

sclm_config.json

@@ -0,0 +1,14 @@
+{
+  "vocab_size": 32000,
+  "hidden_size": 4096,
+  "num_hidden_layers": 32,
+  "latent_state_dim": 256,
+  "n_experts": 2,
+  "n_coherence_heads": 4,
+  "expert_intermediate": 1024,
+  "state_injection_layers": [
+    8,
+    16
+  ],
+  "alpha_inject": 0.02
+}

special_tokens_map.json

@@ -0,0 +1,24 @@
+{
+  "bos_token": {
+    "content": "<s>",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  },
+  "eos_token": {
+    "content": "</s>",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  },
+  "pad_token": "</s>",
+  "unk_token": {
+    "content": "<unk>",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  }
+}

tokenizer.model

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:dadfd56d766715c61d2ef780a525ab43b8e6da4de6865bda3d95fdef5e134055
+size 493443

tokenizer_config.json

@@ -0,0 +1,44 @@
+{
+  "add_bos_token": true,
+  "add_eos_token": false,
+  "add_prefix_space": null,
+  "added_tokens_decoder": {
+    "0": {
+      "content": "<unk>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "1": {
+      "content": "<s>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "2": {
+      "content": "</s>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    }
+  },
+  "additional_special_tokens": [],
+  "bos_token": "<s>",
+  "clean_up_tokenization_spaces": false,
+  "eos_token": "</s>",
+  "extra_special_tokens": {},
+  "legacy": false,
+  "model_max_length": 1000000000000000019884624838656,
+  "pad_token": "</s>",
+  "sp_model_kwargs": {},
+  "spaces_between_special_tokens": false,
+  "tokenizer_class": "LlamaTokenizer",
+  "unk_token": "<unk>",
+  "use_default_system_prompt": false
+}

validation_results.json

@@ -0,0 +1,6 @@
+{
+  "forward": true,
+  "evolution": true,
+  "generation": true,
+  "edit_mode": true
+}