🐍 SOVYTHOS — APEP
The Chaos-Serpent Foundation Model of the Sovythos AI Project
🚧 This repository is a placeholder — training is currently in progress
This repo has been created ahead of time to reserve the model name and document what's coming. No weights, tokenizer, or config files are uploaded yet. This README exists purely to explain the situation to anyone who finds this repo early.
Nothing here is runnable yet.
AutoModelForCausalLM.from_pretrained(...)will fail against this repo until the files below are pushed. Watch/follow the repo, or check back later, for the real release.
🐍 What is APEP?
SOVYTHOS — APEP is a ~550M-parameter, Llama-compatible decoder-only Transformer, trained entirely from scratch (random init — no Llama/GPT/Qwen weights anywhere in its lineage) on a single-stage mixture of:
- Egyptian Arabic + Modern Standard Arabic (authentic dialect sources, not machine-translated)
- English, including high-signal educational web text (FineWeb-Edu, Cosmopedia-v2)
- Mathematics (raw math text + step-by-step solved problems)
- Code (quality-filtered Python + instruction-formatted tasks)
- Instruction-formatted examples mixed in from the very first training step (not a separate SFT stage)
It's the current flagship of the Sovythos AI Project, trained end-to-end — data pipeline, custom tokenizer, architecture, and training loop — on a single consumer-class GPU.
Named after Apep (Apophis), the serpent of chaos in Egyptian mythology who rises nightly against the sun and is beaten back only through relentless, disciplined effort — a fitting name for a model built to stay coherent despite limited compute and a demanding multilingual mixture.
📋 What's happening right now
| Step | Status |
|---|---|
| Architecture finalized (GQA + RoPE + SwiGLU + QK-Norm, Llama-compatible) | ✅ Done |
| Custom tokenizer trained (48K vocab, byte-level BPE) | ✅ Done |
| Dataset assembled (raw + instruct, single interleaved stream) | ✅ Done |
| Model training | 🚧 In progress |
| Checkpoints uploaded to this repo | ⬜ Not yet |
config.json / tokenizer.json / model.safetensors uploaded |
⬜ Not yet |
modeling_sovythos.py / configuration_sovythos.py (trust_remote_code) uploaded |
⬜ Not yet |
| GGUF export (F16/F32) uploaded | ⬜ Not yet |
| Full technical README (architecture, training recipe, benchmarks, usage) | ⬜ Replaces this file once training completes |
📦 What will be uploaded once training finishes
model.safetensors— the trained weightsconfig.json— full architecture config (dim, layers, heads, vocab, etc.)tokenizer.json— the custom-trained tokenizermodeling_sovythos.py+configuration_sovythos.py— so the model loads directly viaAutoModelForCausalLM.from_pretrained(..., trust_remote_code=True)with no external code neededsovythos-apep-550m.f16.gguf— forllama.cpp/llama-cpp-pythoninferencetrain_log.csv— the full training curve (train/val loss over steps), for transparency- A complete, detailed README replacing this placeholder — specs, architecture breakdown, tokenizer design, data sources, training recipe, usage examples, GGUF instructions, limitations, and roadmap
⏳ In the meantime
- This repo will not be usable for inference until the files above land
- Model name and repo ID (
my0919175/Sovythos-APEP-550M) are final and won't change - Training progress is tracked internally via
train_log.csvand will be shared once uploaded - Follow this repository to be notified when the first real checkpoint goes up
👨💻 Creator
Mahmoud Yasser — Founder, Sovythos AI Project
📄 License
Apache License 2.0 (applies once weights are released)
🐍 The serpent is still coiling. Check back soon.