# Build Plan — RSI Content Generation Agent This is the authoritative, phased build plan. The [README](README.md) describes the system **as if finished**; this document is **how we get there without breaking anything**, in an order where each phase is independently testable. The guiding rule: > **Never refactor working code blind.** Copy it intact, wrap it behind a contract, and only > physically move shared modules once the wrapper is proven. Every phase ends at a state you > can run. --- ## Invariants (true at every phase — violating one is "a mistake") 1. **The harness grades; the variants are graded.** `harness/fitness.py`, the scoreboard, and the dispatcher's allocation math are never in the mutator's editable surface. 2. **The agent is blind to credentials.** CI reads a sanitized Mongo `fitness_scoreboard` via a role scoped to that collection only. It never gets `MONGO_URL`, never reads `users`. 3. **Merge is gated.** A mutation only reaches `main` through a PR that (a) has green LOCKED tests and (b) passes the secret-scan. Whether a human or the organism itself clicks merge is the `AUTONOMOUS_MERGE` toggle — but a red or secret-touching build never merges, either way. 4. **3-day leash.** Fitness only ever reads analytics for videos uploaded ≥ 3 days ago. 5. **Death ≠ low score.** A suspended/struck channel HALTs the loop; it is never fed into fitness as a number. 6. **Population ≤ `MAX_LIVING_VARIANTS` (4).** --- ## Build status | Phase | What | State | |-------|------|-------| | 0 | Repo skeleton + seed variant | ✅ done | | 1 | Variant contract (`genome.py`) | ✅ done | | 2 | Scoreboard (Mongo + CSV, sanitized) | ✅ done | | 3 | Fitness scorer + live YouTube fetch + `/fitness/refresh` | ✅ done (live path needs real keys to validate) | | 4 | Dispatcher allocation + `run_day` orchestrator + `/run/daily` + attribution | ✅ done (live render/publish needs real keys) | | 5 | CI: generate / fitness / mutator (scout + self-fix + auto-merge toggle) / deploy | ✅ done | | 6 | Hoist shared publishers/storage/renderer from variant_1 into `harness/` | ⏳ deferred (works as-is via import; pure cleanup) | | 7 | First real mutation & meta-evolution | ▶ runs once deployed with keys | Everything deterministic is unit-tested (70 tests). What the tests **cannot** cover without real credentials: the actual YouTube Analytics HTTP calls, MoviePy rendering, and live publishing. Those run only against real services — wired and syntax-clean, validated by mocks. --- ## Phase 0 — Repository skeleton ✅ ``` content-generator/ ├── README.md # the destination spec ├── PLAN.md # this file ├── .gitignore ├── CODEOWNERS # locks harness/ + fitness + scoreboard to the human ├── .aiderignore # the mutator's blindfold — everything except variants/** ├── EXPERIMENTS_LOG.md # genetic memory (seeded) ├── metrics.csv # scoreboard snapshot (header only, for now) ├── requirements.txt # root deps (superset) ├── Dockerfile # HF Space body (port 7860) ├── harness/ # the locked physics (built in Phases 2–4) ├── variants/ │ └── variant_1/ # faithful copy of meme-generator (the seed strategy) └── .github/workflows/ # fitness.yml + mutator.yml (Phase 5) ``` - `variants/variant_1/` is a **faithful copy** of `meme-generator` (minus `.git`, `scratch`, `queue_state.json`). It still runs exactly as the original does. We do **not** gut it. --- ## Phase 1 — Define the contract (no behaviour change) **Goal:** describe what a "variant" *is* without moving any code yet. - `harness/genome.py` defines: - `VariantManifest` — declared metadata a variant ships in `variants//manifest.json` (id, parent, created_at, the mutable `genome` dict, the tone/strategy knobs). - The `Variant` **protocol**: every variant package must expose `generate_video_plan(budget, *, gemini_api_key) -> list[VideoPlan]` and the assets the shared renderer/publisher consume. Variant 1 satisfies this by adapting its existing `backend_service.video_generator_agent.generate_video_plan_bundle`. - **Test:** `python -c "import harness.genome"` imports cleanly; variant_1 unchanged. --- ## Phase 2 — The scoreboard (read/write the ground truth) **Goal:** a sanitized, secrets-free Mongo collection + a CSV snapshot. - `harness/scoreboard.py`: - `ScoreRow` dataclass: `video_id, upload_date, variant_id, genome_hash, parent_genome, APV, VSA, fitness, channel_status` — **and nothing else** (no tokens, keys, raw tenant id). - `upsert_rows(rows)` — written by the HF body with full `MONGO_URL`. - `read_all()` — used by the mutator via `MONGO_FITNESS_READONLY_URL`. - `snapshot_to_csv(path)` — dump for committed lineage / Aider input. - **One-time ops:** create the `fitnessReadonly` role + `mutator_ro` user (snippet in README). - **Test:** round-trip a fake row through a local/Atlas Mongo; confirm `mutator_ro` can `find` on `fitness_scoreboard` and is **denied** on `users`. --- ## Phase 3 — The fitness function (the locked scorer) **Goal:** turn YouTube Analytics into fitness, safely. Runs **on HF** (it needs per-tenant creds from `users`). - `harness/fitness.py`: - `refresh_scoreboard(*, lab_channel_id, get_channel_credentials, now)`: 1. enumerate the lab channel's videos uploaded **≥ 3 days ago** (the leash), 2. classify `channel_status` (`active` / `terminated` / `suspended` / `no_data`), 3. on `terminated`/`suspended` → **raise `ChannelHalt`** (caller pages the operator; no rows written from a dead channel), 4. fetch APV + VSA via the YouTube Analytics API, 5. `fitness = w_apv * APV + w_vsa * VSA` (weights are constants here, **not** a gene), 6. `scoreboard.upsert_rows(...)`. - The YouTube Analytics call is isolated behind `_fetch_analytics(...)` so it can be mocked. - HF exposes `POST /fitness/refresh` that calls `refresh_scoreboard`. (Or HF self-schedules.) - **Test:** unit-test the leash (a 1-day-old video is excluded), the HALT path, and the fitness formula with mocked analytics. No live API needed. --- ## Phase 4 — The dispatcher (carrying capacity & lifecycle) **Goal:** decide who gets airtime; spawn/retire variants. Pure, deterministic, testable. - `harness/dispatcher.py`: - `living_variants()` — scan `variants/` for valid manifests (cap-aware). - `allocate_slots(budget, fitness_by_variant, *, newborn_floor)`: - newborns (no fitness yet) get a guaranteed floor — never starved to 0, - scored variants split the rest by fitness **rank** (soft taper, best first — ordering, not noisy magnitudes), and no scored variant is forced to 0 while budget covers them, - sum == budget exactly. Daily budget defaults to **5**. - `extinction_candidates(history, K)` — variants at 0 slots for `K` consecutive days. - `enforce_cap(MAX_LIVING_VARIANTS)`. - `run_day(budget)` — for each variant, call its `generate_video_plan`, hand assets to the shared renderer + publisher, then `record_run`. This **replaces** variant 1's standalone `letsDoTodaysJob` loop as the top-level entrypoint. - **Test:** property-test `allocate_slots` (sums to budget, newborn floor honoured, taper by rank); test extinction + cap with synthetic histories. --- ## Phase 5 — The evolutionary engine (CI) **Goal:** wire the two GitHub Actions workflows. - `.github/workflows/fitness.yml` — cron every 3 days: `curl -XPOST $HF_SPACE_URL/fitness/refresh` (optionally with `X-Trigger-Token`). Holds no secrets beyond the URL/token. - `.github/workflows/mutator.yml` — cron every 3 days (offset +6h after the refresh): 1. checkout, set up Python + Aider, 2. `python -m harness.scoreboard --snapshot metrics.csv` using `MONGO_FITNESS_READONLY_URL`, 3. run Aider headless with `--message` from `harness/mutator_prompt.md`, model `$MUTATOR_MODEL` (OpenRouter free DeepSeek → Gemini Flash fallback), allowlist enforced by `.aiderignore`, 4. run `pytest` (liveness gate) inside the container — abort the PR on failure, 5. secret-scan the diff (`gitleaks` + grep) → label, 6. `gh pr create` with the new `EXPERIMENTS_LOG.md` entry as the body. - **Test:** run the mutator workflow manually (`workflow_dispatch`) against a throwaway branch; confirm it opens a PR and **cannot** merge. --- ## Phase 6 — Hoist shared services (the clean split) **Goal:** reach the README's architecture — only now, with tests green. - Physically move `publishers.py`, `storage.py`, and the MoviePy renderer out of `variant_1/backend_service` into `harness/` as `harness.publishers`, `harness.storage`, `harness.video_render`. Variant 1 imports them from `harness`. - Demote variant 1 to *strategy only*: idea/critic/template/music prompts + the meme engine. - Add `.aiderignore` coverage so the hoisted modules are now also off-limits. - **Test:** full `pytest`; one end-to-end dry-run video render with `VIDEO_AGENT_DISABLE_LLM=1`. --- ## Phase 7 — First mutation & meta-evolution - Seed `EXPERIMENTS_LOG.md` with the baseline genome. - Let the mutator spawn `variant_2` from `variant_1` with one change. Two deliberately wacky first experiments to "see what it does": 1. **The Subliminal Frame** — a 1-frame meme at t=0.5s before the real content (bets on VSA). 2. **The TTS Auctioneer** — `tts_rate=+40%`, `seconds_per_image=3` (bets on rewatch). - Watch the scoreboard for ~2 windows; merge survivors; let starvation retire losers. --- ## Resolved decisions (locked) | Decision | Value | |----------|-------| | Evolve cadence | every **3 days** | | Cohorts | multiple, competing for a fixed daily video budget | | Slot allocation | fitness-proportional + juvenile floor; 0 slots for `K=12` days ⇒ extinction | | Population cap | **4** living variants | | Fitness bridge | HF computes → sanitized Mongo `fitness_scoreboard` → mutator reads (read-only role) | | Multi-tenant signal | single **lab channel**; other tenants excluded from selection | | Mutator brain | OpenRouter `deepseek:free` → Gemini Flash fallback (`MUTATOR_MODEL`) | | Merge policy | toggle `AUTONOMOUS_MERGE`: human-in-the-loop (default) or fully autonomous self-merge | | Secret leak control | `.aiderignore` blindfold + CODEOWNERS + CI secret-scan tripwire | ## Open knobs (tunable, not blocking) - Fitness weights `w_apv` / `w_vsa`. - `DAILY_VIDEO_BUDGET` (3–5), `EXTINCTION_DAYS_K` (default 12), juvenile floor size. - `boldness` gene range.