PLAN.md

Build Plan — RSI Content Generation Agent

This is the authoritative, phased build plan. The README describes the system as if finished; this document is how we get there without breaking anything, in an order where each phase is independently testable. The guiding rule:

Never refactor working code blind. Copy it intact, wrap it behind a contract, and only physically move shared modules once the wrapper is proven. Every phase ends at a state you can run.

---

Invariants (true at every phase — violating one is "a mistake")

1. The harness grades; the variants are graded. harness/fitness.py, the scoreboard, and the dispatcher's allocation math are never in the mutator's editable surface.
2. The agent is blind to credentials. CI reads a sanitized Mongo fitness_scoreboard via a role scoped to that collection only. It never gets MONGO_URL, never reads users.
3. Merge is gated. A mutation only reaches main through a PR that (a) has green LOCKED tests and (b) passes the secret-scan. Whether a human or the organism itself clicks merge is the AUTONOMOUS_MERGE toggle — but a red or secret-touching build never merges, either way.
4. 3-day leash. Fitness only ever reads analytics for videos uploaded ≥ 3 days ago.
5. Death ≠ low score. A suspended/struck channel HALTs the loop; it is never fed into fitness as a number.
6. Population ≤ MAX_LIVING_VARIANTS (4).

---

Build status

Phase	What	State
0	Repo skeleton + seed variant	✅ done
1	Variant contract (genome.py)	✅ done
2	Scoreboard (Mongo + CSV, sanitized)	✅ done
3	Fitness scorer + live YouTube fetch + /fitness/refresh	✅ done (live path needs real keys to validate)
4	Dispatcher allocation + run_day orchestrator + /run/daily + attribution	✅ done (live render/publish needs real keys)
5	CI: generate / fitness / mutator (scout + self-fix + auto-merge toggle) / deploy	✅ done
6	Hoist shared publishers/storage/renderer from variant_1 into harness/	⏳ deferred (works as-is via import; pure cleanup)
7	First real mutation & meta-evolution	▶ runs once deployed with keys

Everything deterministic is unit-tested (70 tests). What the tests cannot cover without real credentials: the actual YouTube Analytics HTTP calls, MoviePy rendering, and live publishing. Those run only against real services — wired and syntax-clean, validated by mocks.

---

Phase 0 — Repository skeleton ✅

content-generator/
├── README.md            # the destination spec
├── PLAN.md              # this file
├── .gitignore
├── CODEOWNERS           # locks harness/ + fitness + scoreboard to the human
├── .aiderignore         # the mutator's blindfold — everything except variants/**
├── EXPERIMENTS_LOG.md    # genetic memory (seeded)
├── metrics.csv          # scoreboard snapshot (header only, for now)
├── requirements.txt     # root deps (superset)
├── Dockerfile           # HF Space body (port 7860)
├── harness/             # the locked physics (built in Phases 2–4)
├── variants/
│   └── variant_1/       # faithful copy of meme-generator (the seed strategy)
└── .github/workflows/   # fitness.yml + mutator.yml (Phase 5)

• variants/variant_1/ is a faithful copy of meme-generator (minus .git, scratch, queue_state.json). It still runs exactly as the original does. We do not gut it.

---

Phase 1 — Define the contract (no behaviour change)

Goal: describe what a "variant" is without moving any code yet.

• harness/genome.py defines:
  • VariantManifest — declared metadata a variant ships in variants/<id>/manifest.json (id, parent, created_at, the mutable genome dict, the tone/strategy knobs).
  • The Variant protocol: every variant package must expose generate_video_plan(budget, *, gemini_api_key) -> list[VideoPlan] and the assets the shared renderer/publisher consume. Variant 1 satisfies this by adapting its existing backend_service.video_generator_agent.generate_video_plan_bundle.
• Test: python -c "import harness.genome" imports cleanly; variant_1 unchanged.

---

Phase 2 — The scoreboard (read/write the ground truth)

Goal: a sanitized, secrets-free Mongo collection + a CSV snapshot.

• harness/scoreboard.py:
  • ScoreRow dataclass: video_id, upload_date, variant_id, genome_hash, parent_genome, APV, VSA, fitness, channel_status — and nothing else (no tokens, keys, raw tenant id).
  • upsert_rows(rows) — written by the HF body with full MONGO_URL.
  • read_all() — used by the mutator via MONGO_FITNESS_READONLY_URL.
  • snapshot_to_csv(path) — dump for committed lineage / Aider input.
• One-time ops: create the fitnessReadonly role + mutator_ro user (snippet in README).
• Test: round-trip a fake row through a local/Atlas Mongo; confirm mutator_ro can find on fitness_scoreboard and is denied on users.

---

Phase 3 — The fitness function (the locked scorer)

Goal: turn YouTube Analytics into fitness, safely. Runs on HF (it needs per-tenant creds from users).

• harness/fitness.py:
  • refresh_scoreboard(*, lab_channel_id, get_channel_credentials, now):
    1. enumerate the lab channel's videos uploaded ≥ 3 days ago (the leash),
    2. classify channel_status (active / terminated / suspended / no_data),
    3. on terminated/suspended → raise ChannelHalt (caller pages the operator; no rows written from a dead channel),
    4. fetch APV + VSA via the YouTube Analytics API,
    5. fitness = w_apv * APV + w_vsa * VSA (weights are constants here, not a gene),
    6. scoreboard.upsert_rows(...).
  • The YouTube Analytics call is isolated behind _fetch_analytics(...) so it can be mocked.
• HF exposes POST /fitness/refresh that calls refresh_scoreboard. (Or HF self-schedules.)
• Test: unit-test the leash (a 1-day-old video is excluded), the HALT path, and the fitness formula with mocked analytics. No live API needed.

---

Phase 4 — The dispatcher (carrying capacity & lifecycle)

Goal: decide who gets airtime; spawn/retire variants. Pure, deterministic, testable.

• harness/dispatcher.py:
  • living_variants() — scan variants/ for valid manifests (cap-aware).
  • allocate_slots(budget, fitness_by_variant, *, newborn_floor):
    • newborns (no fitness yet) get a guaranteed floor — never starved to 0,
    • scored variants split the rest by fitness rank (soft taper, best first — ordering, not noisy magnitudes), and no scored variant is forced to 0 while budget covers them,
    • sum == budget exactly. Daily budget defaults to 5.
  • extinction_candidates(history, K) — variants at 0 slots for K consecutive days.
  • enforce_cap(MAX_LIVING_VARIANTS).
  • run_day(budget) — for each variant, call its generate_video_plan, hand assets to the shared renderer + publisher, then record_run. This replaces variant 1's standalone letsDoTodaysJob loop as the top-level entrypoint.
• Test: property-test allocate_slots (sums to budget, newborn floor honoured, taper by rank); test extinction + cap with synthetic histories.

---

Phase 5 — The evolutionary engine (CI)

Goal: wire the two GitHub Actions workflows.

• .github/workflows/fitness.yml — cron every 3 days: curl -XPOST $HF_SPACE_URL/fitness/refresh (optionally with X-Trigger-Token). Holds no secrets beyond the URL/token.
• .github/workflows/mutator.yml — cron every 3 days (offset +6h after the refresh):
  1. checkout, set up Python + Aider,
  2. python -m harness.scoreboard --snapshot metrics.csv using MONGO_FITNESS_READONLY_URL,
  3. run Aider headless with --message from harness/mutator_prompt.md, model $MUTATOR_MODEL (OpenRouter free DeepSeek → Gemini Flash fallback), allowlist enforced by .aiderignore,
  4. run pytest (liveness gate) inside the container — abort the PR on failure,
  5. secret-scan the diff (gitleaks + grep) → label,
  6. gh pr create with the new EXPERIMENTS_LOG.md entry as the body.
• Test: run the mutator workflow manually (workflow_dispatch) against a throwaway branch; confirm it opens a PR and cannot merge.

---

Phase 6 — Hoist shared services (the clean split)

Goal: reach the README's architecture — only now, with tests green.

• Physically move publishers.py, storage.py, and the MoviePy renderer out of variant_1/backend_service into harness/ as harness.publishers, harness.storage, harness.video_render. Variant 1 imports them from harness.
• Demote variant 1 to strategy only: idea/critic/template/music prompts + the meme engine.
• Add .aiderignore coverage so the hoisted modules are now also off-limits.
• Test: full pytest; one end-to-end dry-run video render with VIDEO_AGENT_DISABLE_LLM=1.

---

Phase 7 — First mutation & meta-evolution

• Seed EXPERIMENTS_LOG.md with the baseline genome.
• Let the mutator spawn variant_2 from variant_1 with one change. Two deliberately wacky first experiments to "see what it does":
  1. The Subliminal Frame — a 1-frame meme at t=0.5s before the real content (bets on VSA).
  2. The TTS Auctioneer — tts_rate=+40%, seconds_per_image=3 (bets on rewatch).
• Watch the scoreboard for ~2 windows; merge survivors; let starvation retire losers.

---

Resolved decisions (locked)

Decision	Value
Evolve cadence	every 3 days
Cohorts	multiple, competing for a fixed daily video budget
Slot allocation	fitness-proportional + juvenile floor; 0 slots for K=12 days ⇒ extinction
Population cap	4 living variants
Fitness bridge	HF computes → sanitized Mongo fitness_scoreboard → mutator reads (read-only role)
Multi-tenant signal	single lab channel; other tenants excluded from selection
Mutator brain	OpenRouter deepseek:free → Gemini Flash fallback (MUTATOR_MODEL)
Merge policy	toggle AUTONOMOUS_MERGE: human-in-the-loop (default) or fully autonomous self-merge
Secret leak control	.aiderignore blindfold + CODEOWNERS + CI secret-scan tripwire

Open knobs (tunable, not blocking)

• Fitness weights w_apv / w_vsa.
• DAILY_VIDEO_BUDGET (3–5), EXTINCTION_DAYS_K (default 12), juvenile floor size.
• boldness gene range.