docs: add full end-to-end architectural analysis (5-phase recon, flow, build & stability)

ARCHITECTURE_ANALYSIS.md

@@ -0,0 +1,527 @@
+# Rhodawk AI DevSecOps Engine — End-to-End Architectural Analysis
+
+> Generated by an automated architect/security-researcher pass over commit
+> `14b1bbe` of `Architect8999/rhodawk-ai-devops-engine` on the
+> HuggingFace Spaces repository.
+> Scope: every Python module in the repo root + the new `mythos/` package
+> (~17 kLOC of Python, 30 MCP servers, 1 Dockerfile, 1 Gradio UI).
+
+---
+
+## Phase 1 — High-Level Reconnaissance & Topology
+
+### 1.1 Virtual directory tree (top two levels)
+
+```
+rhodawk-ai-devops-engine/
+├── app.py                       # Gradio control plane + main loop  (~117 KB)
+├── hermes_orchestrator.py       # Multi-phase research brain        (~30 KB)
+├── language_runtime.py          # Polyglot runtime sandbox          (~70 KB)
+├── red_team_fuzzer.py           # CEGIS-style adversarial fuzzer    (~62 KB)
+│
+├── adversarial_reviewer.py      # 3-model concurrent LLM verdict
+├── audit_logger.py              # Hash-chained tamper-evident log
+├── bounty_gateway.py            # H1 / GHSA submission pipeline
+├── chain_analyzer.py            # Commit-history attack graphs
+├── commit_watcher.py            # CAD — silent-patch detector
+├── conviction_engine.py         # Multi-criterion auto-merge gate
+├── cve_intel.py                 # NVD / OSV / Exploit-DB lookups
+├── disclosure_vault.py          # Encrypted finding storage
+├── embedding_memory.py          # sentence-transformers + sqlite-vec
+├── exploit_primitives.py        # ROP / heap / shellcode helpers
+├── formal_verifier.py           # Z3 bounded verification
+├── fuzzing_engine.py            # Hypothesis fuzzer driver
+├── github_app.py                # GitHub App JWT / installation token
+├── harness_factory.py           # Auto-generates fuzzing harnesses
+├── job_queue.py                 # JSON-file-backed job ledger
+├── lora_scheduler.py            # Threshold-triggered LoRA exports
+├── memory_engine.py             # TF-IDF retrieval (legacy)
+├── notifier.py                  # Slack / Discord / email fan-out
+├── public_leaderboard.py        # Public stats endpoint
+├── repo_harvester.py            # Antagonist target picker
+├── sast_gate.py                 # bandit + 16 secret patterns
+├── semantic_extractor.py        # Function / call-graph extraction
+├── supply_chain.py              # pip-audit + typosquat heuristics
+├── swebench_harness.py          # SWE-bench evaluation runner
+├── symbolic_engine.py           # angr planner + Z3 solver glue
+├── taint_analyzer.py            # Source→sink taint tracking
+├── training_store.py            # SQLite/PG attempt ledger + HF push
+├── verification_loop.py         # Retry / prompt-build state machine
+├── vuln_classifier.py           # CWE taxonomy + scoring
+├── webhook_server.py            # GitHub webhook HTTPServer (port 7861)
+├── worker_pool.py               # Process-isolated parallel audits
+│
+├── mythos/                      # ⬅ NEW: Mythos-level upgrade package
+│   ├── MYTHOS_PLAN.md
+│   ├── __init__.py / integration.py
+│   ├── agents/                  # planner / explorer / executor / orchestrator
+│   ├── reasoning/               # probabilistic + attack_graph
+│   ├── static/                  # tree-sitter, joern, codeql, semgrep bridges
+│   ├── dynamic/                 # aflpp, klee, qemu, frida, gdb
+│   ├── exploit/                 # pwntools, ROP, heap, privesc KB
+│   ├── learning/                # rl_planner, mlflow, lora, curriculum, episodic
+│   ├── mcp/                     # 5 new MCP servers
+│   ├── skills/                  # agentskills.io registry
+│   └── api/                     # FastAPI productization (auth, webhooks, schemas)
+│
+├── mcp_config.json              # 30 MCP servers registered
+├── Dockerfile                   # python:3.12-slim + uv + node + Gradio
+├── requirements.txt             # ~30 first-class deps + Mythos optional
+├── pitch_deck/, pitch-deck/     # Marketing collateral (PDF/PPTX/HTML)
+├── FOUNDER_PLAYBOOK.md, SECURITY_RESEARCH_PLAYBOOK.md, README.md
+└── .git/
+```
+
+### 1.2 Structural paradigm
+
+A **flat-module monolith with a side-car package**. There is no explicit
+layered or hexagonal partitioning at the root: every concern (UI, orchestration,
+analysis tools, persistence, networking) lives as a peer `*.py` module
+imported by `app.py`. Communication is **in-process function calls** plus
+**JSON files on `/data`** for cross-process state (jobs, audit chain, memory).
+The new `mythos/` package adds a proper Python package with sub-modules per
+concern, intended as the migration target for a cleaner future architecture.
+
+### 1.3 Technology stack
+
+| Layer | Technology |
+|---|---|
+| Runtime | Python 3.12 (slim Docker), Node.js 20 (npm-installed for some MCP servers) |
+| UI / Control plane | **Gradio 5.29** on port 7860 |
+| Webhooks | Stdlib `http.server.BaseHTTPRequestHandler` on port 7861 |
+| LLM gateway | **OpenRouter** (DeepSeek-R1 / V3 free tier, plus Qwen ∥ Gemma ∥ Mistral consensus); env-driven model tiers in Mythos |
+| Code patching | **Aider 0.86** (driven via subprocess + MCP config) |
+| Static / SAST | bandit, ruff, semgrep, radon, custom 16-pattern secret scanner |
+| Symbolic / Formal | **z3-solver**, **angr**, custom symbolic engine |
+| Fuzzing | Hypothesis (atheris removed — see Dockerfile comments) |
+| Embeddings / Memory | sentence-transformers + sqlite-vec, optional Qdrant |
+| Persistence | SQLite (default) / Postgres (`psycopg2-binary`) for training store |
+| ML / Training | transformers, torch, datasets, custom LoRA scheduler |
+| MCP | `@modelcontextprotocol/server-github` (npm), `mcp-server-fetch` (uvx), and **30** servers in `mcp_config.json` (25 base + 5 new Mythos ones) |
+| Mythos add-ons | FastAPI + uvicorn + pydantic; optional Pyro/PyMC, MLflow, RLlib, pwntools, Frida, tree-sitter-languages |
+| Versioning / VCS | GitPython, PyGithub, PyJWT |
+| Container | Two-stage `python:3.12-slim` Dockerfile, non-root UID 1000, `/data` writable, `EXPOSE 7860`, `CMD ["python","-u","app.py"]` |
+| Deployment target | HuggingFace Spaces (declared in README YAML front-matter) |
+
+### 1.4 Headline metrics
+
+- **Python LOC:** 16 981 across 47 root modules + 47 Mythos modules.
+- **Top three by size:** `app.py` (≈3 200 LOC), `language_runtime.py` (≈70 KB),
+  `red_team_fuzzer.py` (≈62 KB), `hermes_orchestrator.py` (≈900 LOC).
+- **Tests:** none in the repo root — `pytest` is invoked **on the target
+  repository** being audited, not on Rhodawk itself.
+
+---
+
+## Phase 2 — Entry Point & Execution Flow Mapping
+
+### 2.1 Entry points
+
+| Surface | Entry | Listens on | Triggered by |
+|---|---|---|---|
+| **Primary** | `python -u app.py` (Dockerfile `CMD`) | TCP `:7860` (Gradio) | User opens HF Space URL |
+| **Webhook** | `webhook_server.start_webhook_server()` invoked from `app.py` `__main__` | TCP `:7861` (HTTPServer in a daemon thread) | GitHub `push` / `pull_request` events |
+| **Mythos API** | `uvicorn mythos.api.fastapi_server:app` (manual / opt-in) | configurable | `POST /v1/analyze_target` and webhook callbacks |
+| **MCP servers** | Spawned on demand by Aider via `mcp_config.json` (`stdio` JSON-RPC) | stdin/stdout | Aider's tool calls during patching |
+
+### 2.2 Boot sequence (from `if __name__ == "__main__"` in `app.py`)
+
+```
+1. ui_log("Rhodawk AI v3.0 starting …")
+2. Daemon thread → embedding_memory.pre_warm_model()
+       (downloads sentence-transformers model in background to avoid first-call latency)
+3. start_webhook_server()
+       → HTTPServer on 0.0.0.0:7861, daemon thread,
+         registers _webhook_dispatch as the job dispatcher
+4. demo.launch(server_name=0.0.0.0, server_port=$PORT or 7860)
+       → Gradio Blocks UI bound to enterprise_audit_loop, Hermes tabs,
+         job table, audit chain viewer, leaderboard, etc.
+5. gr.Timer(3) ticks every 3 s → get_combined_refresh()
+       (single SSE stream — comment in app.py notes it replaced
+        three concurrent streams that were exhausting connection limits)
+```
+
+### 2.3 Control flow — primary "audit" loop
+
+`enterprise_audit_loop()` in `app.py` (l. 965) is the heart. Per repo:
+
+```
+configure_git_credentials()           # writes ~/.git-credentials from env
+clone target → /tmp/repo
+discover failing tests via pytest --collect-only + run
+for each failing_test:
+  process_audit_test()
+    ├─ retrieve_similar_fixes_v2()    # embedding_memory (fallback: TF-IDF memory_engine)
+    ├─ build_initial_prompt()         # verification_loop
+    ├─ write_mcp_config() → mcp_config.json on disk
+    ├─ run_aider(mcp_config_path, prompt, context_files)   # subprocess Aider with MCP
+    ├─ re-run pytest → VerificationAttempt
+    ├─ if fail and attempts < MAX_RETRIES:
+    │      build_retry_prompt(failure + previous diff) → loop
+    ├─ run_sast_gate()                # bandit + 16 secret patterns
+    ├─ run_supply_chain_gate()        # pip-audit + typosquat
+    ├─ run_adversarial_review()       # 3 LLMs in parallel; ACTS Bayesian score
+    ├─ run_formal_verification()      # Z3 bounded checks
+    ├─ if adversary REJECT:
+    │      retry with critique * ADVERSARIAL_REJECTION_MULTIPLIER
+    ├─ evaluate_conviction()          # multi-criteria gate
+    ├─ if conviction high → auto_merge_pr() else create_github_pr()
+    ├─ record_attempt() / update_test_result()  # training_store (SQLite/PG)
+    ├─ record_fix_outcome()           # memory_engine writes back lesson
+    └─ maybe_trigger_training()       # lora_scheduler exports HF dataset
+```
+
+A parallel **Hermes research mode** (`hermes_orchestrator.run_hermes_research`)
+sits beside the audit loop. It runs the six phases RECON → STATIC → DYNAMIC →
+EXPLOIT → CONSENSUS → DISCLOSURE and produces a `HermesSession` containing
+`VulnerabilityFinding`s with VES / TVG / ACTS / CAD / SSEC scores.
+
+### 2.4 Data flow
+
+```
+GitHub repo  ──clone──►  /tmp/repo/<sha>           (ephemeral)
+                                  │
+            pytest output ────────┤
+                                  ▼
+                         memory_engine ◄── embedding_memory (sqlite-vec @ /data)
+                                  │
+                                  ▼
+                          MCP-equipped Aider ── subprocess ──► fixed source
+                                  │
+            adversarial verdicts ─┤
+                                  ▼
+                         conviction_engine ──► PR / auto-merge
+                                  │
+            attempt row ──────────┴───────────► training_store
+                                                  (SQLite at /data/store.db
+                                                   or Postgres if DATABASE_URL)
+                                                              │
+                                                              ▼
+                                                     lora_scheduler ──► HF dataset
+```
+
+All persistent state lives under `/data` (writable in HF Spaces, mode 777
+in the Dockerfile). Job ledger files: `/data/jobs/<job_id>.json`. Hash-chained
+audit log: `/data/audit_chain.jsonl`. Embeddings vector store: `/data/mem.db`.
+Mythos adds `/data/mythos/{rl_state.json,episodic.sqlite,skills/}`.
+
+---
+
+## Phase 3 — Functional Breakdown
+
+### 3.1 Executive summary
+
+Rhodawk is an **autonomous DevSecOps control plane**. Point it at a GitHub
+repo; it (1) reproduces failing tests, (2) drives an LLM coding agent (Aider,
+through 30 MCP-exposed tools) to write a patch, (3) re-runs the tests in a
+verification loop, (4) gates the patch through SAST + supply-chain + multi-LLM
+adversarial review + Z3 formal verification + a conviction engine, (5) opens
+or auto-merges a pull request, and (6) feeds every attempt into a training
+store so a LoRA fine-tune can be scheduled. A parallel **Hermes** mode flips
+the polarity from "fix bugs" to "find bugs": coordinated multi-phase
+vulnerability research with custom scoring algorithms (VES/TVG/ACTS/CAD/SSEC)
+and a HackerOne / GitHub Security Advisory submission gateway. The new
+**Mythos package** layers a multi-agent (planner / explorer / executor)
+framework, probabilistic Bayesian reasoning, advanced static / dynamic /
+exploit tooling, RL-driven self-improvement, 5 additional MCP servers, and a
+FastAPI productization surface on top of all of the above.
+
+### 3.2 Module responsibilities (selected)
+
+| Module | Responsibility |
+|---|---|
+| `app.py` | Gradio UI, audit loop, Aider subprocess driver, refresh timer, boot sequence |
+| `hermes_orchestrator.py` | Phase state machine, custom security metrics (VES/ACTS/TVG), tool dispatcher |
+| `verification_loop.py` | Retry policy, prompt construction, attempt accounting |
+| `adversarial_reviewer.py` | 3-model parallel verdict → consensus, used by both audit and Hermes |
+| `conviction_engine.py` | Boolean / weighted gate that decides auto-merge vs human review |
+| `sast_gate.py`, `supply_chain.py`, `formal_verifier.py` | Independent gates the patch must pass |
+| `language_runtime.py` | Polyglot sandbox factory — sets up Python venvs, Node, Java, etc. for the target repo |
+| `red_team_fuzzer.py` | CEGIS adversarial fuzzer — counter-example guided refinement |
+| `embedding_memory.py` / `memory_engine.py` | v2 semantic retrieval (sentence-transformers + sqlite-vec); v1 TF-IDF fallback |
+| `training_store.py` | SQLite / Postgres attempt ledger + `export_hf_dataset` for HF push |
+| `lora_scheduler.py` | Threshold-triggered LoRA training-data export |
+| `audit_logger.py` | Append-only hash-chained log + `verify_chain_integrity` |
+| `webhook_server.py` | HMAC-verified GitHub webhook receiver, IP rate-limit, dispatcher hook |
+| `worker_pool.py` | Process-isolated parallel test handling (`MAX_WORKERS`) |
+| `bounty_gateway.py` | Holds findings for human approval → submits to HackerOne / opens GHSA |
+| `vuln_classifier.py` / `cve_intel.py` | CWE taxonomy + NVD/OSV/Exploit-DB lookup |
+| `commit_watcher.py` / `chain_analyzer.py` | Silent-patch detection + per-commit attack-graph diffing |
+| `mythos/agents/*` | Planner produces a probabilistic plan; Explorer enumerates hypotheses; Executor runs tools; Orchestrator routes |
+| `mythos/reasoning/probabilistic.py` | Bayesian hypothesis sampling (Pyro / PyMC / NumPy fallback) |
+| `mythos/learning/rl_planner.py` | Tool-selection policy (RLlib / SB3 / UCB1 fallback), state at `/data/mythos/rl_state.json` |
+| `mythos/api/fastapi_server.py` | `POST /v1/analyze_target`, auth middleware, webhook callbacks |
+| `mythos/mcp/*` | 5 new servers exposed via `python -m mythos.mcp.<name>` and registered in `mcp_config.json` |
+
+### 3.3 Background work / scheduled tasks
+
+- **Embedding pre-warm thread** (daemon, `app.py` `__main__`).
+- **Webhook HTTPServer thread** (daemon).
+- **Gradio refresh timer** (3 s SSE tick) — coalesced into one stream.
+- **Worker-pool subprocesses** for per-test isolation (`worker_pool._run_isolated`).
+- **LoRA scheduler** triggers on attempt-count threshold (no cron — checked at the end of each audit).
+- **MCP server lifecycle** — Aider spawns each declared MCP server on demand and tears it down with the patch session.
+
+There is **no Celery / RQ / APScheduler** — concurrency is purely
+threading + subprocess + ad-hoc daemons.
+
+---
+
+## Phase 4 — Build & Execution Guide
+
+### 4.1 Prerequisites
+
+| Required | Notes |
+|---|---|
+| Python **3.12** | Pinned in README front-matter and Dockerfile |
+| Node.js + npm | Only for `@modelcontextprotocol/server-github` |
+| `uv` (Astral) | Used by `language_runtime` to materialise per-target venvs |
+| `git` | GitPython invokes the system binary |
+| Writable `/data` (Linux) or local equivalent | All persistent state lives here |
+| **Env vars** | `OPENROUTER_API_KEY` (mandatory for any LLM call); `GITHUB_TOKEN` *or* GitHub App creds (`GITHUB_APP_ID`, `GITHUB_APP_PRIVATE_KEY`, `GITHUB_INSTALLATION_ID`); optional `DATABASE_URL` for Postgres training store; `HF_TOKEN` for HF dataset push; `GITHUB_WEBHOOK_SECRET` for webhook HMAC; Slack/Discord URLs for notifier; Mythos tier overrides `MYTHOS_TIER1_PRIMARY` / `MYTHOS_TIER2_PRIMARY` etc.; `RHODAWK_MYTHOS=1` to engage multi-agent loop |
+| Optional native tools | Joern, CodeQL, AFL++, KLEE, QEMU, Frida, GDB, ROPGadget, pwntools — Mythos bridges degrade gracefully if absent |
+
+### 4.2 Setup — local
+
+```bash
+# 1. Clone
+git clone https://huggingface.co/spaces/Architect8999/rhodawk-ai-devops-engine
+cd rhodawk-ai-devops-engine
+
+# 2. Python deps
+python3.12 -m venv .venv && source .venv/bin/activate
+pip install --upgrade pip
+pip install -r requirements.txt mcp-server-fetch
+
+# 3. Node-based MCP server
+npm install -g @modelcontextprotocol/server-github
+
+# 4. (Optional) astral uv for runtime sandboxing
+curl -LsSf https://astral.sh/uv/install.sh | sh
+
+# 5. Persistent state directory
+sudo mkdir -p /data && sudo chmod 777 /data
+
+# 6. Environment
+export OPENROUTER_API_KEY=sk-or-...
+export GITHUB_TOKEN=ghp_...                 # or GITHUB_APP_* trio
+export GITHUB_WEBHOOK_SECRET=whsec_...
+# Optional
+export DATABASE_URL=postgres://...
+export HF_TOKEN=hf_...
+export RHODAWK_MYTHOS=1
+```
+
+### 4.3 Run
+
+```bash
+# Primary control plane (Gradio on :7860, webhook on :7861)
+PORT=7860 python -u app.py
+
+# Or via Docker (the way HF Spaces runs it)
+docker build -t rhodawk-ai .
+docker run -it --rm \
+  -p 7860:7860 -p 7861:7861 \
+  -e OPENROUTER_API_KEY -e GITHUB_TOKEN -e GITHUB_WEBHOOK_SECRET \
+  -v $PWD/data:/data \
+  rhodawk-ai
+
+# Mythos productization API (independent of the Gradio loop)
+uvicorn mythos.api.fastapi_server:app --host 0.0.0.0 --port 8000
+
+# Run a single Mythos MCP server manually (smoke test)
+python -m mythos.mcp.static_analysis_mcp
+```
+
+### 4.4 Verification
+
+```bash
+# 1. Gradio UI returns HTML
+curl -sf http://localhost:7860/ | head -n 5
+
+# 2. Webhook server is up (expect 405 Method Not Allowed on GET)
+curl -sv http://localhost:7861/webhook 2>&1 | grep "HTTP/1"
+
+# 3. Send a synthetic GitHub ping (replace SECRET)
+BODY='{"zen":"hello"}'
+SIG="sha256=$(printf '%s' "$BODY" | openssl dgst -sha256 -hmac "$GITHUB_WEBHOOK_SECRET" | cut -d' ' -f2)"
+curl -sv -X POST http://localhost:7861/webhook \
+  -H "X-GitHub-Event: ping" \
+  -H "X-Hub-Signature-256: $SIG" \
+  -H "Content-Type: application/json" \
+  -d "$BODY"
+
+# 4. Mythos API health
+curl -sf http://localhost:8000/healthz
+curl -sX POST http://localhost:8000/v1/analyze_target \
+  -H "Content-Type: application/json" \
+  -d '{"target":"https://github.com/octocat/Hello-World","mode":"recon"}'
+
+# 5. Verify the audit chain is internally consistent
+python -c "from audit_logger import verify_chain_integrity; print(verify_chain_integrity())"
+
+# 6. Confirm 30 MCP servers register
+jq '.mcpServers | length' mcp_config.json   # → 30
+```
+
+In the Gradio UI you should see live logs ticking every 3 s, the metrics
+row populating, and the **Hermes** and **CWE Reference** tabs available.
+
+---
+
+## Phase 5 — Architectural Thoughts & Stability Assessment
+
+### 5.1 Patterns in use
+
+- **Pipeline / chain-of-responsibility** — the audit loop is a clean
+  sequence of independent gates (`sast_gate → supply_chain → adversarial →
+  formal → conviction`). Each gate exposes a single function returning a
+  decision dataclass; they are trivially composable.
+- **Strategy + graceful-degradation wrappers** — Mythos bridges
+  (`mythos/static/joern_bridge.py`, etc.) all expose `available()` plus a
+  pure-Python fallback. The orchestrator can plan with whatever is on the
+  PATH today.
+- **Tier-routing for LLMs** — Hermes has two model tiers (`HERMES_MODEL`
+  reasoning + `HERMES_FAST_MODEL` for cheap triage); Mythos generalises this
+  to env-driven `MYTHOS_TIER{1,2}_{PRIMARY,FALLBACK}`.
+- **Bayesian / consensus voting** — `compute_acts()` produces a Bayesian
+  multi-model trust score that is reused by both adversarial review and
+  conviction.
+- **Append-only hash chain** — `audit_logger` builds a Merkle-style chain
+  with `verify_chain_integrity()`, which is the right primitive for
+  compliance evidence.
+- **MCP as the universal tool bus** — every external capability (SAST,
+  fuzzers, vuln DBs, web search, GitHub, Postgres) is exposed through
+  `mcp_config.json`; this is by far the strongest piece of architecture in
+  the codebase and is what makes Aider's tool use uniform.
+- **Multi-agent (Mythos)** — Planner emits a probabilistic plan, Explorer
+  enumerates hypotheses, Executor invokes tools, Orchestrator routes — a
+  textbook multi-agent shape with clean message contracts in
+  `mythos/agents/base.py`.
+
+### 5.2 Strengths
+
+- **Sharp separation between "fix mode" (audit loop) and "find mode"
+  (Hermes)** — they share gates and the training store but never tangle.
+- **Robust webhook surface** — HMAC verification, IP rate-limit, dispatcher
+  injection (`set_job_dispatcher`) keeps the receiver pure.
+- **Worker-pool isolation** — `worker_pool._run_isolated` puts each test fix
+  in its own subprocess, which contains LLM/aider blow-ups well.
+- **Mythos add-on is non-invasive** — opt-in via `RHODAWK_MYTHOS=1` and a
+  separate FastAPI surface, so the existing Gradio UX is unchanged unless
+  you want it.
+- **Persistent learning loop** — every attempt, success or failure, lands
+  in `training_store` and feeds `lora_scheduler.maybe_trigger_training`.
+  The flywheel is real and not aspirational.
+
+### 5.3 Bug surfaces & risks
+
+The list below is **prioritised** — items are ordered by likely real-world
+impact on stability or security.
+
+#### High
+
+1. **`app.py` is a 117 KB god-module.** Boot, UI definition, audit loop,
+   subprocess management, MCP-config writing, Hermes UI bindings, and the
+   refresh timer all live in one file. This is the biggest stability risk:
+   any change ripples broadly and there are no unit tests on Rhodawk
+   itself. Recommend extracting the UI definition into `ui/`, the audit
+   loop into `audit/`, and process management into `procctl.py`.
+2. **No test suite for Rhodawk.** `pytest` is invoked only against target
+   repos. There is no CI guard against regressions in the orchestration
+   logic. This is the single highest-leverage fix.
+3. **`/data` mode `777` and shared by every tenant.** The Dockerfile sets
+   `chmod 777 /data` (because of HF Spaces UID quirks). All tenants share
+   `/data/jobs/`, `/data/audit_chain.jsonl`, `/data/store.db`, etc. The
+   `tenant_id` is stamped on each job key, but a buggy import or path-
+   traversal-style filename would let one tenant's data overwrite
+   another's. Add a `pathlib` allowlist + per-tenant subdirectories.
+4. **Shell-out via `git`, `pytest`, `aider`, `npm`** with target-controlled
+   filenames. `run_subprocess_safe` exists, but several callers stitch
+   strings before reaching it. Audit every `subprocess.run` for
+   `shell=True` and for unvalidated repo paths.
+5. **Embedding model pre-warm runs in a daemon thread without back-off.**
+   If `sentence-transformers` fails (rate-limited HF, no disk), every
+   subsequent retrieval call falls back silently to TF-IDF — a real
+   correctness regression that is invisible to operators. Surface this
+   state on the dashboard.
+6. **Hermes phase state is in-process.** `HermesSession` lives in module
+   memory; if `app.py` restarts mid-research the entire session is lost.
+   Persist `HermesSession.asdict()` to `/data/hermes/<session_id>.json`
+   on every phase transition.
+
+#### Medium
+
+7. **`memory_engine.py` (TF-IDF) and `embedding_memory.py` (vec) compete.**
+   `app.py` imports both, and the audit loop calls v2 with v1 as silent
+   fallback. Two stores will drift. Pick one as the source of truth and
+   demote the other to "legacy".
+8. **Webhook server uses stdlib HTTPServer (single-threaded by default).**
+   `start_webhook_server()` should use `ThreadingHTTPServer` (or, ideally,
+   move the same handlers to FastAPI now that uvicorn is a dep).
+9. **`MAX_RETRIES * ADVERSARIAL_REJECTION_MULTIPLIER`** can produce long
+   tail loops where a stubborn adversary blocks the pipeline. Add an
+   absolute wall-clock cap per audit.
+10. **JSON job ledger is read/written without `flock`.** Two parallel
+    workers updating the same job file race. Use SQLite (which is already
+    a dependency) for the job queue too — this also fixes (3) by giving
+    you per-tenant rows instead of files.
+11. **`audit_logger` chain integrity is only verified on demand.** If the
+    chain has been tampered with, no one notices until someone clicks the
+    button. Add a periodic background verifier that pages on a break.
+12. **Mythos optional deps are mostly commented out** — anyone who pip-
+    installs the file gets the FastAPI surface but not Pyro/MLflow/RLlib.
+    Document the on-demand install path more loudly in `MYTHOS_PLAN.md`.
+
+#### Low / hygiene
+
+13. `language_runtime.py` is ~70 KB with deep `if-lang ==` ladders. Replace
+    with a `Runtime` class registry plus `entry_points` so adding Go, Rust,
+    or .NET is a one-file change.
+14. Many modules use `time.sleep()` for backoff instead of `tenacity` even
+    though `tenacity` is already imported in `app.py`.
+15. `red_team_fuzzer.py` (62 KB) duplicates patterns that
+    `fuzzing_engine.py` already has — consolidate.
+16. `notifier.py` swallows exceptions silently; add a structured failure
+    counter.
+17. `pitch_deck/` and `pitch-deck/` (hyphen vs underscore) coexist as
+    sibling directories — pick one.
+
+### 5.4 Memory / leak surfaces
+
+- `_hermes_logs: list[str]` is unbounded — every Hermes run appends without
+  truncation. Add a ring buffer (`collections.deque(maxlen=10_000)`).
+- The Gradio `gr.Timer(3)` keeps building string responses even when no
+  client is connected. The single-stream coalescing helps, but the
+  `live_logs` textbox still grows unboundedly.
+- Aider subprocesses inherit the parent file descriptors; a long-running
+  audit can exhaust FDs. Add `close_fds=True` everywhere.
+- The embedding store grows monotonically. Add a TTL eviction in
+  `embedding_memory`.
+
+### 5.5 Targeted stabilisation roadmap
+
+| Priority | Change | Effort | Payoff |
+|---|---|---|---|
+| 🔥 P0 | Add a `tests/` directory with pytest covering: webhook signature, audit-loop happy path with mocked Aider, audit-chain integrity, MCP-config render | M | Catches future breakage of every gate |
+| 🔥 P0 | Per-tenant subdirectory under `/data/<tenant>/` and a path-allowlist helper | S | Hard isolation between tenants |
+| 🔥 P0 | Migrate `job_queue` from JSON files to the existing SQLite store | S | Removes file-locking races |
+| ⚡ P1 | Carve `app.py` into `ui_blocks.py` + `audit_loop.py` + `procctl.py` (no behaviour change) | M | Drops blast radius of every future change |
+| ⚡ P1 | Switch `webhook_server` to `ThreadingHTTPServer` *or* mount it under the new FastAPI app | S | Concurrency + one fewer port |
+| ⚡ P1 | Surface "embedding model healthy?" on the dashboard | XS | Stops silent fallback to TF-IDF |
+| ⚡ P1 | Persist `HermesSession` after every phase transition | S | Crash-safe research mode |
+| 🛠 P2 | Replace `if-lang ==` ladders in `language_runtime` with a registry | M | Future polyglot support |
+| 🛠 P2 | Bounded ring buffers for `_hermes_logs` + Gradio `live_logs` | XS | Stops slow memory growth |
+| 🛠 P2 | Consolidate `memory_engine` and `embedding_memory` behind one interface | M | One source of retrieval truth |
+| 🛡 P2 | Explicit security pass on every `subprocess.run` for shell injection / path traversal in target-controlled inputs | M | Hardens the polyglot runtime |
+| 🛡 P3 | Background verifier thread for the audit hash-chain that pages on break | S | Compliance evidence stays trustworthy |
+
+### 5.6 Bottom line
+
+The architecture is **ambitious, coherent, and unusually mature for a
+single-Space project** — the gating pipeline, the MCP tool bus, and the
+Mythos multi-agent / RL extension are genuinely well-thought-out. The two
+things holding it back from production-grade are (a) the absence of a
+self-test suite and (b) the size of `app.py`. Both are mechanical, not
+architectural, problems. Address P0 + P1 above and Rhodawk graduates from
+"impressive HuggingFace Space" to "shippable security platform".