# F1 — Tiered Inference Engine > Parent: [../MASTER_PLAN.md](../MASTER_PLAN.md) > Status: **planning** > Owner: TBD ## Problem A single provider is never the right answer for a real coding agent: - A small local model is cheap and private but stalls on hard tasks. - A subscription provider (Ollama Cloud Gemma 4 31B, Claude Max, ChatGPT Plus, etc.) has caps; you don't want every token to count against them. - A per-token provider (Anthropic API, OpenAI API, etc.) works but charges for everything it sees. We want opencoti to start work on the cheapest provider that can plausibly handle it, and **escalate only when needed**. ## Tier 0 vision — primary local execution, two paths Tier 0 is the role-label for "the cheap-and-fast tier we start every turn on." It is *not* hard-wired to a specific runtime. opencoti supports two first-class Tier-0 realizations and the user picks one explicitly at first-run setup (no silent auto-fallback): - **GPU path — the advanced one.** Patched llamafile + `ManniX-ITA/gemma-4-A4B-98e-v5-coder-it` (Gemma 4 26B-A4B base, 98-expert prune of the original 128, ~4 B active per token, 256 K context max, IQ3_M ≈ 9.82 GB, runs end-to-end on a 24 GB GPU with full offload). The patched llamafile is what opens up the advanced features below. - **No-GPU path — the simple one.** Ollama Cloud as an OpenAI-compatible Tier-0 provider with the Gemma 4 31B dense model (free BF16 on Ollama Cloud at time of writing). Plain HTTP request flow, no local compute, no advanced features. Equally first-class for users without a GPU. Anything else the OpenAI-compatible provider interface accepts works too (a remote vLLM, a different Ollama endpoint, etc.). Tier 0 is the abstraction; the realizations are config-driven. Smaller Gemma 4 variants (E2B, E4B) and the stock Gemma 3 lineage are explicitly *not* primary targets — evaluated much later if at all. ### Advanced features tied to the GPU path The patched-llamafile path unlocks features the no-GPU path cannot offer. opencoti **does not** expose this distinction in user-facing configuration: features that need the patched llamafile are present automatically when the GPU path is selected, and silently absent on the no-GPU path. From the user's perspective Tier 0 is Tier 0; internally the engine queries provider capabilities and degrades gracefully. Concretely (see [local_llamafile.md](local_llamafile.md) for detail): - **KV-cache reuse across agentic turns** — turn N+1 reuses turn N's prefix, dramatic latency drop on long contexts. - **Big-context throughput patches** — keep tokens/sec usable at 64K+ contexts. - **Single-download Ollama-blob reuse** — if Ollama is installed locally, the patched llamafile can mount the existing `~/.ollama/models/blobs/sha256-…` GGUF directly. No duplicate download. ## Goals - **G1.** Configure an ordered set of tiers (Tier 0, Tier 1, ...), each with one or more providers. - **G2.** Run a request on the lowest-numbered tier first; escalate on failure-to-progress, hard error, or explicit escalation signal. - **G3.** Support two tier topologies: - **Linear** — try a single provider in this tier, fall through on failure. - **Fan-out** — try multiple providers in this tier in parallel (or as a quorum), pick the best response. - **G4.** Within a tier, prefer subscription providers; fall back to metered providers when subs are exhausted or rate-limited. - **G5.** Make the escalation policy a **user-visible config**, not a hidden heuristic. ## Non-goals (for now) - Reordering tiers automatically based on observed quality. - Cross-session learning of provider quality. - Coordinating tiers across multiple concurrent sessions. ## Design sketch ### Where it lives - `packages/opencoti-tiers/` — new package. Holds the tier schema, the router, the escalation policy, the provider-pool primitives. - **Surgical hook** in opencode's model-call site: when an opencoti tier config is present and enabled, the call is delegated to the router from `opencoti-tiers`. Otherwise opencode behaves as upstream. ### Concepts - **Tier** = an ordered slot in the escalation chain. Tier 0 is the cheapest, tried first. - **Provider Pool** inside a tier = one of: - `linear: [Provider, Provider, ...]` — tried in order - `fanout: { mode: "first-good" | "best-of-n" | "quorum", members: [...] }` - **Billing class** per provider: - `subscription` — has a cap; depleted state is observable. - `metered` — pay per token, no cap. - **Escalation triggers** (configurable, ORed): - hard-fail (provider error, timeout, refused) - stuck-progress (no tool calls / no diff after N turns) - explicit (user keypress, or model emits an "escalate" tool call) - structured-quality (output fails a structured validator — e.g., code did not parse, tests still failing after N attempts) ### Config shape (illustrative, not final) GPU-path example: ```yaml opencoti: tiers: enabled: true effort: balanced # low | balanced | high | max models: # Small local registry — names → entries. Same idea as # claude-hooks' provider registry. Entries are matched by name # from the `chain` below. gemma4-a4b-coder: kind: llamafile gguf: ~/.opencoti/models/gemma-4-A4B-98e-v5-coder-it-IQ3_M.gguf contextLength: 131072 # capped; model supports up to 256K gemma4-31b-cloud: kind: openai-compat baseURL: https://ollama.com/api apiKeyEnv: OLLAMA_CLOUD_API_KEY modelId: gemma4:31b billing: subscription chain: - id: tier0 topology: linear providers: - gemma4-a4b-coder # patched llamafile, advanced features - id: tier1 topology: linear providers: - gemma4-31b-cloud # subscription (Ollama Cloud) - groq/llama-3.3-70b # metered, fallback - id: tier2 topology: fanout mode: first-good providers: - anthropic/claude-sonnet-4-6 # subscription via Max - anthropic-api/claude-sonnet-4-6 # metered, fallback escalation: on: [hard-fail, stuck-progress, structured-quality] stuck-progress: no-progress-turns: 3 structured-quality: validators: [code-parses, tests-pass-or-improve] ``` No-GPU example (Tier 0 is the cloud provider; no llamafile present): ```yaml opencoti: tiers: enabled: true effort: balanced models: gemma4-31b-cloud: kind: openai-compat baseURL: https://ollama.com/api apiKeyEnv: OLLAMA_CLOUD_API_KEY modelId: gemma4:31b billing: subscription chain: - id: tier0 topology: linear providers: - gemma4-31b-cloud - id: tier1 # ... user's preferred escalation provider ... ``` ### Tool surface opencode already injects a strong default tool set into every `streamText` call (15 built-in: `shell`, `read`, `write`, `edit`, `glob`, `grep`, `task`, `fetch`, `search`, `todo`, `skill`, `patch`, `question`, `lsp`, `invalid`), plus full MCP server integration, plus plugin / `.opencode/tool/*` user-defined tools, **all gated by the same permission flow**. That surface already meets-or-exceeds what claude-hooks' caliber-grounding-proxy + consultants skill expose (`survey_project`, `list_files`, `read_file`, `glob`, `grep`, `recall_memory`, and the sandboxed `write_file` for the coder role). What opencode does NOT have today, and that opencoti will port from claude-hooks in a later phase (target: F4 or F5 — out of scope for F1): - **`recall_memory` as a project-injected tool.** Opencode requires the user to wire pgvector/sqlite_vec via MCP. claude-hooks injects recall_memory directly into the tool list so the model can query memory without an MCP roundtrip. The opencoti tiers package already injects synthetic `__tier_*` tools (M5); the same mechanism extends to `__opencoti_recall_memory` cleanly. - **`survey_project` — hierarchical project map.** Composable from `glob` + `read`, but having one shot of "tell me the project shape" saves the model several turns. - **Grounding system-prompt prepend.** A short paragraph that frames the model's task in terms of "you're working in this project at cwd …". The M5 prompt-augmentation primitive (`augmentSystem`) is the right seam. - **Multi-role consultant orchestration.** opencode has `task` (launch one subagent), claude-hooks consultants has a 6-role LangGraph council (planner / researcher / critic / synthesizer / tool_executor / coder). The structured orchestration is the part worth porting; opencode's `task` is the underlying primitive it would build on. Likely a `opencoti-consultants` package consuming opencode's existing subagent + permission infrastructure. These ports are documented here only as the **direction** for later phases. F1 M5.7 (below) covers the configuration plumbing; F4/F5 plans, when written, will own the actual ports. ### "Effort" presets The `effort` knob is a one-line way for the user to pick how willing opencoti is to spend tokens. Conceptual mapping: | Effort | Behavior | | --- | --- | | `low` | Stay on Tier 0 unless hard-fail. Never fan-out. | | `balanced` | Default. Escalate on stuck-progress and structured-quality. Fan-out only on tier-config that explicitly opts in. | | `high` | More aggressive escalation, allows fan-out on Tier 1+. | | `max` | Fan-out on multiple tiers when configured; never silently degrade. | ### Subscription accounting Subscription depletion is observed per-provider: - Provider adapters expose a `subscriptionState()` returning `{ available: bool, resetsAt?: Date, reason?: string }`. - The router consults it before picking a sub-class provider in a pool. On `available: false`, it falls through to the next member (typically metered) until reset. - Adapters are responsible for parsing rate-limit / quota signals from their provider's responses. ## Milestones ### M1 — Schema + dry-run router *(done — 2026-05-21)* - [x] Tier config Zod schema in `packages/opencoti-tiers/src/config.ts` (`Effort`, `Billing`, `FanoutMode`, `EscalationTrigger`, `Provider`, `Tier`, `Escalation`, `TierConfig`). Strings normalize to providers via `z.preprocess`; defaults: `enabled=false`, `effort=balanced`, `escalation.on=[hard-fail]`, fanout `mode=first-good`. - [x] Provider-state model in `src/provider-state.ts` (`ProviderState`, `AVAILABLE`, `exhausted()`). - [x] Effort presets in `src/effort.ts` mapping `low|balanced|high|max` to `{ maxTiersToTry, allowFanout, defaultEscalationTriggers }`. - [x] Pure `plan()` function in `src/router.ts` that produces a `Plan` (ordered `PlanStep`s with `attempts`, `skipped`, `blockedByEffort`, `notes`). No real provider calls. - [x] 17 unit tests (`bun test`) cover: schema parsing + rejects (empty chain, unknown effort, unknown fanout mode), provider normalization (string → object, billing preserved), linear escalation order with subscription-first reordering, subscription-exhausted fallback (single and all-exhausted), fanout modes (`best-of-n`, `quorum`) preserving config order, effort presets (`low` caps to tier 0; `balanced` downgrades fanout to linear; `high` preserves fanout; explicit effort overrides config). - Typecheck (`tsc --noEmit`) clean. ### M2 — Surgical hook into opencode model call *(done — 2026-05-21)* - [x] Mapped the call site to `packages/opencode/src/session/llm.ts` (`run` Effect.fn around the `streamText({...})` call). The file already has a sister experimental-native runtime seam — opencoti's hook is its immediate neighbor, mirroring the same shape (one Effect.fn yield, one conditional early return). - [x] Added `packages/opencoti-tiers/src/runtime.ts` exposing `TierRuntime.maybeRoute(input): Effect` and `TierRuntime.isEnabledIn(cfg)`. M2 implementation is a passthrough (`Effect.succeed(undefined)`); M3 swaps it for the real router → adapter flow without touching upstream again. - [x] `TierRunResult` is typed to match opencode's `run` return shape (`{type: "ai-sdk"; result: ReturnType}` | `{type: "native"; stream: never}`). The `never` on the native stream keeps the inferred return-type union at the call site identical to upstream's — `never` is the absorbed element in discriminated-union narrowing. - [x] Surgical hook in `packages/opencode/src/session/llm.ts`: - import block (line ~29): `import * as TierRuntime from "@opencoti/tiers/runtime"` tagged with the `// opencoti-hook: tiered-inference — see docs/features/tiered_inference.md` marker. - call site (just before the `flags.experimentalNativeLlm` seam): `const tier = yield* TierRuntime.maybeRoute({ input, prepared, cfg }); if (tier) return tier` tagged with the same marker. - [x] `packages/opencode/package.json` gains `"@opencoti/tiers": "workspace:*"`. No comment marker is possible in JSON; the dep is registered in `docs/protocols/UPSTREAM_SYNC.md`. - [x] All three hook entries registered in the UPSTREAM_SYNC.md hook registry. - [x] `bun typecheck` from repo root: 17/17 packages green (including `opencode` with the new hook). - [x] `bun test` in `packages/opencoti-tiers`: 23/23 green, including 6 new tests for `TierRuntime.maybeRoute` + `TierRuntime.isEnabledIn`. Net hook footprint in upstream source: **one import line + a two-line conditional + one JSON dep entry** — the minimum that lets M3 wire real routing without ever editing `llm.ts` again. ### M3 — Tier 0 = llamafile end-to-end *(done — 2026-05-21)* - [x] `TierProvider` interface defined in `packages/opencoti-tiers/src/provider.ts` (`TierProvider`, `OpenedTierProvider`, `TierStreamRequest`). Types-only; no dependency on any concrete provider package — the dep direction is llamafile → tiers, not the other way around. - [x] Llamafile implementation in `packages/opencoti-llamafile/src/tier-provider.ts` (`tierProvider()` factory). Singleton process per host (lazy promise cache); uses `@ai-sdk/openai-compatible` to build a `LanguageModel` against the running llamafile's `/v1/chat/completions` endpoint. - [x] `TierRuntime.maybeRoute` replaced with real routing logic in `packages/opencoti-tiers/src/runtime.ts`: - Env gate (`OPENCOTI_TIERS_ENABLED=true`) — opt-in by design at M3. - Dynamic-imports `@opencoti/llamafile/tier-provider` via a string variable, so `@opencoti/tiers` has no static dep on llamafile (would be circular otherwise — llamafile already depends on tiers for the interface). - All failure modes (provider not installed, launch fails, open fails) fall through to upstream by returning `undefined`. The surgical hook never throws — it's forgiving by design. - Constructs `streamText` with the opened `LanguageModel` and the upstream `prepared` request (messages, system, params, abort). - [x] **No further edits to `packages/opencode/src/session/llm.ts`.** The M2 surgical hook is the last upstream touch needed for this feature. M3 lands entirely inside opencoti packages. - [x] Tests: 24 unit tests passing in `@opencoti/tiers` (env-gate-off, env-gate-on-but-no-llamafile, `isEnabledIn` variants); 2 unit tests in `@opencoti/llamafile` covering `tierProvider()` factory shape. Live `open()` covered by the smoke script (manual). **M3 known limitations** (intentional — addressed by later milestones): - Tool calling not wired through. M3 calls `streamText({model, messages, system, params})` without forwarding `prepared.tools`. Many local llamafile models can't tool-call usefully anyway; tool routing comes when M5 brings escalation triggers and tools matter. - Config is env-var-driven, not opencoti-config-driven. The user-facing tier config (effort, chain, escalation) defined in `@opencoti/tiers/config` is **not yet read** by M3's `maybeRoute` — M4 wires it in (and removes the env gate in favor of `opencoti.tiers.enabled` in the config file). - Single hardcoded llamafile tier. No escalation, no fan-out — those are M4, M5, M6. - Singleton llamafile process for host lifetime. If the user reconfigures the model mid-session, M3 ignores the new model. M3+ refines lifecycle. ### M4 — Tier 1 cloud fallback (linear) **[DONE 2026-05-21]** - One subscription provider + one metered fallback work end-to-end with subscription exhaustion observed correctly. - Implementation: - `cloud-provider.ts` — generic `TierProvider` over any OpenAI-compatible HTTP endpoint. Preflight GET `${baseURL}/models` classifies the response: 200 → AVAILABLE; 401/403 → exhausted (unauthorized); 402 → exhausted (payment-required); 429 → exhausted with `resetsAt` parsed from `Retry-After` (both numeric seconds and HTTP-date forms). State lives in the provider's closure and is exposed via `getState()`. - `registry.ts` — pure resolver mapping a config `Provider` entry to a concrete `TierProvider`. `kind: "llamafile"` (or string shorthand `"llamafile"` / `"llamafile/…"`) dynamic-imports `@opencoti/llamafile/tier-provider`; `kind: "openai-compat"` with `baseURL` + `modelId` builds a `cloudProvider`. - `executor.ts` — walks the `plan()` output across tiers and providers. On each iteration it collects state from each provider's `getState()` plus a module-level overlay (for `open()` failures that don't self-mark) and calls `plan()` again. A bounded loop (`totalProviders + 1`) prevents runaway replanning. - `runtime.ts` — env-var gate **removed**; replaced by `isEnabledIn(cfg)` reading `cfg.opencoti.tiers.enabled === true`, plus `safeParseConfig` on `cfg.opencoti.tiers`. The hook then delegates to `execute()` and wraps the returned `languageModel` in `streamText`. `maybeRouteWith` is a test seam letting tests inject an in-memory resolver. - Schema additions (in `config.ts`): provider entries now accept `kind: "llamafile" | "openai-compat"`, `baseURL`, `apiKey`, `apiKeyEnv` (preferred — read at runtime), `modelId`. Backwards compatible: existing string-shorthand entries (`"llamafile/…"`) still parse. - Test coverage: 47 tests across `config.test.ts`, `router.test.ts`, `runtime.test.ts`, `cloud-provider.test.ts`, `executor.test.ts`. Cloud preflight tests use `fetchImpl` injection — no live HTTP. - Zero upstream edits — M4 is entirely additive. - Out of scope for M4 (deferred to M5+): - **Cross-tier** escalation triggers (hard-fail, stuck-progress, structured-quality). M4 only covers within-tier sub→metered. - Forwarding `prepared.tools` through `streamText`. ### M5 — Escalation triggers v1 **[DONE 2026-05-21]** In/post-stream escalation observer wraps the M4 pre-stream router so the engine can react to bad turns and route the *next* turn to a higher tier. All four configured triggers (`hard-fail`, `stuck-progress`, `structured-quality`, `explicit`) work end-to-end. - Architecture (locked from planning conversation): - **Model-as-judge structured-quality**: the model's in-stream behavior IS the quality verdict — tool errors, no progress, explicit bail-out — NOT a parallel validator registry. The user's tools (test runners, language servers) are already the validators. - **Per-turn escalation unit**: one `streamText` invocation = one chance. Per-session counters survive across turns. When a threshold is met, the provider is overlay-marked; the next turn picks the next tier via the existing executor machinery. - **Distinct executor + escalator primitives** sharing the `stateOverlay`. Executor stays pre-stream (M4). Escalator wraps `streamText` and is in/post-stream. - **No mid-stream stream-swap** — mid-stream errors mark the provider exhausted and let the next turn re-route. Only pre-first-chunk hard-fail triggers a same-turn retry via a `restartStream` callback the runtime supplies. - Implementation: - `escalator.ts` — `wrapWithEscalation({result, opened, cfg, sessionID, telemetry, restartStream?}) → StreamTextResult`. Returns a `Proxy` that overrides only `fullStream`; the wrapped `fullStream` is an async generator that tees events into a `TurnObserver` and re-yields them. The `TurnObserver` counts tool-call / tool-result / tool-error / error events, evaluates triggers at stream finish, calls `_markEscalated(providerName, "trigger:reason")` from `executor.ts` when a threshold is hit, and emits telemetry. - `session-state.ts` — `Map` with 30-min TTL eviction on read. Fields: `stuckTurns`, `qualityFails`, `lastDifficulty`, `lastTier`, `lastAccess`, `explicitEscalationPending`. - `synthetic-tools.ts` — `syntheticTools(sessionID, telemetry?)` returns `{ __tier_report_difficulty, __tier_request_escalation }` as AI-SDK `Tool` objects with `inputSchema` (zod) and `execute` handlers that record into session state and emit telemetry. Both names start with `__tier_` so any consumer can filter them. - `prompt-augmentation.ts` — `augmentSystem(systemLines, enabled)` appends one narrow paragraph telling the model about the two synthetic tools when `enabled` is true. - `telemetry.ts` — `Telemetry` interface with 7 methods (`attemptStart`, `attemptSuccess`, `exhaustionObserved`, `escalationTriggered`, `difficultyReported`, `requestEscalationCalled`, `sessionStateUpdated`) + a `noopTelemetry` default. F3 (`opencoti-server`) will eventually implement this interface; M5 ships the schema only. - `executor.ts` extensions: `ExecuteResult.tierId` (so the escalator knows which tier opened the stream); `telemetry?` in `ExecutorOptions`; `_markEscalated(name, reason)` exported package-internal; telemetry calls at `attemptStart` / `exhaustionObserved` branch points. - `runtime.ts` extensions: merges `prepared.tools` with `syntheticTools(sessionID, telemetry)` (the M3/M4 known limitation is gone); replaces `system: ...join("\n")` with `augmentSystem(...).join("\n")`; provides a `restartStream` callback to the escalator that re-calls `execute()` (the just-marked provider is skipped automatically). - Schema additions in `config.ts`: `escalation.qualityFails.threshold` (defaults to 2). - Trigger semantics: - `hard-fail`: pre-first-chunk error → silently restart THIS turn once via `restartStream`. Mid-stream errors → mark exhausted, escalate NEXT turn. User-AbortSignal abort is excluded. - `stuck-progress`: turn finishes with zero tool calls AND `lastDifficulty != "low"` → increments `stuckTurns`. Escalates when `stuckTurns >= cfg.escalation.stuckProgress.noProgressTurns`. A productive turn (≥1 tool call) resets the counter. - `structured-quality`: any `tool-error` event → increments `qualityFails`. Escalates when `qualityFails >= cfg.escalation.qualityFails.threshold`. A clean turn with ≥1 successful tool result resets the counter. - `explicit`: any call to `__tier_request_escalation` (model bailout) OR an `explicitEscalationPending` flag from a prior turn → immediate escalation regardless of threshold. - Test coverage: 33 new M5 tests (`escalator.test.ts`, `session-state.test.ts`, `synthetic-tools.test.ts`, `prompt-augmentation.test.ts`, `telemetry.test.ts`, + 3 extensions to `runtime.test.ts`). Total `@opencoti/tiers`: 80 tests, all green. Workspace typecheck clean. - Zero upstream edits — M5 is entirely additive in `packages/opencoti-tiers/`. Hook footprint unchanged: still the 2 M2 markers in `packages/opencode/src/session/llm.ts`. ### M5.5 — Effort-level + tiermode schema expansion **[DONE 2026-05-21]** Pure-schema milestone, no runtime change beyond what the new config shape brings. - **Effort enum**: replaced `low|balanced|high|max` with a 5-base set `low|normal|medium|high|max` plus 5 x-prefixed fanout opt-ins `xlow|xnormal|xmedium|xhigh|xmax`. The `x` prefix marks "fan-out at this tier"; aggression is the base level, fanout is the orthogonal bit. No `balanced` legacy alias — pre-1.0, follow CLAUDE.md's "don't keep backwards-compatibility shims" rule. - **`effort.ts` constraints table** rebuilt as `5 × 2` grid: - `low` / `xlow`: cap at one tier; `hard-fail` trigger only. - `normal` / `xnormal`: walk all tiers; `+ stuck-progress`. - `medium` / `xmedium`: walk all tiers; `+ structured-quality`. - `high` / `xhigh`: same trigger set as medium; reserved for later threshold-tuning differentiation. - `max` / `xmax`: same trigger set, most-aggressive thresholds. - `allowFanout` is `false` for the bare base levels, `true` for the `x*` opt-ins. (Behavior change: old `high`/`max` enabled fanout; that capability moved to `xhigh`/`xmax`.) - **Tiermode**: new top-level config field. `tiermodes:` is a map of named profiles; each profile may override `effort`, per-tier `topologyOverrides`, or a partial `escalation` overlay. The top-level `tiermode: string` field points into the map and is validated at parse time (refinement: name must exist). - **`tiermode.ts` resolver**: `resolveTiermode(cfg) → ResolvedTierConfig` pure function. Returns the effective `{ effort, topologyOverrides, escalation, config }` after profile merge. Router and runtime do NOT consume it yet (M5.5 is pure schema); the resolver is package- exported so M6/M7 wiring is trivial. - **Tests**: total `@opencoti/tiers` test count up from 80 to 107 (+27 net: 10 new + parameterized parsing across all 10 effort values + 5 tiermode parsing + 6 resolver tests; 2 existing fanout- tests migrated from `high`/`max` to `xhigh`/`xmax`). Workspace typecheck clean. Zero new upstream surgical hooks. ### M5.7 — Tier-0 provider configuration + model registry **[DONE 2026-05-21]** User-facing first-run setup and a small project-local model registry. Lands after M5.5 (schema is final by then) and before M6 (fan-out needs a working multi-provider Tier 0 to test against). No runtime change beyond what config-read needs. - **Model registry.** A `models:` map in the config (see Config-shape example above) maps short names (`gemma4-a4b-coder`, `gemma4-31b-cloud`, …) to provider entries. The `chain[].providers` field references those names. Same idea as claude-hooks' provider registry. Schema lives in `packages/opencoti-tiers/src/config.ts`. - **First-run setup CLI** (`opencoti tier setup` or equivalent — exact subcommand TBD, runs from the standard opencode CLI surface): 1. GPU detection. Reads `nvidia-smi`, `rocm-smi`, or an equivalent probe. Reports detected VRAM. 2. **GPU path branch.** Prompts: - Confirm Gemma 4 A4B-98e-v5-coder as the Tier-0 model (yes by default, with the option to point to a different GGUF). - Quant pick — IQ3_M default for ≥10 GB VRAM, IQ4_XS or CD-IQ4_K_M for ~11 GB, Q4_K_S for ~12 GB, Q5_K_L for ~15 GB, Q8_0 for 24 GB. Recommended pick is computed from detected VRAM and shown as the default. - GGUF source — three options: - **Download from HuggingFace.** Downloads to `$XDG_DATA_HOME/opencoti/models/` (`~/.local/share/opencoti/models/` on Linux default) using the user's HF auth if any. - **Use existing file.** User points at a GGUF path. Validated (magic bytes + metadata read) before being recorded. - **Import from local Ollama.** If `~/.ollama/models/` is detected, parse manifests under `~/.ollama/models/manifests/registry.ollama.ai/library//` to find the blob digest; the patched llamafile mounts `~/.ollama/models/blobs/sha256-` directly. **Single download, zero duplication.** 3. **No-GPU path branch.** Prompts: - Confirm Ollama Cloud as the Tier-0 provider. - Recommend `gemma4:31b` (free BF16 on Ollama Cloud at time of writing); allow override. - Capture `OLLAMA_CLOUD_API_KEY` via env-var reference (never stored in config — `apiKeyEnv` indirection). 4. **No silent auto-fallback.** If GPU detection fails and the user hasn't picked a path, setup exits with a clear next-step prompt. No defaulting to cloud. 5. Writes the chosen entries into the project's opencoti config (`opencoti.tiers.models[...]` and `chain[0].providers[0]`). - **Ollama local registry detection.** Helper module reads the manifest JSON, follows the `digest` field to the blobs/ entry, returns `{path, sha256, sizeBytes}`. Pure read-only — never modifies the Ollama store. Tests use a fixture `~/.ollama/` tree. - **Reuse pattern in llamafile launcher.** When a registry entry points at an Ollama blob (kind: `llamafile`, gguf path inside `~/.ollama/`), the launcher passes the blob path directly to the patched llamafile — no copy, no symlink, just read. Verified by a smoke test. - **Test coverage:** new test files in `packages/opencoti-tiers/test/` (`model-registry.test.ts`, `ollama-detect.test.ts`) and `packages/opencoti-llamafile/test/` (`ollama-blob-reuse.test.ts`). GPU detection itself is shimmed in tests (the real probe lives outside `@opencoti/tiers` to keep the package testable in CI). Zero new upstream surgical hooks. M5.7 is entirely additive in opencoti packages + a new CLI subcommand exposed via opencode's existing command surface. **M5.7 implementation notes (what actually shipped):** - Schema: `ModelEntry` + `models: Record` on `TierConfig`. Refinements enforce "llamafile requires exactly one of `gguf`/`ollamaBlob`" and "openai-compat requires `baseURL`". - `src/registry/ollama-detect.ts` — read-only manifest walker (`registry.ollama.ai///`), follows the GGUF layer digest to `~/.ollama/models/blobs/sha256-…`. Tolerant of malformed manifests. - `src/registry/gpu-detect.ts` — async `detectGPU()` probes `nvidia-smi --query-gpu=...` first, falls back to `rocm-smi --showmeminfo vram --json`. 5 s default timeout. `runImpl` test seam. - `src/registry/quant-recommender.ts` — pure table-driven picker encoding the Gemma 4 A4B-98e quant table from `local_llamafile.md`. Reserves ~3 GB KV per 32 K context + 1.5 GB safety; picks highest-quality (HE+) quant that fits. - `src/registry/hf-download.ts` — minimal HF Hub HTTP client. Resume via `${destPath}.partial` + `Range: bytes=N-`, optional `$HF_TOKEN` auth, SHA256 verify (deletes the partial on mismatch), per-chunk progress callback. - `src/cli/setup-flow.ts` — pure-logic flow with injected deps (`SetupDeps`). `runSetupFlow(deps)` returns `cancelled` or `configured` with the `ConfigPatch` to write. Testable via scripted-answers stub deps. - `src/cli/setup.ts` — live wiring: @clack/prompts for input, `jsonc-parser` `modify`/`applyEdits` for comment-preserving config write to `~/.config/opencode/opencode.jsonc`. Exports `tierCommand` (yargs parent) and `tierSetupCommand` (subcommand). - Surgical hook in `packages/opencode/src/index.ts`: 1 import + 1 `.command(tierCommand)` call, both marker-tagged. Both registered in `docs/protocols/UPSTREAM_SYNC.md`. - `package.json`: adds `@clack/prompts`, `jsonc-parser`, `yargs` as deps (same pinned versions opencode uses) and `@types/yargs` as devDep. New `./cli` export. - Test coverage: 35 net new tests across 5 new files (`ollama-detect.test.ts` 6, `gpu-detect.test.ts` 6, `quant-recommender.test.ts` 10, `hf-download.test.ts` 7, `cli/setup-flow.test.ts` 8). Total `@opencoti/tiers`: 144 tests. Workspace typecheck clean. Hook conformance: exactly 4 markers (2 M2 + 2 M5.7). ### M6 — Fan-out tier (first-good) **[DONE 2026-05-21]** Goal: when the plan picks a fanout tier (xhigh / xmax effort, or explicit `tiermode` override), open every provider in that tier in parallel and let the fastest useful response through. Losers are aborted; losers that fail before being aborted are overlay-marked the same way the M5 linear path marks them. Behavior knobs the user locked this turn: - **`winnerTrigger` (per tier)**: `"content"` (default) or `"finish"`. `content` races on the first real-content event (same predicate the M5 escalator uses — `text-delta`, `tool-call`, `tool-result`, `file`, etc.). `finish` races on the first racer to complete a clean turn. - **Pre-trigger loser failures** → overlay-marked via `_markEscalated(name, "fanout-prefail")`. Aborts-by-us are NOT marked (the racer was cut off through no fault of its own — tracked via a `abortedByUs` flag). - **`maxConcurrency` (per tier, optional)**: caps how many providers actually race. The rest are recorded in `skippedByConcurrencyCap` on the executor result and are NOT overlay-marked — they just sit out this turn. The fanout `mode` enum was expanded to include `"synthesizer"` and `"synthesizer-critic"` for forward-compatibility with the consultants- council pattern (Tier 0 as synthesizer + separate critic model). M6 ships `"first-good"` only; `"best-of-n"`, `"quorum"`, `"synthesizer"`, and `"synthesizer-critic"` throw a typed `FanoutModeNotImplemented` error that the runtime catches as a forgiving fall-through. Implementation surface (all additive in `packages/opencoti-tiers/`): - `src/config.ts` — `FanoutTier` gets `winnerTrigger` + `maxConcurrency`; `FanoutMode` enum expanded. - `src/executor.ts` — `executeFanout()` opens every router-emitted attempt for the first fanout step in parallel via `Promise.all`, applying `maxConcurrency` and overlay-marking open-failures the same way `execute()` does. - `src/fanout.ts` (new) — `raceFanout()` runs N per-racer drain goroutines, exposes `{proxy, outcome, settled}`. The proxy is a `Proxy` whose `fullStream` getter replays the winner's buffered pre-trigger events then continues from the underlying iterator. Per-racer buffer cap (default 4 MB) catches pathological non-content streams. - `src/runtime.ts` — `peekTopology()` decides linear vs fanout branch from `plan()`. The fanout branch builds per-racer streamText invocations with child `AbortController`s, calls `raceFanout`, then wraps the proxy with `wrapWithEscalation` so M5's escalation triggers apply on the winner's stream. The `restartStream` callback handles the `all-failed` outcome the same way M5's hard-fail-pre-stream path does. - `src/escalator.ts` — `isRealContent` was exported so `fanout.ts` shares the exact same predicate. Test coverage: 20 net new tests across `test/{config,executor,runtime,fanout}.test.ts`. Total `@opencoti/tiers` count: **168 tests**. Zero new surgical hooks. Hook count stays at 4 (2 M2 + 2 M5.7). ### M7 — TUI effort + tiermode pickers **[DONE 2026-05-21]** Goal: in-TUI dials for the M5.5 `effort` (10 values) and `tiermode` (named profiles) so users can change them mid-session without re-editing `~/.config/opencode/opencode.jsonc`. Plus a runtime-side override seam so a TUI pick actually takes effect on the very next request, plus auto-wire of the new picker plugin into the M5.7 setup CLI. Three additive surfaces: - **`@opencoti/tiers/src/active-config.ts`** — module-level overlay (`setActiveEffort` / `setActiveTiermode` / `getActiveOverride`) matching the existing `stateOverlay` pattern. Both the TUI plugin and the M2 hook run in the same Bun process, so module-level singleton state is genuinely shared. - **`runtime.ts` overlay + tiermode wiring** — `routeImpl` now merges the overlay onto the parsed config, calls `resolveTiermode` (which M5.5 left unwired), patches each tier's topology per the resolver output, and forwards the effective effort to the executor via `ExecutorOptions.effort`. Precedence: **TUI overlay > tiermode profile > base `cfg.effort`**. - **`@opencoti/tui-tiers`** — new sibling package. Ships an opencode TUI plugin (`TuiPluginModule`) that registers two commands (`opencoti.effort`, `opencoti.tiermode`), opens `DialogSelect`s, and persists picks via `api.kv` under a per-project key (`opencoti..{effort|tiermode}`) so each project remembers its own selection. On plugin start, the persisted values replay into the runtime overlay — TUI restart preserves the last pick. Setup CLI extension (M5.7 → M7): - `setup-flow.ts` got a `tuiPickerInstalled` sentinel on `SetupDeps` and an `installTuiPicker` boolean on `ConfigPatch`. After the GPU/no-GPU branch builds its patch, a `finalizePatch` helper asks "Install the TUI picker plugin?" (skipped when the sentinel is true — re-runs don't re-prompt). - The live `setup.ts` reads the sentinel from the existing config (best-effort `parseTree` + `findNodeAtLocation`) and pre-populates `tuiPickerInstalled` on the live deps. When `installTuiPicker` is true on the patch, the writer appends `"@opencoti/tui-tiers"` to the user's `plugin: []` (deduped against the existing array) and flips `opencoti.tiers.tuiPickerInstalled = true`. Schema: `TierConfig` got one new optional sentinel field `tuiPickerInstalled: z.boolean().default(false)`. Pure schema — defining it in Zod makes a typo a clear error instead of silent. Test coverage: **16 net new tests** across `@opencoti/tiers` (active-config, runtime, executor, setup-flow) + **12 tests** in the new `@opencoti/tui-tiers` package. Totals: tiers **184 tests**, tui-tiers **12 tests**, workspace 18/18 typecheck clean. Zero new surgical hooks. The TUI plugin loads via opencode's existing `plugin: []` discovery path (the setup-flow auto-wire writes the entry). Hook footprint stays at 4 markers. **Behavior change worth noting**: `resolveTiermode` was unwired since M5.5. Wiring it in M7 means users who set `tiermode` in their config — silently ignored pre-M7 — now actually see the profile apply. This is a fix, not a regression, but anyone debugging tiermode behavior should know the activation date is 2026-05-21. ### M8 — Structured-quality validators v1 - Pluggable validators (code-parses, tests-pass-or-improve). Trigger escalation when validator fails N times on the current tier. ## Open questions - ~~Where exactly is the upstream call site?~~ Resolved at M2: `packages/opencode/src/session/llm.ts`, just before the existing experimental-native-llm seam. - **Streaming under fan-out.** First-good is fine; quorum and best-of-n are not naturally streaming. Defer their UX to M6+. - **Cost accounting.** Should opencoti show per-tier token spend? Probably yes, but it's not on the critical path. The M5 telemetry interface already carries the data; F3's web dashboard is the obvious place to surface it. - **Capability negotiation between escalator and provider.** The patched llamafile exposes capabilities the no-GPU path lacks (KV reuse, big-context throughput). Today the executor doesn't ask providers what they can do — it just opens them. M5.7+ may want to add a `capabilities()` method to `TierProvider` so the escalator can know whether KV-prefix-reuse is on the table for the next turn. Not blocking; revisit when the second patched-llamafile feature lands. ## Risks - The opencode provider abstraction may not factor cleanly enough for a single hook. If so, we widen the hook to a small adapter shim in the upstream call site, and keep the adapter implementation in `opencoti-tiers`. The hook stays small; the implementation stays ours. - Provider rate-limit signal parsing is fragile. We isolate it per provider adapter so a broken adapter never poisons the router.