docs/features/tiered_inference.md

F1 — Tiered Inference Engine

Parent: ../MASTER_PLAN.md Status: planning Owner: TBD

Problem

A single provider is never the right answer for a real coding agent:

• A small local model is cheap and private but stalls on hard tasks.
• A subscription provider (Ollama Cloud Gemma 4 31B, Claude Max, ChatGPT Plus, etc.) has caps; you don't want every token to count against them.
• A per-token provider (Anthropic API, OpenAI API, etc.) works but charges for everything it sees.

We want opencoti to start work on the cheapest provider that can plausibly handle it, and escalate only when needed.

Tier 0 vision — primary local execution, two paths

Tier 0 is the role-label for "the cheap-and-fast tier we start every turn on." It is not hard-wired to a specific runtime. opencoti supports two first-class Tier-0 realizations and the user picks one explicitly at first-run setup (no silent auto-fallback):

• GPU path — the advanced one. Patched llamafile + ManniX-ITA/gemma-4-A4B-98e-v5-coder-it (Gemma 4 26B-A4B base, 98-expert prune of the original 128, ~4 B active per token, 256 K context max, IQ3_M ≈ 9.82 GB, runs end-to-end on a 24 GB GPU with full offload). The patched llamafile is what opens up the advanced features below.
• No-GPU path — the simple one. Ollama Cloud as an OpenAI-compatible Tier-0 provider with the Gemma 4 31B dense model (free BF16 on Ollama Cloud at time of writing). Plain HTTP request flow, no local compute, no advanced features. Equally first-class for users without a GPU.

Anything else the OpenAI-compatible provider interface accepts works too (a remote vLLM, a different Ollama endpoint, etc.). Tier 0 is the abstraction; the realizations are config-driven.

Smaller Gemma 4 variants (E2B, E4B) and the stock Gemma 3 lineage are explicitly not primary targets — evaluated much later if at all.

Advanced features tied to the GPU path

The patched-llamafile path unlocks features the no-GPU path cannot offer. opencoti does not expose this distinction in user-facing configuration: features that need the patched llamafile are present automatically when the GPU path is selected, and silently absent on the no-GPU path. From the user's perspective Tier 0 is Tier 0; internally the engine queries provider capabilities and degrades gracefully.

Concretely (see local_llamafile.md for detail):

• KV-cache reuse across agentic turns — turn N+1 reuses turn N's prefix, dramatic latency drop on long contexts.
• Big-context throughput patches — keep tokens/sec usable at 64K+ contexts.
• Single-download Ollama-blob reuse — if Ollama is installed locally, the patched llamafile can mount the existing ~/.ollama/models/blobs/sha256-… GGUF directly. No duplicate download.

Goals

• G1. Configure an ordered set of tiers (Tier 0, Tier 1, ...), each with one or more providers.
• G2. Run a request on the lowest-numbered tier first; escalate on failure-to-progress, hard error, or explicit escalation signal.
• G3. Support two tier topologies:
  • Linear — try a single provider in this tier, fall through on failure.
  • Fan-out — try multiple providers in this tier in parallel (or as a quorum), pick the best response.
• G4. Within a tier, prefer subscription providers; fall back to metered providers when subs are exhausted or rate-limited.
• G5. Make the escalation policy a user-visible config, not a hidden heuristic.

Non-goals (for now)

• Reordering tiers automatically based on observed quality.
• Cross-session learning of provider quality.
• Coordinating tiers across multiple concurrent sessions.

Design sketch

Where it lives

• packages/opencoti-tiers/ — new package. Holds the tier schema, the router, the escalation policy, the provider-pool primitives.
• Surgical hook in opencode's model-call site: when an opencoti tier config is present and enabled, the call is delegated to the router from opencoti-tiers. Otherwise opencode behaves as upstream.

Concepts

• Tier = an ordered slot in the escalation chain. Tier 0 is the cheapest, tried first.
• Provider Pool inside a tier = one of:
  • linear: [Provider, Provider, ...] — tried in order
  • fanout: { mode: "first-good" | "best-of-n" | "quorum", members: [...] }
• Billing class per provider:
  • subscription — has a cap; depleted state is observable.
  • metered — pay per token, no cap.
• Escalation triggers (configurable, ORed):
  • hard-fail (provider error, timeout, refused)
  • stuck-progress (no tool calls / no diff after N turns)
  • explicit (user keypress, or model emits an "escalate" tool call)
  • structured-quality (output fails a structured validator — e.g., code did not parse, tests still failing after N attempts)

Config shape (illustrative, not final)

GPU-path example:

opencoti:
  tiers:
    enabled: true
    effort: balanced     # low | balanced | high | max
    models:
      # Small local registry — names → entries. Same idea as
      # claude-hooks' provider registry. Entries are matched by name
      # from the `chain` below.
      gemma4-a4b-coder:
        kind: llamafile
        gguf: ~/.opencoti/models/gemma-4-A4B-98e-v5-coder-it-IQ3_M.gguf
        contextLength: 131072    # capped; model supports up to 256K
      gemma4-31b-cloud:
        kind: openai-compat
        baseURL: https://ollama.com/api
        apiKeyEnv: OLLAMA_CLOUD_API_KEY
        modelId: gemma4:31b
        billing: subscription
    chain:
      - id: tier0
        topology: linear
        providers:
          - gemma4-a4b-coder       # patched llamafile, advanced features
      - id: tier1
        topology: linear
        providers:
          - gemma4-31b-cloud       # subscription (Ollama Cloud)
          - groq/llama-3.3-70b     # metered, fallback
      - id: tier2
        topology: fanout
        mode: first-good
        providers:
          - anthropic/claude-sonnet-4-6      # subscription via Max
          - anthropic-api/claude-sonnet-4-6  # metered, fallback
    escalation:
      on: [hard-fail, stuck-progress, structured-quality]
      stuck-progress:
        no-progress-turns: 3
      structured-quality:
        validators: [code-parses, tests-pass-or-improve]

No-GPU example (Tier 0 is the cloud provider; no llamafile present):

opencoti:
  tiers:
    enabled: true
    effort: balanced
    models:
      gemma4-31b-cloud:
        kind: openai-compat
        baseURL: https://ollama.com/api
        apiKeyEnv: OLLAMA_CLOUD_API_KEY
        modelId: gemma4:31b
        billing: subscription
    chain:
      - id: tier0
        topology: linear
        providers:
          - gemma4-31b-cloud
      - id: tier1
        # ... user's preferred escalation provider ...

Tool surface

opencode already injects a strong default tool set into every streamText call (15 built-in: shell, read, write, edit, glob, grep, task, fetch, search, todo, skill, patch, question, lsp, invalid), plus full MCP server integration, plus plugin / .opencode/tool/* user-defined tools, all gated by the same permission flow. That surface already meets-or-exceeds what claude-hooks' caliber-grounding-proxy + consultants skill expose (survey_project, list_files, read_file, glob, grep, recall_memory, and the sandboxed write_file for the coder role).

What opencode does NOT have today, and that opencoti will port from claude-hooks in a later phase (target: F4 or F5 — out of scope for F1):

• recall_memory as a project-injected tool. Opencode requires the user to wire pgvector/sqlite_vec via MCP. claude-hooks injects recall_memory directly into the tool list so the model can query memory without an MCP roundtrip. The opencoti tiers package already injects synthetic __tier_* tools (M5); the same mechanism extends to __opencoti_recall_memory cleanly.
• survey_project — hierarchical project map. Composable from glob + read, but having one shot of "tell me the project shape" saves the model several turns.
• Grounding system-prompt prepend. A short paragraph that frames the model's task in terms of "you're working in this project at cwd …". The M5 prompt-augmentation primitive (augmentSystem) is the right seam.
• Multi-role consultant orchestration. opencode has task (launch one subagent), claude-hooks consultants has a 6-role LangGraph council (planner / researcher / critic / synthesizer / tool_executor / coder). The structured orchestration is the part worth porting; opencode's task is the underlying primitive it would build on. Likely a opencoti-consultants package consuming opencode's existing subagent + permission infrastructure.

These ports are documented here only as the direction for later phases. F1 M5.7 (below) covers the configuration plumbing; F4/F5 plans, when written, will own the actual ports.

"Effort" presets

The effort knob is a one-line way for the user to pick how willing opencoti is to spend tokens. Conceptual mapping:

Effort	Behavior
low	Stay on Tier 0 unless hard-fail. Never fan-out.
balanced	Default. Escalate on stuck-progress and structured-quality. Fan-out only on tier-config that explicitly opts in.
high	More aggressive escalation, allows fan-out on Tier 1+.
max	Fan-out on multiple tiers when configured; never silently degrade.

Subscription accounting

Subscription depletion is observed per-provider:

• Provider adapters expose a subscriptionState() returning { available: bool, resetsAt?: Date, reason?: string }.
• The router consults it before picking a sub-class provider in a pool. On available: false, it falls through to the next member (typically metered) until reset.
• Adapters are responsible for parsing rate-limit / quota signals from their provider's responses.

Milestones

M1 — Schema + dry-run router (done — 2026-05-21)

• Tier config Zod schema in packages/opencoti-tiers/src/config.ts (Effort, Billing, FanoutMode, EscalationTrigger, Provider, Tier, Escalation, TierConfig). Strings normalize to providers via z.preprocess; defaults: enabled=false, effort=balanced, escalation.on=[hard-fail], fanout mode=first-good.
• Provider-state model in src/provider-state.ts (ProviderState, AVAILABLE, exhausted()).
• Effort presets in src/effort.ts mapping low|balanced|high|max to { maxTiersToTry, allowFanout, defaultEscalationTriggers }.
• Pure plan() function in src/router.ts that produces a Plan (ordered PlanSteps with attempts, skipped, blockedByEffort, notes). No real provider calls.
• 17 unit tests (bun test) cover: schema parsing + rejects (empty chain, unknown effort, unknown fanout mode), provider normalization (string → object, billing preserved), linear escalation order with subscription-first reordering, subscription-exhausted fallback (single and all-exhausted), fanout modes (best-of-n, quorum) preserving config order, effort presets (low caps to tier 0; balanced downgrades fanout to linear; high preserves fanout; explicit effort overrides config).
• Typecheck (tsc --noEmit) clean.

M2 — Surgical hook into opencode model call (done — 2026-05-21)

• Mapped the call site to packages/opencode/src/session/llm.ts (run Effect.fn around the streamText({...}) call). The file already has a sister experimental-native runtime seam — opencoti's hook is its immediate neighbor, mirroring the same shape (one Effect.fn yield, one conditional early return).
• Added packages/opencoti-tiers/src/runtime.ts exposing TierRuntime.maybeRoute(input): Effect<TierRunResult | undefined> and TierRuntime.isEnabledIn(cfg). M2 implementation is a passthrough (Effect.succeed(undefined)); M3 swaps it for the real router → adapter flow without touching upstream again.
• TierRunResult is typed to match opencode's run return shape ({type: "ai-sdk"; result: ReturnType<typeof streamText>} | {type: "native"; stream: never}). The never on the native stream keeps the inferred return-type union at the call site identical to upstream's — never is the absorbed element in discriminated-union narrowing.
• Surgical hook in packages/opencode/src/session/llm.ts:
  • import block (line ~29): import * as TierRuntime from "@opencoti/tiers/runtime" tagged with the // opencoti-hook: tiered-inference — see docs/features/tiered_inference.md marker.
  • call site (just before the flags.experimentalNativeLlm seam): const tier = yield* TierRuntime.maybeRoute({ input, prepared, cfg }); if (tier) return tier tagged with the same marker.
• packages/opencode/package.json gains "@opencoti/tiers": "workspace:*". No comment marker is possible in JSON; the dep is registered in docs/protocols/UPSTREAM_SYNC.md.
• All three hook entries registered in the UPSTREAM_SYNC.md hook registry.
• bun typecheck from repo root: 17/17 packages green (including opencode with the new hook).
• bun test in packages/opencoti-tiers: 23/23 green, including 6 new tests for TierRuntime.maybeRoute + TierRuntime.isEnabledIn.

Net hook footprint in upstream source: one import line + a two-line conditional + one JSON dep entry — the minimum that lets M3 wire real routing without ever editing llm.ts again.

M3 — Tier 0 = llamafile end-to-end (done — 2026-05-21)

• TierProvider interface defined in packages/opencoti-tiers/src/provider.ts (TierProvider, OpenedTierProvider, TierStreamRequest). Types-only; no dependency on any concrete provider package — the dep direction is llamafile → tiers, not the other way around.
• Llamafile implementation in packages/opencoti-llamafile/src/tier-provider.ts (tierProvider() factory). Singleton process per host (lazy promise cache); uses @ai-sdk/openai-compatible to build a LanguageModel against the running llamafile's /v1/chat/completions endpoint.
• TierRuntime.maybeRoute replaced with real routing logic in packages/opencoti-tiers/src/runtime.ts:
  • Env gate (OPENCOTI_TIERS_ENABLED=true) — opt-in by design at M3.
  • Dynamic-imports @opencoti/llamafile/tier-provider via a string variable, so @opencoti/tiers has no static dep on llamafile (would be circular otherwise — llamafile already depends on tiers for the interface).
  • All failure modes (provider not installed, launch fails, open fails) fall through to upstream by returning undefined. The surgical hook never throws — it's forgiving by design.
  • Constructs streamText with the opened LanguageModel and the upstream prepared request (messages, system, params, abort).
• No further edits to packages/opencode/src/session/llm.ts. The M2 surgical hook is the last upstream touch needed for this feature. M3 lands entirely inside opencoti packages.
• Tests: 24 unit tests passing in @opencoti/tiers (env-gate-off, env-gate-on-but-no-llamafile, isEnabledIn variants); 2 unit tests in @opencoti/llamafile covering tierProvider() factory shape. Live open() covered by the smoke script (manual).

M3 known limitations (intentional — addressed by later milestones):

• Tool calling not wired through. M3 calls streamText({model, messages, system, params}) without forwarding prepared.tools. Many local llamafile models can't tool-call usefully anyway; tool routing comes when M5 brings escalation triggers and tools matter.
• Config is env-var-driven, not opencoti-config-driven. The user-facing tier config (effort, chain, escalation) defined in @opencoti/tiers/config is not yet read by M3's maybeRoute — M4 wires it in (and removes the env gate in favor of opencoti.tiers.enabled in the config file).
• Single hardcoded llamafile tier. No escalation, no fan-out — those are M4, M5, M6.
• Singleton llamafile process for host lifetime. If the user reconfigures the model mid-session, M3 ignores the new model. M3+ refines lifecycle.

M4 — Tier 1 cloud fallback (linear) [DONE 2026-05-21]

• One subscription provider + one metered fallback work end-to-end with subscription exhaustion observed correctly.
• Implementation:
  • cloud-provider.ts — generic TierProvider over any OpenAI-compatible HTTP endpoint. Preflight GET ${baseURL}/models classifies the response: 200 → AVAILABLE; 401/403 → exhausted (unauthorized); 402 → exhausted (payment-required); 429 → exhausted with resetsAt parsed from Retry-After (both numeric seconds and HTTP-date forms). State lives in the provider's closure and is exposed via getState().
  • registry.ts — pure resolver mapping a config Provider entry to a concrete TierProvider. kind: "llamafile" (or string shorthand "llamafile" / "llamafile/…") dynamic-imports @opencoti/llamafile/tier-provider; kind: "openai-compat" with baseURL + modelId builds a cloudProvider.
  • executor.ts — walks the plan() output across tiers and providers. On each iteration it collects state from each provider's getState() plus a module-level overlay (for open() failures that don't self-mark) and calls plan() again. A bounded loop (totalProviders + 1) prevents runaway replanning.
  • runtime.ts — env-var gate removed; replaced by isEnabledIn(cfg) reading cfg.opencoti.tiers.enabled === true, plus safeParseConfig on cfg.opencoti.tiers. The hook then delegates to execute() and wraps the returned languageModel in streamText. maybeRouteWith is a test seam letting tests inject an in-memory resolver.
• Schema additions (in config.ts): provider entries now accept kind: "llamafile" | "openai-compat", baseURL, apiKey, apiKeyEnv (preferred — read at runtime), modelId. Backwards compatible: existing string-shorthand entries ("llamafile/…") still parse.
• Test coverage: 47 tests across config.test.ts, router.test.ts, runtime.test.ts, cloud-provider.test.ts, executor.test.ts. Cloud preflight tests use fetchImpl injection — no live HTTP.
• Zero upstream edits — M4 is entirely additive.
• Out of scope for M4 (deferred to M5+):
  • Cross-tier escalation triggers (hard-fail, stuck-progress, structured-quality). M4 only covers within-tier sub→metered.
  • Forwarding prepared.tools through streamText.

M5 — Escalation triggers v1 [DONE 2026-05-21]

In/post-stream escalation observer wraps the M4 pre-stream router so the engine can react to bad turns and route the next turn to a higher tier. All four configured triggers (hard-fail, stuck-progress, structured-quality, explicit) work end-to-end.

• Architecture (locked from planning conversation):
  • Model-as-judge structured-quality: the model's in-stream behavior IS the quality verdict — tool errors, no progress, explicit bail-out — NOT a parallel validator registry. The user's tools (test runners, language servers) are already the validators.
  • Per-turn escalation unit: one streamText invocation = one chance. Per-session counters survive across turns. When a threshold is met, the provider is overlay-marked; the next turn picks the next tier via the existing executor machinery.
  • Distinct executor + escalator primitives sharing the stateOverlay. Executor stays pre-stream (M4). Escalator wraps streamText and is in/post-stream.
  • No mid-stream stream-swap — mid-stream errors mark the provider exhausted and let the next turn re-route. Only pre-first-chunk hard-fail triggers a same-turn retry via a restartStream callback the runtime supplies.
• Implementation:
  • escalator.ts — wrapWithEscalation({result, opened, cfg, sessionID, telemetry, restartStream?}) → StreamTextResult. Returns a Proxy<StreamTextResult> that overrides only fullStream; the wrapped fullStream is an async generator that tees events into a TurnObserver and re-yields them. The TurnObserver counts tool-call / tool-result / tool-error / error events, evaluates triggers at stream finish, calls _markEscalated(providerName, "trigger:reason") from executor.ts when a threshold is hit, and emits telemetry.
  • session-state.ts — Map<sessionID, SessionTierState> with 30-min TTL eviction on read. Fields: stuckTurns, qualityFails, lastDifficulty, lastTier, lastAccess, explicitEscalationPending.
  • synthetic-tools.ts — syntheticTools(sessionID, telemetry?) returns { __tier_report_difficulty, __tier_request_escalation } as AI-SDK Tool objects with inputSchema (zod) and execute handlers that record into session state and emit telemetry. Both names start with __tier_ so any consumer can filter them.
  • prompt-augmentation.ts — augmentSystem(systemLines, enabled) appends one narrow paragraph telling the model about the two synthetic tools when enabled is true.
  • telemetry.ts — Telemetry interface with 7 methods (attemptStart, attemptSuccess, exhaustionObserved, escalationTriggered, difficultyReported, requestEscalationCalled, sessionStateUpdated) + a noopTelemetry default. F3 (opencoti-server) will eventually implement this interface; M5 ships the schema only.
  • executor.ts extensions: ExecuteResult.tierId (so the escalator knows which tier opened the stream); telemetry? in ExecutorOptions; _markEscalated(name, reason) exported package-internal; telemetry calls at attemptStart / exhaustionObserved branch points.
  • runtime.ts extensions: merges prepared.tools with syntheticTools(sessionID, telemetry) (the M3/M4 known limitation is gone); replaces system: ...join("\n") with augmentSystem(...).join("\n"); provides a restartStream callback to the escalator that re-calls execute() (the just-marked provider is skipped automatically).
  • Schema additions in config.ts: escalation.qualityFails.threshold (defaults to 2).
• Trigger semantics:
  • hard-fail: pre-first-chunk error → silently restart THIS turn once via restartStream. Mid-stream errors → mark exhausted, escalate NEXT turn. User-AbortSignal abort is excluded.
  • stuck-progress: turn finishes with zero tool calls AND lastDifficulty != "low" → increments stuckTurns. Escalates when stuckTurns >= cfg.escalation.stuckProgress.noProgressTurns. A productive turn (≥1 tool call) resets the counter.
  • structured-quality: any tool-error event → increments qualityFails. Escalates when qualityFails >= cfg.escalation.qualityFails.threshold. A clean turn with ≥1 successful tool result resets the counter.
  • explicit: any call to __tier_request_escalation (model bailout) OR an explicitEscalationPending flag from a prior turn → immediate escalation regardless of threshold.
• Test coverage: 33 new M5 tests (escalator.test.ts, session-state.test.ts, synthetic-tools.test.ts, prompt-augmentation.test.ts, telemetry.test.ts, + 3 extensions to runtime.test.ts). Total @opencoti/tiers: 80 tests, all green. Workspace typecheck clean.
• Zero upstream edits — M5 is entirely additive in packages/opencoti-tiers/. Hook footprint unchanged: still the 2 M2 markers in packages/opencode/src/session/llm.ts.

M5.5 — Effort-level + tiermode schema expansion [DONE 2026-05-21]

Pure-schema milestone, no runtime change beyond what the new config shape brings.

• Effort enum: replaced low|balanced|high|max with a 5-base set low|normal|medium|high|max plus 5 x-prefixed fanout opt-ins xlow|xnormal|xmedium|xhigh|xmax. The x prefix marks "fan-out at this tier"; aggression is the base level, fanout is the orthogonal bit. No balanced legacy alias — pre-1.0, follow CLAUDE.md's "don't keep backwards-compatibility shims" rule.
• effort.ts constraints table rebuilt as 5 × 2 grid:
  • low / xlow: cap at one tier; hard-fail trigger only.
  • normal / xnormal: walk all tiers; + stuck-progress.
  • medium / xmedium: walk all tiers; + structured-quality.
  • high / xhigh: same trigger set as medium; reserved for later threshold-tuning differentiation.
  • max / xmax: same trigger set, most-aggressive thresholds.
  • allowFanout is false for the bare base levels, true for the x* opt-ins. (Behavior change: old high/max enabled fanout; that capability moved to xhigh/xmax.)
• Tiermode: new top-level config field. tiermodes: is a map of named profiles; each profile may override effort, per-tier topologyOverrides, or a partial escalation overlay. The top-level tiermode: string field points into the map and is validated at parse time (refinement: name must exist).
• tiermode.ts resolver: resolveTiermode(cfg) → ResolvedTierConfig pure function. Returns the effective { effort, topologyOverrides, escalation, config } after profile merge. Router and runtime do NOT consume it yet (M5.5 is pure schema); the resolver is package- exported so M6/M7 wiring is trivial.
• Tests: total @opencoti/tiers test count up from 80 to 107 (+27 net: 10 new + parameterized parsing across all 10 effort values + 5 tiermode parsing + 6 resolver tests; 2 existing fanout- tests migrated from high/max to xhigh/xmax). Workspace typecheck clean. Zero new upstream surgical hooks.

M5.7 — Tier-0 provider configuration + model registry [DONE 2026-05-21]

User-facing first-run setup and a small project-local model registry. Lands after M5.5 (schema is final by then) and before M6 (fan-out needs a working multi-provider Tier 0 to test against). No runtime change beyond what config-read needs.

• Model registry. A models: map in the config (see Config-shape example above) maps short names (gemma4-a4b-coder, gemma4-31b-cloud, …) to provider entries. The chain[].providers field references those names. Same idea as claude-hooks' provider registry. Schema lives in packages/opencoti-tiers/src/config.ts.
• First-run setup CLI (opencoti tier setup or equivalent — exact subcommand TBD, runs from the standard opencode CLI surface):
  1. GPU detection. Reads nvidia-smi, rocm-smi, or an equivalent probe. Reports detected VRAM.
  2. GPU path branch. Prompts:
     • Confirm Gemma 4 A4B-98e-v5-coder as the Tier-0 model (yes by default, with the option to point to a different GGUF).
     • Quant pick — IQ3_M default for ≥10 GB VRAM, IQ4_XS or CD-IQ4_K_M for ~11 GB, Q4_K_S for ~12 GB, Q5_K_L for ~15 GB, Q8_0 for 24 GB. Recommended pick is computed from detected VRAM and shown as the default.
     • GGUF source — three options:
       • Download from HuggingFace. Downloads to $XDG_DATA_HOME/opencoti/models/ (~/.local/share/opencoti/models/ on Linux default) using the user's HF auth if any.
       • Use existing file. User points at a GGUF path. Validated (magic bytes + metadata read) before being recorded.
       • Import from local Ollama. If ~/.ollama/models/ is detected, parse manifests under ~/.ollama/models/manifests/registry.ollama.ai/library/<model>/<tag> to find the blob digest; the patched llamafile mounts ~/.ollama/models/blobs/sha256-<digest> directly. Single download, zero duplication.
  3. No-GPU path branch. Prompts:
     • Confirm Ollama Cloud as the Tier-0 provider.
     • Recommend gemma4:31b (free BF16 on Ollama Cloud at time of writing); allow override.
     • Capture OLLAMA_CLOUD_API_KEY via env-var reference (never stored in config — apiKeyEnv indirection).
  4. No silent auto-fallback. If GPU detection fails and the user hasn't picked a path, setup exits with a clear next-step prompt. No defaulting to cloud.
  5. Writes the chosen entries into the project's opencoti config (opencoti.tiers.models[...] and chain[0].providers[0]).
• Ollama local registry detection. Helper module reads the manifest JSON, follows the digest field to the blobs/ entry, returns {path, sha256, sizeBytes}. Pure read-only — never modifies the Ollama store. Tests use a fixture ~/.ollama/ tree.
• Reuse pattern in llamafile launcher. When a registry entry points at an Ollama blob (kind: llamafile, gguf path inside ~/.ollama/), the launcher passes the blob path directly to the patched llamafile — no copy, no symlink, just read. Verified by a smoke test.
• Test coverage: new test files in packages/opencoti-tiers/test/ (model-registry.test.ts, ollama-detect.test.ts) and packages/opencoti-llamafile/test/ (ollama-blob-reuse.test.ts). GPU detection itself is shimmed in tests (the real probe lives outside @opencoti/tiers to keep the package testable in CI).

Zero new upstream surgical hooks. M5.7 is entirely additive in opencoti packages + a new CLI subcommand exposed via opencode's existing command surface.

M5.7 implementation notes (what actually shipped):

• Schema: ModelEntry + models: Record<string, ModelEntry> on TierConfig. Refinements enforce "llamafile requires exactly one of gguf/ollamaBlob" and "openai-compat requires baseURL".
• src/registry/ollama-detect.ts — read-only manifest walker (registry.ollama.ai/<ns>/<model>/<tag>), follows the GGUF layer digest to ~/.ollama/models/blobs/sha256-…. Tolerant of malformed manifests.
• src/registry/gpu-detect.ts — async detectGPU() probes nvidia-smi --query-gpu=... first, falls back to rocm-smi --showmeminfo vram --json. 5 s default timeout. runImpl test seam.
• src/registry/quant-recommender.ts — pure table-driven picker encoding the Gemma 4 A4B-98e quant table from local_llamafile.md. Reserves ~3 GB KV per 32 K context + 1.5 GB safety; picks highest-quality (HE+) quant that fits.
• src/registry/hf-download.ts — minimal HF Hub HTTP client. Resume via ${destPath}.partial + Range: bytes=N-, optional $HF_TOKEN auth, SHA256 verify (deletes the partial on mismatch), per-chunk progress callback.
• src/cli/setup-flow.ts — pure-logic flow with injected deps (SetupDeps). runSetupFlow(deps) returns cancelled or configured with the ConfigPatch to write. Testable via scripted-answers stub deps.
• src/cli/setup.ts — live wiring: @clack/prompts for input, jsonc-parser modify/applyEdits for comment-preserving config write to ~/.config/opencode/opencode.jsonc. Exports tierCommand (yargs parent) and tierSetupCommand (subcommand).
• Surgical hook in packages/opencode/src/index.ts: 1 import + 1 .command(tierCommand) call, both marker-tagged. Both registered in docs/protocols/UPSTREAM_SYNC.md.
• package.json: adds @clack/prompts, jsonc-parser, yargs as deps (same pinned versions opencode uses) and @types/yargs as devDep. New ./cli export.
• Test coverage: 35 net new tests across 5 new files (ollama-detect.test.ts 6, gpu-detect.test.ts 6, quant-recommender.test.ts 10, hf-download.test.ts 7, cli/setup-flow.test.ts 8). Total @opencoti/tiers: 144 tests. Workspace typecheck clean. Hook conformance: exactly 4 markers (2 M2 + 2 M5.7).

M6 — Fan-out tier (first-good) [DONE 2026-05-21]

Goal: when the plan picks a fanout tier (xhigh / xmax effort, or explicit tiermode override), open every provider in that tier in parallel and let the fastest useful response through. Losers are aborted; losers that fail before being aborted are overlay-marked the same way the M5 linear path marks them.

Behavior knobs the user locked this turn:

• winnerTrigger (per tier): "content" (default) or "finish". content races on the first real-content event (same predicate the M5 escalator uses — text-delta, tool-call, tool-result, file, etc.). finish races on the first racer to complete a clean turn.
• Pre-trigger loser failures → overlay-marked via _markEscalated(name, "fanout-prefail"). Aborts-by-us are NOT marked (the racer was cut off through no fault of its own — tracked via a abortedByUs flag).
• maxConcurrency (per tier, optional): caps how many providers actually race. The rest are recorded in skippedByConcurrencyCap on the executor result and are NOT overlay-marked — they just sit out this turn.

The fanout mode enum was expanded to include "synthesizer" and "synthesizer-critic" for forward-compatibility with the consultants- council pattern (Tier 0 as synthesizer + separate critic model). M6 ships "first-good" only; "best-of-n", "quorum", "synthesizer", and "synthesizer-critic" throw a typed FanoutModeNotImplemented error that the runtime catches as a forgiving fall-through.

Implementation surface (all additive in packages/opencoti-tiers/):

• src/config.ts — FanoutTier gets winnerTrigger + maxConcurrency; FanoutMode enum expanded.
• src/executor.ts — executeFanout() opens every router-emitted attempt for the first fanout step in parallel via Promise.all, applying maxConcurrency and overlay-marking open-failures the same way execute() does.
• src/fanout.ts (new) — raceFanout() runs N per-racer drain goroutines, exposes {proxy, outcome, settled}. The proxy is a Proxy<StreamTextResult> whose fullStream getter replays the winner's buffered pre-trigger events then continues from the underlying iterator. Per-racer buffer cap (default 4 MB) catches pathological non-content streams.
• src/runtime.ts — peekTopology() decides linear vs fanout branch from plan(). The fanout branch builds per-racer streamText invocations with child AbortControllers, calls raceFanout, then wraps the proxy with wrapWithEscalation so M5's escalation triggers apply on the winner's stream. The restartStream callback handles the all-failed outcome the same way M5's hard-fail-pre-stream path does.
• src/escalator.ts — isRealContent was exported so fanout.ts shares the exact same predicate.

Test coverage: 20 net new tests across test/{config,executor,runtime,fanout}.test.ts. Total @opencoti/tiers count: 168 tests.

Zero new surgical hooks. Hook count stays at 4 (2 M2 + 2 M5.7).

M7 — TUI effort + tiermode pickers [DONE 2026-05-21]

Goal: in-TUI dials for the M5.5 effort (10 values) and tiermode (named profiles) so users can change them mid-session without re-editing ~/.config/opencode/opencode.jsonc. Plus a runtime-side override seam so a TUI pick actually takes effect on the very next request, plus auto-wire of the new picker plugin into the M5.7 setup CLI.

Three additive surfaces:

• @opencoti/tiers/src/active-config.ts — module-level overlay (setActiveEffort / setActiveTiermode / getActiveOverride) matching the existing stateOverlay pattern. Both the TUI plugin and the M2 hook run in the same Bun process, so module-level singleton state is genuinely shared.
• runtime.ts overlay + tiermode wiring — routeImpl now merges the overlay onto the parsed config, calls resolveTiermode (which M5.5 left unwired), patches each tier's topology per the resolver output, and forwards the effective effort to the executor via ExecutorOptions.effort. Precedence: TUI overlay > tiermode profile > base cfg.effort.
• @opencoti/tui-tiers — new sibling package. Ships an opencode TUI plugin (TuiPluginModule) that registers two commands (opencoti.effort, opencoti.tiermode), opens DialogSelects, and persists picks via api.kv under a per-project key (opencoti.<basename-hash12>.{effort|tiermode}) so each project remembers its own selection. On plugin start, the persisted values replay into the runtime overlay — TUI restart preserves the last pick.

Setup CLI extension (M5.7 → M7):

• setup-flow.ts got a tuiPickerInstalled sentinel on SetupDeps and an installTuiPicker boolean on ConfigPatch. After the GPU/no-GPU branch builds its patch, a finalizePatch helper asks "Install the TUI picker plugin?" (skipped when the sentinel is true — re-runs don't re-prompt).
• The live setup.ts reads the sentinel from the existing config (best-effort parseTree + findNodeAtLocation) and pre-populates tuiPickerInstalled on the live deps. When installTuiPicker is true on the patch, the writer appends "@opencoti/tui-tiers" to the user's plugin: [] (deduped against the existing array) and flips opencoti.tiers.tuiPickerInstalled = true.

Schema: TierConfig got one new optional sentinel field tuiPickerInstalled: z.boolean().default(false). Pure schema — defining it in Zod makes a typo a clear error instead of silent.

Test coverage: 16 net new tests across @opencoti/tiers (active-config, runtime, executor, setup-flow) + 12 tests in the new @opencoti/tui-tiers package. Totals: tiers 184 tests, tui-tiers 12 tests, workspace 18/18 typecheck clean.

Zero new surgical hooks. The TUI plugin loads via opencode's existing plugin: [] discovery path (the setup-flow auto-wire writes the entry). Hook footprint stays at 4 markers.

Behavior change worth noting: resolveTiermode was unwired since M5.5. Wiring it in M7 means users who set tiermode in their config — silently ignored pre-M7 — now actually see the profile apply. This is a fix, not a regression, but anyone debugging tiermode behavior should know the activation date is 2026-05-21.

M8 — Structured-quality validators v1

• Pluggable validators (code-parses, tests-pass-or-improve). Trigger escalation when validator fails N times on the current tier.

Open questions

• Where exactly is the upstream call site? Resolved at M2: packages/opencode/src/session/llm.ts, just before the existing experimental-native-llm seam.
• Streaming under fan-out. First-good is fine; quorum and best-of-n are not naturally streaming. Defer their UX to M6+.
• Cost accounting. Should opencoti show per-tier token spend? Probably yes, but it's not on the critical path. The M5 telemetry interface already carries the data; F3's web dashboard is the obvious place to surface it.
• Capability negotiation between escalator and provider. The patched llamafile exposes capabilities the no-GPU path lacks (KV reuse, big-context throughput). Today the executor doesn't ask providers what they can do — it just opens them. M5.7+ may want to add a capabilities() method to TierProvider so the escalator can know whether KV-prefix-reuse is on the table for the next turn. Not blocking; revisit when the second patched-llamafile feature lands.

Risks

• The opencode provider abstraction may not factor cleanly enough for a single hook. If so, we widen the hook to a small adapter shim in the upstream call site, and keep the adapter implementation in opencoti-tiers. The hook stays small; the implementation stays ours.
• Provider rate-limit signal parsing is fragile. We isolate it per provider adapter so a broken adapter never poisons the router.