# F3 — opencoti-server (Go companion daemon) > Parent: [../MASTER_PLAN.md](../MASTER_PLAN.md) > Status: **planning** > Owner: TBD > Reference: `/shared/dev/claude-hooks` (the inspiration; mirror its > goals, then go further) ## Problem `claude-hooks` is a Python daemon that wraps Claude Code with a hook API and adds persistent memory, multi-session coordination, tool glue, and a small zoo of integrations. It works very well *for Claude Code*. opencoti has the same shape of needs — and a few extra because opencoti has its own tier engine, its own local llamafile, and its own session model. We want a Go daemon that is to opencoti what claude-hooks is to Claude Code, **but deeper** because we control opencoti's source. Instead of bolting onto a hook API from outside, opencoti-server can be a first-class peer of opencoti's runtime. ## Goals - **G1.** Single Go binary, `opencoti-server`, that exposes a small, well-versioned API to opencoti. - **G2.** Persistent memory (vector + KG) shared across opencoti sessions on a host. - **G3.** Multi-session coordination: opencoti CLI instances on the same host see each other, share context where the user wants it. - **G4.** Tool glue: a place to register external tools (search, filesystem extensions, MCP-like) once and have them appear in all opencoti sessions. - **G5.** Event sink for tier-engine routing decisions (so we can log, audit, and later learn from them). - **G6.** Hook fan-out compatible with the existing claude-hooks ecosystem where it makes sense — but not bound by it. ## Non-goals (for now) - Replacing claude-hooks. opencoti-server is opencoti-specific. - Becoming a generic "MCP host". MCP integration is a feature, not the architecture. - Multi-host federation. Single-host first. ## Design sketch ### Where it lives - `opencoti/server/` — top-level non-bun source root. Go module. Putting it outside `packages/` keeps the bun workspace clean. - `packages/opencoti-server-client/` — TypeScript client used by opencoti to talk to the server. - **Surgical hooks** in opencode session lifecycle (session start, session end, turn complete, tool invocation, model call result): each hook is a one-line "if a client is available, notify it". Listed in `docs/protocols/UPSTREAM_SYNC.md`. ### Transport Unix domain socket on Linux/macOS, named pipe on Windows. Loopback TCP as a fallback. HTTP/JSON wire format for simplicity (no gRPC build dependency for the client). **Daemon-launch policy (applies to opencoti-server and every opencoti daemon)**: - Listen port MUST come from opencoti's reserved **47000-48000 range** — never 38000-39000 or 18790-18811 (claude-hooks territory). Tentative reservation table to keep adjacent ports clear of accidental collision: - **47092** — `@opencoti/embedder` (F4) - **47190** — opencoti-server daemon (proposed default; revisit when F3 M1 ships) - **47191** — opencoti-server dashboard (proposed default) - First-run setup MUST ask the user explicitly where the daemon should bind: `127.0.0.1` (default), all interfaces, or a specific IP. The default port from the reservation table is proposed; the user may override either field within the 47000-48000 range. Setup writes the validated choice to the user's config. ### Persistence Two backends, picked per-deployment: - **pgvector** — same as solidPC's `claude-hooks` setup. Best when a Postgres is already on the host. - **sqlite-vec** — single-file, no external dependency. Default for fresh installs. The backend choice is config; the memory API is identical. ### Surface (illustrative) ``` POST /v1/sessions # register a session POST /v1/sessions/:id/turns # record a turn POST /v1/memory # store a memory (M2) POST /v1/memory/search # vector recall (M2 — POST because the embedding is multi-KB) POST /v1/memory/collections # collections CRUD (M2) PUT /v1/memory/collections/:name/acl # per-session ACL (M2) GET /v1/memory/count # count (M2) POST /v1/memory/kg/entities # KG ops POST /v1/memory/kg/relations POST /v1/tier-events # log a tier-engine decision GET /v1/tools # list registered tools POST /v1/tools/:name/invoke # invoke (server-side glue) GET /v1/healthz # M1; M2 extended with store_ok/store_path/embedding_dim/store_error ``` The server is **embedder-agnostic** at M2: clients pass vectors, not text. The text→vector convenience (e.g., `GET /v1/memory?q=` proxying to the embedder daemon) is a deferred follow-up — adding it requires a runtime dep on `@opencoti/embedder` (F4 M5) and is most naturally landed once the TS client (M3) demonstrates the convenience is wanted at the HTTP layer vs done client-side. API is **versioned** (`/v1`) and the wire schema is owned by `packages/opencoti-server-client/` (so opencoti can rev independently of the server binary, within compatible versions). ### Mirroring claude-hooks goals Same goals, mapped: | claude-hooks | opencoti-server | | --- | --- | | `claude-hooks` recall hooks | implicit recall on `/v1/memory` query, surfaced by opencoti's prompt builder | | `mcp__pgvector__*` MCP tools | `/v1/memory` (and a thin MCP shim if external MCP clients want it) | | Stop hook auto-ingest | server records turn-complete events, applies same heuristics | | `claude-hooks` companion tools (Episodic etc.) | first-class tools registered via `/v1/tools` | ### Deeper integration than claude-hooks Because we own opencoti, we can: - Receive **tier-engine** decisions directly (provider chosen, escalation reasons, cost) without scraping logs. - Receive **diff and tool-output** events as structured payloads, not parsed from a transcript. - Push back: server can suggest a memory recall payload that opencoti **injects directly** into the prompt builder, instead of appending to a transcript. ## Milestones ### M1 — Server skeleton + healthz *(2026-05-23 — shipped)* - Go module under `opencoti/server/` (module `github.com/mann1x/opencoti/server`, Go 1.22). Single dep: `golang.org/x/sys` for the Windows `svc` packages. - `opencoti-server serve` binds a UDS by default on Unix (`$XDG_RUNTIME_DIR/opencoti/server.sock` or `~/.opencoti/server.sock` fallback) or `tcp://127.0.0.1:47190` on Windows. `--addr unix:///path` and `--addr tcp://host:port` override. - One HTTP endpoint: `GET /v1/healthz` returns 200 + JSON `{status, version, started_at, uptime_seconds}`. All other paths return a 404 envelope with code + message so the wire format is consistent for M2+. `POST /v1/healthz` returns 405 with the same envelope. - Graceful shutdown on SIGTERM/SIGINT (Unix) or SCM Stop/Shutdown (Windows). UDS file unlinked on Unix shutdown. - **Windows service support shipped in M1 (per user directive 2026-05-23):** - `install` subcommand wraps SCM (`golang.org/x/sys/windows/svc/mgr`): `--name`, `--display-name`, `--description`, `--start-type=auto|manual|disabled`, `--addr`. Registers the Event Log source under the same name. - `uninstall` removes the SCM entry AND the Event Log source. - `start` / `stop` wrap `mgr.Service.Start` / `Service.Control(svc.Stop)`. `stop` waits up to 10s for the state to become Stopped. - When `svc.IsWindowsService()` returns true, `serve` enters `svc.Run` with a handler that translates SCM Stop / Shutdown into a graceful `http.Server.Shutdown`. Log records flow to the Windows Application Event Log via `eventlog.Open(name)`. - Single source tree, build-tagged platform splits (`*_windows.go` / `*_nonwindows.go`). Same binary surface on every platform — `install` / `uninstall` / `start` / `stop` on Unix print "Windows-only command" and exit 2. - 7 cross-platform tests (parser + listener + healthz/404/405 + UDS roundtrip + UDS unlink-on-cleanup + double-shutdown safety) + 3 Windows-tagged tests for the `BuildMgrConfig` install-config builder. All green on Linux; Windows cross-compile clean. - **Hook footprint: ZERO.** M1 is purely additive — no `opencoti-hook:` markers, no UPSTREAM_SYNC.md registry rows. The autostart hook lands in M3. - **Deferred from M1:** Windows named-pipe TRANSPORT (use TCP on Windows in M1; named-pipe support is a follow-up); systemd/launchd unit files on Unix; CI workflows (the repo has no CI yet — separate concern); the first-run setup wizard step ("install as Windows service?" — comes when the setup-flow picks up F3 concerns). ### M2 — Memory backend (sqlite-vec) + recall API *(2026-05-23 — shipped)* - Go SQLite stack: `github.com/mattn/go-sqlite3` (cgo) + `github.com/asg017/sqlite-vec-go-bindings/cgo`, sqlite-vec statically linked via `sqlite_vec.Auto()` so the daemon ships as a single binary (no `vec0.so` to bundle). CGO becomes a hard build requirement at M2; cross-compile to Windows from Linux needs MinGW-w64 (`apt install gcc-mingw-w64-x86-64`). - `internal/store/store.go` defines a `Store` interface mirroring the TS `MemoryStore` in `packages/opencoti-memory/src/types.ts` 1:1. M7's pgvector backend will implement the same interface, so the HTTP layer doesn't change. - `internal/store/sqlitevec/` is the M2 backend. DDL mirrors `packages/opencoti-memory/src/schema.ts` byte-functionally: four tables (`meta`, `collections`, `memories`, `session_acl`) plus the `memory_vecs` vec0 virtual table; `SCHEMA_VERSION=1`; `embedding_dim` parameter-substituted into the vec0 DDL at first creation and stored in `meta` for validation on later opens. Content idempotency: SHA256-hex with `UNIQUE(collection_name, content_hash)`. Pure `ResolveAccessMode` mirrors the TS resolver (global=r, session-owner=rw, others=none; explicit `session_acl` rows override). Recall is vec0 MATCH + overscan (`k * min(8, max(2, len(candidates)))`) + join-back + ACL filter, truncated to k. - **DB schema is wire-compatible with the in-process `@opencoti/memory` package** — a DB created by either side opens cleanly under the other. This is the load-bearing invariant for M3's path-B refactor. - HTTP surface (eight new routes, all under `/v1/memory/*`): - `POST /v1/memory/collections` — create (201 / 409 collection_exists / 400 invalid_collection_name). - `GET /v1/memory/collections?session_id=...` — list, optionally filtered through the ACL resolver. - `DELETE /v1/memory/collections/{name}` — cascade-deletes memories + vec0 rows + session_acl entries. - `PUT /v1/memory/collections/{name}/acl` — set per-session ACL (`r`/`w`/`rw`/`none`). - `GET /v1/memory/collections/{name}/acl?session_id=...` — read the resolved access mode. - `POST /v1/memory` — store. 422 dim_mismatch on length mismatch; 403 write_denied on ACL deny. - `POST /v1/memory/search` — vector recall (POST because a 1024-element Float32 array doesn't fit a query string). - `GET /v1/memory/count?collection=...` — count, optionally scoped to one collection. - All error responses use M1's `{error: {code, message}}` envelope. Embeddings on the wire are JSON `[float32, ...]` arrays (verbose but trivial for the M3 TS client; base64 raw bytes is a follow-up if wire size measurably matters). - `/v1/healthz` extended with `store_ok`, `store_path`, `embedding_dim`, and `store_error` (omitempty). When the store fails to open, the daemon still serves `/v1/healthz` with `store_ok=false` and `/v1/memory/*` returns 503 `store_unavailable` — partial degradation beats refuse-to-start so the daemon stays observable when something's wrong with the DB. - `serve` gets `--db-path` (default `$HOME/.opencoti/memory/state.db` — matches the TS `defaultDbPath()` exactly so both implementations point at the same file by default) and `--embedding-dim` (default 1024, matches `DEFAULT_EMBEDDING_DIM`). `install` bakes both into the SCM service-start arguments. - 30 tests cover the surface: 18 unit tests on the store conformance (collection lifecycle, ACL resolver, idempotent store, recall ordering, count, close+reopen, dim-mismatch paths) + 12 HTTP integration tests via `httptest.NewServer` (every endpoint, happy + 4xx + 5xx envelopes). - `testdata/crosslang/` ships a Go↔Bun schema-compat probe: Go writes a DB → optionally invokes a Bun script that opens the same DB via `@opencoti/memory` and confirms readback. Skips cleanly when bun or the workspace package isn't on PATH. `make test-cross-lang` is always safe to run. - **Hook footprint: ZERO** (unchanged from M1). M2 is purely additive — the API exists but no surgical hooks into opencode yet. - **Deferred from M2:** text→vector convenience (`GET /v1/memory?q=`); streaming recall (SSE); pagination on list-collections; authentication for TCP transport (UDS owner-only on Unix is the M2 security boundary); pgvector backend (M7); migration tools (schemas are identical — the DB just opens); server-side embedder cache; bulk-store / batch endpoints; base64 raw-byte embedding wire format. ### M3 — TS client + opencoti hook to start the server *(2026-05-23 — shipped)* - `@opencoti/opencoti-server-client` published in-workspace under `packages/opencoti-server-client/`. Single class `OpencotiServerClient` implements `MemoryStore` over HTTP — all eight `/v1/memory/*` routes from M2 plus a `healthz()` probe. Wire format is M2's verbatim: JSON `[float32, ...]` for embeddings; M1/M2's `{error: {code, message}}` envelope decoded into a sentinel `OpencotiServerError({status, code, message})` so callers can match on `code === "store_unavailable"`, `dim_mismatch`, etc. Per-request `AbortController` + 5 s default timeout. `close()` is a documented no-op (no per-instance handle to close). - **Transport** uses Bun's native `fetch` — `tcp://host:port` is rewritten to `http://...` in `toFetchBase`; `unix:///path` passes through verbatim (Bun supports UDS fetch natively, no custom adapter needed). `defaultAddress(platform)` mirrors the Go `DefaultAddress()` exactly: `$XDG_RUNTIME_DIR/opencoti/server.sock` on Linux (`$HOME/.opencoti/server.sock` fallback), and `tcp://127.0.0.1:47190` on Windows. - **Autostart** lives in `src/autostart.ts`: `startAutostart({address, binaryPath?, dbPath?, embeddingDim?, ...})` probes `/v1/healthz` first (200 → `already_running`), locates the binary via `Bun.which` if none was passed (→ `no_binary` when missing), spawns with `stdio: "ignore"` (mirroring the `@opencoti/embedder` manager pattern), and polls `/v1/healthz` every 200 ms until the deadline (default 5 s; → `ready_timeout` if never recovers). Returns a discriminated `AutostartOutcome` so consumers can log five distinct states (`already_running | spawned | no_binary | spawn_failed | ready_timeout`) without exception handling. Test seams: `spawnImpl`, `fetchImpl`, `whichImpl`, `sleepImpl`. - **Surgical hook footprint: 3 markers + 1 package.json dep.** All three TS markers land in `packages/opencode/src/config/config.ts` (the same file that already carries `opencoti-default-plugins`): the import next to it, the `opencoti.server.*` schema struct (`autostart: boolean`, `address: string`, `binary_path: string`, `db_path: string`, `embedding_dim: PositiveInt`), and the fire-and-forget `void maybeStartOpencotiServer(...)` call site right after the existing `applyDefaultPlugins(...)` call. The hook's full substance lives in `@opencoti/opencoti-server-client/autostart-hook` so the opencode-side surface stays minimal — three lines plus the package.json dep (no anchor comment in JSON, registered in the table). - **Plugin swap.** Both `@opencoti/memory-plugin` and `@opencoti/tui-memory` accept a new optional `server_address?: string`. The default `StoreFactory`: if `serverAddress` is set, instantiate `OpencotiServerClient`, call `healthz()`, and return the client when `storeOk === true`; otherwise silently fall through to the existing in-process `SqliteVecStore.open(...)`. The `__setStoreFactory` test seam is unchanged — existing tests continue to inject stubs that ignore `serverAddress`. - **Health envelope mismatch detection.** Client validates that `embedding_dim` from the server matches the configured dim; on mismatch, `healthz()` returns `storeOk: false` with a `dim mismatch` error so the plugin falls back to the in-process store rather than corrupting the wire format. This is **client-side**; the server's own dim check still fires on `POST /v1/memory`. - **Tests.** 56 new tests in the client package: 33 mocked-fetch client conformance (every endpoint, happy + 4xx + 5xx envelopes, dedup, dim mismatch, store_unavailable propagation), 11 pure tests for `defaultAddress` / `toFetchBase` / `joinURL` / `probeReady`, 6 mocked-spawn `startAutostart` cases (already running, no binary, spawned-then-ready, ready timeout, spawn failed, --addr passthrough). Plus 2 new cases in the memory-plugin tests and 3 new in the tui-memory tests for the `server_address` swap path. All 23 workspace packages typecheck clean; 82 tests pass across the three affected packages. - **Live verification.** Manual end-to-end smoke against the F3 M2 binary (TCP loopback): `healthz` returns the expected shape (`store_ok: true`, `embedding_dim: 64`), `createCollection` + `setSessionAcl(rw)` + `store(...)` (with `deduplicated: false` then `true` on idempotency) + `recall(...)` (distance 0 on exact match) + `count(...)` (returns 1) + `deleteCollection` all return the expected shapes. UDS path tested at the discover / autostart layer; full UDS smoke deferred to M4 when session-event hooks bring it under day-to-day use. - **Path-B loop closed.** With M3 in, the F4 M5 in-process-then-daemon migration path described in [memory_embedder.md](memory_embedder.md) is mechanically complete: setting `opencoti.server.address` (or `autostart: true`) in `opencode.jsonc` is the only change a user makes to switch from the in-process store to the Go daemon. No code paths change in `@opencoti/memory` itself. - **Deferred from M3:** detached / setsid daemon lifecycle (M3's spawn dies with opencode — cross-session sharing requires running the daemon externally, e.g. the M1 Windows service installer or a future systemd unit); the first-run "install as a service?" wizard step (a follow-up once the setup-flow picks up F3 concerns); a text-`q=` convenience endpoint (`GET /v1/memory?q=`) — every M3 caller has the embedder in-process and can pre-embed; retry / backoff on transient HTTP errors (single attempt, the plugin's silent fallback handles the failure case); bulk-store / batch endpoints; TLS / shared-secret authentication for TCP transport (UDS owner-only remains the M3 security boundary). ### M4 — Session + turn events flowing *(2026-05-23 — shipped)* **Design pivot vs the original spec.** M4 was originally specced as *surgical hooks at session start/end/turn complete*. Phase-1 exploration confirmed opencode's `@opencode-ai/plugin` API already exposes those events fully typed via `Hooks.event` (session.created/updated/deleted/error/idle plus session.status). Following the precedent set by M5-D2 (`@opencoti/memory-plugin`), M4 ships as a plugin with **zero opencoti-hook footprint** — strictly richer than a surgical hook for this use case (typed payloads, zero upstream-source touch, no UPSTREAM_SYNC.md row to maintain on every sync). Hook count stays at 9 source markers + 4 JSON deps (unchanged from F3 M3). - **Per-feature schema versioning.** `meta.schema_version` stays at 1 (memory tables — kept stable so M2/M3 TypeScript `@opencoti/memory` clients still open M4 DBs cleanly). New `meta.sessions_schema_version = 1` is written by M4+ daemons and reported on `/v1/healthz`. M3 clients without the key treat the daemon as pre-M4; M4 plugins disable forwarding silently when `sessionsSchemaVersion < 1`. - **Three new tables** in the existing `state.db` (no new `--flag`, no migration tool): - `sessions` — soft-delete via `deleted_at`. Indices on `parent_id` (fork-tree queries) and `deleted_at` (cheap `WHERE deleted_at IS NULL`). Upsert is idempotent on `id`; `INSERT ... ON CONFLICT(id) DO UPDATE SET ...` overwrites all columns from the latest snapshot. - `session_events` — append-only audit log. `FOREIGN KEY ... ON DELETE CASCADE` would normally wipe history on `DELETE FROM sessions`; we use soft-delete instead so the audit trail outlives the session row. - `session_messages` — UPSERT on opencode `MessageID`. The plugin re-sends the full latest snapshot on every turn boundary; SQLite UPSERT handles dedup. - **SessionStore interface** (`internal/store/store.go`): 8 methods + sentinel errors (`ErrSessionNotFound`, `ErrInvalidSessionID`, `ErrSessionPayloadTooLarge`). The same `*sqlitevec.Store` implements both `Store` (memory) and `SessionStore`; server.go type-asserts at each entry point. M7 pgvector will implement the same interface. - **HTTP surface** under `/v1/sessions/*`: - `POST /v1/sessions` — upsert (201 on insert, 200 on update, 400 invalid_session_id, 413 payload_too_large). - `GET /v1/sessions[?parent_id=&include_deleted=&limit=&offset=]` — list, default newest-first by `updated_at`, omitting soft-deleted rows. - `GET /v1/sessions/{id}` / `DELETE /v1/sessions/{id}` — fetch / soft-delete. - `POST /v1/sessions/{id}/events` / `GET /v1/sessions/{id}/events[?type=&limit=&offset=]` — append + list audit entries. - `POST /v1/sessions/{id}/messages` / `GET /v1/sessions/{id}/messages[?limit=&offset=&since=]` — bulk-upsert (idempotent on message id) + list. - `GET /v1/sessions/{id}/turns` — coarse server-side projection grouping messages by user-message boundaries. Authoritative turn-boundary logic stays in opencode's `compaction.ts`; this view is for UI / debugging. - **healthz extension**: `sessions_schema_version` field added; M3 clients without the field treat the daemon as pre-M4. - **Payload size guards**: 8 MiB hard cap on `session_events.payload` and the bulk-upsert messages body (413 `payload_too_large`); the plugin chunks at a softer 1 MiB default before posting. - **TS client extension** (`@opencoti/opencoti-server-client`): 8 new `OpencotiServerClient` methods (`upsertSession`, `listSessions`, `getSession`, `deleteSession`, `appendSessionEvent`, `listSessionEvents`, `upsertSessionMessages`, `listSessionMessages`, `listSessionTurns`). Wire format snake_case → camelCase via private mappers (same pattern as M2's `mapHit` / `mapCollectionInfo`). `healthz()` return type + `ProbeResult` gain `sessionsSchemaVersion?: number`. - **New plugin `@opencoti/session-events-plugin`**: subscribes to `session.created/updated/deleted/error/idle` and `session.status` (idle transition) via opencode's typed `Hooks.event`; does **not** subscribe to `message.updated` (fires thousands of times per turn — text deltas, tool calls, reasoning chunks — and forwarding every delta would melt the daemon). On idle, the plugin fetches the session's latest messages via `input.client.session.messages.list` and bulk-upserts in byte-bounded chunks. Maintains a per-session merge cache so opencode's partial `session.updated` payloads are reconciled into a full snapshot before reaching the daemon. Best-effort, never-throws: a forwarding plugin must not propagate daemon failures into the opencode session. - **session.error before sessionID exists** lands on a synthetic `__pre_session_errors__` session (lazily upserted on first such error) so the audit log captures pre-creation failures without losing context. - **Auto-wire**: one-line addition to `@opencoti/tiers/default-plugins` `OPENCOTI_DEFAULT_PLUGINS`. Any user with opencoti config in `opencode.jsonc` gets the plugin automatically; without an `opencoti.server.address` the plugin's healthz probe fails, the internal client stays `undefined`, and every event handler early-returns — same no-op-when-unconfigured ergonomic the memory-plugin uses. - **Tests**: 26 new Go tests (16 store unit + 10 HTTP integration) — total Go test count climbs from 30 → 63. 14 new TS client tests (33 → 47). 16 new plugin tests in `@opencoti/session-events-plugin`. All 24 workspace packages typecheck clean. - **Live verification**: end-to-end smoke against a real M4 daemon (TCP loopback) confirmed healthz reports `sessions_schema_version: 1`, full upsert → update → get → list → events → messages → turns → soft-delete cycle works, soft-deleted session's events + messages remain queryable through `include_deleted=true`. - **Deferred from M4**: - **`message.updated` real-time forwarding** — coalescing on idle is the right granularity; per-delta forwarding would mean O(text-deltas) HTTP roundtrips per turn. - **Per-tool-invocation events** as first-class rows — captured today inside the message's `parts` JSON; can be promoted to a `session_tool_calls` table later if needed. - **WebSocket / SSE push from daemon → client** — request-response only in M4. - **TLS / authentication for `/v1/sessions/*` TCP transport** — UDS owner-only on Unix remains the M4 security boundary. ### M5 — Tier-engine event sink (shipped 2026-05-23) - **Daemon side: `/v1/tier-events`.** - `POST /v1/tier-events` appends a row (201; 400 invalid_tier_event; 413 payload_too_large at the same 8 MiB cap as `/v1/sessions/*`). Body: `{event_type, ts?, session_id?, payload}` where `payload` is opaque JSON — the daemon does not introspect it. - `GET /v1/tier-events?session_id=&event_type=&since=&limit=&offset=` returns the audit log newest-first. - One new `tier_events` table: `(id PK AUTO, session_id TEXT, event_type TEXT, payload TEXT, ts INTEGER)` + two indices `(session_id, ts)` and `(event_type, ts)`. **`session_id` is a soft reference — no FK, no cascade.** Tier events fire intra-turn and can land before `session.created` reaches the daemon; the soft reference guarantees no event is dropped. - Per-feature `meta.tier_events_schema_version = 1`. The cross-language sqlite_vec contract continues to live on `meta.schema_version = 1` (still M2-compatible with the TS `@opencoti/memory` reader); the new key is independent. - **TS plumbing.** `@opencoti/opencoti-server-client` gains `appendTierEvent` + `listTierEvents` (camelCase→snake_case wire mapping mirroring `mapSessionMessage`). `healthz()` surfaces `tierEventsSchemaVersion?: number`; plugins use it as their feature gate. - **Design pivot continues: plugin path > surgical hook.** The `Telemetry` interface in `@opencoti/tiers` (designed for this milestone — its docstring at `telemetry.ts:1-4` said so) is already threaded end-to-end through `runtime.ts:99 → executor.ts → escalator.ts → fanout.ts`. M5 plugs a real sink into the existing seam via a new module-level `registered-telemetry.ts` slot (mirrors `active-config.ts`). Precedence chain in `runtime.ts:99` becomes: ```ts const telemetry = hook.telemetry ?? execOpts.telemetry ?? getRegisteredTelemetry() ?? noopTelemetry ``` The new `@opencoti/tier-events-plugin` calls `setRegisteredTelemetry(impl)` during plugin boot when the daemon's healthz reports `tierEventsSchemaVersion >= 1`. **Zero new opencoti-hook source markers** — count unchanged from F3 M4 (9 source + 4 JSON deps + 3 prose). See the F1 page for where the Telemetry interface itself was added. - **Fire-and-forget posture.** The plugin's Telemetry impl maps each method to a `client.appendTierEvent(...).catch(() => {})` — synchronous to the caller, HTTP swallowed on failure. The tier engine never blocks on the daemon. Cf. M4's `@opencoti/session-events-plugin`, same design. - **Auto-wire via defaults.** `@opencoti/tier-events-plugin` is in `OPENCOTI_DEFAULT_PLUGINS` alongside the M4 session plugin. Any user with `opencoti: {...}` in their config gets the audit log for free; if the daemon is unreachable, the healthz gate fails and the runtime stays on noop. - **Cross-version compat.** An M5 plugin against an M4 daemon sees `sessions_schema_version: 1` but no `tier_events_schema_version` in healthz, so `setRegisteredTelemetry` is never called and the runtime stays on noop. An M5 daemon against M4 clients keeps shipping the same `/v1/sessions/*` surface untouched. - **Explicitly deferred to later:** aggregation endpoints (`/v1/tier-events/stats?...`), real-time push (WS/SSE), server-side retention policy, client-side batching/coalescing, FK on `session_id`. M5 ships raw events + filters; everything on top of that is composable later. ### M6 — Tool registry + invocation (shipped 2026-05-23) - **Daemon side: `/v1/tools`.** - `GET /v1/tools` lists every registered tool sorted by name: `{ tools: [{ name, description, params_schema, handler, created_at, updated_at }] }`. `params_schema` is a string-serialised JSON Schema; `handler` is a stable opaque dispatch ID (debug-only — invocation always goes by name). - `GET /v1/tools/{name}` fetches one (404 tool_not_found). - `POST /v1/tools/{name}/invoke` runs it. Body `{ arguments: {...}, session_id? }` → `{ result: {...} }` (200) or `{ error: { code, message } }` (400 invalid_arguments / 404 tool_not_found / 500 handler_failed / 413 payload_too_large at the same 8 MiB cap as `/v1/sessions/*` and `/v1/tier-events`). - One `tools` table: `(name PK, description, params_schema, handler, created_at, updated_at)`. Process-scoped, no session FK. Dispatch is a daemon-internal `map[string]ToolHandler` in `toolhandlers.go`; **no external registration in M6** (no POST/DELETE on `/v1/tools` itself) — the catalog is seeded into the binary at boot. - Per-feature `meta.tools_schema_version = 1`, independent of the cross-language `meta.schema_version = 1` (still M2-compatible with the TS `@opencoti/memory` reader). - **Two seed tools** are `UpsertTool`'d at startup (idempotent — `created_at` fixed on first insert, `updated_at` bumps on re-seed): - **`opencoti_episodic_search`** — lexical `LIKE` search across the M4 `session_messages` table joined with `sessions`, newest-first. Params `{ query, limit?, session_id_excludes? }`. The exclude list lets the plugin pass the live session ID so the model recalls *other* sessions, not echoes of the current one. The integrated analog of claude-hooks's `episodic_server` — same SQLite file the sessions API writes, no second daemon, no shell-out. FTS5/vector ranking is an M7+ optimisation. - **`opencoti_recall`** — vector recall over the M2 memory store, delegating to the same `Store.Recall` path `/v1/memory/search` uses. Params `{ embedding, collection?, k?, session_id? }`. **Takes a pre-computed embedding, not a query string** — the daemon bundles no embedder, so the caller (the plugin, or an operator via curl) embeds the query and POSTs the float vector; length must equal the daemon's `embedding_dim`. `session_id` drives the M2 per-session ACL check. Coexists intentionally with `@opencoti/memory`'s in-process `__memory_recall`: that one survives a daemon being down; `opencoti_recall` gives the same surface to plugins without direct DB access. - **TS plumbing.** `@opencoti/opencoti-server-client` gains `listTools` + `getTool` + `invokeTool`; `healthz()` surfaces `toolsSchemaVersion?: number` as the plugin's feature gate. `ServerTool` is camelCase (`paramsSchema` ← `params_schema`), `paramsSchema` left as `unknown` (interpreted in the plugin). - **Design pivot continues: plugin path > surgical hook.** A new module-level `registered-server-tools.ts` slot in `@opencoti/tiers` (mirrors M5's `registered-telemetry.ts`) holds the daemon tools as a `Record`; `runtime.ts` merges it **last** into the tier tool list, next to synthetic-tier and memory-bridge tools: ```ts const tools = mergeTools(hook.prepared.tools, { ...syntheticTools(hook.input.sessionID, telemetry), ...memoryTools, ...(getRegisteredServerTools() ?? {}), }) ``` Server tools merge last so user-explicit + memory tools win on a name collision (none expected — server tools are `opencoti_*`, memory `__memory_*`, synthetic-tier `__tier_*`). **Zero new opencoti-hook source markers** — count unchanged from F3 M4/M5 (9 source + 4 JSON deps + 3 prose). - **The `@opencoti/server-tools-plugin`** probes `/v1/healthz` (gated on `toolsSchemaVersion >= 1`), fetches `/v1/tools` once on boot, and builds an AI SDK `dynamicTool` per entry: `params_schema` round-trips through `ai`'s `jsonSchema()` (no JSON-Schema-to-Zod reimplementation); each `execute` forwards to `/v1/tools/{name}/invoke`. Tool names are prefixed (`opencoti_`, idempotently). The session ID arrives via `experimental_context` on the single `openStream` `streamText` call and is forwarded as `session_id`. Invocations are awaited (not fire-and-forget — tool calls are model-blocking), but errors crash the *tool call*, not the session: 404→`tool_not_found`, 400→`invalid_arguments`, 5xx/network→`tool_handler_failed`, deadline→`timeout` (plugin-owned `Promise.race`, separate `invoke_timeout_ms`). Args over 1 MiB are pre-rejected client-side. - **Auto-wire via defaults.** `@opencoti/server-tools-plugin` is in `OPENCOTI_DEFAULT_PLUGINS`. No reachable daemon → healthz gate fails → `setRegisteredServerTools` never called → runtime tool set unchanged. - **Cross-version compat.** An M6 plugin against an M5 daemon sees no `tools_schema_version` in healthz, so the gate fails and the tool set is unchanged — a model query expecting the tool gets "I don't have that tool", no crash. An M6 daemon serves M5 clients the `/v1/sessions/*` and `/v1/tier-events` surfaces untouched. - **Explicitly deferred:** external tool registration (POST/DELETE on `/v1/tools` — needs a per-tool ACL/owner model); streaming tool results; tool invocations as `tier_events` audit rows; an MCP wrapper for the registry; FTS5/vector ranking in `episodic_search`; an in-daemon embedder so `opencoti_recall` can take a `query` string directly. ### M7 — pgvector backend parity (shipped 2026-05-23) opencoti-server gains its **second storage backend**: PostgreSQL + the pgvector extension, via `github.com/jackc/pgx/v5` (pure Go, no new cgo). `*pgvector.Store` implements the **full** interface family — `Store` + `SessionStore` + `TierEventStore` + `ToolStore` + `SchemaInspector` — so the HTTP layer, `/v1/healthz`, and the TS client are **untouched**. sqlite-vec stays the zero-dependency default; pgvector is for hosts that already run Postgres. **Schema-phrasing correction.** The original M7 stub said *"/v1/memory works against pgvector with the same schema as claude-hooks."* That predated the M4–M6 buildout. opencoti's binding contract is now its **own** `Store` interface (collections + per-session ACL + sessions + tier_events + tools), which is a different data model from claude-hooks's flat `memories` + `kg_*` schema. So M7 implements **opencoti's own model on a dedicated `opencoti` Postgres database**, isolated from claude-hooks. "Same as claude-hooks" is reread as *same storage technology (Postgres + pgvector)*, not the same tables. - **`internal/store/pgvector/`** mirrors `internal/store/sqlitevec/` table-for-table in PG dialect, with three dialect differences: - the embedding lives **inline** on `memories` as a `vector(dim)` column (no separate vec0 virtual table); an HNSW `vector_l2_ops` index is created when `dim ≤ 2000` (pgvector's HNSW ceiling) — correctness-neutral, perf-positive. - `Recall` pushes the ACL filter into the query (`WHERE collection_name = ANY($readable) ORDER BY embedding <-> $q LIMIT k`) — Postgres can filter + rank in one statement where sqlite-vec must overscan + join-back + post-filter. The `<->` operator is **L2**, matching sqlite-vec's vec0 default, so `MemoryHit.Distance` stays comparable across backends. - `Store` dedups via `INSERT … ON CONFLICT (collection_name, content_hash) DO NOTHING RETURNING id` in one round trip. - **Per-feature schema versions** (`schema_version`, `sessions_/tier_events_/tools_schema_version`, all `1`) live in a `meta` key/value table, validated on open exactly as the sqlite-vec backend does — but they are pgvector's own contract (no cross-language reader, since pgvector is daemon-only). - **Shared conformance harness** `internal/store/storetest/` is the parity guarantee: a single `RunConformance` body (Memory, Sessions, TierEvents, Tools sub-suites) that **both** backends opt into via a thin `conformance_test.go`. Either backend drifting from the contract fails the same assertions. sqlite-vec runs it with `t.TempDir`; pgvector with a testcontainers fixture. - **Pure-logic lifted to `internal/store/common.go`** (package `store`): the ACL resolver (`ResolveAccessMode`/`CanRead`/`CanWrite`), the collection/session validators, `Sha256Hex`, and the dim bounds — one cgo-free source of truth both backends share. sqlite-vec keeps its exported names as thin delegating wrappers, so the HTTP layer's `sqlitevec.ValidateCollectionName`/`ValidateSessionID` call sites are byte-identical. - **Backend selection** is a serve-time flag, not a build flag: `--backend sqlite|pgvector` (default `sqlite`) + `--pg-dsn` (falling back to `$OPENCOTI_PG_DSN`). `--db-path` stays sqlite-only; `--embedding-dim` applies to both (locks the `vector(dim)` column). The interface var is assigned only on a successful open (avoids the typed-nil trap), so a pgvector open failure degrades to `store_ok=false` + a clear `store_error` rather than crashing. The Windows `install` command bakes `--backend`/`--pg-dsn` into the service args alongside the existing flags. - **healthz** reports a **redacted** `store_path` for pgvector (`pg://user@host:port/db`, never the password). - **Tests**: `pgtest` spins an ephemeral `pgvector/pgvector:pg17` container via testcontainers-go and skips cleanly when Docker is absent (`testcontainers.SkipIfProviderIsNotHealthy`), so `make test` stays green on a Docker-less host. Both backends pass the full shared conformance suite; an end-to-end smoke (daemon → pgvector container) confirms collection create, store, and `<->` - ranked recall over HTTP. - **Toolchain**: the modern pgx / pgvector-go / testcontainers-go releases require **Go 1.25**, so the module's `go` directive and the host toolchain moved to go1.25 (latest stable). No build-posture regression — pgx is pure Go; sqlite-vec's existing cgo requirement is unchanged. - **Zero new surgical hooks.** pgvector is additive Go inside `opencoti/server/`; no TS changes. The surgical-hook grep count stays at 18 (see `docs/protocols/UPSTREAM_SYNC.md`). **Deferred from M7:** data migration between backends (a `migrate` subcommand — "Migration tools later"); a TS pgvector backend for the in-process `@opencoti/memory` (pgvector is daemon-only); external tool registration over HTTP (still security-gated, from M6); pgxpool tuning / read replicas. ### M8 — Multi-session coordination (shipped 2026-05-23) Goal **G3**: opencoti instances/sessions on one host see each other and cooperate. M8 ships the *live-coordination* half as an **in-memory, ephemeral** hub — deliberately store-independent (neither the `Store` interface nor either backend is touched), because presence and locks are runtime state that should not survive a daemon restart. - **`internal/coord.Hub`** — backend-agnostic, one `sync.RWMutex`: - **Presence**: `RegisterPeer` / `Heartbeat` / `DeregisterPeer` / `ListPeers`, with a background TTL sweep that reaps peers whose heartbeat lapsed (default 45s) and emits `peer.left`. `ListPeers` also filters expired peers lazily. - **Broadcasts**: a pub/sub bus — `Publish` assigns a monotonic `seq`, appends to a bounded replay ring, and fans out non-blockingly to subscribers (a full subscriber channel is dropped + closed so the client reconnects with its last seq). `Subscribe(since)` atomically snapshots the replay backlog and registers for live events. - **Advisory locks**: try-only `AcquireLock` (reentrant-by-holder refresh; expired locks reclaimable) / `ReleaseLock` (holder-checked) / `ListLocks`. Lock transitions emit `lock.acquired` / `lock.released`. - **HTTP** (`internal/server/coord.go`), hub injected via `Options.Hub`, 503 `coord_unavailable` when absent: | Method + path | Purpose | | --- | --- | | `POST /v1/coord/peers` | register/upsert presence → PeerInfo | | `POST /v1/coord/peers/{id}/heartbeat` | refresh TTL | | `DELETE /v1/coord/peers/{id}` | deregister | | `GET /v1/coord/peers` | list live peers | | `POST /v1/coord/broadcast` | publish `{peer_id,topic,payload}` → `{seq}` | | `GET /v1/coord/events?since=&peer_id=` | **SSE** event stream (the daemon's first) | | `POST /v1/coord/locks/{name}` | acquire `{holder,ttl_ms?}`; 200 or 409 `lock_held` | | `DELETE /v1/coord/locks/{name}` | release `{holder}`; 200 / 404 `lock_not_held` / 409 `lock_not_holder` | | `GET /v1/coord/locks` | list held locks | SSE is viable because the `http.Server` sets no `WriteTimeout`; the handler exits on request-context cancellation so graceful shutdown releases it within the grace window. `/v1/healthz` gains `coord_ok` + `coord_peers`. - **TS client**: `registerPeer` / `heartbeatPeer` / `deregisterPeer` / `listPeers` / `broadcast` / `acquireLock` / `releaseLock` / `listLocks`, plus `subscribeCoordEvents` — the client's first streaming method (reads `response.body`, parses `data:` frames). - **`@opencoti/coordination-plugin`**: registers each session as a peer, heartbeats while active, deregisters on delete, subscribes to the event stream (SSE) to keep a live peer view, and advertises "N other active opencoti session(s)" in the system prompt. Auto-wired via the `@opencoti/tiers` default-plugins list — **no surgical hook** (grep count stays 18). **Deferred to F3 M9:** opt-in *shared context* (a thin convention atop existing M2 global collections + per-session ACL — a session opts to expose a collection to peers); the sqlite↔pgvector `migrate` subcommand (open from M7); blocking/queued lock acquire (M8 is try-only). Cross-host federation remains an explicit F3 non-goal. ### M9 — Opt-in shared context (shipped 2026-05-23) Goal **G3**, the *persisted-context* half: a session exposes one of its session-scoped memory collections to peer sessions on the same host. The storage model from M2 already supports the grant, so M9 adds **no new `Store` method, no new schema, no new surgical hook** (grep count stays 18). It is a thin convention bridging two things that already exist — the persisted per-session ACL (`SetSessionACL`) and M8's ephemeral coord bus. - **`internal/share.Manager`** — in-memory registry + one goroutine: - `Share(collection, owner, mode)` records the share, grants read-ACL to every live peer `!= owner` via `SetSessionACL`, and publishes `collection.shared` on the hub. - It **subscribes to the hub** and, on `peer.joined`, grants every active share to the newcomer — so peers that join *after* a share still get access (auto-grant via the bus, dogfooding M8's SSE). - `Unshare(collection, owner)` revokes live peers (writes mode `none`, the non-owner default) and publishes `collection.unshared`. - **Persisted vs ephemeral:** the ACL grants persist (they survive a restart); the "keep auto-granting new joiners" intent is in-memory and lost on restart by design — existing grants remain, but the owner must re-share to resume auto-granting. This keeps M9 storage-free. - **Recall "just works":** once a peer holds an `r` grant, its no-filter `Recall` includes the shared collection automatically (M2's readable-collections resolution), so no recall-path change is needed. - **HTTP** (`internal/server/share.go`), Manager constructed in `server.New()` when `Store`+`Hub` are present, released on `Shutdown`; 503 `share_unavailable` when the hub is absent: | Method + path | Purpose | | --- | --- | | `POST /v1/memory/collections/{name}/share` | owner-only share `{owner_session, mode?}` (mode default `r`) → `{granted}` | | `DELETE /v1/memory/collections/{name}/share` | withdraw `{owner_session}` → 200 | | `GET /v1/memory/shares` | list active shares | **Owner-only:** the handler looks the collection up via `ListCollections` (no filter) and requires `scope == session` and `session_id == owner_session`; global collections (already `r`-for-all) are rejected `not_shareable`, a different owner `not_owner` (403). - **TS client**: `shareCollection` / `unshareCollection` / `listShares` + `SharedCollectionInfo`. - **`@opencoti/coordination-plugin`** (extended in place — no new plugin): consumes `collection.shared` / `collection.unshared` to advertise peer-shared collections in the system prompt, and gains an opt-in `share_session_collection` flag (default **false**) that shares the session's own collection (`sessionCollectionName(id)`) on `session.created` and unshares it on `session.deleted` (best-effort; a not-yet-created collection's 404 is swallowed). **Deferred to F3 M10:** the sqlite↔pgvector `migrate` subcommand — **not** a cutover: both backends are first-class and may run in parallel; migrate is an idempotent, re-runnable, either-direction copy (`--from`/`--to`, dedupe on `content_hash`, upsert PKs) that tops up a parallel target, never abandoning the source. Also deferred: revoke-on-peer-leave cleanup (M9 keeps grants on `peer.left`, since peers may return) and wildcard / group ACLs. ### M10 — sqlite↔pgvector `migrate` (shipped 2026-05-24) A daemon-internal `migrate` subcommand that copies data between the two first-class backends. It is **not a cutover**: both backends stay first-class and may run in parallel (two daemons, or alternating `--backend`). `migrate` is an **idempotent, re-runnable, either-direction, selectable copy** that *tops up* a target (dedupe on natural keys), never "move then abandon source". Additive Go only — **no plugin, no HTTP surface, no TS client, no surgical hook** (grep count stays 18). **Use cases (maximum flexibility):** 1. **Scale-up (primary).** Start on sqlite; migrate *everything* (collections, memories+embeddings, ACL, sessions, session_events, session_messages, tier_events, tools) into pgvector; then **switch the primary backend** by changing the serve flag to `--backend pgvector`. "Switch primary backend" is operational, no extra code: run the full migrate, then change `--backend`. 2. **Way back.** Same command with `--from`/`--to` swapped. 3. **Single memory container / additional partial backend.** Copy only specific collection(s) with `--collection`, so e.g. pgvector holds just certain memories while sqlite keeps the rest. The two backends coexist, each holding different data. 4. **Additive top-up.** Re-running, or copying into a populated target, merges idempotently (natural-key dedupe). **Selection model — two orthogonal selectors:** - `--include ` of sections: `memory,sessions,tier-events,tools`. Default (unset) = **all four** (full dataset). The `memory` section carries collections + their memories + their ACL rows. - `--collection ` (repeatable) restricts the `memory` section to those collections only. When `--collection` is given and `--include` is unset, the default narrows to **memory-only** (the single-container case). So: full switch = no selectors; single container = `--collection notes`; logs-only = `--include sessions,tier-events`. **Copy phases (FK-respecting; each gated by the selection):** | Section | Phase | Export | Import | | --- | --- | --- | --- | | memory | collections | `ListCollections` (filtered) | `PutCollection` | | memory | memories | `ListMemories` (per coll, keyset-paged) | `PutMemory` | | memory | session_acl | `ListSessionACLs` | `SetSessionACL` | | sessions | sessions | `ListSessions(+deleted)` | `UpsertSession` | | sessions | session_events | `ListSessionEvents`/sess | `PutSessionEvent` | | sessions | session_messages | `ListSessionMessages`/sess | `UpsertSessionMessages` | | tier-events | tier_events | `ListTierEvents` (batched) | `PutTierEvent` | | tools | tools | `ListTools` | `UpsertTool` | Both backends are the same concrete `*Store` implementing every feature interface, so migrate opens each side as `storepkg.Store`, type-asserts the optional `MigrationStore`, and copies a section only if **both** sides implement it. Per-phase counts (scanned / inserted / skipped) print at the end; `--dry-run` reads sources and reports would-copy counts, no writes. **Idempotency / dedup.** Memories dedupe on `(collection_name, content_hash)`. The append-only audit logs (`session_events`, `tier_events`) have **no natural key** (autoinc id only), so their idempotent import dedups on an **insert-if-no-identical-row** check (all business columns match) — re-running produces no duplicates. A second full run reports 0 inserts everywhere. **Two backend-specific wrinkles (both resolved):** - **sqlite-vec embeddings are an opaque BLOB.** vec0 `memory_vecs` stores the embedding as a blob and the Go binding ships `SerializeFloat32` but no deserialize. The on-disk format is plain little-endian `float32` (`binary.Write(buf, LittleEndian, vector)`), so `deserializeFloat32` reverses it with `binary.Read` at the locked dim — **byte-faithful, no re-embedding**. pgvector exports via `pgvector.Vector.Scan` + `.Slice()`. - **The public `Store()` write path enforces ACL** (a global collection is `r`-for-all → `ErrWriteDenied`), so migrate cannot reuse it for imports. The new `MigrationStore.PutMemory` is **ACL-free and timestamp- preserving**. **Fidelity contract (documented caveat).** Preserved exactly: collection `created_at`, memory `ts`+`content`+embedding (byte-identical), session `created_at`/`deleted_at`, message `created_at`/`finished_at`, event `occurred_at`/`ts`, all ACL modes. Rewritten to migrate-time (reused upserts stamp `now()`): `sessions.updated_at`, `tools.created_at`/ `updated_at` — acceptable "last-written" fields (tools are also re-seeded at startup). A byte-identical `PutSession`/`PutTool` is deferred. The new `MigrationStore` optional interface (in `store.go`, mirroring the `SchemaInspector`/`SessionStore` convention) is exercised by the `storetest` conformance harness on **both** backends — the round-trip asserts the exported embedding is byte-identical, validating the sqlite LE deserialize and the pgvector scan together. Verified end-to-end across a `sqlite→pgvector→sqlite` hop: recall on the round-tripped file returns `distance 0` for the exact source vector. **Deferred to F3 M11+:** continuous-sync / daemon mode (migrate is one-shot CLI only), cross-host federation (explicit F3 non-goal), per-session selective log copy (selective granularity is per-collection for memory + section toggles for the rest), and the byte-identical `updated_at`/tool-timestamp imports noted above. ## Open questions - **Do we ship the server inside the opencoti binary or alongside it?** Alongside (separate binary) is cleaner; explore a single-fat-binary option as an opt-in. - **Authentication.** Local UDS owner is the user; over loopback TCP we need a shared secret. Default to UDS. - **claude-hooks coexistence.** If a user runs both Claude Code (with claude-hooks) and opencoti (with opencoti-server) on the same host, the two memories should *not* collide. They use different stores and different schemas by default; a separate migration tool can bridge them if the user wants. ## Risks - Two persistence backends doubles the test matrix. Mitigation: a storage trait + a shared conformance test. - Drift from claude-hooks features. Mitigation: don't try to replicate feature-for-feature; replicate goals. claude-hooks remains the reference for Claude Code; opencoti-server can diverge where it makes opencoti better.