Sync Space with current realtime demo

README.md

@@ -10,12 +10,12 @@ short_description: Talk directly to Inkling, with replies voiced by Qwen3-TTS
 hf_oauth: true
 ---
 
-# Minimal Conversation App (S2S backend, **WebSocket** transport)
+# Realtime Voice Demo — Inkling
 
-Drop-in alternative to [`amir-tfrere/minimal-conversation-app-s2s-backend`](https://huggingface.co/spaces/amir-tfrere/minimal-conversation-app-s2s-backend)
-that uses the **WebSocket** route of the Hugging Face speech-to-speech
-backend instead of the WebRTC SDP proxy. Same load balancer, same
-`/session` handshake, same UI, same orb. Just a different wire.
+Browser voice-chat UI for the
+[huggingface/speech-to-speech](https://github.com/huggingface/speech-to-speech)
+backend, speaking the OpenAI Realtime **GA** protocol over **WebSocket**
+(the deployed Space path) or **WebRTC** (env-pinned direct deployments only).
 
 ## How it works
 
@@ -43,26 +43,26 @@ backend instead of the WebRTC SDP proxy. Same load balancer, same
 The backend exposes one concurrent session per compute (same as WebRTC
 mode); the LB pins the session via a signed `session_token`.
 
-## Why WebSocket instead of WebRTC
+## WebRTC transport
 
-| | WebRTC (original) | WebSocket (this) |
-|---|---|---|
-| Transport | UDP + Opus 48 kHz + ICE/STUN | TCP + raw PCM16 |
-| NAT traversal | needs STUN, can fail on corporate / cellular | none, works everywhere TCP is allowed |
-| Audio quality | excellent (Opus, jitter buffer, FEC) | good (raw PCM, simple ring buffer) |
-| Latency | lowest (~50-150 ms) | low (~150-300 ms typical) |
-| Echo cancellation | browser AEC active on the WebRTC track | browser AEC active via `getUserMedia` constraints |
-| Debuggability | needs `chrome://webrtc-internals` | `wscat` / DevTools network tab |
-| Mobile data | sometimes blocked (UDP) | always works (HTTPS+WSS) |
+With `SPEECH_TO_SPEECH_URL` set, **Settings → Transport** offers WebRTC as an
+alternative to WebSocket. The browser sends its SDP offer through the
+same-origin `/api/calls` proxy; negotiated audio and the OpenAI Realtime event
+data channel then flow directly between browser and backend.
+
+The proxy forwards only to the env-pinned URL, never to a client-supplied
+target. This is why user-typed URLs and load-balancer mode remain WebSocket-only.
+Set `RTC_ICE_SERVERS` on this app and `SPEECH_TO_SPEECH_ICE_SERVERS` on the
+backend when remote connections need STUN/TURN.
+
+The WebSocket-only noise gate is hidden during WebRTC calls. Camera snapshots
+are also re-encoded to fit the data-channel message limit.
 
 ## Backend requirement
 
-This app talks to the WebSocket route `@app.websocket("/v1/realtime")`
-defined in
-[`websocket_router.py`](https://github.com/huggingface/speech-to-speech/blob/feat/webrtc-transport/src/speech_to_speech/api/openai_realtime/websocket_router.py)
-on the **`feat/webrtc-transport`** branch. The same compute serves both
-the WebRTC POST and the WebSocket upgrade on the same path; no backend
-change required.
+The backend must be installed with the `webrtc` extra to serve
+`POST /v1/realtime/calls`; otherwise WebRTC responds with 501. The WebSocket
+route remains `/v1/realtime`.
 
 Smoke-test from the shell:
 
@@ -94,6 +94,7 @@ Three modes, picked by env (`/api/config` tells the client which one is active):
   **directly** to this realtime WebSocket URL; it's shown read-only in Settings.
   Setting it disables the load-balancer logic entirely (no `/api/session` proxy,
   no queue, no metering, no sign-in). Unlike the LB address it is not a secret.
+  This is the only mode in which WebRTC can be selected.
 - **`LOAD_BALANCER_URL` env** — the original flow: the browser POSTs the
   same-origin `/api/session` proxy, the server forwards to the LB, and the
   browser dials the per-session compute URL the LB hands back. The LB address
@@ -106,12 +107,12 @@ Three modes, picked by env (`/api/config` tells the client which one is active):
   `connect_url` (`wss://host/v1/realtime?...`) or a bare host like `localhost:8080`
   (the app adds `/v1/realtime`), and the browser connects to it directly.
 
-| `SPEECH_TO_SPEECH_URL` | `LOAD_BALANCER_URL` | `SPACE_ID` | Connection | URL field | Metering |
-|:---:|:---:|:---:|---|---|---|
-| ✅ | any | any | direct → pinned URL | visible, locked | off |
-| – | ✅ | ✅ | LB proxy | hidden | **on** |
-| – | ✅ | – | LB proxy | hidden | off |
-| – | – | any | direct → user URL | editable | off |
+| `SPEECH_TO_SPEECH_URL` | `LOAD_BALANCER_URL` | `SPACE_ID` | Connection | URL field | Transport | Metering |
+|:---:|:---:|:---:|---|---|---|---|
+| ✅ | any | any | direct → pinned URL | visible, locked | WS or WebRTC | off |
+| – | ✅ | ✅ | LB proxy | hidden | WS only | **on** |
+| – | ✅ | – | LB proxy | hidden | WS only | off |
+| – | – | any | direct → user URL | editable | WS only | off |
 
 **Settings → Restart** reconnects with the current voice, instructions and URL.
 
@@ -158,12 +159,15 @@ Then open <http://localhost:7860/>, click the orb, allow the mic, talk.
 
 | Key | What |
 |-----|------|
-| Load balancer URL | Base URL of your S2S deployment. App POSTs `<lb>/session`. |
+| Speech-to-speech server URL | Direct realtime WebSocket URL (hidden/locked when deployment-owned) |
+| Transport | WebSocket (default) or WebRTC; selectable only with an env-pinned URL |
+| Microphone | Input device for capture. Applies on the next conversation / Restart. |
+| Speakers | Output device for assistant audio. Chrome/Edge can switch live; other browsers keep the system default. |
 | Voice | Qwen3-TTS speaker name (Aiden, Ryan, Dylan, Eric, Ono_Anna, Serena, Sohee, Uncle_Fu, Vivian) |
-| Instructions | System prompt sent in `session.update` once the WS opens |
+| Instructions | System prompt sent in `session.update` once the connection opens |
 
-LocalStorage keys are namespaced `s2s.ws.*` so this app's settings do
-NOT collide with the WebRTC variant.
+LocalStorage keys are namespaced `s2s.ws.*`, plus `s2s.transport` and
+`s2s.audio.inputId` / `s2s.audio.outputId`.
 
 ## Files
 
@@ -177,6 +181,7 @@ NOT collide with the WebRTC variant.
 | `auth.py` | HF OAuth + per-request identity (tier, hashed keys) |
 | `limiter.py` | SQLite per-day talk-time budget (chunked server-clock reservation) |
 | `ws/s2s-ws-client.js` | WebSocket handshake + OpenAI Realtime GA protocol |
+| `rtc/s2s-rtc-client.js` | WebRTC sibling: SDP via `/api/calls`, events over data channel, track audio |
 | `ws/codec.js` | base64 <-> PCM helpers + transcript extraction (pure) |
 | `ws/orb-visualizer.js` | `OrbVisualiser`: FFT bands -> orb CSS custom properties |
 | `worklets/mic-capture.js` | AudioWorklet: 48 kHz Float32 -> 16 kHz Int16 PCM, posts ~40 ms chunks |
@@ -200,5 +205,5 @@ NOT collide with the WebRTC variant.
 
 ## Credits
 
-- Backend: [huggingface/speech-to-speech](https://github.com/huggingface/speech-to-speech) on `feat/webrtc-transport`
+- Backend: [huggingface/speech-to-speech](https://github.com/huggingface/speech-to-speech)
 - UI verbatim from `amir-tfrere/minimal-conversation-app-s2s-backend` (Pollen Robotics × Hugging Face)

index.html

@@ -236,6 +236,21 @@
               </small>
             </label>
 
+            <!-- Transport picker. Hidden in LB mode; disabled (locked to
+                 WebSocket) unless the deploy pins SPEECH_TO_SPEECH_URL — the
+                 WebRTC handshake goes through the server-side /api/calls
+                 proxy, which only forwards to the pinned URL. -->
+            <label class="field" id="transport-field" hidden>
+              <span>Transport</span>
+              <select id="transport">
+                <option value="ws" selected>WebSocket</option>
+                <option value="webrtc">WebRTC</option>
+              </select>
+              <small id="transport-hint">
+                How audio travels to the server. Applies on the next conversation.
+              </small>
+            </label>
+
             <div class="field-row">
               <label class="field">
                 <span>Voice</span>
@@ -253,7 +268,27 @@
               </label>
             </div>
 
-            <div class="field">
+            <label class="field">
+              <span>Microphone</span>
+              <select id="audio-input">
+                <option value="">System default</option>
+              </select>
+              <small>Applies on the next conversation (or Restart).</small>
+            </label>
+
+            <label class="field" id="audio-output-field">
+              <span>Speakers</span>
+              <select id="audio-output">
+                <option value="">System default</option>
+              </select>
+              <small id="audio-output-hint">
+                Where assistant audio plays. Chrome/Edge support switching outputs.
+              </small>
+            </label>
+
+            <!-- WS-only: the gate runs in the WebSocket capture worklet; the
+                 WebRTC mic path sends the raw track, so this hides there. -->
+            <div class="field" id="gate-field">
               <span class="field-head">
                 Noise gate
                 <span id="gate-value" class="field-value">Off</span>

main.js

@@ -1,22 +1,24 @@
 // @ts-check
 /**
  * Minimal voice conversation app, talking to a Hugging Face speech-to-speech
- * backend over **WebSocket** (drop-in alternative to the WebRTC variant).
+ * backend over **WebSocket** or **WebRTC**.
  *
- * Click the orb -> we ask for the mic, POST a session on the LB, open a
- * WebSocket on the routed compute endpoint, push session.update + mic
- * audio, play back the TTS audio. The orb visually reflects the live
- * state (idle, connecting, listening, user-speaking, processing,
- * ai-speaking).
+ * Click the orb -> we ask for the mic, connect (WS dial, or SDP handshake via
+ * the /api/calls proxy), push session.update + mic audio, play back the TTS
+ * audio. The orb visually reflects the live state (idle, connecting,
+ * listening, user-speaking, processing, ai-speaking).
  *
- * The only meaningful difference vs. the WebRTC main.js is that the
- * client owns its own AudioContext (no `attachOutputTrack`), so we hand
- * it the MediaStream directly.
+ * The two client classes expose the same events and methods, so everything
+ * below except the constructor pick is transport-agnostic. WebRTC is offered
+ * only in env-pinned direct mode (the /api/calls proxy forwards exclusively
+ * to SPEECH_TO_SPEECH_URL); LB mode and user-typed URLs stay on WebSocket.
  *
  * @typedef {"idle" | "connecting" | "queued" | "your-turn" | "listening" | "user-speaking" | "processing" | "ai-speaking" | "error"} AppState
+ * @typedef {S2sWsRealtimeClient | S2sRtcRealtimeClient} RealtimeClient
  */
 
 import { S2sWsRealtimeClient } from "./ws/s2s-ws-client.js";
+import { S2sRtcRealtimeClient } from "./rtc/s2s-rtc-client.js";
 import { $, truncateError, DEBUG } from "./ui/dom.js";
 import { ChatView } from "./ui/chat.js";
 import { Account } from "./ui/account.js";
@@ -47,6 +49,11 @@ const STORAGE_KEYS = {
   tools: "s2s.ws.tools",
   searchKey: "s2s.ws.searchKey",
   noiseGate: "s2s.ws.noiseGate",
+  // "ws" | "webrtc". Not under the historical "s2s.ws." prefix — it selects
+  // between the transports rather than configuring the WS one.
+  transport: "s2s.transport",
+  audioInputId: "s2s.audio.inputId",
+  audioOutputId: "s2s.audio.outputId",
 };
 
 // ── Noise gate ──────────────────────────────────────────────────────────────
@@ -98,6 +105,21 @@ const TOOL_DEFS = {
 /** Longest edge of the snapshot sent to the VLM, in px (keeps payload sane). */
 const SNAPSHOT_MAX_EDGE = 768;
 const SNAPSHOT_QUALITY = 0.7;
+// Over WebRTC the snapshot travels inside ONE data-channel message, and SCTP
+// messages above the negotiated max (64 KiB on the aiortc side) fail to send.
+// Budget for the data-URL portion, leaving headroom for the JSON envelope.
+const SNAPSHOT_DC_BUDGET_CHARS = 60_000;
+// Re-encode ladder walked until the frame fits the budget: quality first
+// (cheap wins), then resolution. The last rung is ~15–25 KB for any content,
+// so a frame that "fits" is deterministic, not content-dependent luck.
+const SNAPSHOT_LADDER = /** @type {[number, number][]} */ ([
+  [SNAPSHOT_MAX_EDGE, SNAPSHOT_QUALITY],
+  [SNAPSHOT_MAX_EDGE, 0.5],
+  [640, 0.45],
+  [512, 0.4],
+  [448, 0.35],
+  [384, 0.3],
+]);
 
 function loadSettings() {
   return {
@@ -105,6 +127,10 @@ function loadSettings() {
     voice: localStorage.getItem(STORAGE_KEYS.voice) || DEFAULT_VOICE,
     instructions: localStorage.getItem(STORAGE_KEYS.instructions) || DEFAULT_INSTRUCTIONS,
     noiseGate: loadGateThreshold(),
+    // Default WebSocket: the proven path stays the first-run experience.
+    transport: localStorage.getItem(STORAGE_KEYS.transport) === "webrtc" ? "webrtc" : "ws",
+    audioInputId: localStorage.getItem(STORAGE_KEYS.audioInputId) || "",
+    audioOutputId: localStorage.getItem(STORAGE_KEYS.audioOutputId) || "",
   };
 }
 
@@ -127,6 +153,9 @@ function saveSettings(s) {
   localStorage.setItem(STORAGE_KEYS.voice, s.voice);
   localStorage.setItem(STORAGE_KEYS.instructions, s.instructions);
   localStorage.setItem(STORAGE_KEYS.noiseGate, String(s.noiseGate));
+  localStorage.setItem(STORAGE_KEYS.transport, s.transport);
+  localStorage.setItem(STORAGE_KEYS.audioInputId, s.audioInputId || "");
+  localStorage.setItem(STORAGE_KEYS.audioOutputId, s.audioOutputId || "");
 }
 
 /** @returns {{ web_search: boolean, camera_snapshot: boolean }} */
@@ -239,8 +268,22 @@ const inputLbUrl = $("#lb-url");
 const connField = $("#conn-field");
 /** @type {HTMLElement} */
 const connHint = $("#conn-hint");
+/** @type {HTMLElement} */
+const transportField = $("#transport-field");
+/** @type {HTMLSelectElement} */
+const inputTransport = $("#transport");
+/** @type {HTMLElement} */
+const transportHint = $("#transport-hint");
+/** @type {HTMLElement} */
+const gateField = $("#gate-field");
 /** @type {HTMLSelectElement} */
 const inputVoice = $("#voice");
+/** @type {HTMLSelectElement} */
+const inputAudioInput = $("#audio-input");
+/** @type {HTMLSelectElement} */
+const inputAudioOutput = $("#audio-output");
+/** @type {HTMLElement} */
+const audioOutputHint = $("#audio-output-hint");
 /** @type {HTMLTextAreaElement} */
 const inputInstructions = $("#instructions");
 /** @type {HTMLInputElement} */
@@ -283,6 +326,17 @@ let allowDirect = true;
 // Deploy-pinned s2s URL (SPEECH_TO_SPEECH_URL). Non-empty -> locked direct
 // mode: the field displays it read-only and the saved user URL is untouched.
 let pinnedUrl = "";
+// Whether the deploy offers the WebRTC transport (/api/config `rtc`; true
+// exactly when the URL is env-pinned, since /api/calls only forwards there).
+let rtcAvailable = false;
+/** @type {RTCIceServer[]} STUN/TURN servers for the browser peer connection
+ * (deploy-provided via RTC_ICE_SERVERS; empty -> host candidates only). */
+let iceServers = [];
+// Transport of the LIVE (or starting) conversation — as opposed to
+// `settings.transport`, which is what the NEXT one will use. Drives the
+// camera-snapshot size budget while a call is running.
+/** @type {"ws" | "webrtc"} */
+let activeTransport = "ws";
 
 // ── Tool state ──────────────────────────────────────────────────────────────
 let toolsEnabled = loadTools();
@@ -339,7 +393,7 @@ let trackedTier = "";
 // queue on teardown / tab-close so we don't hold a phantom place.
 let queuedTicketId = "";
 
-/** @type {S2sWsRealtimeClient | null} */
+/** @type {RealtimeClient | null} */
 let client = null;
 /** @type {MediaStream | null} */
 let micStream = null;
@@ -411,6 +465,7 @@ function openSettings() {
   inputInstructions.value = settings.instructions;
   syncGateUi();
   updateRestartAvailability();
+  void refreshAudioDeviceLists();
   settingsModal.showModal();
 }
 
@@ -712,22 +767,44 @@ async function watchCameraPermission() {
  * Grab the current webcam frame as a downscaled JPEG data URL. The preview is
  * mirrored in CSS for a natural self-view, but we draw the raw (un-mirrored)
  * video here so the model sees the scene in its true orientation.
+ *
+ * Over WebRTC the frame must fit one data-channel message, so it's re-encoded
+ * down the SNAPSHOT_LADDER until it's under SNAPSHOT_DC_BUDGET_CHARS; over
+ * WebSocket the first (full-quality) rung is used as before.
  * @returns {string | null}
  */
 function captureSnapshot() {
   if (!cameraStream || !camVideo.videoWidth) return null;
   const vw = camVideo.videoWidth;
   const vh = camVideo.videoHeight;
-  const scale = Math.min(1, SNAPSHOT_MAX_EDGE / Math.max(vw, vh));
-  const w = Math.max(1, Math.round(vw * scale));
-  const h = Math.max(1, Math.round(vh * scale));
-  const canvas = document.createElement("canvas");
-  canvas.width = w;
-  canvas.height = h;
-  const ctx = canvas.getContext("2d");
-  if (!ctx) return null;
-  ctx.drawImage(camVideo, 0, 0, w, h);
-  return canvas.toDataURL("image/jpeg", SNAPSHOT_QUALITY);
+
+  /** @param {number} maxEdge @param {number} quality @returns {string | null} */
+  const encode = (maxEdge, quality) => {
+    const scale = Math.min(1, maxEdge / Math.max(vw, vh));
+    const w = Math.max(1, Math.round(vw * scale));
+    const h = Math.max(1, Math.round(vh * scale));
+    const canvas = document.createElement("canvas");
+    canvas.width = w;
+    canvas.height = h;
+    const ctx = canvas.getContext("2d");
+    if (!ctx) return null;
+    ctx.drawImage(camVideo, 0, 0, w, h);
+    return canvas.toDataURL("image/jpeg", quality);
+  };
+
+  if (activeTransport !== "webrtc") {
+    return encode(SNAPSHOT_MAX_EDGE, SNAPSHOT_QUALITY);
+  }
+  let dataUrl = null;
+  for (const [edge, quality] of SNAPSHOT_LADDER) {
+    dataUrl = encode(edge, quality);
+    if (!dataUrl) return null;
+    if (dataUrl.length <= SNAPSHOT_DC_BUDGET_CHARS) return dataUrl;
+  }
+  // Even the last rung overflowed (shouldn't happen in practice) — send it
+  // anyway; the client logs the failed send rather than killing the session.
+  console.warn(`[tool] snapshot exceeds the data-channel budget after the full ladder (${dataUrl?.length} chars)`);
+  return dataUrl;
 }
 
 /** Brief shutter flash on the preview so the user sees a snapshot was taken. */
@@ -840,6 +917,10 @@ async function fetchConfig() {
       allowDirect = json.allowDirect ?? !lbMode;
       // Deploy-pinned direct URL (overrides the LB server-side already).
       pinnedUrl = (json.s2sUrl || "").trim();
+      // WebRTC transport: offered only when the deploy pins the URL (the
+      // /api/calls proxy refuses to forward anywhere else).
+      rtcAvailable = !!json.rtc;
+      iceServers = Array.isArray(json.iceServers) ? json.iceServers : [];
       // The conversation-time limiter rides on the LB being present.
       limiterOn = lbMode;
     }
@@ -916,13 +997,22 @@ function createResumedAudioContext() {
 
 /** Read the editable settings out of the form. The URL field is only honoured
  *  in free direct mode — in LB mode it's hidden, and when the deploy pins a
- *  URL it's read-only, so the user's saved URL survives either way. */
+ *  URL it's read-only, so the user's saved URL survives either way. The
+ *  transport select is only honoured while selectable, so a saved "webrtc"
+ *  survives a visit to a deploy that can't offer it. */
 function readSettingsFromForm() {
   return {
     directUrl: allowDirect && !pinnedUrl ? inputLbUrl.value.trim() : settings.directUrl,
     voice: inputVoice.value || DEFAULT_VOICE,
     instructions: inputInstructions.value.trim() || DEFAULT_INSTRUCTIONS,
     noiseGate: readGateThreshold(),
+    transport: /** @type {"ws" | "webrtc"} */ (
+      transportSelectable()
+        ? (inputTransport.value === "webrtc" ? "webrtc" : "ws")
+        : settings.transport
+    ),
+    audioInputId: inputAudioInput.value || "",
+    audioOutputId: inputAudioOutput.value || "",
   };
 }
 
@@ -933,8 +1023,37 @@ function readGateThreshold() {
   return Math.min(GATE_MAX_DB, Math.max(GATE_OFF_DB, v));
 }
 
+/** Whether the transport picker is live: env-pinned direct mode only. The
+ *  /api/calls proxy forwards exclusively to SPEECH_TO_SPEECH_URL, so without
+ *  the pin there is nowhere safe to send a WebRTC offer. */
+function transportSelectable() {
+  return allowDirect && !!pinnedUrl && rtcAvailable;
+}
+
+/** The transport the next conversation will actually use. */
+function effectiveTransport() {
+  return transportSelectable() && settings.transport === "webrtc" ? "webrtc" : "ws";
+}
+
+/** Reflect transport availability + selection into Settings, and hide the
+ *  noise gate when WebRTC is picked (the gate lives in the WS capture
+ *  worklet; the WebRTC mic path sends the raw track). */
+function syncTransportUi() {
+  // Hidden in LB mode (nothing to choose); visible-but-locked in un-pinned
+  // direct mode so the option is discoverable along with what unlocks it.
+  transportField.hidden = !allowDirect;
+  const selectable = transportSelectable();
+  inputTransport.disabled = !selectable;
+  inputTransport.value = selectable && settings.transport === "webrtc" ? "webrtc" : "ws";
+  transportHint.textContent = selectable
+    ? "How audio travels to the server. Applies on the next conversation."
+    : "WebRTC needs a server URL pinned by the deployment (SPEECH_TO_SPEECH_URL).";
+  gateField.hidden = effectiveTransport() === "webrtc";
+}
+
 /** Adapt the connection field to the mode learned from /api/config. */
 function syncConnectionUi() {
+  syncTransportUi();
   if (pinnedUrl) {
     // Deploy-pinned URL: show it, but locked — the deployment owns it.
     connField.hidden = false;
@@ -981,9 +1100,14 @@ settingsForm.addEventListener("submit", (event) => {
   saveSettings(settings);
 
   // Voice + instructions can apply to a live session without reconnecting; a
-  // changed connection URL only takes effect on the next restart.
+  // changed connection URL only takes effect on the next restart. Speaker
+  // output can switch live when the browser supports AudioContext.setSinkId;
+  // mic device changes need a Restart (new getUserMedia stream).
   if (client && LIVE_STATES.has(currentState)) {
     client.updateSession({ voice: settings.voice, instructions: effectiveInstructions() });
+    if (typeof client.setAudioOutputDevice === "function") {
+      void client.setAudioOutputDevice(settings.audioOutputId);
+    }
   }
 });
 
@@ -993,6 +1117,15 @@ inputNoiseGate.addEventListener("input", () => {
   setGateThreshold(readGateThreshold());
 });
 
+// Transport persists on change (like the gate) and takes effect on the next
+// conversation; the gate field previews its WS-only availability right away.
+inputTransport.addEventListener("change", () => {
+  if (!transportSelectable()) return;
+  settings.transport = inputTransport.value === "webrtc" ? "webrtc" : "ws";
+  localStorage.setItem(STORAGE_KEYS.transport, settings.transport);
+  syncTransportUi();
+});
+
 restartBtn.addEventListener("click", async () => {
   if (currentState === "connecting") return; // a connect is already underway
   settings = readSettingsFromForm();
@@ -1084,16 +1217,112 @@ joinQueueBtn.addEventListener("click", () => {
   if (client) client.join();
 });
 
-const MIC_CONSTRAINTS = {
-  audio: { echoCancellation: true, noiseSuppression: true, autoGainControl: true },
+const MIC_CONSTRAINTS_BASE = {
+  echoCancellation: true,
+  noiseSuppression: true,
+  autoGainControl: true,
 };
 
+/** @returns {MediaStreamConstraints} */
+function micConstraints() {
+  /** @type {MediaTrackConstraints} */
+  const audio = { ...MIC_CONSTRAINTS_BASE };
+  if (settings.audioInputId) {
+    // ideal (not exact): if the saved device was unplugged, fall back quietly.
+    audio.deviceId = { ideal: settings.audioInputId };
+  }
+  return { audio };
+}
+
+/** True when Web Audio can route playback to a chosen output device. */
+function supportsAudioOutputSelection() {
+  const Ctx = window.AudioContext || /** @type {any} */ (window).webkitAudioContext;
+  return typeof Ctx?.prototype?.setSinkId === "function";
+}
+
+/**
+ * Rebuild the mic/speaker <select>s from enumerateDevices. Labels are blank
+ * until mic permission has been granted at least once.
+ */
+async function refreshAudioDeviceLists() {
+  const canPickOutput = supportsAudioOutputSelection();
+  inputAudioOutput.disabled = !canPickOutput;
+  audioOutputHint.textContent = canPickOutput
+    ? "Where assistant audio plays. Can change live while connected."
+    : "Speaker selection needs a browser with AudioContext.setSinkId (Chrome/Edge).";
+
+  /** @type {MediaDeviceInfo[]} */
+  let devices = [];
+  try {
+    devices = await navigator.mediaDevices.enumerateDevices();
+  } catch (err) {
+    console.warn("[main] enumerateDevices failed:", err);
+  }
+
+  const inputs = devices.filter((d) => d.kind === "audioinput");
+  const outputs = devices.filter((d) => d.kind === "audiooutput");
+  const labelsReady = devices.some((d) => d.label);
+
+  fillDeviceSelect(inputAudioInput, inputs, settings.audioInputId, "Microphone");
+  fillDeviceSelect(inputAudioOutput, outputs, settings.audioOutputId, "Speaker");
+
+  if (!labelsReady) {
+    // Permission unlocks real device names; keep it quiet — user can tap Start
+    // or we unlock when they already connected once this session.
+    const hint = inputAudioInput.parentElement?.querySelector("small");
+    if (hint) {
+      hint.textContent =
+        "Allow microphone access (tap Start once) to see device names. Mic changes apply on Restart.";
+    }
+  } else {
+    const hint = inputAudioInput.parentElement?.querySelector("small");
+    if (hint) hint.textContent = "Applies on the next conversation (or Restart).";
+  }
+}
+
+/**
+ * @param {HTMLSelectElement} select
+ * @param {MediaDeviceInfo[]} devices
+ * @param {string} selectedId
+ * @param {string} fallbackLabel
+ */
+function fillDeviceSelect(select, devices, selectedId, fallbackLabel) {
+  const prev = selectedId || select.value || "";
+  select.replaceChildren();
+  const def = document.createElement("option");
+  def.value = "";
+  def.textContent = "System default";
+  select.appendChild(def);
+  devices.forEach((d, i) => {
+    const opt = document.createElement("option");
+    opt.value = d.deviceId;
+    opt.textContent = d.label || `${fallbackLabel} ${i + 1}`;
+    select.appendChild(opt);
+  });
+  // Keep a saved id even if it isn't currently listed (unplugged); browser
+  // will fall back via ideal constraints / setSinkId errors.
+  if (prev && ![...select.options].some((o) => o.value === prev)) {
+    const missing = document.createElement("option");
+    missing.value = prev;
+    missing.textContent = `${fallbackLabel} (saved, not found)`;
+    select.appendChild(missing);
+  }
+  select.value = prev;
+  if (select.value !== prev) select.value = "";
+}
+
+if (navigator.mediaDevices?.addEventListener) {
+  navigator.mediaDevices.addEventListener("devicechange", () => {
+    if (settingsModal.open) void refreshAudioDeviceLists();
+  });
+}
+
 /** Prompt for mic permission up front, then immediately release the tracks so no
  *  recording indicator lingers during a queue wait. Throws a friendly error if the
  *  user denies. */
 async function primeMicPermission() {
   try {
-    const s = await navigator.mediaDevices.getUserMedia(MIC_CONSTRAINTS);
+    const s = await navigator.mediaDevices.getUserMedia(micConstraints());
     for (const track of s.getTracks()) track.stop();
   } catch (err) {
     throw new Error(
@@ -1105,7 +1334,7 @@ async function primeMicPermission() {
 /** Acquire the live capture stream once a slot is granted. Permission was primed
  *  in the tap gesture, so this is silent. Stored module-side for mute + teardown. */
 async function acquireMicStream() {
-  micStream = await navigator.mediaDevices.getUserMedia(MIC_CONSTRAINTS);
+  micStream = await navigator.mediaDevices.getUserMedia(micConstraints());
   return micStream;
 }
 
@@ -1153,9 +1382,16 @@ function stopJoinCountdown() {
  * @param {AudioContext | null} [audioContext]
  */
 async function doStart(audioContext = null) {
+  const transport = effectiveTransport();
   // Resolve the target before touching mic/audio so a misconfiguration (e.g.
-  // direct mode with no URL) fails fast with a clear message.
-  const target = connectionTarget();
+  // direct mode with no URL) fails fast with a clear message. Over WebRTC the
+  // browser never dials the s2s server itself — the offer goes to the
+  // same-origin /api/calls proxy — so there is no target to resolve.
+  const target = transport === "webrtc" ? null : connectionTarget();
+  activeTransport = transport;
+  // The radial gate arc (threshold handle around the mic button) is a WS
+  // feature; over WebRTC only the mute button remains.
+  document.body.classList.toggle("rtc-live", transport === "webrtc");
 
   chat.clear();
   chat.reset();
@@ -1182,16 +1418,26 @@ async function doStart(audioContext = null) {
   // The webcam is started on arrival (autoStartCamera), so nothing to do here;
   // a still-pending grant just means the snapshot tool isn't ready yet.
 
-  const c = new S2sWsRealtimeClient({
-    ...target,
+  const common = {
     voice: settings.voice,
     instructions: effectiveInstructions(),
     startupGreeting: STARTUP_GREETING_PROMPT,
     acquireMic: acquireMicStream,
     tools: activeToolDefs(),
-    noiseGate: gateParams(settings.noiseGate),
+    audioOutputId: settings.audioOutputId || "",
     ...(audioContext ? { audioContext } : {}),
-  });
+  };
+  const c = target === null
+    ? new S2sRtcRealtimeClient({
+        callsUrl: "api/calls",
+        iceServers,
+        ...common,
+      })
+    : new S2sWsRealtimeClient({
+        ...target,
+        noiseGate: gateParams(settings.noiseGate),
+        ...common,
+      });
   client = c;
 
   c.addEventListener("queue", (e) => {
@@ -1410,6 +1656,7 @@ async function teardown() {
   // on the page), so we leave it on here — only the camera toggle stops it.
   micMuted = false;
   micBtn.classList.remove("muted");
+  document.body.classList.remove("rtc-live");
   setState("idle");
   // Refresh the chip's remaining-today after the budget moved.
   if (limiterOn) void account.refresh();

rtc/s2s-rtc-client.js

@@ -0,0 +1,864 @@
+// @ts-check
+/**
+ * Minimal WebRTC client for a speech-to-speech realtime endpoint.
+ *
+ * Sibling of `ws/s2s-ws-client.js` with the SAME public surface (constructor
+ * options subset, methods, dispatched events), so `main.js` can pick either
+ * class from the transport setting and wire it identically. Direct mode only:
+ * there is no load-balancer/queue path over WebRTC yet.
+ *
+ * Handshake (OpenAI Realtime GA "calls" endpoint, via the same-origin proxy):
+ *
+ *   1. getUserMedia -> add the mic track + create the `oai-events` data
+ *      channel on an RTCPeerConnection, `createOffer`, wait for ICE gathering.
+ *   2. POST the offer SDP (Content-Type: application/sdp) to `callsUrl`
+ *      (usually the same-origin `api/calls` proxy, which forwards it to the
+ *      s2s server's /v1/realtime/calls) -> 201 + answer SDP.
+ *   3. `setRemoteDescription(answer)`; once the data channel opens the server
+ *      pushes `session.created` and we reply with `session.update`.
+ *
+ * After that the JSON protocol on the data channel is the one the WebSocket
+ * transport speaks, with the audio moved out of it:
+ *
+ *   - Mic audio rides the RTP media track (Opus), NOT
+ *     `input_audio_buffer.append` — the server rejects `append` over WebRTC.
+ *   - Assistant audio arrives as a remote media track, NOT as
+ *     `response.output_audio.delta` events. There is no client playback
+ *     buffer: barge-in flushing happens server-side, so `speech_started`
+ *     only needs to flip the UI state here.
+ *
+ * Because no audio events exist, "the assistant is audibly speaking" is
+ * detected from the output analyser's RMS (with a short hang time), and
+ * `response.done` / `speech_started` remain the authoritative exits — the
+ * same status contract the WS client exposes.
+ *
+ * @typedef {"idle" | "connecting" | "connected" | "user-speaking" |
+ *           "processing" | "ai-speaking" | "closed" | "error"
+ * } RtcStatus
+ *
+ * @typedef {Object} RtcClientOptions
+ * @property {string} callsUrl URL to POST the SDP offer to (same-origin proxy
+ *   like `api/calls`, or a direct `/v1/realtime/calls` URL when CORS allows).
+ * @property {RTCIceServer[]} [iceServers] STUN/TURN servers for the peer
+ *   connection (from /api/config). Empty/absent -> browser defaults (host
+ *   candidates only — fine locally, may not traverse NATs).
+ * @property {string} voice
+ * @property {string} instructions
+ * @property {string} [startupGreeting] Hidden user prompt that asks the model
+ *   to greet once after the initial session configuration.
+ * @property {MediaStream} [micStream] Live mic stream. Provide this OR `acquireMic`.
+ * @property {() => Promise<MediaStream>} [acquireMic] Lazily obtain the mic
+ *   stream once connect() actually runs (same contract as the WS client).
+ * @property {AudioContext} [audioContext] Pre-created (and resumed) context —
+ *   created inside the tap gesture so iOS lets it start.
+ * @property {import("../ws/s2s-ws-client.js").ToolDef[]} [tools] Function tools
+ *   declared in the initial `session.update`.
+ * @property {string} [audioOutputId] MediaDeviceInfo.deviceId for speakers
+ *   (AudioContext.setSinkId when supported).
+ */
+
+import { extractResponseTranscript } from "../ws/codec.js";
+import { OrbVisualiser, VIS_FFT_SIZE } from "../ws/orb-visualizer.js";
+
+// The server's data channel label (aiortc side ignores any other label).
+const DATA_CHANNEL_LABEL = "oai-events";
+// Give up on the handshake if the data channel hasn't opened this long after
+// the SDP answer was applied (ICE failed silently, e.g. NAT without STUN).
+// The server holds its slot behind a 30 s watchdog; stay under it.
+const DC_OPEN_TIMEOUT_MS = 20_000;
+// Don't wait forever for ICE gathering before POSTing the offer — host
+// candidates land near-instantly; STUN answers within a couple of seconds.
+const ICE_GATHERING_TIMEOUT_MS = 3_000;
+// Output-RMS gate for "the assistant is audibly speaking": open above the
+// threshold, and hang on briefly so inter-word gaps don't flap the status.
+const SPEAKING_OPEN_DB = -50;
+const SPEAKING_HANG_MS = 250;
+const LEVEL_POLL_MS = 50;
+
+/** Build an Error carrying a `code` so callers can branch on the failure kind.
+ *  @param {string} message @param {string} code */
+function _codedError(message, code) {
+  const err = /** @type {Error & { code?: string }} */ (new Error(message));
+  err.code = code;
+  return err;
+}
+
+export class S2sRtcRealtimeClient extends EventTarget {
+  /** @param {RtcClientOptions} options */
+  constructor(options) {
+    super();
+    /** @type {RtcClientOptions} */
+    this.options = options;
+    /** @type {import("../ws/s2s-ws-client.js").ToolDef[]} */
+    this._tools = options.tools ?? [];
+    /** @type {(() => Promise<MediaStream>) | null} */
+    this._acquireMic = options.acquireMic ?? null;
+    /** @type {boolean} Set by close() so late async steps unwind quietly. */
+    this._closed = false;
+    /** @type {RTCPeerConnection | null} */
+    this._pc = null;
+    /** @type {RTCDataChannel | null} */
+    this._dc = null;
+    /** @type {AudioContext | null} */
+    this._ctx = null;
+    /** @type {MediaStreamAudioSourceNode | null} */
+    this._micSrc = null;
+    /** @type {MediaStreamAudioSourceNode | null} */
+    this._remoteSrc = null;
+    /** @type {HTMLAudioElement | null} Muted sink for the Chrome quirk: a
+     * remote WebRTC track stays silent in WebAudio unless the stream is ALSO
+     * attached to an <audio> element. Never added to the DOM. */
+    this._quirkAudio = null;
+    /** @type {AnalyserNode | null} */
+    this._micAnalyser = null;
+    /** @type {AnalyserNode | null} */
+    this._outAnalyser = null;
+    /** @type {OrbVisualiser | null} */
+    this._visualiser = null;
+    /** @type {number} Level/status poll timer (setInterval id). */
+    this._levelTimer = 0;
+    /** @type {Uint8Array} Scratch buffer for time-domain RMS reads. */
+    this._levelBuf = new Uint8Array(VIS_FFT_SIZE);
+    /** @type {number} Last time the output RMS was above the speaking gate. */
+    this._lastAudibleAt = 0;
+    /** @type {RtcStatus} */
+    this._status = "idle";
+    this._aiSpeaking = false;
+    /** @type {string} The response currently holding the backend slot (from
+     * response.created), so RMS-detected audio can be attributed to it. */
+    this._activeResponseId = "";
+    /** @type {Set<string>} response_ids that audibly played, so the UI can
+     * tell a barge-in cut (keep it) from a never-heard speculative response
+     * (drop it). Attribution is by "RMS opened while this response was
+     * active" — sound can't be tied to a response id without audio events. */
+    this._audibleResponses = new Set();
+    /** @type {Map<string, string>} CURRENT assistant transcript segment per
+     * response, accumulated from streamed deltas (reset on segment done). */
+    this._asstTranscriptByResp = new Map();
+    /** @type {Map<string, string>} Completed segments per response, joined. */
+    this._asstFullByResp = new Map();
+    this._muted = false;
+    // ── Response lock ────────────────────────────────────────────────────
+    // Same scheme as the WS client: the backend allows ONE response in
+    // flight, so response.create is serialized. `_openResponses` counts
+    // confirmed-but-unfinished responses; `_createInFlight` covers the window
+    // between our create and its response.created echo; extra creates queue.
+    this._openResponses = 0;
+    this._createInFlight = false;
+    /** @type {{ image?: string }[]} */
+    this._createQueue = [];
+    this._sessionConfigured = false;
+    this._startupGreeting = options.startupGreeting?.trim() ?? "";
+    this._startupGreetingSent = false;
+    this._debug = (() => { try { return localStorage.getItem("s2s.debug") === "1"; } catch { return false; } })();
+  }
+
+  get status() {
+    return this._status;
+  }
+
+  /** @param {RtcStatus} status */
+  _setStatus(status) {
+    if (this._status === status) return;
+    this._status = status;
+    this.dispatchEvent(new CustomEvent("status", { detail: { status } }));
+  }
+
+  /** Full assistant transcript so far for a response: completed segments plus
+   *  the in-progress one. @param {string} rid @returns {string} */
+  _asstDisplay(rid) {
+    const full = this._asstFullByResp.get(rid) || "";
+    const seg = this._asstTranscriptByResp.get(rid) || "";
+    if (!seg) return full;
+    return full ? `${full} ${seg}` : seg;
+  }
+
+  _markAudible() {
+    if (this._status === "ai-speaking") return;
+    if (this._status === "closed" || this._status === "error") return;
+    this._setStatus("ai-speaking");
+  }
+
+  /**
+   * Full handshake. Resolves once the data channel is open AND the audio
+   * graph is wired (mic track sending, remote track feeding the analysers).
+   * @returns {Promise<void>}
+   */
+  async connect() {
+    if (this._pc) throw new Error("Already connected");
+    this._setStatus("connecting");
+
+    if (!this.options.micStream && this._acquireMic) {
+      this.options.micStream = await this._acquireMic();
+    }
+    if (this._closed) throw _codedError("connect aborted", "aborted");
+
+    this._setupAudioGraph();
+
+    const pc = new RTCPeerConnection(
+      this.options.iceServers?.length ? { iceServers: this.options.iceServers } : {},
+    );
+    this._pc = pc;
+
+    pc.addEventListener("track", (e) => this._onRemoteTrack(e));
+    pc.addEventListener("connectionstatechange", () => this._onConnectionState());
+
+    const micTrack = this.options.micStream?.getAudioTracks()[0];
+    if (!micTrack) throw new Error("No microphone track available");
+    micTrack.enabled = !this._muted;
+    pc.addTrack(micTrack, /** @type {MediaStream} */ (this.options.micStream));
+
+    // The client opens the events channel (OpenAI convention); the server
+    // waits for it and ignores channels with any other label.
+    const dc = pc.createDataChannel(DATA_CHANNEL_LABEL, { ordered: true });
+    this._dc = dc;
+    dc.addEventListener("message", (e) => this._onDcMessage(e.data));
+    dc.addEventListener("close", () => this._onDcClose());
+
+    await pc.setLocalDescription(await pc.createOffer());
+    // No trickle ICE over the one-shot HTTP handshake: the offer must carry
+    // our candidates, so wait for gathering (bounded — host candidates are
+    // near-instant, and a STUN timeout shouldn't stall the connect).
+    await this._waitIceGathering(pc);
+    if (this._closed) throw _codedError("connect aborted", "aborted");
+
+    const answerSdp = await this._postOffer(pc.localDescription?.sdp ?? "");
+    if (this._closed) throw _codedError("connect aborted", "aborted");
+    await pc.setRemoteDescription({ type: "answer", sdp: answerSdp });
+
+    await this._waitDataChannelOpen(dc);
+    this._startLevelLoop();
+    // The server pushes session.created once it sees the channel open; the
+    // session.update reply is sent from that handler (mirrors the WS client).
+  }
+
+  /** @param {RTCPeerConnection} pc */
+  _waitIceGathering(pc) {
+    if (pc.iceGatheringState === "complete") return Promise.resolve();
+    return new Promise((resolve) => {
+      const done = () => {
+        pc.removeEventListener("icegatheringstatechange", check);
+        clearTimeout(timer);
+        resolve(undefined);
+      };
+      const check = () => {
+        if (pc.iceGatheringState === "complete") done();
+      };
+      const timer = setTimeout(() => {
+        console.warn("[rtc] ICE gathering timed out; sending offer with partial candidates");
+        done();
+      }, ICE_GATHERING_TIMEOUT_MS);
+      pc.addEventListener("icegatheringstatechange", check);
+    });
+  }
+
+  /**
+   * POST the SDP offer, return the answer SDP. Non-2xx bodies are surfaced as
+   * errors ("all slots in use" arrives as a JSON error event with a 503).
+   * @param {string} offerSdp @returns {Promise<string>}
+   */
+  async _postOffer(offerSdp) {
+    console.log("[rtc] POST", this.options.callsUrl);
+    const response = await fetch(this.options.callsUrl, {
+      method: "POST",
+      headers: { "Content-Type": "application/sdp" },
+      body: offerSdp,
+    });
+    if (!response.ok) {
+      const text = await response.text().catch(() => "");
+      if (response.status === 503) {
+        // The s2s server rejects with a JSON error event when the pool is full.
+        try {
+          const j = JSON.parse(text);
+          const msg = j?.error?.message;
+          if (msg) throw _codedError(msg, "busy");
+        } catch (err) {
+          if (err instanceof Error && /** @type {any} */ (err).code === "busy") throw err;
+        }
+        throw _codedError("The speech service is busy — try again shortly.", "busy");
+      }
+      throw new Error(`WebRTC handshake failed (${response.status}): ${text.slice(0, 200)}`);
+    }
+    return response.text();
+  }
+
+  /** @param {RTCDataChannel} dc */
+  _waitDataChannelOpen(dc) {
+    if (dc.readyState === "open") return Promise.resolve();
+    return new Promise((resolve, reject) => {
+      const cleanup = () => {
+        clearTimeout(timer);
+        dc.removeEventListener("open", onOpen);
+        dc.removeEventListener("close", onClose);
+        dc.removeEventListener("error", onClose);
+      };
+      const onOpen = () => {
+        cleanup();
+        resolve(undefined);
+      };
+      const onClose = () => {
+        cleanup();
+        reject(new Error("WebRTC data channel closed before opening"));
+      };
+      const timer = setTimeout(() => {
+        cleanup();
+        reject(new Error(
+          "WebRTC connection timed out. If the server is remote, it may need a STUN/TURN server (RTC_ICE_SERVERS).",
+        ));
+      }, DC_OPEN_TIMEOUT_MS);
+      dc.addEventListener("open", onOpen);
+      dc.addEventListener("close", onClose);
+      dc.addEventListener("error", onClose);
+    });
+  }
+
+  // ── Audio graph ───────────────────────────────────────────────────────────
+  // Mic:    micStream ─→ micAnalyser (orb bars + level meter; the track itself
+  //         goes to the peer connection untouched).
+  // Output: remote track ─→ remoteSrc ─→ outAnalyser ─→ destination, plus the
+  //         hidden muted <audio> that makes Chrome actually run the track.
+
+  _setupAudioGraph() {
+    const ctx = this.options.audioContext ?? new AudioContext({ latencyHint: "interactive" });
+    this._ctx = ctx;
+    if (ctx.state === "suspended") {
+      // Best-effort here — on iOS the resume that counts happened in the tap.
+      void ctx.resume().catch((err) => console.warn("[rtc] AudioContext resume failed:", err));
+    }
+
+    const micAnalyser = ctx.createAnalyser();
+    micAnalyser.fftSize = VIS_FFT_SIZE;
+    micAnalyser.smoothingTimeConstant = 0;
+    const micSrc = ctx.createMediaStreamSource(/** @type {MediaStream} */ (this.options.micStream));
+    micSrc.connect(micAnalyser);
+    this._micSrc = micSrc;
+    this._micAnalyser = micAnalyser;
+
+    const outAnalyser = ctx.createAnalyser();
+    outAnalyser.fftSize = VIS_FFT_SIZE;
+    outAnalyser.smoothingTimeConstant = 0.3;
+    outAnalyser.connect(ctx.destination);
+    this._outAnalyser = outAnalyser;
+
+    void this.setAudioOutputDevice(this.options.audioOutputId || "");
+
+    this._visualiser = new OrbVisualiser(micAnalyser, outAnalyser, () => this._aiSpeaking);
+    this._visualiser.start();
+  }
+
+  /**
+   * Route Web Audio playback to a specific output device (Chrome/Edge).
+   * @param {string} [deviceId]
+   * @returns {Promise<boolean>}
+   */
+  async setAudioOutputDevice(deviceId = "") {
+    const ctx = this._ctx;
+    if (!ctx || typeof /** @type {any} */ (ctx).setSinkId !== "function") return false;
+    try {
+      await /** @type {any} */ (ctx).setSinkId(deviceId || "");
+      return true;
+    } catch (err) {
+      console.warn("[rtc] setSinkId failed:", err);
+      return false;
+    }
+  }
+
+  /** @param {RTCTrackEvent} e */
+  _onRemoteTrack(e) {
+    if (e.track.kind !== "audio" || !this._ctx || !this._outAnalyser) return;
+    const stream = e.streams[0] ?? new MediaStream([e.track]);
+    // Chrome quirk: without an <audio> sink the remote track produces silence
+    // in WebAudio. Muted so playback happens once, through the analyser path.
+    const quirk = new Audio();
+    quirk.muted = true;
+    quirk.srcObject = stream;
+    this._quirkAudio = quirk;
+    this._remoteSrc?.disconnect();
+    this._remoteSrc = this._ctx.createMediaStreamSource(stream);
+    this._remoteSrc.connect(this._outAnalyser);
+    if (this._debug) console.debug("[rtc] remote audio track attached");
+  }
+
+  // ── Output-RMS status + mic level ─────────────────────────────────────────
+
+  _startLevelLoop() {
+    if (this._levelTimer) return;
+    this._levelTimer = window.setInterval(() => this._pollLevels(), LEVEL_POLL_MS);
+  }
+
+  /** RMS (0..1) of an analyser's current time-domain buffer.
+   *  @param {AnalyserNode} analyser */
+  _rmsOf(analyser) {
+    analyser.getByteTimeDomainData(this._levelBuf);
+    let sum = 0;
+    for (let i = 0; i < this._levelBuf.length; i++) {
+      const s = (this._levelBuf[i] - 128) / 128;
+      sum += s * s;
+    }
+    return Math.sqrt(sum / this._levelBuf.length);
+  }
+
+  _pollLevels() {
+    if (!this._micAnalyser || !this._outAnalyser) return;
+
+    // Mic level for the Settings meter / gate arc (raw, pre-mute).
+    this.dispatchEvent(
+      new CustomEvent("input-level", { detail: { rms: this._rmsOf(this._micAnalyser) } }),
+    );
+
+    // Speaking gate on the output: RMS opens the state; `response.done` and
+    // `speech_started` (the protocol's authoritative signals) close it.
+    const rms = this._rmsOf(this._outAnalyser);
+    const db = rms > 0 ? 20 * Math.log10(rms) : -Infinity;
+    const now = performance.now();
+    if (db > SPEAKING_OPEN_DB) this._lastAudibleAt = now;
+    const audible = now - this._lastAudibleAt < SPEAKING_HANG_MS;
+    if (audible && !this._aiSpeaking) {
+      this._aiSpeaking = true;
+      if (this._activeResponseId) this._audibleResponses.add(this._activeResponseId);
+      this._markAudible();
+    }
+  }
+
+  // ── Data channel protocol ─────────────────────────────────────────────────
+
+  /** @param {unknown} raw */
+  _onDcMessage(raw) {
+    if (typeof raw !== "string") return;
+    let event;
+    try {
+      event = JSON.parse(raw);
+    } catch {
+      return;
+    }
+
+    const type = event?.type;
+    if (typeof type !== "string") return;
+    if (this._debug) {
+      const extra = type.startsWith("conversation.item.input_audio_transcription")
+        ? ` item=${event.item_id} ${event.delta ?? event.transcript ?? ""}`
+        : type.startsWith("response.")
+          ? ` resp=${event.response_id ?? event.response?.id ?? ""} status=${event.response?.status ?? ""}`
+          : "";
+      console.debug(`[rtc] ${type}${extra}`);
+    }
+
+    switch (type) {
+      case "session.created":
+        // Server-side defaults are already what we want; push only the
+        // user-tunable bits (voice, instructions, tools) — same as WS.
+        this._sendSessionUpdate();
+        this._sessionConfigured = true;
+        // Data-channel messages are ordered, so the hidden item and
+        // response.create are handled after the session.update above.
+        this._sendStartupGreeting();
+        if (this._status === "connecting") this._setStatus("connected");
+        break;
+
+      case "session.updated":
+        break;
+
+      case "input_audio_buffer.speech_started":
+        // Barge-in: unlike WS there is no client playback buffer to clear —
+        // the server flushes its track buffer — so this is UI state only.
+        this._aiSpeaking = false;
+        this._lastAudibleAt = 0;
+        this._setStatus("user-speaking");
+        break;
+
+      case "input_audio_buffer.speech_stopped":
+        if (this._status === "user-speaking") this._setStatus("processing");
+        break;
+
+      case "response.created":
+        this._openResponses++;
+        this._createInFlight = false;
+        this._activeResponseId = event.response?.id ?? "";
+        if (this._status === "connected" || this._status === "user-speaking") {
+          this._setStatus("processing");
+        }
+        break;
+
+      case "response.output_item.added":
+        if (this._status === "connected" || this._status === "user-speaking") {
+          this._setStatus("processing");
+        }
+        break;
+
+      case "response.done": {
+        this._aiSpeaking = false;
+        this._lastAudibleAt = 0;
+        this._openResponses = Math.max(0, this._openResponses - 1);
+        if (this._status === "ai-speaking" || this._status === "processing") {
+          this._setStatus("connected");
+        }
+        // Completion AND cancellation both arrive as response.done (status
+        // "cancelled") — same contract the WS client documents.
+        const status = event.response?.status ?? "completed";
+        const responseId = event.response?.id ?? "";
+        if (this._activeResponseId === responseId) this._activeResponseId = "";
+        const audible = responseId ? this._audibleResponses.has(responseId) : false;
+        this._audibleResponses.delete(responseId);
+        const transcript =
+          extractResponseTranscript(event.response) ||
+          this._asstDisplay(responseId) ||
+          "";
+        this._asstTranscriptByResp.delete(responseId);
+        this._asstFullByResp.delete(responseId);
+        this.dispatchEvent(new CustomEvent("response-finished", {
+          detail: { responseId, status, audible, transcript },
+        }));
+        this._flushQueuedCreate();
+        break;
+      }
+
+      case "response.function_call_arguments.done": {
+        const name = typeof event.name === "string" ? event.name : "";
+        const args = typeof event.arguments === "string" ? event.arguments : "{}";
+        const callId = typeof event.call_id === "string" ? event.call_id : "";
+        if (name) {
+          this.dispatchEvent(new CustomEvent("toolcall", {
+            detail: { name, arguments: args, callId },
+          }));
+        } else {
+          console.warn(`[rtc] function_call_arguments.done with no name (call_id=${callId}); cannot run tool — turn may stall`);
+        }
+        break;
+      }
+
+      case "conversation.item.input_audio_transcription.delta": {
+        const delta = typeof event.delta === "string" ? event.delta : "";
+        if (delta) {
+          // The delta carries the full cumulative transcript so far; itemId is
+          // reused across a speculative continuation so the UI groups them.
+          this.dispatchEvent(
+            new CustomEvent("transcript", {
+              detail: {
+                role: "user",
+                text: delta,
+                partial: true,
+                itemId: typeof event.item_id === "string" ? event.item_id : "",
+              },
+            }),
+          );
+        }
+        break;
+      }
+
+      case "conversation.item.input_audio_transcription.completed": {
+        const transcript = typeof event.transcript === "string" ? event.transcript : "";
+        if (transcript) {
+          this.dispatchEvent(
+            new CustomEvent("transcript", {
+              detail: {
+                role: "user",
+                text: transcript,
+                partial: false,
+                itemId: typeof event.item_id === "string" ? event.item_id : "",
+              },
+            }),
+          );
+        }
+        break;
+      }
+
+      case "response.audio_transcript.delta":
+      case "response.output_audio_transcript.delta": {
+        this._markAudible();
+        const rid = typeof event.response_id === "string" ? event.response_id : "";
+        const delta = typeof event.delta === "string" ? event.delta : "";
+        if (delta) {
+          this._asstTranscriptByResp.set(rid, (this._asstTranscriptByResp.get(rid) || "") + delta);
+          this.dispatchEvent(
+            new CustomEvent("transcript", {
+              detail: { role: "assistant", text: this._asstDisplay(rid), partial: true, responseId: rid },
+            }),
+          );
+        }
+        break;
+      }
+
+      case "response.audio_transcript.done":
+      case "response.output_audio_transcript.done": {
+        const rid = typeof event.response_id === "string" ? event.response_id : "";
+        // ONE completed segment; a response can emit several — concatenate
+        // until response.done clears the accumulator.
+        const segment =
+          (typeof event.transcript === "string" && event.transcript) ||
+          this._asstTranscriptByResp.get(rid) ||
+          "";
+        this._asstTranscriptByResp.delete(rid);
+        if (segment) {
+          const prev = this._asstFullByResp.get(rid) || "";
+          this._asstFullByResp.set(rid, prev ? `${prev} ${segment}` : segment);
+        }
+        const full = this._asstFullByResp.get(rid) || "";
+        if (full) {
+          this.dispatchEvent(
+            new CustomEvent("transcript", {
+              detail: { role: "assistant", text: full, partial: false, responseId: rid },
+            }),
+          );
+        }
+        break;
+      }
+
+      case "error": {
+        const err = event.error;
+        console.error("[rtc] server error:", err);
+        // Our optimistic create collided with a still-running response: clear
+        // the guard and re-queue for the next response.done (never retried
+        // immediately — that would just collide again).
+        if (err?.type === "conversation_already_has_active_response" ||
+            err?.code === "conversation_already_has_active_response") {
+          if (this._createInFlight) {
+            this._createInFlight = false;
+            this._createQueue.push({});
+          }
+          break;
+        }
+        // Every other server error is non-fatal: surface for logging, keep
+        // the session alive. Transport failures come through their own paths.
+        this.dispatchEvent(
+          new CustomEvent("server-error", { detail: { error: new Error(err?.message ?? "Server error") } }),
+        );
+        break;
+      }
+    }
+  }
+
+  _onDcClose() {
+    if (this._closed) return;
+    if (this._status === "closed" || this._status === "error") return;
+    console.log("[rtc] data channel closed by server");
+    this.dispatchEvent(
+      new CustomEvent("error", { detail: { error: new Error("Connection closed by the server") } }),
+    );
+    this._setStatus("error");
+  }
+
+  _onConnectionState() {
+    const state = this._pc?.connectionState;
+    if (this._debug) console.debug(`[rtc] connection state: ${state}`);
+    if (this._closed) return;
+    if (state === "disconnected") {
+      // Transient: browsers fire this on brief packet loss and usually recover
+      // to "connected" on their own; the terminal state is "failed", which
+      // fires this handler again if recovery doesn't happen.
+      console.warn("[rtc] connection disconnected — waiting for recovery or failure");
+      return;
+    }
+    if (state === "failed") {
+      if (this._status === "closed" || this._status === "error") return;
+      this.dispatchEvent(
+        new CustomEvent("error", { detail: { error: new Error(`WebRTC connection ${state}`) } }),
+      );
+      this._setStatus("error");
+    }
+  }
+
+  _sendSessionUpdate() {
+    // Minimal payload, exactly like the WS client: the server's pydantic
+    // validator rejects the whole event on unknown sub-field shapes.
+    /** @type {Record<string, any>} */
+    const session = {
+      type: "realtime",
+      instructions: this.options.instructions,
+      audio: {
+        output: { voice: this.options.voice },
+      },
+    };
+    if (this._tools.length) {
+      session.tools = this._tools;
+      session.tool_choice = "auto";
+    }
+    this._send({ type: "session.update", session });
+  }
+
+  /** Update voice/instructions on a live session without tearing down.
+   *  @param {{ voice?: string; instructions?: string }} patch */
+  updateSession(patch) {
+    /** @type {Record<string, any>} */
+    const session = { type: "realtime" };
+    if (patch.instructions) session.instructions = patch.instructions;
+    if (patch.voice) session.audio = { output: { voice: patch.voice } };
+    if (Object.keys(session).length > 1) {
+      this._send({ type: "session.update", session });
+    }
+  }
+
+  /** Replace the declared tool set on a live session. Always sends `tools` —
+   *  an empty array clears them. @param {import("../ws/s2s-ws-client.js").ToolDef[]} tools */
+  setTools(tools) {
+    this._tools = tools;
+    this._send({
+      type: "session.update",
+      session: { type: "realtime", tools, tool_choice: tools.length ? "auto" : "none" },
+    });
+  }
+
+  /** Return a tool's result to the model. @param {string} callId @param {string} output */
+  sendToolOutput(callId, output) {
+    if (!callId) return;
+    this._send({
+      type: "conversation.item.create",
+      item: { type: "function_call_output", call_id: callId, output },
+    });
+  }
+
+  /** Add an image to the conversation as user content (camera tool). The
+   *  caller keeps `dataUrl` under the data-channel message budget — SCTP
+   *  messages above the negotiated max (64 KiB on aiortc) fail to send.
+   *  @param {string} dataUrl */
+  sendUserImage(dataUrl) {
+    this._send({
+      type: "conversation.item.create",
+      item: {
+        type: "message",
+        role: "user",
+        content: [{ type: "input_image", image_url: dataUrl }],
+      },
+    });
+  }
+
+  /**
+   * Ask the model to open the conversation exactly once. The synthetic user
+   * prompt stays in conversation history, warming the prompt prefix reused by
+   * the first spoken turn.
+   */
+  _sendStartupGreeting() {
+    if (!this._startupGreeting || this._startupGreetingSent) return;
+    this._startupGreetingSent = true;
+    this._send({
+      type: "conversation.item.create",
+      item: {
+        type: "message",
+        role: "user",
+        content: [{ type: "input_text", text: this._startupGreeting }],
+      },
+    });
+    this.requestResponse();
+    if (this._debug) console.debug("[rtc] startup greeting queued");
+  }
+
+  /** Ask the model to respond now (after tool results). Serialized behind the
+   *  single-response slot, same as the WS client.
+   *  @param {{ image?: string }} [opts] */
+  requestResponse(opts = {}) {
+    if (this._responseActive()) {
+      this._createQueue.push(opts);
+      if (this._debug) console.debug(`[rtc] response.create queued; pending=${this._createQueue.length}`);
+      return;
+    }
+    this._createResponseNow(opts);
+  }
+
+  _responseActive() {
+    return this._openResponses > 0 || this._createInFlight;
+  }
+
+  /** @param {{ image?: string }} [opts] */
+  _createResponseNow(opts = {}) {
+    if (!this._dc || this._dc.readyState !== "open") return;
+    if (opts.image) this.sendUserImage(opts.image);
+    this._createInFlight = true;
+    this._send({ type: "response.create" });
+  }
+
+  _flushQueuedCreate() {
+    if (this._createQueue.length > 0 && !this._responseActive()) {
+      const opts = this._createQueue.shift();
+      if (this._debug) console.debug(`[rtc] replaying queued response.create; remaining=${this._createQueue.length}`);
+      this._createResponseNow(opts);
+    }
+  }
+
+  /** @param {boolean} muted */
+  setMuted(muted) {
+    this._muted = muted;
+    // The caller (main.js) also toggles the shared micStream tracks; doing it
+    // here too keeps the client correct when driven standalone. A disabled
+    // track makes the browser transmit silence — the server VAD stays quiet.
+    for (const track of this.options.micStream?.getAudioTracks() ?? []) {
+      track.enabled = !muted;
+    }
+  }
+
+  /** Noise gate is a WebSocket-transport feature (implemented in its capture
+   *  worklet); the WebRTC mic path sends the raw track, so this is a no-op.
+   *  @param {import("../ws/s2s-ws-client.js").NoiseGate} _gate */
+  setNoiseGate(_gate) {}
+
+  /** Queue "join" gate — LB-mode only, which WebRTC doesn't support. Present
+   *  so the two clients keep the same surface for the caller. */
+  join() {}
+
+  /** @param {Record<string, unknown>} event */
+  _send(event) {
+    if (!this._dc || this._dc.readyState !== "open") return;
+    try {
+      this._dc.send(JSON.stringify(event));
+    } catch (err) {
+      // A message over the SCTP limit (or a racing close) lands here; keep
+      // the session alive and let the caller's flow recover.
+      console.error("[rtc] data channel send failed:", err);
+    }
+  }
+
+  async close() {
+    this._closed = true;
+    if (this._levelTimer) {
+      clearInterval(this._levelTimer);
+      this._levelTimer = 0;
+    }
+    this._visualiser?.stop();
+    this._visualiser = null;
+    try {
+      this._dc?.close();
+    } catch {
+      // ignored
+    }
+    this._dc = null;
+    try {
+      this._pc?.close();
+    } catch {
+      // ignored
+    }
+    this._pc = null;
+    if (this._quirkAudio) {
+      this._quirkAudio.srcObject = null;
+      this._quirkAudio = null;
+    }
+    try {
+      this._remoteSrc?.disconnect();
+    } catch {
+      // ignored
+    }
+    try {
+      this._micSrc?.disconnect();
+    } catch {
+      // ignored
+    }
+    try {
+      this._micAnalyser?.disconnect();
+    } catch {
+      // ignored
+    }
+    try {
+      this._outAnalyser?.disconnect();
+    } catch {
+      // ignored
+    }
+    try {
+      await this._ctx?.close();
+    } catch {
+      // ignored
+    }
+    this._ctx = null;
+    this._remoteSrc = null;
+    this._micSrc = null;
+    this._micAnalyser = null;
+    this._outAnalyser = null;
+    this._setStatus("closed");
+  }
+}

server.py

@@ -23,9 +23,10 @@ disabled entirely (no session proxy, no queue, no metering, no sign-in) and the
 browser connects directly to that URL, shown read-only in Settings.
 
 Endpoints:
-  GET  /api/config           -> { search, lb, allowDirect, s2sUrl, auth }
+  GET  /api/config           -> { search, lb, allowDirect, s2sUrl, rtc, iceServers, auth }
   GET  /api/me               -> login + tier + remaining budget (LB mode only)
   POST /api/search           -> { results, answer }  Google via Serper.dev
+  POST /api/calls            -> proxies the WebRTC SDP offer to <s2s>/v1/realtime/calls
   POST /api/session          -> proxies <LB>/session: a grant, or a queue ticket
   GET  /api/queue/{id}       -> proxies <LB>/queue/{id}: position, or a grant on claim
   DELETE /api/queue/{id}     -> leave the queue (explicit "Leave queue" button)
@@ -41,11 +42,13 @@ the moment a slot is actually claimed (a grant), never while queued.
 """
 
 import asyncio
+import json
 import logging
 import os
+from urllib.parse import urlsplit, urlunsplit
 
 import httpx
-from fastapi import FastAPI, HTTPException, Request
+from fastapi import FastAPI, HTTPException, Request, Response
 from fastapi.responses import JSONResponse
 from fastapi.staticfiles import StaticFiles
 from pydantic import BaseModel
@@ -77,6 +80,48 @@ if SPEECH_TO_SPEECH_URL:
 # but nothing is metered: no budget, no reservations, no sign-in gating.
 SPACE_ID = os.environ.get("SPACE_ID", "").strip()
 LIMITER_ENABLED = bool(LOAD_BALANCER_URL) and bool(SPACE_ID)
+
+
+def _parse_ice_servers(raw: str) -> list:
+    """ICE servers for the browser's RTCPeerConnection, from RTC_ICE_SERVERS.
+
+    Accepts a JSON list of RTCIceServer dicts (same format as the s2s
+    server's SPEECH_TO_SPEECH_ICE_SERVERS, e.g.
+    ``[{"urls": "turn:t.example.com", "username": "u", "credential": "c"}]``),
+    a single such dict, or a plain comma-separated list of STUN/TURN URLs.
+    Empty when unset — host candidates only, which is fine for local use."""
+    raw = raw.strip()
+    if not raw:
+        return []
+    try:
+        data = json.loads(raw)
+        if isinstance(data, list):
+            return data
+        if isinstance(data, dict):
+            return [data]
+    except ValueError:
+        pass
+    return [{"urls": u.strip()} for u in raw.split(",") if u.strip()]
+
+
+RTC_ICE_SERVERS = _parse_ice_servers(os.environ.get("RTC_ICE_SERVERS", ""))
+
+
+def _webrtc_calls_url(s2s_url: str) -> str:
+    """Derive the WebRTC handshake URL from the pinned realtime URL.
+
+    ``ws://host:port/v1/realtime`` -> ``http://host:port/v1/realtime/calls``
+    (ws->http, wss->https; a bare host gets the default /v1/realtime path,
+    mirroring the client's buildDirectWsUrl normalisation)."""
+    s = s2s_url.strip()
+    if not s.startswith(("ws://", "wss://", "http://", "https://")):
+        s = "http://" + s
+    parts = urlsplit(s)
+    scheme = {"ws": "http", "wss": "https"}.get(parts.scheme, parts.scheme)
+    path = parts.path if parts.path not in ("", "/") else "/v1/realtime"
+    return urlunsplit((scheme, parts.netloc, path.rstrip("/") + "/calls", parts.query, ""))
+
+
 SERPER_URL = "https://google.serper.dev/search"
 # Cap results so the tool output stays small enough to feed back to the model.
 MAX_RESULTS = 5
@@ -128,6 +173,11 @@ def config():
         # Deploy-pinned direct s2s URL (empty when unset). Not a secret: the
         # browser dials it itself, and Settings shows it locked.
         "s2sUrl": SPEECH_TO_SPEECH_URL,
+        # WebRTC transport availability: the /api/calls proxy only forwards to
+        # the env-pinned URL (never a client-supplied one), so the toggle is
+        # offered exactly when that URL exists.
+        "rtc": bool(SPEECH_TO_SPEECH_URL),
+        "iceServers": RTC_ICE_SERVERS,
         "auth": AUTH_ENABLED,
     }
 
@@ -215,6 +265,46 @@ async def search(req: SearchRequest):
     return JSONResponse({"query": query, "answer": answer, "results": results})
 
 
+@app.post("/api/calls")
+async def calls(request: Request):
+    """Proxy the WebRTC SDP handshake to the pinned s2s server.
+
+    The browser can't POST /v1/realtime/calls cross-origin (the s2s server has
+    no CORS middleware, and an application/sdp POST is preflighted), so it
+    posts the offer here and we forward it server-side. Only the signaling hop
+    goes through this proxy — the negotiated audio/data-channel media flows
+    directly between the browser and the s2s server.
+
+    Deliberately forwards ONLY to SPEECH_TO_SPEECH_URL: honouring a
+    client-supplied target would make this an open proxy (SSRF). No env pin,
+    no WebRTC — the client keeps such setups on the WebSocket transport."""
+    if not SPEECH_TO_SPEECH_URL:
+        raise HTTPException(status_code=404, detail="Not found.")
+
+    offer = await request.body()
+    url = _webrtc_calls_url(SPEECH_TO_SPEECH_URL)
+    try:
+        # Generous timeout: the s2s server waits for its own ICE gathering
+        # (up to ~5 s) before returning the answer.
+        async with httpx.AsyncClient(timeout=15.0) as http:
+            resp = await http.post(url, headers={"Content-Type": "application/sdp"}, content=offer)
+    except httpx.RequestError as exc:
+        logger.warning("s2s calls endpoint unreachable: %r", exc)
+        raise HTTPException(status_code=502, detail="Speech service unreachable.")
+
+    # Relay the answer (or the error body) as-is; keep the Location header the
+    # s2s server sets on success (the call id, per the OpenAI GA contract).
+    headers = {}
+    if "location" in resp.headers:
+        headers["Location"] = resp.headers["location"]
+    return Response(
+        content=resp.content,
+        status_code=resp.status_code,
+        media_type=resp.headers.get("content-type", "application/sdp"),
+        headers=headers,
+    )
+
+
 @app.post("/api/session")
 async def session(request: Request):
     """Proxy the session handshake to the load balancer, keeping its URL secret,

style.css

@@ -1009,6 +1009,9 @@ a:hover {
   transition: opacity 0.25s ease;
 }
 .orb-wrap.live .mic-gate-arc { opacity: 1; }
+/* The gate is a WebSocket-transport feature (it lives in the WS capture
+ * worklet). During a WebRTC call only the mute button remains. */
+body.rtc-live .mic-gate-arc { display: none; }
 .mga-track {
   stroke: rgba(255, 255, 255, 0.1); /* hairline; recedes until needed */
   stroke-width: 1.5;

ws/s2s-ws-client.js

@@ -67,6 +67,8 @@
  *   executes and replies via `sendToolOutput` + `requestResponse`.
  * @property {NoiseGate} [noiseGate] Client-side noise gate applied to the mic
  *   before it's sent. Tunable live via `setNoiseGate`.
+ * @property {string} [audioOutputId] MediaDeviceInfo.deviceId for speakers.
+ *   Applied via AudioContext.setSinkId when the browser supports it.
  *
  * @typedef {Object} NoiseGate
  * @property {boolean} enabled
@@ -530,10 +532,30 @@ export class S2sWsRealtimeClient extends EventTarget {
     this._outAnalyser = outAnalyser;
     this._playbackNode = playbackNode;
 
+    await this.setAudioOutputDevice(this.options.audioOutputId || "");
+
     this._visualiser = new OrbVisualiser(micAnalyser, outAnalyser, () => this._aiSpeaking);
     this._visualiser.start();
   }
 
+  /**
+   * Route Web Audio playback to a specific output device (Chrome/Edge).
+   * Empty string restores the system default.
+   * @param {string} [deviceId]
+   * @returns {Promise<boolean>}
+   */
+  async setAudioOutputDevice(deviceId = "") {
+    const ctx = this._ctx;
+    if (!ctx || typeof /** @type {any} */ (ctx).setSinkId !== "function") return false;
+    try {
+      await /** @type {any} */ (ctx).setSinkId(deviceId || "");
+      return true;
+    } catch (err) {
+      console.warn("[ws] setSinkId failed:", err);
+      return false;
+    }
+  }
+
   /** @param {string} connectUrl */
   _openWebSocket(connectUrl) {
     return new Promise((resolve, reject) => {