{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "intro",
   "metadata": {},
   "source": [
    "# Colab → VSCode bridge\n",
    "\n",
    "Run this notebook **on Colab** (T4 GPU runtime). It clones the stocker repo,\n",
    "installs deps, then starts a Jupyter server tunneled via cloudflared.\n",
    "\n",
    "Then on your **local machine**, open VSCode → Command Palette →\n",
    "*Jupyter: Specify Jupyter Server for Connections* → paste the URL printed\n",
    "by the last cell. Open `training/train_grpo.ipynb` locally; pick the remote\n",
    "kernel; cells now execute on Colab's GPU while you edit in VSCode.\n",
    "\n",
    "Caveat: the kernel runs on Colab, so file paths must match what the kernel\n",
    "sees. The launcher `cd`s into `/content/stocker` and adds it to `sys.path`\n",
    "before starting the server, so `import app...` works out of the box.\n",
    "Files you save inside the notebook write to Colab's filesystem — push them\n",
    "back via `git push` from a Colab cell, or mount Drive (cell 0)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "drive",
   "metadata": {},
   "outputs": [],
   "source": [
    "# OPTIONAL — mount Drive so artifacts under training/runs/ persist across sessions.\n",
    "# Skip if you'll just download artifacts at the end of the run.\n",
    "# from google.colab import drive\n",
    "# drive.mount('/content/drive')\n",
    "pass"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "clone",
   "metadata": {},
   "outputs": [],
   "source": [
    "import os\n",
    "REPO_URL = 'https://github.com/CRIMSONHydra/stocker.git'   # <-- edit me\n",
    "WORKDIR  = '/content/stocker'\n",
    "if not os.path.isdir(WORKDIR):\n",
    "    assert '<your-username>' not in REPO_URL, 'Edit REPO_URL first.'\n",
    "    !git clone {REPO_URL} {WORKDIR}\n",
    "%cd {WORKDIR}\n",
    "!git pull --rebase 2>/dev/null || true"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "deps",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Same dep set as train_grpo.ipynb's install cell.\n",
    "!pip install -q -U 'transformers>=4.55' 'trl>=0.11' 'peft>=0.13' 'accelerate>=1.0' 'bitsandbytes>=0.43' 'datasets>=3.0'\n",
    "!pip install -q --upgrade-strategy=only-if-needed yfinance mplfinance pyarrow 'pydantic>=2' pydantic-settings 'openai>=1' tensorboard 'huggingface_hub>=1.10' jupyter-server"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "cloudflared",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Install cloudflared (free trycloudflare.com tunnel, no signup).\n",
    "!wget -q -O /usr/local/bin/cloudflared https://github.com/cloudflare/cloudflared/releases/latest/download/cloudflared-linux-amd64\n",
    "!chmod +x /usr/local/bin/cloudflared\n",
    "!cloudflared --version"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "build-data",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Pre-build the bundled dataset on Colab so VSCode-driven cells can\n",
    "# `from app.data import loader` immediately. Idempotent.\n",
    "!python scripts/build_dataset.py\n",
    "!python scripts/validate_tasks.py"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "launch",
   "metadata": {},
   "outputs": [],
   "source": "import os, re, secrets, signal, socket, subprocess, sys, time, pathlib\n\nTOKEN = secrets.token_urlsafe(24)\nPORT  = 9988\nLOG_J = pathlib.Path('/content/jupyter.log')\nLOG_C = pathlib.Path('/content/cloudflared.log')\nWORKDIR = '/content/stocker'\n\n# We track the PIDs of OUR own children so re-running this cell can clean up\n# without `pkill -f jupyter-server` (which would also kill Colab's backend\n# and disconnect the runtime!).\nPID_DIR = pathlib.Path('/tmp/stocker_launcher_pids'); PID_DIR.mkdir(exist_ok=True)\n\ndef _kill_prev(name: str) -> None:\n    pf = PID_DIR / f'{name}.pid'\n    if not pf.exists():\n        return\n    try:\n        pid = int(pf.read_text().strip())\n        os.kill(pid, signal.SIGKILL)\n    except (ProcessLookupError, ValueError):\n        pass\n    pf.unlink(missing_ok=True)\n\n_kill_prev('jupyter'); _kill_prev('cloudflared')\nLOG_J.write_text(''); LOG_C.write_text('')\n\ndef _port_free(p: int) -> bool:\n    with socket.socket(socket.AF_INET, socket.SOCK_STREAM) as s:\n        return s.connect_ex(('127.0.0.1', p)) != 0\n\nif not _port_free(PORT):\n    raise RuntimeError(\n        f'Port {PORT} still in use. Re-run after a moment, or restart the '\n        f'runtime (Runtime > Restart session) and re-run all cells.'\n    )\n\n# --- jupyter-server -------------------------------------------------------\nos.environ['PYTHONPATH'] = WORKDIR\nj = subprocess.Popen(\n    [\n        sys.executable, '-m', 'jupyter', 'server',\n        f'--ServerApp.token={TOKEN}',\n        '--ServerApp.password=',\n        f'--ServerApp.port={PORT}',\n        '--ServerApp.ip=127.0.0.1',\n        '--ServerApp.allow_origin=*',\n        '--ServerApp.disable_check_xsrf=True',\n        '--ServerApp.allow_remote_access=True',\n        '--no-browser',\n        f'--ServerApp.root_dir={WORKDIR}',\n    ],\n    stdout=open(LOG_J, 'w'),\n    stderr=subprocess.STDOUT,\n    cwd=WORKDIR,\n)\n(PID_DIR / 'jupyter.pid').write_text(str(j.pid))\ntime.sleep(3)\nif j.poll() is not None:\n    raise RuntimeError(f'jupyter-server died:\\n{LOG_J.read_text()[-2000:]}')\nprint(f'jupyter-server up on 127.0.0.1:{PORT}  (pid={j.pid})')\n\n# --- cloudflared (with --logfile, NOT shell redirection) ------------------\nc = subprocess.Popen(\n    ['cloudflared', 'tunnel',\n     '--url', f'http://127.0.0.1:{PORT}',\n     '--no-autoupdate',\n     '--loglevel', 'info',\n     '--logfile', str(LOG_C)],\n    stdout=subprocess.DEVNULL,\n    stderr=subprocess.DEVNULL,\n)\n(PID_DIR / 'cloudflared.pid').write_text(str(c.pid))\n\nprint('cloudflared starting (typical: 5-30s) ', end='', flush=True)\nurl, deadline = None, time.time() + 120\nURL_RE = re.compile(r'https://[a-z0-9-]+\\.trycloudflare\\.com')\nwhile time.time() < deadline:\n    if c.poll() is not None:\n        print(f'\\ncloudflared exited unexpectedly:\\n{LOG_C.read_text()[-2000:]}', file=sys.stderr)\n        break\n    txt = LOG_C.read_text() if LOG_C.exists() else ''\n    m = URL_RE.search(txt)\n    if m:\n        url = m.group(0)\n        break\n    print('.', end='', flush=True)\n    time.sleep(2)\nprint()\n\nif not url:\n    raise RuntimeError(\n        'cloudflared did not produce a URL within 120s.\\n'\n        '----- last 2KB of cloudflared log -----\\n'\n        + LOG_C.read_text()[-2000:]\n        + '\\n----- last 1KB of jupyter log -----\\n'\n        + LOG_J.read_text()[-1000:]\n    )\n\nFULL = f'{url}/?token={TOKEN}'\nprint('=' * 70)\nprint('Paste this into VSCode -> Jupyter: Specify Jupyter Server for Connections')\nprint()\nprint('   ', FULL)\nprint()\nprint('Smoke-test from anywhere:')\nprint(f'   curl -s \"{FULL}/api/status\" | head -c 200')\nprint('=' * 70)\nprint('\\nKeep this Colab tab open — the tunnel dies when the runtime disconnects.')"
  },
  {
   "cell_type": "markdown",
   "id": "vscode-steps",
   "metadata": {},
   "source": [
    "## Connect from VSCode\n",
    "\n",
    "1. **Install the Jupyter extension** in VSCode (Microsoft, ID `ms-toolsai.jupyter`) if you don't have it.\n",
    "2. Open `training/train_grpo.ipynb` locally.\n",
    "3. Command Palette → **`Jupyter: Specify Jupyter Server for Connections`** → *Existing* → paste the URL printed above.\n",
    "4. In the kernel picker (top-right of the notebook), pick the remote runtime — VSCode shows it as `Python 3 (ipykernel)` under the trycloudflare host.\n",
    "5. Run cells. Imports like `from app.council.llm import TransformersLLMClient` resolve because the kernel cwd is `/content/stocker`.\n",
    "\n",
    "## Heartbeat\n",
    "\n",
    "If you idle, Colab eventually evicts the runtime and the URL stops working.\n",
    "Re-run the **`launch`** cell to get a new URL and re-specify the server in\n",
    "VSCode. Long training runs should run inside a `nohup`-style guard or use\n",
    "Colab Pro for the 24-hour runtime.\n",
    "\n",
    "## Pulling artifacts back\n",
    "\n",
    "Trained LoRAs and plots land in `/content/stocker/training/runs/<id>/`. To\n",
    "get them onto your laptop:\n",
    "\n",
    "```python\n",
    "# in a Colab cell, after training\n",
    "from google.colab import files\n",
    "import shutil, glob\n",
    "run = sorted(glob.glob('/content/stocker/training/runs/grpo_*'))[-1]\n",
    "shutil.make_archive('/content/run', 'zip', run)\n",
    "files.download('/content/run.zip')\n",
    "```\n",
    "\n",
    "Or `git add` + `git push` from a Colab cell to your fork."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "name": "python"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}