Comprehensive fix: newer dep stack + HF-dataset cache + verified locally

Dep cascade resolved by jumping forward instead of holding back:
- spaces/train/Dockerfile: pytorch:2.6.0-cuda12.4-cudnn9-runtime base
(has torch.int1 needed by torchao>=0.16). Smoke-test layer asserts
torch.int1 presence + imports torch/transformers/peft/trl/
GRPOTrainer/GRPOConfig/AutoTokenizer at build time, so any future
dep mismatch fails the build in 30s instead of crashing phase 3.
- pyproject.toml [train] extras: trl>=0.22, transformers>=4.55,
peft>=0.16, accelerate>=1.6, bitsandbytes>=0.45, torchao>=0.16,
huggingface_hub>=1.0. Verified locally: torch 2.10 / transformers
5.6 / peft 0.19 / trl 1.2 all import cleanly and Gemma-4 tokenizer
loads (the bug that crashed phase 3).

Cache architecture migrated from in-repo to HF Hub dataset:
- .gitignore: revert to ignoring .cache/ entirely.
- root Dockerfile: drop COPY .cache/ — env Space pulls from hub.
- app/main.py: lifespan startup hook _hydrate_cache_from_hub() calls
snapshot_download(STOCKER_CACHE_REPO) when env var is set. Idempotent,
never crashes startup. No-op locally without the env var.
- scripts/upload_cache.py: new utility (HfApi.upload_folder) to push
.cache/council to Hydr473/stocker-cache after running precache.
- .github/workflows/deploy_spaces.yml: STOCKER_CACHE_REPO added to
workflow env block + pushed as Space secret in deploy-env-space.
- README: paragraph documenting precache → upload → restart workflow.

Verified locally before pushing:
- 22/22 pytest pass with new dep set
- AutoTokenizer.from_pretrained('google/gemma-4-E4B-it') succeeds
- create_app() spawns 22 routes (includes /, /web, /blog, /council,
/reset, /step, /state, /meta, /ohlcv, /training/runs, /training/metrics)
- torch.int1 present, all imports clean

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

.github/workflows/deploy_spaces.yml

@@ -7,12 +7,13 @@ on:
 # Non-sensitive config — hardcoded here so the only required GitHub secret
 # is HF_TOKEN. Edit these in-place if your endpoint URL or model changes.
 env:
-  HF_USERNAME:    Hydr473
-  ENV_SPACE:      Hydr473/stocker-env
-  TRAIN_SPACE:    Hydr473/stocker-train
-  API_BASE_URL:   https://at0e6z2u64774tc7.us-east-1.aws.endpoints.huggingface.cloud/v1
-  MODEL_NAME:     ggml-org/gemma-4-26B-A4B-it-GGUF
-  RESULTS_REPO:   Hydr473/stocker-results
+  HF_USERNAME:        Hydr473
+  ENV_SPACE:          Hydr473/stocker-env
+  TRAIN_SPACE:        Hydr473/stocker-train
+  API_BASE_URL:       https://at0e6z2u64774tc7.us-east-1.aws.endpoints.huggingface.cloud/v1
+  MODEL_NAME:         ggml-org/gemma-4-26B-A4B-it-GGUF
+  RESULTS_REPO:       Hydr473/stocker-results
+  STOCKER_CACHE_REPO: Hydr473/stocker-cache
 
 jobs:
   deploy-env-space:
@@ -36,7 +37,7 @@ jobs:
           repo = os.environ["ENV_SPACE"]
           api.create_repo(repo_id=repo, repo_type="space", space_sdk="docker", exist_ok=True)
           # Optional — env Space only needs LLM creds if /inference is hit
-          for k in ("HF_TOKEN", "API_BASE_URL", "MODEL_NAME"):
+          for k in ("HF_TOKEN", "API_BASE_URL", "MODEL_NAME", "STOCKER_CACHE_REPO"):
               v = os.environ.get(k, "")
               if v:
                   api.add_space_secret(repo_id=repo, key=k, value=v)

.gitignore

@@ -32,13 +32,9 @@ coverage.xml
 .nox/
 
 # --- Stocker runtime artifacts --------------------------------------------
-# Council cache: ignore everything under .cache/ EXCEPT the demo bundle
-# (specialists/base for the 3 stable tasks). Allows shipping a warm cache
-# in the env Space so judges' UI clicks hit instant responses.
-.cache/*
-!.cache/council/
-.cache/council/*/lora_*/
-.cache/moderator/
+# Council vote cache lives in an HF dataset repo (Hydr473/stocker-cache),
+# downloaded at Space startup. Never commit it to git.
+.cache/
 # GRPO runs, eval rollouts, plots, LoRA adapters
 training/runs/
 !training/runs/.gitkeep

Dockerfile

@@ -24,10 +24,6 @@ COPY server/ /app/server/
 COPY data/ /app/data/
 COPY inference.py client.py README.md /app/
 
-# Pre-built council cache — gives judges instant UI responses for the 3
-# stable tasks. Optional: COPY only fires if .cache/ exists in build context.
-COPY .cache/ /app/.cache/
-
 # Pull in the built frontend so FastAPI mounts the React SPA at /web
 COPY --from=frontend-builder /frontend/dist /app/frontend/dist
 

README.md

@@ -265,6 +265,26 @@ pytest tests/ -q
 To run the council against your own endpoint, set `API_BASE_URL`,
 `MODEL_NAME`, and `HF_TOKEN` (see [.env.example](.env.example)).
 
+### Council vote cache
+
+Each `(role, ticker, date)` triple is computed once and cached as JSON. The
+env Space pulls a pre-warmed cache from the HF dataset
+[Hydr473/stocker-cache](https://huggingface.co/datasets/Hydr473/stocker-cache)
+on startup so judges' UI clicks are instant. To re-warm:
+
+```bash
+# 1. Compute votes via your endpoint (writes .cache/council/<role>/base/...)
+python scripts/precache_endpoint.py --tasks task_easy,task_medium,task_hard
+
+# 2. Upload to the dataset repo
+python scripts/upload_cache.py     # uses $STOCKER_CACHE_REPO or default
+
+# 3. Restart the env Space — its lifespan hook downloads the new cache.
+```
+
+The startup hook is no-op (and never crashes) when `STOCKER_CACHE_REPO` is
+unset, so local dev still works without HF Hub access.
+
 ---
 
 ## Evaluation

app/main.py

@@ -3,9 +3,11 @@
 from __future__ import annotations
 
 import logging
+import os
 import sys
 import time
 import traceback
+from contextlib import asynccontextmanager
 from pathlib import Path
 
 from fastapi import FastAPI, Request
@@ -19,6 +21,7 @@ from app.config import settings
 
 FRONTEND_DIST = Path(__file__).resolve().parents[1] / "frontend" / "dist"
 TRAINING_RUNS = Path(__file__).resolve().parents[1] / "training" / "runs"
+CACHE_DIR    = Path(__file__).resolve().parents[1] / ".cache"
 
 logging.basicConfig(
     level=logging.INFO,
@@ -28,6 +31,39 @@ logging.basicConfig(
 logger = logging.getLogger(__name__)
 
 
+def _hydrate_cache_from_hub() -> None:
+    """If STOCKER_CACHE_REPO is set, download the cache from that HF dataset
+    into .cache/ on startup. Idempotent — skipped when .cache/ already has
+    council/ entries. Never fails startup; logs and continues on error."""
+    repo = os.getenv("STOCKER_CACHE_REPO", "").strip()
+    if not repo:
+        return
+    if (CACHE_DIR / "council").exists() and any((CACHE_DIR / "council").iterdir()):
+        logger.info("Cache already populated; skipping hub download.")
+        return
+    try:
+        from huggingface_hub import snapshot_download
+        token = os.getenv("HF_TOKEN") or None
+        logger.info("Downloading council cache from %s ...", repo)
+        snapshot_download(
+            repo_id=repo,
+            repo_type="dataset",
+            local_dir=str(CACHE_DIR),
+            token=token,
+            allow_patterns=["council/**/*.json"],
+        )
+        n = sum(1 for _ in CACHE_DIR.rglob("*.json"))
+        logger.info("Cache hydrated: %d entries from %s", n, repo)
+    except Exception as e:
+        logger.warning("Cache hydration failed (%s); will fall back to live calls / mock.", e)
+
+
+@asynccontextmanager
+async def lifespan(app: FastAPI):
+    _hydrate_cache_from_hub()
+    yield
+
+
 def create_app() -> FastAPI:
     app = FastAPI(
         title="Stocker - OpenEnv",
@@ -36,6 +72,7 @@ def create_app() -> FastAPI:
         openapi_url="/openapi.json",
         docs_url="/docs",
         redoc_url="/redoc",
+        lifespan=lifespan,
     )
 
     @app.exception_handler(Exception)

pyproject.toml

@@ -39,17 +39,19 @@ serve = [
     "vllm>=0.6.0",
 ]
 train = [
-    "trl>=0.11.0,<0.20.0",
-    # transformers <4.50 avoids the torchao quant import path that hits
-    # `torch.int1` (added in torch 2.6). Matches torch 2.5.1 base image.
-    "transformers>=4.45.0,<4.50.0",
-    # peft <0.13 doesn't probe torchao at LoRA-injection time, so it works
-    # without torchao>=0.16. Stays compatible with torch 2.5.1.
-    "peft>=0.11.0,<0.13.0",
-    "accelerate>=1.0.0",
-    "bitsandbytes>=0.43.0",
+    # Newer coherent stack: torch 2.6 (has torch.int1) → torchao>=0.16 →
+    # peft>=0.13 (LoRA-injection probe) → transformers>=4.55 (Gemma-4
+    # tokenizer fix, has BloomPreTrainedModel only via lazy import which
+    # works) → trl>=0.22 → huggingface_hub>=1.0.
+    "trl>=0.22.0",
+    "transformers>=4.55.0",
+    "peft>=0.16.0",
+    "accelerate>=1.6.0",
+    "bitsandbytes>=0.45.0",
     "datasets>=3.0.0",
     "tensorboard>=2.18.0",
+    "torchao>=0.16.0",
+    "huggingface_hub>=1.0.0",
 ]
 
 [tool.setuptools]

scripts/upload_cache.py

@@ -0,0 +1,57 @@
+#!/usr/bin/env python3
+"""Push the local .cache/council/ directory to an HF dataset repo so the
+env Space can download it at startup (instead of bundling in git).
+
+Usage:
+    export HF_TOKEN=hf_xxx
+    python scripts/upload_cache.py                          # default repo
+    python scripts/upload_cache.py --repo Hydr473/stocker-cache
+"""
+from __future__ import annotations
+
+import argparse
+import os
+import sys
+from pathlib import Path
+
+ROOT = Path(__file__).resolve().parent.parent
+CACHE_DIR = ROOT / ".cache"
+
+
+def main() -> None:
+    p = argparse.ArgumentParser()
+    p.add_argument("--repo", default=os.getenv("STOCKER_CACHE_REPO", "Hydr473/stocker-cache"),
+                   help="HF dataset repo (default: Hydr473/stocker-cache)")
+    p.add_argument("--token", default=os.getenv("HF_TOKEN"),
+                   help="HF write token (default: $HF_TOKEN env)")
+    args = p.parse_args()
+
+    if not args.token:
+        sys.exit("HF_TOKEN not set (and no --token provided). Aborting.")
+
+    if not (CACHE_DIR / "council").is_dir():
+        sys.exit(f"No cache to upload — {CACHE_DIR/'council'} doesn't exist. "
+                 "Run scripts/precache_endpoint.py first.")
+
+    n_files = sum(1 for _ in (CACHE_DIR / "council").rglob("*.json"))
+    if n_files == 0:
+        sys.exit("Cache directory is empty. Run scripts/precache_endpoint.py first.")
+
+    from huggingface_hub import HfApi
+
+    api = HfApi(token=args.token)
+    api.create_repo(repo_id=args.repo, repo_type="dataset", exist_ok=True)
+    print(f"Uploading {n_files} cache entries → https://huggingface.co/datasets/{args.repo}")
+
+    api.upload_folder(
+        folder_path=str(CACHE_DIR),
+        repo_id=args.repo,
+        repo_type="dataset",
+        allow_patterns=["council/**/*.json"],
+        commit_message=f"Update council cache ({n_files} entries)",
+    )
+    print(f"Done. The env Space will pull this on next startup if STOCKER_CACHE_REPO={args.repo}.")
+
+
+if __name__ == "__main__":
+    main()

spaces/train/Dockerfile

@@ -10,13 +10,14 @@
 # Layer order is deliberate:
 #   1. OS packages         (rare changes — cached forever)
 #   2. Heavy pip install   (cached unless THIS Dockerfile changes)
-#   3. ADD commits/main    (cache-busts everything below)
-#   4. git clone           (~10s)
-#   5. editable install    (~5s)
-#   6. dataset build       (~30s)
-# So a normal git push only triggers steps 4-6 (~45s total).
+#   3. Smoke test          (fails build immediately on dep mismatch)
+#   4. ADD commits/main    (cache-busts everything below)
+#   5. git clone           (~10s)
+#   6. editable install    (~5s)
+#   7. dataset build       (~30s)
+# Normal git push triggers only steps 5-7 (~45s total).
 
-FROM pytorch/pytorch:2.5.1-cuda12.4-cudnn9-runtime
+FROM pytorch/pytorch:2.6.0-cuda12.4-cudnn9-runtime
 
 ENV PYTHONDONTWRITEBYTECODE=1 \
     PYTHONUNBUFFERED=1 \
@@ -31,16 +32,11 @@ RUN apt-get update -qq && \
     rm -rf /var/lib/apt/lists/*
 
 # ─── Layer 2: heavy pip install (cached) ────────────────────────────────
-# Pinned for torch 2.5.1 compatibility:
-#   transformers<4.50 → avoids torch.int1 import path in newer torchao
-#   peft<0.13         → avoids the torchao>=0.16 LoRA-injection probe
-#   trl<0.20          → compatible with peft<0.13
-# huggingface_hub is intentionally NOT pinned — pip resolves to <1.0 to
-# satisfy transformers' constraint.
-# NOTE: gradio is intentionally NOT installed here. gradio<5 pulls
-# gradio-client==1.3.0 which constrains websockets<13, but openenv-core
-# requires websockets>=15.0.1 — incompatible. The training Space UI uses
-# minimal FastAPI + HTML instead (see spaces/train/app.py).
+# Newer coherent stack: torch 2.6 (has torch.int1) → torchao>=0.16 →
+# peft>=0.16 (LoRA-injection probe satisfied) → transformers>=4.55
+# (correct Gemma-4 tokenizer behavior) → trl>=0.22 → huggingface_hub>=1.0.
+# Gradio is NOT installed — it conflicts with openenv-core's websockets>=15.
+# The training Space UI uses minimal FastAPI + HTML (spaces/train/app.py).
 RUN pip install --no-cache-dir uv && \
     uv pip install --system --no-cache \
         "fastapi>=0.115.0" "uvicorn[standard]>=0.30.0" \
@@ -48,27 +44,36 @@ RUN pip install --no-cache-dir uv && \
         "openai>=1.0.0" "openenv-core>=0.2.0" \
         "pandas>=2.0.0" "pyarrow>=14.0.0" "pillow>=10.0.0" "markdown-it-py>=3.0.0" \
         "yfinance>=0.2.0" "mplfinance>=0.12.0" "matplotlib>=3.8.0" \
-        "trl>=0.11.0,<0.20.0" "transformers>=4.45.0,<4.50.0" \
-        "peft>=0.11.0,<0.13.0" "accelerate>=1.0.0" "bitsandbytes>=0.43.0" \
-        "datasets>=3.0.0" "tensorboard>=2.18.0"
+        "trl>=0.22.0" "transformers>=4.55.0" "peft>=0.16.0" \
+        "accelerate>=1.6.0" "bitsandbytes>=0.45.0" "datasets>=3.0.0" \
+        "tensorboard>=2.18.0" "torchao>=0.16.0" "huggingface_hub>=1.0.0"
 
-# Fail-fast smoke test: if dep resolution produced an incompatible set,
-# this layer fails the build immediately instead of crashing at phase 3.
-RUN python -c "import torch, transformers, peft, trl, bitsandbytes, accelerate; \
-from peft import LoraConfig; from trl import GRPOConfig, GRPOTrainer; \
+# ─── Layer 3: build-time smoke test ────────────────────────────────────
+# If any dep combo is broken (import error, version mismatch, missing
+# attribute, tokenizer regression), this fails at BUILD time in 30 sec
+# instead of crashing 25 minutes into phase 3 of training.
+RUN python -c "\
+import torch, transformers, peft, trl, bitsandbytes, accelerate; \
 import huggingface_hub as hub; \
-print(f'OK torch={torch.__version__} transformers={transformers.__version__} peft={peft.__version__} trl={trl.__version__} hub={hub.__version__}')"
+from peft import LoraConfig; \
+from trl import GRPOConfig, GRPOTrainer; \
+from transformers import AutoTokenizer; \
+print(f'torch={torch.__version__} hub={hub.__version__}'); \
+print(f'transformers={transformers.__version__} peft={peft.__version__} trl={trl.__version__}'); \
+assert hasattr(torch, 'int1'), 'torch.int1 missing — torchao>=0.16 will crash'; \
+print('smoke test passed')\
+"
 
-# ─── Layer 3: cache-bust on every commit to main ────────────────────────
+# ─── Layer 4: cache-bust on every commit to main ────────────────────────
 ADD https://api.github.com/repos/CRIMSONHydra/stocker/commits/main /tmp/.commit-sha
 
-# ─── Layer 4: fresh clone of the repo ───────────────────────────────────
+# ─── Layer 5: fresh clone of the repo ───────────────────────────────────
 RUN git clone --depth=1 https://github.com/CRIMSONHydra/stocker.git /app
 
-# ─── Layer 5: register the local package (no dep resolution) ────────────
+# ─── Layer 6: register the local package (no dep resolution) ────────────
 RUN cd /app && pip install --no-deps -e .
 
-# ─── Layer 6: build the bundled dataset ─────────────────────────────────
+# ─── Layer 7: build the bundled dataset ─────────────────────────────────
 RUN python scripts/build_dataset.py && python scripts/build_ideal_profit.py
 
 EXPOSE 7860