Spaces:
Sleeping
Sleeping
| # SchemaShift Build Log | |
| Running log of everything built, tested, trained, and deployed. Append-only — never delete. | |
| **Team Tripod:** Yashash Sheshagiri (lead), Gajanand V Dhayagode, Likith B S | |
| **Event:** Meta × HF × PyTorch OpenEnv Hackathon 2026 · Round 2 · Bangalore · April 25-26 | |
| --- | |
| ## PHASES COMPLETED | |
| ### Phase 0 — Project Scaffolding | |
| - **Date:** Tuesday April 21, 2026 (evening) | |
| - **Commit:** `d4ab0f1` | |
| - **Tests:** 0 (scaffolding only — no logic yet) | |
| - **Time:** ~30 min | |
| - **Notes:** 29 files created. Python venv built. Deps installed cleanly (fastapi 0.136, pydantic 2.13, openai 2.32, openenv-core 0.2.3, gradio 6.13). Repo pushed to https://github.com/Yashash4/SchemaShift. | |
| ### Phase 1 — Pydantic Models | |
| - **Date:** Tuesday April 21, 2026 | |
| - **Commit:** `8cdb02b` | |
| - **Tests:** 6 passing (0.22s runtime) | |
| - **Line counts:** models.py = 135 lines, test_models.py = ~120 lines | |
| - **Time:** ~20 min | |
| - **Judgment calls:** | |
| 1. 12 classes + 2 Literal type aliases (not 13 classes as prompt suggested) | |
| - **Notes:** Spec code copied verbatim. No deviations. | |
| ### Phase 2 — Mail Tool | |
| - **Date:** Tuesday April 21, 2026 | |
| - **Commit:** `8410ff1` | |
| - **Tests:** 7 new (13 total), 0.16s runtime | |
| - **Line counts:** tools/base.py = 60, tools/mail.py = 121, tests/test_mail.py = 149 | |
| - **Time:** ~90 min | |
| - **Judgment calls:** | |
| 1. Absolute imports (`from models import`) over relative imports — matches Phase 1 convention | |
| 2. `get_schema()` returns `{}` for unknown endpoints post-deprecation | |
| 3. 501 fallback in `BaseTool.call` for schema-without-handler (dev-time safety) | |
| 4. `messages.send` is deep-copy rename of `send_message` after deprecation drift | |
| 5. Pagination tokens: `"tok_xyz"` / `"cur_abc123"` when >10 results, `None` otherwise | |
| - **Notes:** 3 endpoints (list_messages, send_message, get_message), 3 drifts (field_rename, endpoint_deprecation, new_required_param). | |
| ### Phase 3 — Calendar Tool + DriftInjector | |
| - **Date:** Tuesday April 21, 2026 | |
| - **Commit:** `27dd958` | |
| - **Tests:** 10 new (23 total), 0.24s runtime | |
| - **Line counts:** tools/calendar.py = 164, drift.py = 28, tests/test_calendar.py = 182, tests/test_drift.py = 122 | |
| - **Time:** ~100 min | |
| - **Judgment calls:** | |
| 1. Overlap predicate for `list_events` (inclusive both sides, ISO string sort) | |
| 2. `_create_event` handles both attendees and participants gracefully | |
| 3. `_update_event` accepts both attendees and participants (no second drift handler needed) | |
| 4. Cancelled events remain in storage (matches Google/Outlook behavior) | |
| 5. `_fired` sentinel in `event.details` (underscore prefix avoids collision) | |
| 6. `rate_limit_tightening` chosen for unknown-drift test (semantically meaningful) | |
| - **Notes:** 4 endpoints (list/create/update/delete_event), 2 drifts (field_rename attendees→participants, tool_removal delete_event). DriftInjector is stateless. | |
| ### Phase 4 — CRM Tool + Scenarios E1/E2/E3 | |
| - **Date:** Tuesday April 21, 2026 | |
| - **Commit:** `bc8f184` | |
| - **Tests:** 11 new (34 total), 0.24s runtime | |
| - **Line counts:** tools/crm.py = 213, scenarios.py = 124, tests/test_crm.py = 142, tests/test_scenarios.py = 65 | |
| - **Time:** ~100 min | |
| - **Judgment calls:** | |
| 1. Strict param validation in `BaseTool.call` — rejects unknown params with 400 "Unknown params" (matches Stripe/Google API behavior) | |
| 2. Internal storage stays `customer_email`; projection on read via `_project_contact` | |
| 3. `search_contacts()` with no filters returns ALL contacts (REST convention) | |
| 4. Rate-limit counter resets on drift fire (pre-drift calls don't count against post-drift budget) | |
| 5. Drift-order commutativity for `contacts.patch` (either order produces same final state) | |
| 6. E2 task description doesn't disclose deprecation (agent must discover via 410 response) | |
| 7. E3 drift fires at step 2 (gives agent one clean baseline call) | |
| 8. `DriftEvent.details` populated with hints (`from`, `to`) for grader/scenario auditing | |
| - **Notes:** 4 endpoints (search/get/create/update_contact), 3 drifts (tool-wide field_rename, rate_limit_tightening, endpoint_deprecation update→contacts.patch). | |
| ### Phase 5 — Composable Grader + Dense Step Shaping | |
| - **Date:** Tuesday April 21, 2026 | |
| - **Commit:** `46b8c56` | |
| - **Tests:** 11 new (45 total), 0.25s runtime | |
| - **Line counts:** graders.py = 267, tests/test_graders.py = 224 | |
| - **Time:** ~150 min | |
| - **Judgment calls:** | |
| 1. Binary threshold 0.95 (not 1.0) — floating point safety | |
| 2. `AdaptationRubric` iterates all fired drifts (TODO: revisit for Phase 11 M-scenarios with multi-drift-per-tool) | |
| 3. `_final_state_acceptable` lenient mid-episode, strict at terminal — preserves training signal | |
| 4. `compute_step_shaping` checks `not d.detected_by_agent` — anti-farming guard | |
| 5. Dumb-retry penalty requires same tool AND same endpoint | |
| 6. `assert` for weight-sum invariant (dev-time, not runtime) | |
| 7. Rubric names coupled to `RewardBreakdown` field names (intentional reward contract) | |
| 8. Bonus test asserts `step_shaping = 0` in grader output (env fills it, not grader) | |
| - **Notes:** 4 rubrics (Completion, DriftDetection, Adaptation, Efficiency) + 2 gates (catastrophic, correct_final) + WeightedSum + Gate + `compute_step_shaping` + `build_grader`. | |
| ### Phase 6 — Environment Scheduler | |
| - **Date:** Tuesday April 21, 2026 | |
| - **Commit:** `c618a61` | |
| - **Tests:** 8 new (53 total), 0.29s runtime | |
| - **Line counts:** server/environment.py = 363, tests/test_environment.py = 242 | |
| - **Time:** ~120 min | |
| - **E1 FULL-EPISODE BENCHMARK:** | |
| - `shaped_total = 0.934375` (expected ~0.93, exact match on weighted sum) | |
| - `task_completion = 1.0` | |
| - `drift_detection = 1.0` | |
| - `adaptation_quality = 1.0` | |
| - `efficiency = 0.5625` (used 7 of 8 max steps) | |
| - `catastrophic_gate = 1.0, correct_final_gate = 1.0` | |
| - `binary = 1.0` | |
| - `cumulative_reward = 4.2975` across 7 steps | |
| - **Dense shaping fired correctly:** +0.10 at step 4 (inspect-after-failure), +0.15 at step 6 (drift report) | |
| - **Judgment calls:** | |
| 1. Shaping-before-marking ordering: dispatch→compute_step_shaping→mark detected→grader | |
| 2. `tokens_used` clamped to non-negative with `max(0, remaining - tokens_used)` | |
| 3. Lazy tool imports in `_instantiate_tool` — stretch tools don't break core | |
| 4. `deepcopy(drift_plan)` on reset — prevents state leak across episodes | |
| 5. Observation history windowed to last 5 steps for LLM prompt budget | |
| 6. Calendar body parser handles both attendees (strings) and participants (dicts) | |
| 7. `_completion_summary` stored with underscore prefix (no GT key collision) | |
| 8. E2 three-recipient check tracked in `_update_agent_state` | |
| 9. `done` flag set AFTER dispatch but BEFORE grader (enables strict gate at terminal) | |
| 10. Dispatch returns `None` for `report_drift` and `complete_task` (no tool response) | |
| - **Notes:** Round 1 `_engine is None` bug pattern explicitly prevented via `RuntimeError` in step() when state is None. | |
| ### Phase 7 — FastAPI Server + Docker | |
| - **Date:** Tuesday April 21, 2026 (late evening) | |
| - **Commit:** `cb33205` | |
| - **Tests:** 10 new (63 total), 0.94s runtime | |
| - **Line counts:** server/app.py = 120, tests/test_server.py = 129, Dockerfile = 14, openenv.yaml = 14 | |
| - **Time:** ~60 min | |
| - **Live server smoke tests:** uvicorn on 127.0.0.1:7860 verified /health, /tasks (count=3), /reset (E1 → step=0, tools=['mail', 'calendar']) all 200 OK | |
| - **`pip install -e .` confirmation:** Editable wheel built successfully. Docker container will resolve absolute imports correctly. | |
| - **Judgment calls:** | |
| 1. Removed root `__init__.py` — conflict with `py-modules` in pyproject | |
| 2. `py-modules` + `packages` split in pyproject (preserves `from models import` convention) | |
| 3. Dropped unused imports from spec's server code | |
| 4. TestClient monkeypatch fixture — fresh env per test prevents cross-contamination | |
| 5. `/step` catches `RuntimeError` → 400 not 500 (client bug not server bug) | |
| 6. Read-only endpoints (`/state`, `/grader`) don't auto-reset | |
| 7. `/tasks` trims description to 120 chars (lightweight listing) | |
| 8. No CORS middleware yet (defer to Phase 10 if needed) | |
| 9. Hardcoded version "0.1.0" in two places (pragmatic duplication) | |
| 10. `urllib.request` for smoke test (Windows-safe, no curl dependency) | |
| - **Notes:** Endpoints: /, /health, /reset, /step, /state, /tasks, /grader. Port 7860 for HF Spaces compatibility. | |
| ### Phase 8 — Env Client + Training Skeleton | |
| - **Date:** Tuesday April 21, 2026 (late evening) | |
| - **Commit:** `2464e9e` | |
| - **Tests:** 4 new (67 total), 1.52s runtime | |
| - **Line counts:** client.py = 91, training/grpo_smoke.py = 143, training/grpo_kaggle.ipynb = 10 cells (1 markdown + 9 code), tests/test_client.py = 70 | |
| - **Smoke test result:** Step 4 step_shaping = 0.1000 ✅ EXACTLY as expected (dense shaping preserved across HTTP roundtrip: env → JSON serialize → HTTP → JSON deserialize → client → caller) | |
| - **Time:** ~90 min | |
| - **Judgment calls:** | |
| 1. Brace-balanced JSON extractor in `parse_completion_to_actions` (depth-counting; handles arbitrarily nested JSON, unlike flat regex which would miss `call_tool.tool_call.params` structures) | |
| 2. Mocked httpx tests over live TestClient (faster, no real network, no port conflicts; live integration covered by grpo_smoke.py) | |
| 3. Notebook uses explicit Cell 4 / Cell 5 / Cell 6 separation (variant block uncommenting is cleaner than runtime flags — one account owner toggles exactly one block) | |
| 4. Smoke test returns exit code 2 (not 1) when step_shaping is wrong, distinguishing from "server unreachable" exit 1 | |
| 5. Model name in Cell 4 left as Qwen 2.5 1.5B Instruct (Account 3 Coder ablation is documented as a manual edit, not conditional code) | |
| 6. Procedural drift scheduler lives in its own Cell 6 (Account 3 rebuilds dataset each ~25 steps without editing reward_fn) | |
| 7. GRPO `hub_model_id` uses `$HF_USERNAME` env var fallback to `Yashash4` (each account member pushes to their own HF namespace — prevents checkpoint collisions) | |
| 8. Notebook Cell 3 uses `git clone || git pull` (idempotent on warm Kaggle sessions) | |
| 9. Client tests mock `_client.get/post` directly, not `httpx.Client` (surgical, exercises same code path runtime hits) | |
| 10. Deferred live HTTP client tests to Phase 10 (grpo_smoke.py is the live integration check — passed cleanly) | |
| - **Notes:** This is the bridge to training. After this phase, Kaggle can load Qwen 1.5B, generate action JSON, submit via HTTP, receive shaped rewards, and feed GRPO. The 0.1000 step_shaping on smoke Step 4 is the "signal is alive" proof — the most important number from Phase 8. | |
| ### Phase 9 — Baseline Eval Harness | |
| - **Date:** Tuesday April 21, 2026 (late night) | |
| - **Commit:** `4d2f869` | |
| - **Tests:** 6 new (73 total), 1.53s runtime | |
| - **Line counts:** eval.py = 674, tests/test_eval.py = 132 | |
| - **Time:** ~2 hours | |
| - **Live eval discriminability result:** | |
| - naive_heuristic: 0.000 shaped, 0.235 cumulative, 0% binary | |
| - policy_aware_heuristic: 0.348 shaped, 1.284 cumulative, 66.67% binary | |
| - **GAP: 0.348 shaped (threshold was 0.2 → PASS)** | |
| - Binary rate gap: 66.67 percentage points | |
| - **Per-scenario behavior:** | |
| - E1: policy_aware finishes in 3 steps, beats drift at step 3, binary=1 | |
| - E2: policy_aware hits unavoidable drift at step 1, recovers but only sends 1/3 emails, binary=0 — intentional ceiling for rule-based agents | |
| - E3: policy_aware search+update completes cleanly, binary=1 | |
| - **Judgment calls:** | |
| 1. Retry guard to prevent infinite loop (caught upfront) | |
| 2. Task-progress awareness via obs.known_state (prevents duplicate tool calls) | |
| 3. CRM search→update two-stage flow with company name capture | |
| 4. _adapt_endpoint handles mail messages.send and crm contacts.patch | |
| 5. Email regex bug fix (`r"[\w.+-]+@[\w-]+(?:\.[\w-]+)+"` — can't swallow trailing period; caught via debug, would have broken E3 silently) | |
| 6. Added cumulative_reward to EpisodeResult output (captures dense shaping even when gate zeros terminal) | |
| 7. Strict-validation-aware _adapt_params (strips keys not in current schema) | |
| 8. LLM agents deferred to Phase 10+ (API keys not ready) | |
| - **Notes:** Discriminability proves env rewards the correct meta-skill and isn't gameable. E2's design feature (unreachable for rule-based agents) is critical for RL training story — demonstrates the policy ceiling that a trained agent must exceed. | |
| ### Phase 10 — HF Space Deploy | |
| - **Date:** Tuesday April 21, 2026 (late night) | |
| - **Commit:** `fc82aca` (merge with HF initial commit) | |
| - **Tests:** 4 new deploy smoke tests (73 local + 4 conditional = 77 total) | |
| - **Line counts:** README.md (HF frontmatter + body), DEPLOY.md, tests/test_deploy_smoke.py | |
| - **Time:** < 30 minutes (fastest phase so far) | |
| - **Deploy timing:** HF Docker base cached → APP_STARTING at t=0s, RUNNING at t=15s, /health 200 immediately | |
| - **Live URLs:** | |
| - HF Space: https://yashash045-schemashift.hf.space | |
| - HF Space management: https://huggingface.co/spaces/yashash045/schemashift | |
| - GitHub: https://github.com/Yashash4/SchemaShift | |
| - **CRITICAL VERIFICATION — Step 4 step_shaping on production:** | |
| - Local Phase 8 smoke: 0.1000 exact | |
| - **Production Phase 10 smoke: 0.1000 exact** — dense shaping preserved through full prod HTTP path | |
| - **Heuristic eval on production (matches local exactly):** | |
| - naive: 0.000 shaped, 0.235 cumulative, 0% binary | |
| - policy_aware: 0.348 shaped, 1.284 cumulative, 66.67% binary | |
| - Gap: +0.348 shaped / +66.67pp binary (matches local to 3 decimal places) | |
| - **Deploy smoke test suite:** 4/4 passing (test_deployed_health, test_deployed_tasks_list, test_deployed_reset_and_step, test_deployed_step_shaping_fires) — 6.32s runtime | |
| - **Judgment calls:** | |
| 1. Used existing HF creds (yashash045) found at ~/.cache/huggingface/token instead of re-auth | |
| 2. GitHub Yashash4 vs HF yashash045 — different handles, both remotes working | |
| 3. Merge HF initial commit with `-X ours` strategy, not force-push (audit trail preserved) | |
| 4. Deploy smoke tests skippable via SCHEMASHIFT_DEPLOY_URL env var | |
| 5. Monitor tool for build-status polling (context-efficient) | |
| 6. Tokenized URL for git push (portable across Win/Unix) | |
| 7. Dual remote setup — both origin + space pushed at fc82aca | |
| 8. Deploy took < 1 min end-to-end; redeployments will be < 30s (base image cached) | |
| - **Notes:** Kaggle training notebooks can now set SCHEMASHIFT_URL=https://yashash045-schemashift.hf.space for Phase 13. README.md has HF frontmatter with patronus+scaler sub-theme tags for judge discoverability. | |
| ### Phase 11 — Medium Scenarios M1/M2/M3 | |
| - **Date:** Wednesday April 22, 2026 (early morning) | |
| - **Commits:** `7828dcd` (Phase 11 core) + `0fd93c4` (deploy smoke count fixup) | |
| - **Tests:** 5 new (78 local pass + 4 deploy skip = 82 collected), 4/4 deploy smoke pass on prod | |
| - **Line counts:** scenarios.py 124→283 (+159), server/environment.py 363→390 (+27), tests/test_scenarios.py 65→95 (+30), tests/test_graders.py 224→295 (+71) | |
| - **Time:** ~2 hours | |
| - **M-tier discriminability result (policy_aware_heuristic, 3 seeds):** | |
| - M1_customer_escalation: shaped=0.000, cumul=2.510, binary=0% | |
| - M2_weekly_report: shaped=0.000, cumul=3.013, binary=0% | |
| - M3_event_cleanup: shaped=0.000, cumul=0.441, binary=0% | |
| - **M-tier overall: shaped=0.000, cumul=1.988 (HIGHER than E-tier's 1.284), binary=0%** | |
| - **Interpretation:** Gate-zeroed terminal rewards + higher cumulative = dense shaping fires correctly during multi-drift adaptation but rule-based agent cannot complete M-tier tasks. This is the designed ceiling — M-tier requires planning ahead across 10-15 steps with 2 drifts per episode, which no keyword-triggered heuristic can do. **M-tier is the "requires RL" tier.** | |
| - **E-tier regression check:** policy_aware on E1/E2/E3 still returns 0.348 shaped / 66.67% binary — identical to Phases 9/10. Env extensions didn't regress easy scenarios. | |
| - **AdaptationRubric multi-drift test (judgment call #2 from Phase 5 resolved):** | |
| - Test constructed synthetic M3 state: 2 calendar drifts fired, 1 adaptation opportunity succeeded | |
| - Score: 0.5 (1/2 opportunities adapted) — partial credit for partial adaptation | |
| - Decision: NOT modified. Conservative partial-credit rubric + dense step_shaping (+0.20 for successful retry) together provide clean decomposable training signal. Change only if Phase 13 training shows convergence issues. | |
| - **Production verification:** HF Space /tasks now returns 6 scenarios. Zero-downtime redeploy. M-tier prod eval matches local to 3 decimals. | |
| - **Judgment calls:** | |
| 1. Extended `calendar.last_event_has_both_attendees` to OR-match priya+alex AND bob+alex (shared GT key across E1 and M1) | |
| 2. Added 3 new subject-contains checks (priority_support, weekly, calendar_updated) following existing pattern | |
| 3. Dynamic `calendar.evt_{X}_status` tracking on update_event status param — good pattern for entity-specific GT | |
| 4. `calendar.events_count_new_friday_wrapup` counter — case-insensitive title substring | |
| 5. Renamed `test_all_three_scenarios_present` → `test_all_scenarios_present` (6 scenarios now) | |
| 6. Updated `test_tasks_endpoint` and `test_deployed_tasks_list` count assertions 3→6 | |
| 7. AdaptationRubric NOT modified — partial-credit behavior is correct (prevents double-counting, preserves signal) | |
| 8. Heuristic (eval.py) NOT modified — M-tier shaped=0 is the designed ceiling, not a heuristic bug to fix | |
| - **Notes:** After Phase 11, SCHEMASHIFT_URL/tasks returns 6 scenarios (3 easy + 3 medium). Phase 13 training uses all 6 as training data. H-tier (hard) scenarios deferred to stretch. | |
| --- | |
| ## PHASES REMAINING | |
| - [x] Phase 11 — Medium scenarios (M1/M2/M3) ✅ commit 7828dcd + 0fd93c4 | |
| - [ ] Phase 12 — Insurance video recording (60s core cut + 2min full) | |
| - [ ] Phase 13 — Kaggle training runs (Stage 1 single account → Stage 2 parallelize) | |
| - [ ] Phase 14 — Pitch + blog + video | |
| - [ ] Phase 15 — Onsite Saturday-Sunday | |
| --- | |
| ## TRAINING RUNS / EVAL RUNS — see TRAINING_LOG.md | |
| All training runs (per-checkpoint evals, reward curves, config snapshots, iteration history) are logged in `TRAINING_LOG.md` — a dedicated append-only file with 7 structured sections. This separation keeps BUILD_LOG focused on build/phase tracking. For Phase 9 eval results, Phase 13 training data, and head-to-head comparison tables, see TRAINING_LOG.md. | |
| --- | |
| ## DEPLOYMENTS | |
| (Will be populated during Phase 10+.) | |
| ### Deploy v[N] — [Date] | |
| - **Target:** HF Space / local / Kaggle | |
| - **Commit:** | |
| - **URL:** | |
| - **Health check:** | |
| - **Config secrets set:** | |
| - **Notes:** | |
| --- | |
| ## DECISIONS LOG (strategic, append-only) | |
| ### Tuesday ~9 PM — Domain lock-in | |
| - **Decision:** Schema Drift Adaptation Arena (SaaS Admin Workflows) | |
| - **Rationale:** Direct Patronus sub-theme hit; Scaler secondary; no competitor finalist has it; authentic to Yashash's APEX OS/VisionX experience; cleanest before/after RL story | |
| ### Tuesday ~10 PM — Staged Kaggle approach (v2.3) | |
| - **Decision:** Run ONE Kaggle account (Gajanand/Account 2, main config) first. Parallelize only if Stage 1 converges. | |
| - **Rationale:** If reward function or parser has a bug, 3 simultaneous accounts waste 3× GPU quota. Debug once on one account. Main config = primary pitch claim, so prove that first. | |
| ### Tuesday ~10 PM — Qwen 2.5 Coder 1.5B ablation on Account 3 (v2.2) | |
| - **Decision:** Account 3 uses Coder-1.5B instead of same model + curriculum only. | |
| - **Rationale:** More informative ablation — tests whether code-pretraining helps schema-adaptation learning. Gives blog/pitch a secondary result regardless of outcome. | |
| ### Tuesday ~10 PM — Two videos in Phase 12 (v2.1) | |
| - **Decision:** Produce 60-second core cut + 2-minute full version, not one 90-second video. | |
| - **Rationale:** 60s fits pitch timing with hook+close room. 2min fits HF blog context depth. Recording once (Thursday), using twice. | |
| ### Tuesday ~10 PM — One-sentence pitch opener (v2.1) | |
| - **Decision:** Open README/pitch/video with: *"SchemaShift teaches agents to recover when the tool schema changes under them."* | |
| - **Rationale:** Gallery winners (Kube SRE Gym, GAIA) lead with one-sentence claims. Judges remember one clean idea, not four themes. | |
| ### Tuesday ~10 PM — Procedural drift scheduler on Account 3 (v2.1) | |
| - **Decision:** Account 3 variant adds ~30 LOC scheduler that rotates drift steps and adds secondary drift after step 150. | |
| - **Rationale:** Kube SRE Gym uses Claude-powered adversarial designer; we achieve similar curriculum story without API credit burn. | |
| --- | |
| ## REJECTED IDEAS (do not revive — posterity record) | |
| ### Validator / oversight agent — rejected Tuesday | |
| - **Why rejected:** Would break 63 passing tests, adds scope at wrong phase, theme-benefit marginal (multi-agent claim already defensible via scripted tool servers) | |
| ### Uncertainty injection across failure types — rejected Tuesday | |
| - **Why rejected:** Dilutes pitch (becomes "uncertainty" not "schema drift"), risks GRPO convergence by introducing probabilistic reward, already solved by dense shaping + policy-aware heuristic baseline proving non-gameability | |
| ### Claude-powered adversarial drift designer — rejected Tuesday | |
| - **Why rejected:** Time sink, API credit burn, provides marginal benefit over procedural scheduler | |
| ### Pivot to "decision-making under uncertainty" framing — rejected Tuesday | |
| - **Why rejected:** Breaks Patronus sub-theme fit, muddies demo story, opens scope unnecessarily | |
| ### Qwen3 family migration (primary training) — rejected Tuesday | |
| - **Why rejected:** Newer but less stable, prompt-behavior churn, migration risk for 48-hour build. Stick with Qwen 2.5 1.5B. | |
| ### 3B+ primary model — rejected Tuesday | |
| - **Why rejected:** Slow iteration (2x fewer experiments per Kaggle hour), tight T4 memory, doesn't improve pitch. "1.5B beats GPT-4o-mini" is a stronger claim than "3B beats GPT-4o-mini". | |
| --- | |
| ## OPEN QUESTIONS / TODOS | |
| - [ ] Phase 8: verify step_shaping survives HTTP serialization roundtrip | |
| - [ ] Phase 9: do baseline LLMs hit at least 0.30 on drifted E1? (if too high, env is too easy) | |
| - [ ] Phase 10: deploy to HF Space, verify remote /health works | |
| - [ ] Phase 11: when adding M1/M2/M3, verify AdaptationRubric denominator still makes sense (flagged in Phase 5 judgment call #2) | |
| - [ ] Phase 13 Stage 1: go/no-go decision at 20 steps — document the reward curve image | |
| - [ ] Phase 14: draft strong-claim and softer-claim versions of pitch, pick based on training results | |
| - [ ] Phase 15 onsite: bring a backup video file on USB in case HF Space is slow to load | |
| - [ ] Phase 13 Stage 1 Eval: when evaluating trained checkpoint, include M-tier in the scenarios list. M-tier is where the "beats GPT-4o-mini" claim lives. | |
| - [ ] Pitch Phase 14: frame heuristic brittleness on M-tier ("can't parse 'schedule'/'check-in call' as calendar intent") as evidence that env rewards real language understanding, not regex matching. | |
| --- | |
| ## ONSITE LOGISTICS | |
| **Event:** Saturday-Sunday April 25-26, 2026 · SST Campus Bangalore | |
| **Travel:** (add details) | |
| **Accommodation:** (add details) | |
| **Equipment to bring:** | |
| - 2 laptops (primary + backup) | |
| - USB with video files + repo backup | |
| - HDMI adapter for presentation | |
| - Chargers + extension cord | |
| - Both phones with hotspot capability (in case venue wifi fails) | |
| --- | |