docs: add comprehensive PROJECT.md knowledge base for AI agent context

PROJECT.md

@@ -0,0 +1,464 @@
+# Kasualdad LFED — Project Knowledge Base
+
+> **Last updated:** 2026-06-06 (Session 3)
+> **Target:** HF Build Small Hackathon — "Backyard AI" chapter
+> **Space:** https://huggingface.co/spaces/build-small-hackathon/Kasualdad_LFED
+> **Remote:** `git@hf.co:spaces/build-small-hackathon/Kasualdad_LFED` (remote name: `space`)
+
+---
+
+## What It Does
+
+A Gradio app that lets school district admins ask plain-English questions about student data. A local LLM (llama.cpp + Qwen2.5-Coder-7B GGUF) generates DuckDB SQL, which is validated and executed on in-memory seed data. **Nothing leaves the machine — no API calls, no cloud.**
+
+---
+
+## Architecture & Data Flow
+
+```
+User types question
+  ↓
+app.py  →  handle_query()  [@spaces.GPU on HF Zero GPU]
+  ↓
+model_inference.py  →  build_prompt() + llama.cpp inference
+  ↓  raw LLM output (JSON or ```sql```)
+data_engine.py  →  extract_sql() → validate_sql() → execute_safe()
+  ↓  pandas DataFrame
+app.py  →  renders SQL + table in Gradio UI
+```
+
+**Key design rule:** `app.py` is a thin controller. All logic lives in the three engine modules.
+
+---
+
+## File Map
+
+| File | Purpose | Lines |
+|---|---|---|
+| `app.py` | Gradio UI, CSS, `@spaces.GPU` wiring, Parquet bootstrap | ~380 |
+| `model_inference.py` | llama.cpp wrapper, CUDA preload, model cache, SQL generation + streaming | ~290 |
+| `data_engine.py` | DuckDB lifecycle, Parquet loading, SQL extraction/validation/execution | ~280 |
+| `prompts.py` | System prompt, schema docs, few-shot examples, prompt assembler | ~130 |
+| `data/generate_seed.py` | Generates 2,900 students, enrollment, attendance | ~200 |
+| `data/export_parquet.py` | One-shot script: seed → Parquet files | ~60 |
+| `data/enrollment.parquet` | Pre-generated enrollment table (1,526 bytes) | — |
+| `data/attendance.parquet` | Pre-generated attendance table (31,482 bytes) | — |
+| `requirements.txt` | Pinned Python deps (see below) | 9 lines |
+| `README.md` | Public Space README | ~280 |
+| `docs/HANDOFF.md` | Developer session handoff doc | ~190 |
+| `docs/PLAN.md` | Original build plan | ~170 |
+| `tests/` | pytest suite (81 tests: execution guard, data engine, model inference) | 4 files |
+| `modal_train/` | Modal fine-tuning pipeline (synthetic data, Unsloth QLoRA, GGUF export) | 5 files |
+
+---
+
+## Model & Inference
+
+### Active Model
+
+- **Repo:** `mradermacher/Qwen2.5-Coder-7B-Instruct-GGUF`
+- **File:** `Qwen2.5-Coder-7B-Instruct.Q4_K_M.gguf` (~4.68 GB)
+- **Quantization:** Q4_K_M
+- **Context:** 4096 tokens (`n_ctx`)
+- **GPU layers:** `-1` (all layers; Zero GPU handles context switching)
+
+### Model Resolution
+
+`model_inference.py:_resolve_model_path()` [model_inference.py:124]
+1. Check local cache: `/tmp/lfed-models/qwen/...` (fast if pre-downloaded)
+2. Fallback: download from HF Hub via `huggingface_hub.hf_hub_download()`
+
+### Inference Config
+
+| Parameter | Local | HF Zero GPU | Note |
+|---|---|---|---|
+| `n_ctx` | 4096 | 4096 | |
+| `n_threads` | 2 | 4 | Detected via `SPACES_ZERO_GPU` env |
+| `n_gpu_layers` | -1 | -1 | Always GPU; Zero GPU emulation at module level |
+| `max_tokens` | 256 | 256 | |
+| `temperature` | 0.0 | 0.0 | Deterministic |
+| `stop` | `\n\n`, `Question:`, `User:` | same | |
+
+### Thread Safety
+
+`model_inference.py` caches the model in a module-level `_llm` global with a `threading.Lock` (double-check pattern). `generate_sql()` auto-loads if `llm=None`.
+
+---
+
+## Zero GPU / HF Space Configuration
+
+### Space Identity
+- **URL:** `https://huggingface.co/spaces/build-small-hackathon/Kasualdad_LFED`
+- **SDK:** Gradio 6.16.0
+- **Python:** 3.12
+- **Hardware:** Zero GPU (NVIDIA RTX Pro 6000 Blackwell, half — 48 GB VRAM)
+- **Git remote:** `space` → `https://huggingface.co/spaces/build-small-hackathon/Kasualdad_LFED`
+
+### Zero GPU Architecture
+
+Zero GPU uses **CUDA emulation** at module level and **real GPU** inside `@spaces.GPU` functions. This means:
+- Model loads with `n_gpu_layers=-1` at startup (module level, emulated CUDA)
+- Inference runs inside `@spaces.GPU` decorated `handle_query()` (real GPU)
+- Zero GPU transparently switches between emulated and real CUDA contexts
+
+### `@spaces.GPU` Decorator
+
+`app.py:37` — `handle_query()` is the only decorated function. Required for Zero GPU to recognize the Space as GPU-enabled.
+
+### Startup Sequence (app.py:16-30)
+
+1. Print banner
+2. **Ensure Parquet seed files exist** — if `/data/enrollment.parquet` and `/data/attendance.parquet` aren't found, generate them via `export_parquet.py` into `/data/` (persistent Space storage) or `data/` (local dev)
+3. Load model (`load_model()`) — downloads from HF Hub if not cached, loads into GPU/RAM
+4. Launch Gradio UI
+
+---
+
+## CUDA Dependency Story (Full Debugging History)
+
+### Problem
+`llama-cpp-python` installed with CUDA 12.1 wheel (`--extra-index-url .../cu121`). At import time, `libllama.so` needs multiple CUDA shared libraries (`libcudart.so.12`, `libcublas.so.12`, `libcublasLt.so.12`, etc.), but they aren't on the system linker path in the HF Space container.
+
+### Failed Attempt #1 — `LD_LIBRARY_PATH` via `os.environ`
+**Why it failed:** The C dynamic linker (`ld.so`) reads `LD_LIBRARY_PATH` once at process start. Setting `os.environ["LD_LIBRARY_PATH"]` in Python at runtime has zero effect on `dlopen()`.
+
+### Failed Attempt #2 — Load only `libcudart.so.12`
+**Why it failed:** `libllama.so` also needs `libcublas.so.12`, `libcublasLt.so.12`, and other CUDA libs. Loading just `libcudart` got us past the first error but hit the next missing library.
+
+### Working Solution — Bulk ctypes preload
+`model_inference.py:20-78`
+
+```python
+import ctypes
+
+def _preload_dir(lib_dir):
+    """Preload every lib*.so* in a directory with RTLD_GLOBAL."""
+    for f in sorted(os.listdir(lib_dir)):
+        if f.startswith("lib") and ".so" in f:
+            ctypes.CDLL(os.path.join(lib_dir, f), mode=ctypes.RTLD_GLOBAL)
+```
+
+**Why it works:** `ctypes.CDLL(full_path, mode=RTLD_GLOBAL)` calls `dlopen()` with the full path + `RTLD_GLOBAL` flag. This loads the library into the process and registers it under its SONAME. When `libllama.so` is subsequently `dlopen()`'d, the linker finds all CUDA dependencies already resident.
+
+**Search order:**
+1. System CUDA: `/usr/local/cuda/lib64`, `/usr/local/cuda-12.1/lib64`, `/usr/local/cuda-12.4/lib64`
+2. Pip packages: iterate `sys.path`, find `nvidia/*/lib/` directories, preload all `.so` files
+
+### Required CUDA Pip Packages
+
+| Package | Version | Provides |
+|---|---|---|
+| `nvidia-cuda-runtime-cu12` | 12.1.105 | `libcudart.so.12` |
+| `nvidia-cublas-cu12` | 12.1.3.1 | `libcublas.so.12`, `libcublasLt.so.12` |
+| `nvidia-cusparse-cu12` | 12.1.0.106 | `libcusparse.so.12` |
+
+These are in `requirements.txt`. On the HF Space, the bulk preload picked up **39 total CUDA shared libraries** from the transitive dependencies of these packages.
+
+### Import Guard
+
+`model_inference.py:81-91` — If the `from llama_cpp import Llama` still fails (truly missing CUDA), a `RuntimeError` with a clear diagnostic is raised:
+
+```
+Failed to load llama-cpp-python — CUDA runtime not found.
+If this is a CPU-only machine, install the CPU wheel:
+  pip uninstall llama-cpp-python -y && pip install llama-cpp-python
+```
+
+---
+
+## UI Design System
+
+### Current Palette (Cool Professional — slate + indigo)
+
+| Token | Value | Usage |
+|---|---|---|
+| `--bg` | `#f1f5f9` (slate-100) | Page background |
+| `--surface` | `#ffffff` | Cards, inputs, code blocks, tables |
+| `--surface-alt` | `#f8fafc` (slate-50) | Table headers |
+| `--border` | `#e2e8f0` (slate-200) | All borders |
+| `--text` | `#1e293b` (slate-800) | Primary body text |
+| `--text-muted` | `#64748b` (slate-500) | Secondary text, labels |
+| `--action` | `#4f46e5` (indigo-600) | Primary CTA buttons |
+| `--action-hover` | `#4338ca` (indigo-700) | Button hover state |
+| `--error` | `#b91c1c` (red-700) | Error messages |
+| `--success` | `#059669` (emerald-600) | Success messages |
+| `--radius` | `12px` | All containers, inputs |
+| `--radius-lg` | `20px` | (defined but not heavily used) |
+| `--transition` | `120ms ease-out` | Hover/focus transitions |
+
+### Fonts
+- **Headings:** Cormorant Garamond (serif) — `--font-display`
+- **Body/UI:** Atkinson Hyperlegible (sans) — `--font-ui`
+- **Code:** JetBrains Mono — `--font-mono`
+
+All loaded via Google Fonts in `HEAD_HTML` (`app.py:345`).
+
+### Gradio Dark-Theme Overrides
+`app.py:77-82` — A "nuke block" strips Gradio's dark-theme defaults before our rules apply:
+```css
+.gr-textbox, .gr-code, .gr-dataframe, .gr-accordion {
+    background: transparent !important;
+    border-color: transparent !important;
+}
+```
+
+The CSS uses aggressive descendant selectors (`> div > div`, `[data-testid]`) because Gradio nests components in multiple wrapper divs.
+
+### Design Evolution
+1. **Original:** Linear/Vercel inspired — teal accent (`#14b8a6`), Inter font, white backgrounds
+2. **WellFull Collective:** Warm terracotta + sage palette — rejected (looked "bad" for a data tool)
+3. **Current:** Cool slate + indigo professional palette with the WellFull structure (fonts, radii, spacing)
+
+---
+
+## Data Engine & Parquet Optimization
+
+### Per-Request Lifecycle
+`data_engine.py:create_session()` → `get_connection()` + `seed_database()`
+
+Each query creates a **fresh in-memory DuckDB** connection. This ensures:
+- Thread safety (no shared state between requests)
+- Query isolation (can't affect other requests)
+- Clean state (no stale data)
+
+### Seeding Priority
+`data_engine.py:seed_database()` [data_engine.py:62-96]
+
+1. **Parquet files** (fastest — 33 KB, single-digit ms):
+   - `/data/enrollment.parquet` + `/data/attendance.parquet` (Space persistent storage)
+   - `data/enrollment.parquet` + `data/attendance.parquet` (local dev)
+   - Loaded via DuckDB `read_parquet()` → `CREATE TABLE ... AS SELECT *`
+2. **`data/seed.sql`** (custom overrides)
+3. **Python generator** `data/generate_seed.py` (slow fallback — ~2,900 students, ~11,600 attendance rows)
+
+### Parquet Bootstrap
+`app.py:22-29` — On startup, if no Parquet files are found in any of the search dirs, `export_parquet.py` is called to generate them. On the Space, they go to `/data/` which persists across restarts.
+
+### SQL Safety Pipeline
+`data_engine.py:execute_safe()` [data_engine.py:235-260]
+
+1. `extract_sql()` — Parse JSON envelope → ```sql``` block → raw fallback
+2. `validate_sql()` — Forbidden token check (DROP, DELETE, INSERT, UPDATE, etc.) + schema-aware `EXPLAIN` validation
+3. Wrap: `SELECT * FROM (<user_query>) AS _safe LIMIT 1000`
+4. Execute directly on DuckDB
+5. Return `(cleaned_sql, DataFrame)`
+
+### Forbidden Tokens
+`data_engine.py:22-25` — `DROP`, `DELETE`, `INSERT`, `UPDATE`, `ALTER`, `TRUNCATE`, `CREATE`, `ATTACH`, `DETACH`, `PRAGMA`
+
+---
+
+## Database Schema
+
+### `enrollment`
+
+| Column | Type | Description |
+|---|---|---|
+| `school_year` | VARCHAR | School year, format `'YYYY-YYYY'` |
+| `school_name` | VARCHAR | One of 5 schools |
+| `grade_level` | INTEGER | Grade level (K=0 through 12) |
+| `student_count` | INTEGER | Students enrolled in that grade/year/school |
+
+### `attendance`
+
+| Column | Type | Description |
+|---|---|---|
+| `student_id` | INTEGER | Unique student identifier |
+| `school_name` | VARCHAR | School the student attends |
+| `school_year` | VARCHAR | School year, format `'YYYY-YYYY'` |
+| `absence_count` | INTEGER | Total absences for that year |
+| `is_chronically_absent` | BOOLEAN | TRUE if missed ≥10% of school days |
+
+### Schools
+
+| School | Grades | Base Enrollment |
+|---|---|---|
+| Lincoln Elementary | K–5 | 520 |
+| Washington Middle | 6–8 | 480 |
+| Jefferson High | 9–12 | 900 |
+| Roosevelt Academy | K–8 | 380 |
+| Kennedy Prep | 6–12 | 620 |
+
+### Seed Data Stats
+- **Students:** 2,900
+- **Chronic absenteeism rate:** 15% (435 students)
+- **Enrollment rows:** 116 (5 schools × 4 years × variable grades)
+- **Attendance rows:** 11,600 (2,900 students × 4 years)
+- **School years:** 2021-2022, 2022-2023, 2023-2024, 2024-2025
+- **YOY growth:** 3% per year with ±15% random grade-size variance
+- **Reproducible:** `random.seed(42)` in `generate_seed.py`
+
+---
+
+## Prompt Engineering
+
+`prompts.py` assembles the full LLM prompt:
+
+### Structure
+```
+SYSTEM_PROMPT (rules, constraints, output format)
+  ↓
+Schema documentation (table + column list)
+  ↓
+Few-shot examples (4 question→SQL pairs)
+  ↓
+User question
+  ↓
+"SQL:"
+```
+
+### System Prompt Rules
+`prompts.py:SYSTEM_PROMPT` [prompts.py:9-18]
+1. Only SELECT statements
+2. Exact table/column names from schema
+3. Proper DuckDB syntax (VARCHAR → single quotes, BOOLEAN → TRUE/FALSE)
+4. Use column aliases for aggregations
+5. Join on logical columns
+6. Make reasonable assumptions if ambiguous
+7. Output ONLY the ```sql``` block, no explanation
+
+### Few-Shot Examples
+`prompts.py:FEW_SHOT_EXAMPLES` [prompts.py:56-76]
+
+| Question | SQL pattern |
+|---|---|
+| "How many chronically absent in 2023-2024?" | `COUNT(*)` with `WHERE ... AND is_chronically_absent = TRUE` |
+| "Show enrollment per school sorted" | `SUM(student_count) GROUP BY school_name ORDER BY ... DESC` |
+| "What percentage at Lincoln Elementary?" | `COUNT(CASE WHEN ...) * 100.0 / COUNT(*)` |
+| "Enrollment trend since 2021" | `GROUP BY school_year ORDER BY school_year` |
+
+---
+
+## Fine-Tuning Pipeline (Modal)
+
+### Location: `modal_train/`
+
+| Script | Purpose |
+|---|---|
+| `generate_synthetic.py` | Creates 1,289 NL→SQL pairs from 32 query templates |
+| `train.py` | Unsloth QLoRA on Qwen2.5-Coder-7B (r=16, 4-bit, 3 epochs, A10G) |
+| `export_gguf.py` | Merges LoRA → converts to GGUF Q4_K_M → pushes to HF Hub |
+| `modal_app.py` | Modal orchestration (`modal.App("kasualdad-lfed-train")`) |
+| `train.jsonl` | Training data (1,289 pairs) |
+
+### How to Run
+```bash
+modal secret create huggingface HF_TOKEN=<token>
+modal run modal_train/modal_app.py
+```
+
+### Post-Training
+After the fine-tuned model is pushed to HF Hub, swap `HF_REPO_ID` and `HF_MODEL_FILE` in `model_inference.py:103-107`. The exact repo ID is documented in `docs/HANDOFF.md`.
+
+**Status:** ⏳ Not yet run (needs HF_TOKEN Modal secret)
+
+---
+
+## Dependencies
+
+### Python (`requirements.txt`)
+
+| Package | Version | Why |
+|---|---|---|
+| `gradio` | 6.16.0 | Web UI |
+| `duckdb` | 1.5.3 | In-memory SQL engine |
+| `llama-cpp-python` | 0.3.26 | Local LLM inference (CUDA 12.1 wheel) |
+| `huggingface_hub` | 1.18.0 | Model download, HF Hub API |
+| `spaces` | latest | `@spaces.GPU` decorator for Zero GPU |
+| `nvidia-cuda-runtime-cu12` | 12.1.105 | `libcudart.so.12` |
+| `nvidia-cublas-cu12` | 12.1.3.1 | `libcublas.so.12`, `libcublasLt.so.12` |
+| `nvidia-cusparse-cu12` | 12.1.0.106 | `libcusparse.so.12` |
+
+**Wheel index:** `--extra-index-url https://abetlen.github.io/llama-cpp-python/whl/cu121` — required to get the pre-built CUDA 12.1 wheel (avoids 15-minute source compile).
+
+---
+
+## Hackathon Badges
+
+| Badge | Status | Implementation |
+|---|---|---|
+| **Off the Grid** | ✅ | llama.cpp + local GGUF, no API calls |
+| **Well-Tuned** | ✅ | Fine-tuning pipeline ready (Modal, 1,289 pairs) |
+| **Llama Champion** | ✅ | llama.cpp sole inference backend, Q4_K_M, streaming |
+| **Off-Brand** | ✅ | Custom design system (Atkinson Hyperlegible + Cormorant Garamond, slate + indigo, WCAG AA) |
+
+---
+
+## Gotchas & Pitfalls
+
+### 1. `LD_LIBRARY_PATH` doesn't work at Python runtime
+The dynamic linker caches it at process start. Use `ctypes.CDLL(path, mode=RTLD_GLOBAL)` to preload libraries instead.
+
+### 2. Zero GPU has CUDA emulation at module level
+Load models with GPU layers at startup (module level), not lazily inside `@spaces.GPU` functions. The docs explicitly say lazy loading is "significantly less efficient."
+
+### 3. Gradio dark theme bleeds through CSS
+Gradio 6 uses a dark theme by default. You need a "nuke block" (`background: transparent !important`) at the top of your CSS before applying light-theme colors, and you need to target nested wrapper divs (`> div > div`) because Gradio wraps everything.
+
+### 4. HF Spaces auto-link models detected in repo text
+Any `username/repo-name` string in README, code, or docs gets linked on the Space page. Keep commented-out future model references out of the repo, or use a non-detectable format (e.g., point to HANDOFF.md instead of spelling out the repo ID).
+
+### 5. The fine-tuned model doesn't exist yet
+`kasualdad/lfed-qwen2.5-coder-7b-sql-gguf` is a placeholder. All references to it were scrubbed from the repo to prevent double-linking on the HF Space. The exact repo ID lives only in `docs/HANDOFF.md` for when training completes.
+
+### 6. Parquet files are gitignored but force-committed
+The `.gitignore` has `*.parquet`, but the seed Parquet files (33 KB total) were force-added (`git add -f`) because they're needed for local dev and Space builds. Update `export_parquet.py` and re-export if the seed data changes.
+
+### 7. macOS vs Linux CUDA divergence
+Locally (macOS), `llama-cpp-python` uses Metal or CPU — the CUDA preload code is harmless (it searches for `.so` files that don't exist and skips silently). On the HF Space (Linux), it preloads 39 CUDA libraries.
+
+---
+
+## Quick Start
+
+```bash
+# Clone
+cd Kasualdad_LFED
+
+# Virtual env
+python3.12 -m venv .venv && source .venv/bin/activate
+
+# Install
+pip install -r requirements.txt
+
+# Generate Parquet seed files (first time only)
+python data/export_parquet.py
+
+# Run
+python app.py
+# → http://localhost:7860
+
+# Tests
+pytest tests/ -v
+
+# Deploy
+git push space main
+```
+
+---
+
+## Git History (this session)
+
+```
+0e9c140 perf: load seed data from Parquet instead of generating on every query
+9bae1e9 fix: remove remaining fine-tuned model repo ID from HANDOFF.md
+14ee1d7 fix: remove nonexistent fine-tuned model reference to prevent double-linking on HF Space
+9618e4b design: hybrid cool palette (slate+indigo), aggressive overrides for Gradio dark-theme boxes
+47f650c fix: restore GPU offload on Zero GPU (emulation mode handles module-level CUDA)
+827143b design: apply WellFull Collective palette + Cormorant Garamond + Atkinson Hyperlegible
+7a51b3e fix: CPU-only on Zero GPU (model persists in RAM), 8 threads, restore eager load
+9ba7b17 fix: add @spaces.GPU decorator, defer model load to first query (Zero GPU)
+a963ee6 fix: preload ALL CUDA .so files, add nvidia-cublas-cu12 + nvidia-cusparse-cu12
+adf4793 fix: preload libcudart.so.12 via ctypes (LD_LIBRARY_PATH has no effect at runtime)
+a6f6cbd fix: resolve CUDA runtime from pip package, add GPU→CPU fallback
+```
+
+---
+
+## Future Work
+
+- [ ] **Run Modal fine-tuning** — needs `modal secret create huggingface HF_TOKEN=<token>`, then `modal run modal_train/modal_app.py`
+- [ ] **Swap to fine-tuned model** — after GGUF is pushed to HF Hub, update `HF_REPO_ID`/`HF_MODEL_FILE` in `model_inference.py:103-107`
+- [ ] **Add streaming back** — `generate_sql_streaming()` exists but is unused (switched to synchronous for HF Space reliability). Could re-enable with proper Zero GPU timeout handling.
+- [ ] **HF Space smoke test** — verify all 6 example queries return correct results after each deploy
+- [ ] **PRO subscription** — if 5 min/day GPU quota is too tight, upgrade to PRO (40 min/day)