--- license: mit base_model: - Qwen/Qwen3-0.6B tags: - llm-router - model-routing - openrouter - safetensors - research --- # BenchGen Fugu-Lite Fugu-Lite is a small, runnable model router for one local GPU. A frozen local language model reads a task, a custom linear head selects one worker, and the selected worker is called through OpenRouter (or through a local OpenAI-compatible server). This is **Fugu-inspired research code, not Sakana AI's proprietary Fugu implementation**. The useful overlap is the learned worker selection, offline trajectory/reward collection, and an optional separable CMA-ES stage. Version 0.1 performs one learned routing decision; it does not yet learn recursive, multi-turn delegation or answer synthesis. ## Project documentation - [How the system works](docs/HOW_IT_WORKS.md) - [V1 experiment and progress record](docs/V1_PROGRESS.md) - [Use the private Hugging Face repository](docs/HUGGING_FACE.md) - [Documentation index](docs/README.md) The repository includes the V1 task/reward snapshot and trained routing-head artifacts so the experiment can be audited and resumed. The real API key is never stored in the project. ## What trains where | Part | Runs on | What changes | API cost during training? | |---|---|---|---| | Reward-data generation | OpenRouter/local workers | Nothing; records every worker's answer, score, cost, and latency | Yes, once per task × worker | | SFT warm-start | Your GPU | Router LayerNorm + classification head | No | | Contextual-bandit RL | Your GPU | Same small routing head | No | | Sep-CMA-ES (optional) | CPU after one GPU embedding pass | Final linear routing layer | No | | Inference | Your GPU + selected worker | Nothing | One worker call per prompt, unless fallback is needed | The default backbone is `Qwen/Qwen3-0.6B`. With three workers, its trainable head is only a few thousand parameters. The backbone stays frozen, so the project is practical on modest GPUs. The exact memory depends on sequence length and software, but the default normally fits comfortably above roughly 4 GiB of VRAM. ## Architecture ```mermaid flowchart TD A[BenchGen tasks] --> B[Call every worker once] B --> C[Quality, cost, latency rewards] C --> D[Reward matrix JSONL] D --> E[SFT warm-start] E --> F[Contextual-bandit RL] F --> G[Optional Sep-CMA-ES] G --> H[Local router checkpoint] H --> I[Single API endpoint] I --> J[Selected OpenRouter or local worker] ``` ## 1. Install on the remote GPU server Python 3.10–3.13 and a working NVIDIA/PyTorch setup are expected. Install the PyTorch CUDA build recommended for your server's driver from first. If `python -c "import torch; print(torch.cuda.is_available())"` already prints `True`, keep that installation. ```bash unzip BenchGen_Fugu_Lite.zip cd fugu-lite python3 -m venv .venv source .venv/bin/activate python -m pip install --upgrade pip pip install -e '.[dev]' cp .env.example .env chmod 600 .env ``` Edit `.env` on the server and set `OPENROUTER_API_KEY`. Never paste the real key into source code, YAML, a Git commit, or chat. Check the machine: ```bash fugu-lite doctor ``` If PyTorch cannot see CUDA, fix the server's NVIDIA driver/PyTorch installation before training. The router can run on CPU, but it will be much slower. ## 2. Prove the whole data path without spending credits The mock pool contains a math, code, and general expert. It deterministically creates a reward matrix from the sample tasks. ```bash fugu-lite generate \ --workers configs/workers.mock.yaml \ --tasks data/tasks.sample.jsonl \ --output data/rewards.mock.jsonl ``` Run the tests: ```bash pytest -q ``` ## 3. Choose real OpenRouter workers OpenRouter model availability changes, so query the live catalog rather than relying on an old list: ```bash fugu-lite list-models --contains qwen --limit 30 fugu-lite list-models --contains free --limit 30 ``` Copy and edit the worker file: ```bash cp configs/workers.openrouter.example.yaml configs/workers.yaml ``` Choose **meaningfully different** workers—for example, a low-cost fast model, a strong reasoning model, and a coding specialist. If all workers produce the same reward on every task, the router has no learning signal. Three or four workers are a good first experiment. To include a model hosted on the same server, add an `openai_compatible` worker. Any server that exposes `/v1/chat/completions` can be used; an example vLLM entry is already commented in the YAML. ## 4. Prepare real BenchGen tasks Input is JSON Lines, one object per task: ```json {"task_id":"math-101","prompt":"Solve ...","domain":"math","reference_answer":"42","grader":{"type":"numeric","tolerance":0.0},"split":"train","tags":["algebra"]} ``` Supported local graders are `exact`, `contains`, `numeric`, and `regex`. For an open-ended task, use `llm_judge`, add a clear `grader.rubric`, and configure `reward.judge_model` in the worker YAML. LLM judging adds one judge call for each worker response and should be audited on a human-labeled sample. Do not place answers, benchmark labels, or worker scores in `prompt`; only the grader sees the reference answer. Keep benchmark test sets out of training. A useful first real run is: - 200–1,000 training tasks covering your production distribution; - at least 50 validation and 100 held-out test tasks; - balanced domains, including tasks on which each worker can actually win; - deterministic executable graders whenever possible. If BenchGen already produces worker rewards, skip API generation and write the matrix shape shown in `data/rewards.schema.example.jsonl`. `worker_ids` order must remain identical in every row, and `rewards[i]` must belong to `worker_ids[i]`. ## 5. Generate the real reward matrix First use a small limit to verify prompts, grading, and billing: ```bash fugu-lite generate \ --workers configs/workers.yaml \ --tasks data/tasks.jsonl \ --output data/rewards.real.jsonl \ --limit 10 ``` Inspect the JSONL. When it looks correct, run the complete job (preferably to a new output path). Records are appended after each task, so an interrupted job can continue safely with `--resume`. The number of primary calls is `number_of_tasks × number_of_workers`; `llm_judge` doubles that count. OpenRouter's non-streaming usage object is recorded when it provides token and cost fields. ```bash fugu-lite generate \ --workers configs/workers.yaml \ --tasks data/tasks.jsonl \ --output data/rewards.real.jsonl \ --resume ``` The default utility is: \[ U = w_q Q - w_c \min(C/C_0,5) - w_l \min(L/L_0,5) \] where quality \(Q\) is 0–1, cost \(C\) is the reported request cost, and latency \(L\) is seconds. Adjust the weights and scales in the worker YAML to match your product. If quality matters much more than price, keep the cost and latency weights small. ## 6. Train the routing head Start with SFT. It converts each reward vector into a soft target distribution and trains the custom head with cross-entropy: ```bash fugu-lite train-sft \ --config configs/train_sft.yaml \ --data data/rewards.real.jsonl \ --output artifacts/router-sft ``` Then run real offline contextual-bandit RL. The default REINFORCE estimator samples worker actions and uses the full-information expected reward as a baseline: ```bash fugu-lite train-rl \ --config configs/train_rl.yaml \ --data data/rewards.real.jsonl \ --checkpoint artifacts/router-sft \ --output artifacts/router-rl ``` For smaller or noisy data, `estimator: expected_reward` in `train_rl.yaml` is lower variance. For a closer analogue to the reported Sakana optimization style, optionally run separable CMA-ES. It caches backbone features once and evolves only the last linear layer: ```bash fugu-lite train-es \ --config configs/train_es.yaml \ --data data/rewards.real.jsonl \ --checkpoint artifacts/router-rl \ --output artifacts/router-es ``` Use SFT alone as the first baseline. Add RL only when workers have different rewards and the validation utility improves. Add CMA-ES when you want to optimize a discrete or otherwise non-differentiable end metric; keep it only if held-out performance improves. ## 7. Evaluate correctly ```bash fugu-lite evaluate \ --checkpoint artifacts/router-rl \ --data data/rewards.real.jsonl \ --split test \ --output artifacts/router-rl/test_metrics.json ``` The report includes: | Metric | Meaning | |---|---| | `router_utility` | Mean reward obtained by the learned greedy route | | `oracle_utility` | Upper bound if the best worker were known for every task | | `best_fixed_utility` | Strongest single worker used for every task | | `random_utility` | Uniform random routing baseline | | `regret` | Oracle utility minus router utility; lower is better | | `oracle_route_accuracy` | Fraction of tasks routed to an oracle-best worker | Do not deploy merely because route accuracy looks high. The router should beat the best fixed worker on held-out utility and on your actual cost/latency guardrails. ## 8. Use the trained router Route without calling any worker: ```bash fugu-lite route \ --checkpoint artifacts/router-rl \ --domain code \ --prompt "Find the bug in this Python function ..." ``` Route and call the selected worker: ```bash fugu-lite ask \ --checkpoint artifacts/router-rl \ --workers configs/workers.yaml \ --domain code \ --prompt "Find the bug in this Python function ..." ``` Serve a single OpenAI-compatible endpoint: ```bash fugu-lite serve \ --checkpoint artifacts/router-rl \ --workers configs/workers.yaml \ --host 0.0.0.0 \ --port 8080 ``` ```bash curl http://127.0.0.1:8080/v1/chat/completions \ -H 'Content-Type: application/json' \ -d '{"model":"fugu-lite","domain":"code","messages":[{"role":"user","content":"Explain this stack trace..."}]}' ``` The response includes a `fugu_lite` field with route probabilities, the actual served model, latency, cost when available, and fallback errors. In production, put authentication and TLS in front of this service; do not expose it directly to the public internet. ## What to improve next Version 0.1 is deliberately narrow. The next meaningful step toward a fuller Fugu-like system is to define a small action grammar such as `CALL(worker, subtask)`, `VERIFY(worker)`, `SYNTHESIZE`, and `STOP`; collect complete multi-turn trajectories in a sandbox; and optimize the final verified task reward plus cost/latency penalties. Do that only after this one-step router reliably beats fixed-worker baselines—otherwise multi-agent complexity hides basic data and reward problems. ## References - Sakana AI Fugu release: - OpenRouter quickstart: - OpenRouter usage accounting: - Qwen3-0.6B model card: - `cmaes` Sep-CMA implementation: