Gemma-4-31B Rovo Chat Orchestrator — LoRA r=64 (LR 5e-5), merged
LoRA SFT (rank=64, alpha=128, LR=5e-5, seq=16384, 1 epoch) merged into google/gemma-4-31B-it. One point in a rank x LR sweep on the Rovo Chat Orchestrator tool-calling task.
Eval (725 held-out examples, Claude Opus 4.8 judge)
| metric | this model | base gemma-4-31B-it |
|---|---|---|
| Text AA (shallow) | 90.3% | 96.1% |
| Tool AA (shallow) | 51.6% | 53.2% |
| Overall AA | 80.1% | 84.8% |
| Text AA Deep (binary, 1-5 rubric) | 75.7% | 86.7% |
| Deep quality mean (1-5) | 4.09 | 4.27 |
| False-tool-trigger rate | 6.9% | 1.9% |
| Runaway rate | 1.5% | 0.3% |
Sweep finding
Across the rank x LR grid, learning rate dominates both accuracy and stability: every LR=5e-5 config beats every LR=1e-4 config on overall AA and runaway rate. Instability is approximately (adapter capacity x LR) = how far the weights travel from the RLHF-aligned base. The sweet spot is r=16, LR=5e-5 (sibling repo tzchen07/gemma4-31b-rovochat-lora-r16-lr5e5).
No fine-tune beats the base -it model overall — narrow-domain tuning erodes the base's tool-vs-text calibration faster than it gains tool accuracy. Base wins on text AA, overall, and stability; the best tune improves only tool AA (+3 pts). Companion full-FT analysis (LR is the killing factor): tzchen07/gemma4-31b-rovochat-sft-v3.
- Downloads last month
- 18