jdthewlis commited on
Commit
6733293
Β·
verified Β·
1 Parent(s): 0fe04a0

Add science-gym-bio sweep figure + shared run link; balance training-effect section

Browse files
Files changed (3) hide show
  1. .gitattributes +1 -0
  2. README.md +25 -19
  3. science_gym_bio_sweep.png +3 -0
.gitattributes CHANGED
@@ -34,3 +34,4 @@ saved_model/**/* filter=lfs diff=lfs merge=lfs -text
34
  *.zst filter=lfs diff=lfs merge=lfs -text
35
  *tfevents* filter=lfs diff=lfs merge=lfs -text
36
  header.png filter=lfs diff=lfs merge=lfs -text
 
 
34
  *.zst filter=lfs diff=lfs merge=lfs -text
35
  *tfevents* filter=lfs diff=lfs merge=lfs -text
36
  header.png filter=lfs diff=lfs merge=lfs -text
37
+ science_gym_bio_sweep.png filter=lfs diff=lfs merge=lfs -text
README.md CHANGED
@@ -54,7 +54,8 @@ why the adapter is large (~4.6 GB) despite being rank-16.
54
 
55
  ## Training
56
 
57
- Trained on [Prime Intellect](https://app.primeintellect.ai) Hosted Training:
 
58
 
59
  - **Algorithm:** GRPO
60
  - **Reward:** binary final-answer correctness (1.0 correct / 0.0 wrong) β€” using
@@ -63,16 +64,21 @@ Trained on [Prime Intellect](https://app.primeintellect.ai) Hosted Training:
63
  - **Rollouts per example:** 16
64
  - **Batch size:** 128
65
  - **Max tokens:** 4096, thinking enabled
66
- - Stopped early once held-out eval saturated at 100% (β‰ˆ30 steps).
67
 
68
- ## What training actually changed
69
 
70
- The base model already *knew how to use the tools* β€” its failures were almost
71
- always a behavioural one: it would gather evidence and then **forget to commit a
72
- final answer**, running out of turns at reward 0.
73
 
74
- **Concrete before/after.** The single question the base model got wrong in the
75
- first batch (step 0, before any gradient update) was this Veber-filter problem:
 
 
 
 
 
76
 
77
  > *Among C1–C4, find the single candidate passing Veber (`veber_pass == 1`) with
78
  > `tpsa ≀ 120` and `rotatable_bonds ≀ 6`.*
@@ -82,20 +88,20 @@ first batch (step 0, before any gradient update) was this Veber-filter problem:
82
  > - **C3 `COc1ccc(CCN(C)C)cc1` ← correct answer**
83
  > - C4 `NCCCCCCCCCCCCNCC(=O)O`
84
 
85
- The base model made **12 tool calls** (`mol_descriptors`, `veber_pass`) and even
86
- measured C3 β€” seeing it clearly passes (TPSA 12.5, 4 rotatable bonds) β€” but it
87
- **never called `submit_answer`**. It ran out of turns and scored **0**.
 
88
 
89
- After RL, the model reliably does the *measure-then-commit* loop:
90
 
91
- | | Base model (step 0) | Trained (step 30) |
92
- |---|---|---|
93
- | Held-out eval `pass@1` (n=20) | **0.95** (19/20) | **1.00** (20/20) |
94
- | Sampled training rollouts solved | one Veber question lost to "no answer submitted" | **every** sampled rollout submitted a correct answer (reward mean 0.99) |
 
95
 
96
- So the headline effect is not new chemistry knowledge β€” it's **discipline**: the
97
- adapter teaches the model to stop dithering with tools and actually commit the
98
- answer the evidence supports.
99
 
100
  ## Usage
101
 
 
54
 
55
  ## Training
56
 
57
+ Trained on [Prime Intellect](https://app.primeintellect.ai) Hosted Training β€”
58
+ **[shared run dashboard](https://app.primeintellect.ai/training/shared/buoc18kj8s9odmkdorp0kto6)**:
59
 
60
  - **Algorithm:** GRPO
61
  - **Reward:** binary final-answer correctness (1.0 correct / 0.0 wrong) β€” using
 
64
  - **Rollouts per example:** 16
65
  - **Batch size:** 128
66
  - **Max tokens:** 4096, thinking enabled
67
+ - Stopped early once held-out eval reached 100% (β‰ˆ30 steps).
68
 
69
+ ## What training changed
70
 
71
+ RL taught the model to run the **measure-then-commit** loop reliably: gather the
72
+ tool evidence a question needs, then commit a final answer. Held-out eval rose to
73
+ a perfect score and training stayed stable throughout (no collapse).
74
 
75
+ | | Base model (step 0) | Trained (step 30) |
76
+ |---|---|---|
77
+ | Held-out eval `pass@1` (n=20) | 0.95 (19/20) | **1.00 (20/20)** |
78
+ | Sampled training rollouts | a Veber question lost to "no answer submitted" | **every** sampled rollout submitted a correct answer (reward mean 0.99) |
79
+
80
+ **A concrete win.** The one question the base model got wrong in the first batch
81
+ (step 0, before any gradient update) was this Veber-filter problem:
82
 
83
  > *Among C1–C4, find the single candidate passing Veber (`veber_pass == 1`) with
84
  > `tpsa ≀ 120` and `rotatable_bonds ≀ 6`.*
 
88
  > - **C3 `COc1ccc(CCN(C)C)cc1` ← correct answer**
89
  > - C4 `NCCCCCCCCCCCCNCC(=O)O`
90
 
91
+ The base model made 12 tool calls and even measured C3 β€” seeing it clearly passes
92
+ (TPSA 12.5, 4 rotatable bonds) β€” but **never called `submit_answer`**, ran out of
93
+ turns, and scored 0. The trained adapter handles exactly this case: it measures
94
+ the candidates and commits the answer the evidence supports.
95
 
96
+ ## Choosing the training recipe
97
 
98
+ The stable hyperparameters above didn't come for free β€” they're the output of a
99
+ sweep on a precursor environment, `allan/science-gym-bio`. The lesson: **learning
100
+ rate is the stability lever** (5e-5 peaks then collapses; 1e-5 holds), and larger
101
+ rollout groups (8 β†’ 16) cut GRPO advantage variance. That recipe β€” **LR 1e-5,
102
+ 16 rollouts/example, thinking on** β€” is what we carried into the protein-ligand run.
103
 
104
+ ![science-gym-bio hyperparameter sweep](science_gym_bio_sweep.png)
 
 
105
 
106
  ## Usage
107
 
science_gym_bio_sweep.png ADDED

Git LFS Details

  • SHA256: 3e1e3ded07e2182d01ed98b4babcf73995e0df0ca38e82df39a3ac3db9a47918
  • Pointer size: 131 Bytes
  • Size of remote file: 199 kB