GoodStartLabs
/

ginrummy-qwen3-8b-grpo-lora

@@ -1,39 +1,190 @@
 ---
-license: apache-2.0
 base_model: Qwen/Qwen3-8B
-tags: [gin-rummy, grpo, reinforcement-learning, lora, game-playing]
-datasets: [self-play]
 library_name: peft
 ---
-# Gin Rummy GRPO LoRA — Run 1 (Qwen3-8B)
-LoRA trained via GRPO on Gin Rummy self-play. Baseline run (no reasoning, no tools).
-| Field | Value |
-|-------|-------|
-| **Base model** | [Qwen/Qwen3-8B](https://huggingface.co/Qwen/Qwen3-8B) |
 | **Method** | GRPO (TRL GRPOTrainer) |
-| **LoRA** | r=16, alpha=32, q/k/v/o_proj |
-| **Steps** | 200 (800 games) |
-| **Opponent** | Algorithmic bot (GinRummyBot) |
-| **Win rate** | 16.8% |
-| **Infra** | Together AI 8x H100 80GB, 11.9 min |
-| **Git commit** | `2a987af docs: add Phase 3 (V2 migration) to experiment log` |
-| **W&B** | N/A (logging disabled in this run) |
-| **Dataset** | Self-play rollouts vs algorithmic bot |
-## Config
-- `enable_thinking=False`, minimal prompt (no tools), `num_generations=4`, `lr=1e-6`
-- Sparse terminal reward only (+1/-1/0)
-## Limitations
-Baseline run — no reasoning tokens, no tool use. See [experiment log](https://github.com/GoodStartLabs/GinRummy_RLenv/blob/main/grpo_training/experiment_log.md) for the full 9-run iteration history.
 ## Usage
 ```python
 from peft import PeftModel
-from transformers import AutoModelForCausalLM
-base = AutoModelForCausalLM.from_pretrained("Qwen/Qwen3-8B", torch_dtype="bfloat16", device_map="auto")
 model = PeftModel.from_pretrained(base, "GoodStartLabs/ginrummy-qwen3-8b-grpo-lora")
 ```

 ---
 base_model: Qwen/Qwen3-8B
+datasets:
+- self-play
 library_name: peft
+license: apache-2.0
+tags:
+- gin-rummy
+- grpo
+- reinforcement-learning
+- lora
+- game-playing
+- qwen3
+pipeline_tag: text-generation
+model-index:
+- name: ginrummy-qwen3-8b-grpo-lora
+  results:
+  - task:
+      type: text-generation
+      name: Math Reasoning (GSM8K)
+    dataset:
+      name: GSM8K
+      type: gsm8k
+    metrics:
+    - type: accuracy
+      value: 91.1
+      name: Accuracy
+      verified: false
+  - task:
+      type: text-generation
+      name: Science Reasoning (ARC Challenge)
+    dataset:
+      name: ARC Challenge
+      type: ai2_arc
+    metrics:
+    - type: accuracy
+      value: 96.3
+      name: Accuracy
+      verified: false
+  - task:
+      type: text-generation
+      name: Factual Accuracy (TruthfulQA)
+    dataset:
+      name: TruthfulQA
+      type: truthful_qa
+    metrics:
+    - type: accuracy
+      value: 69.0
+      name: Accuracy
+      verified: false
+  - task:
+      type: text-generation
+      name: Broad Knowledge (MMLU-Pro)
+    dataset:
+      name: MMLU-Pro
+      type: TIGER-Lab/MMLU-Pro
+    metrics:
+    - type: accuracy
+      value: 59.7
+      name: Accuracy
+      verified: false
+  - task:
+      type: text-generation
+      name: Commonsense Reasoning (HellaSwag)
+    dataset:
+      name: HellaSwag
+      type: hellaswag
+    metrics:
+    - type: accuracy
+      value: 73.3
+      name: Accuracy
+      verified: false
 ---
+# ginrummy-qwen3-8b-grpo-lora
+A LoRA adapter trained via **GRPO** (Group Relative Policy Optimization) on Gin Rummy self-play, built on [Qwen/Qwen3-8B](https://huggingface.co/Qwen/Qwen3-8B).
+## Evaluation Results vs Base Model
+Benchmarked against the base Qwen3-8B (via OpenRouter) with n=300 samples per benchmark and Wilson 95% confidence intervals.
+| Benchmark | Base Qwen3-8B | This Model | Delta | 95% CI (this model) | Significant? |
+|-----------|:---:|:---:|:---:|:---:|:---:|
+| **GSM8K** (math) | **96.3%** | 91.1% | -5.2% | [87.9%, 93.5%] | Borderline |
+| **ARC Challenge** (science) | 70.7% | **96.3%** | **+25.6%** | [93.5%, 97.9%] | Yes |
+| **TruthfulQA** (factual) | 65.7% | **69.0%** | +3.3% | [63.6%, 74.0%] | No |
+| **MMLU-Pro** (knowledge) | **70.3%** | 59.7% | -10.6% | [54.1%, 65.1%] | Yes |
+| **HellaSwag** (commonsense) | 69.0% | **73.3%** | +4.3% | [68.0%, 78.0%] | No |
+**Key findings:**
+- Massive improvement on ARC Challenge (+25.6%), likely due to improved strategic reasoning from RL training
+- Regression on MMLU-Pro (-10.6%) and GSM8K (-5.2%), consistent with RL fine-tuning trading broad knowledge for task-specific reasoning
+- TruthfulQA and HellaSwag differences are within noise
+## Training Details
+| Parameter | Value |
+|-----------|-------|
 | **Method** | GRPO (TRL GRPOTrainer) |
+| **Base model** | Qwen/Qwen3-8B |
+| **Training steps** | 200 (800 games) |
+| **Learning rate** | 1e-6 |
+| **Training setup** | Self-play vs algorithmic bot (GinRummyBot) |
+| **Win rate achieved** | 16.8% |
+| **Hardware** | Together AI 8x H100 80GB |
+| **Training time** | 11.9 minutes |
+### LoRA Configuration
+```
+r = 16
+lora_alpha = 32
+target_modules = [q_proj, k_proj, v_proj, o_proj]
+task_type = CAUSAL_LM
+```
+### Training Hyperparameters
+- `enable_thinking=False` (no reasoning tokens)
+- `num_generations=4`
+- Sparse terminal reward only (+1/-1/0)
+- Minimal prompt (no tool use)
 ## Usage
+### With PEFT (direct loading)
 ```python
 from peft import PeftModel
+from transformers import AutoModelForCausalLM, AutoTokenizer
+import torch
+base_model = AutoModelForCausalLM.from_pretrained(
+    "Qwen/Qwen3-8B",
+    torch_dtype=torch.bfloat16,
+    device_map="auto",
+)
+model = PeftModel.from_pretrained(base_model, "GoodStartLabs/ginrummy-qwen3-8b-grpo-lora")
+tokenizer = AutoTokenizer.from_pretrained("Qwen/Qwen3-8B")
+```
+### With vLLM (serving)
+```bash
+vllm serve Qwen/Qwen3-8B \
+  --enable-lora \
+  --max-lora-rank 16 \
+  --lora-modules ginrummy=GoodStartLabs/ginrummy-qwen3-8b-grpo-lora \
+  --max-model-len 4096 \
+  --enforce-eager \
+  --port 8000
+```
+Then query via OpenAI-compatible API:
+```python
+from openai import OpenAI
+client = OpenAI(base_url="http://localhost:8000/v1", api_key="unused")
+response = client.chat.completions.create(
+    model="ginrummy",
+    messages=[{"role": "user", "content": "Your prompt here"}],
+    max_tokens=512,
+)
+```
+### Merge into base model (standalone)
+```python
+from peft import PeftModel
+from transformers import AutoModelForCausalLM, AutoTokenizer
+import torch
+base = AutoModelForCausalLM.from_pretrained("Qwen/Qwen3-8B", torch_dtype=torch.bfloat16, device_map="auto")
 model = PeftModel.from_pretrained(base, "GoodStartLabs/ginrummy-qwen3-8b-grpo-lora")
+model = model.merge_and_unload()
+model.save_pretrained("./qwen3-8b-ginrummy-merged")
+AutoTokenizer.from_pretrained("Qwen/Qwen3-8B").save_pretrained("./qwen3-8b-ginrummy-merged")
 ```
+## Limitations
+- This is a **baseline run** (run 9 of iteration series) with no reasoning tokens or tool use
+- Win rate of 16.8% indicates early-stage training; further iterations expected
+- See the [experiment log](https://github.com/GoodStartLabs/GinRummy_RLenv/blob/main/grpo_training/experiment_log.md) for the full iteration history
+## Eval Methodology
+Evaluations run using [Inspect AI](https://inspect.ai-safety-institute.org.uk/) (v0.3.x). Fine-tuned model served via vLLM 0.18.0 on A100-80GB. Base model accessed via OpenRouter. Full results with Wilson CIs available at [GoodStartLabs/huggingface-evals](https://github.com/GoodStartLabs/huggingface-evals).