pratimassaravanan
/

clinical-qwen3-4b-sft-lora

@@ -1,210 +1,112 @@
 ---
-base_model: unsloth/Qwen3-4B-unsloth-bnb-4bit
 library_name: peft
 pipeline_tag: text-generation
 tags:
-- base_model:adapter:unsloth/Qwen3-4B-unsloth-bnb-4bit
 - lora
-- sft
-- transformers
-- trl
-- unsloth
 ---
-# Model Card for Model ID
-<!-- Provide a quick summary of what the model is/does. -->
-## Model Details
-### Model Description
-<!-- Provide a longer summary of what this model is. -->
-- **Developed by:** [More Information Needed]
-- **Funded by [optional]:** [More Information Needed]
-- **Shared by [optional]:** [More Information Needed]
-- **Model type:** [More Information Needed]
-- **Language(s) (NLP):** [More Information Needed]
-- **License:** [More Information Needed]
-- **Finetuned from model [optional]:** [More Information Needed]
-### Model Sources [optional]
-<!-- Provide the basic links for the model. -->
-- **Repository:** [More Information Needed]
-- **Paper [optional]:** [More Information Needed]
-- **Demo [optional]:** [More Information Needed]
-## Uses
-<!-- Address questions around how the model is intended to be used, including the foreseeable users of the model and those affected by the model. -->
-### Direct Use
-<!-- This section is for the model use without fine-tuning or plugging into a larger ecosystem/app. -->
-[More Information Needed]
-### Downstream Use [optional]
-<!-- This section is for the model use when fine-tuned for a task, or when plugged into a larger ecosystem/app -->
-[More Information Needed]
-### Out-of-Scope Use
-<!-- This section addresses misuse, malicious use, and uses that the model will not work well for. -->
-[More Information Needed]
-## Bias, Risks, and Limitations
-<!-- This section is meant to convey both technical and sociotechnical limitations. -->
-[More Information Needed]
-### Recommendations
-<!-- This section is meant to convey recommendations with respect to the bias, risk, and technical limitations. -->
-Users (both direct and downstream) should be made aware of the risks, biases and limitations of the model. More information needed for further recommendations.
-## How to Get Started with the Model
-Use the code below to get started with the model.
-[More Information Needed]
-## Training Details
-### Training Data
-<!-- This should link to a Dataset Card, perhaps with a short stub of information on what the training data is all about as well as documentation related to data pre-processing or additional filtering. -->
-[More Information Needed]
-### Training Procedure
-<!-- This relates heavily to the Technical Specifications. Content here should link to that section when it is relevant to the training procedure. -->
-#### Preprocessing [optional]
-[More Information Needed]
-#### Training Hyperparameters
-- **Training regime:** [More Information Needed] <!--fp32, fp16 mixed precision, bf16 mixed precision, bf16 non-mixed precision, fp16 non-mixed precision, fp8 mixed precision -->
-#### Speeds, Sizes, Times [optional]
-<!-- This section provides information about throughput, start/end time, checkpoint size if relevant, etc. -->
-[More Information Needed]
-## Evaluation
-<!-- This section describes the evaluation protocols and provides the results. -->
-### Testing Data, Factors & Metrics
-#### Testing Data
-<!-- This should link to a Dataset Card if possible. -->
-[More Information Needed]
-#### Factors
-<!-- These are the things the evaluation is disaggregating by, e.g., subpopulations or domains. -->
-[More Information Needed]
-#### Metrics
-<!-- These are the evaluation metrics being used, ideally with a description of why. -->
-[More Information Needed]
-### Results
-[More Information Needed]
-#### Summary
-## Model Examination [optional]
-<!-- Relevant interpretability work for the model goes here -->
-[More Information Needed]
-## Environmental Impact
-<!-- Total emissions (in grams of CO2eq) and additional considerations, such as electricity usage, go here. Edit the suggested text below accordingly -->
-Carbon emissions can be estimated using the [Machine Learning Impact calculator](https://mlco2.github.io/impact#compute) presented in [Lacoste et al. (2019)](https://arxiv.org/abs/1910.09700).
-- **Hardware Type:** [More Information Needed]
-- **Hours used:** [More Information Needed]
-- **Cloud Provider:** [More Information Needed]
-- **Compute Region:** [More Information Needed]
-- **Carbon Emitted:** [More Information Needed]
-## Technical Specifications [optional]
-### Model Architecture and Objective
-[More Information Needed]
-### Compute Infrastructure
-[More Information Needed]
-#### Hardware
-[More Information Needed]
-#### Software
-[More Information Needed]
-## Citation [optional]
-<!-- If there is a paper or blog post introducing the model, the APA and Bibtex information for that should go in this section. -->
-**BibTeX:**
-[More Information Needed]
-**APA:**
-[More Information Needed]
-## Glossary [optional]
-<!-- If relevant, include terms and calculations in this section that can help readers understand the model or model card. -->
-[More Information Needed]
-## More Information [optional]
-[More Information Needed]
-## Model Card Authors [optional]
-[More Information Needed]
-## Model Card Contact
-[More Information Needed]
-### Framework versions
-- PEFT 0.19.1

 ---
+base_model: Qwen/Qwen3-4B
 library_name: peft
 pipeline_tag: text-generation
 tags:
+- clinical-trial
+- reinforcement-learning
+- REINFORCE
 - lora
+- qwen3
+- openenv
+- hackathon
+- theme-2
+license: mit
 ---
+# Clinical Trial Recruitment Agent — Qwen3-4B + RL
+A clinical trial recruitment agent trained with **REINFORCE** on a 180-step long-horizon environment. Theme 2 submission for the OpenEnv Hackathon.
+## Overview
+| Component | Detail |
+|---|---|
+| **Base Model** | Qwen/Qwen3-4B (4-bit quantized) |
+| **Method** | Fresh LoRA (r=16, alpha=32) + REINFORCE policy gradient |
+| **Environment** | Clinical Recruitment Env (8 action types, 37 observation features, 180-step horizon) |
+| **Training GPU** | NVIDIA L40S (48GB) on Lightning AI |
+| **HF Space** | [pratimassaravanan/clinical-recruitment](https://huggingface.co/spaces/pratimassaravanan/clinical-recruitment) |
+## Training Pipeline
+### Phase 1: SFT (Supervised Fine-Tuning)
+- **Data**: 2,000 heuristic-generated traces (observation → JSON action pairs)
+- **Result**: 100% JSON parse rate but **complete policy collapse** — model only outputs `adjust_strategy`
+- **Loss**: 3.14 → 0.015 (perfect memorization, zero generalization)
+- **Finding**: SFT teaches format but not observation-conditional behavior
+### Phase 2: REINFORCE (Policy Gradient RL)
+- **Key fix**: Observation parsing bug (`result.get("observation")` → `result` — API returns flat dict)
+- **Key fix**: Heuristic override prevents degenerate actions when candidates exist
+- **Reward shaping**: +0.20 for allocate, +0.15 for recontact, +0.10 for screen, -0.10 for adjust_strategy when productive actions available
+**Debug Trial Results (3 episodes, 15 steps each):**
+| Episode | Task | Enrolled | Target | Reward | Action Distribution |
+|---|---|---|---|---|---|
+| 0 | easy_bench | **5** | 80 | 6.61 | screen=6, allocate=9 |
+| 1 | easy_bench | **5** | 80 | 6.61 | screen=6, allocate=9 |
+| 2 | easy_bench | **1** | 80 | 7.41 | screen=14, allocate=1 |
+**Improvement over SFT**: 0 → 5 enrolled patients, diverse action distribution vs. 100% adjust_strategy collapse.
+### Phase 3: Full REINFORCE Run (30 episodes)
+- 30 episodes across easy/medium/hard benchmarks
+- Results uploading upon completion
+## Key Findings
+1. **SFT collapse is fundamental**: Two independent SFT runs (2K and 16K traces) both produced identical policy collapse. More data doesn't help — the model memorizes the most common action without learning observation-conditional behavior.
+2. **Observation parsing was the root cause of 0 enrollment**: The HF Space API returns a flat dict (not nested under `observation`), causing all candidate lists to appear empty.
+3. **Heuristic override + RL reward shaping enables real enrollment**: The combination of a smart fallback (allocate > recontact > screen > adjust) with REINFORCE reward shaping produces agents that actually progress through the recruitment funnel.
+4. **Fresh LoRA outperforms SFT LoRA**: Starting from base Qwen3-4B with a fresh LoRA adapter (instead of loading the collapsed SFT adapter) allows the model to explore and learn productive actions.
+## Repository Structure
+```
+clinical-recruitment-env/
+├── env.py                  # 180-step clinical trial environment
+├── models.py               # Action/Observation Pydantic models
+├── graders.py              # Task-specific scoring (easy/medium/hard)
+├── app.py                  # FastAPI server
+├── openenv_adapter.py      # OpenEnv protocol adapter
+├── train.py                # SFT training script (local, uses Python API)
+├── _lightning_reinforce.py  # REINFORCE v3 (Lightning AI, uses HTTP API)
+├── _reinforce_v4.py         # REINFORCE v4 (gentler reward shaping)
+└── _debug_trial.py          # Debug trial (3 episodes, full logging)
+```
+## Usage
+```python
+from peft import PeftModel
+from transformers import AutoModelForCausalLM, AutoTokenizer
+base = AutoModelForCausalLM.from_pretrained("Qwen/Qwen3-4B", load_in_4bit=True)
+model = PeftModel.from_pretrained(base, "pratimassaravanan/clinical-qwen3-4b-sft-lora/rl_v3_adapter")
+tokenizer = AutoTokenizer.from_pretrained("Qwen/Qwen3-4B")
+```
+## Environment Details
+- **Actions**: screen_patient, recontact, allocate_to_site, adjust_strategy, plan_next_phase, summarize_and_index, retrieve_relevant_history, stop_recruitment
+- **Observation**: 37 features including patient lists, site performance, funnel metrics, world_type
+- **Reward**: Enrollment (+0.50), screening (+0.30), dropout (-0.35), milestone bonuses, hypothesis accuracy (+0.10)
+- **Tasks**: easy_bench (80 target), medium_bench (100 target), hard_bench (150 target)
+## Compute & Cost
+| Job | GPU | Duration | Cost |
+|---|---|---|---|
+| SFT v1 (2K traces) | L40S | ~30 min | ~$2.50 |
+| SFT v2 (16K traces) | L40S | ~60 min | ~$5.00 |
+| RL debug trial | L40S | ~5 min | $0.44 |
+| RL v3-fixed (30 eps) | L40S | ~60 min | ~$5.00 |
+## License
+MIT