---
license: apache-2.0
language:
- en
library_name: transformers
pipeline_tag: image-text-to-text
base_model:
- Qwen/Qwen3.5-9B
tags:
- agent
- tool-use
- medical
- healthcare
- reinforcement-learning
- distillation
- qwen3_5
---

# TT-OPD-Qwen3.5-9B

**TT-OPD-Qwen3.5-9B** is a Qwen3.5-9B checkpoint post-trained with **multi-turn agentic
reinforcement learning on medical tasks**, using turn-level on-policy self-distillation
(**TT-OPD**) on top of GRPO.

This card documents **only what we changed** — the training direction, the objective, and the
behaviour we measured. Everything about the underlying architecture, tokenizer, multilingual
coverage and general capability comes unchanged from
[Qwen/Qwen3.5-9B](https://huggingface.co/Qwen/Qwen3.5-9B); read that card for the base model's
properties and benchmarks.

---

## What it was trained to do

The model is trained as a **medical agent**, not as a question answerer:

- Each episode is a clinical or biomedical task solved over **multiple tool-use turns** against
  a medical tool environment (literature/evidence search, knowledge lookup, structured record
  queries) with a final `submit_answer` call.
- Reward is on the **outcome of the episode**, so the policy is optimised for *reaching a
  committed answer through the tool loop*, not for producing fluent prose.

The single clearest behavioural change is **answer commitment**. On the same harness, the rate
of episodes that end without ever producing an answer drops sharply:

| no-answer rate ↓ | base | **this model** |
|---|---|---|
| MedQA | 9.30% | **1.15%** |
| MMLU-Med | 6.73% | **0.73%** |

## Objective

TT-OPD = **turn-level on-policy distillation**, added to GRPO. As configured in this run:

- **Gradient-free EMA teacher** — the teacher is an exponential moving average of the student
  itself, so there is no external teacher model and no separate teacher training.
- **Bidirectional KL** between student and EMA teacher, applied with a sign flip so that correct
  trajectories are stabilised and incorrect ones are actively pushed away.
- **Turn-level span** — the KL covers the **whole trajectory (all turns)**, not just the final
  response, which is what makes the regulariser act on the agent loop rather than on one answer.
- **Top-K position filtering** — the distillation term is applied only at the highest-signal
  token positions.
- **Cosine length shaping** on the reward.

Released weights are **step 660**, the end of the run.

## Measured behaviour

Agentic multi-turn evaluation on our own harness (5-turn budget, tools + knowledge base),
**3 independent runs per cell**, mean ± sd. The base row is the same Qwen3.5-9B run through the
identical harness, so this is a controlled comparison, not a comparison against published
single-turn numbers.

| Benchmark | n | base | GRPO | **TT-OPD (this model)** |
|---|---|---|---|---|
| MedQA (USMLE) | 1273 | 80.96 ± 0.39 | 83.69 ± 0.24 | **83.97 ± 0.14** |
| MMLU-Med (6 subtypes) | 1089 | 85.89 ± 0.47 | 85.83 ± 0.75 | **86.69 ± 0.24** |

**Read this honestly:**
- **MedQA +3.0 pp over base** is the one solid gain, and most of it is the answer-commitment
  effect above rather than better per-answer reasoning.
- **MMLU-Med +0.8 pp** is inside ~1.5 sd — treat it as "no regression", not as a gain. It is
  worth stating plainly that this checkpoint shows **no catastrophic forgetting** of parametric
  medical knowledge on this harness.

### Degeneration checks

Measured over 1,273 episodes / 6,334 generated turns, against the base model on the same harness:

| | base | **this model** |
|---|---|---|
| episodes with a repeated identical turn | 2.7% | **1.3%** |
| distinct turns per episode (of 5) | 4.93 | **4.96** |
| output length p50 / p95 / max (chars) | 206 / 1796 / 6587 | **187 / 1282 / 5966** |
| answer-option distribution (MedQA A/B/C/D) | 25.9/25.7/27.4/21.0 | 25.2/25.6/28.2/21.0 |

No repetition collapse, no length runaway, no answer-option collapse, and no tool-call format
degradation (99.5% of turns emit a well-formed tool call).

## Known limitations

- **It is a tool-using agent.** It was trained and evaluated entirely inside a multi-turn tool
  loop that ends in a `submit_answer` call. We have **not** measured it as a plain single-turn
  chat model without tools, and we would not expect the numbers above to carry over to that
  setting. Serve it with its tool loop.
- **Short answers on open-ended questions.** On free-text medical questions the median answer
  this model submits is about 55 characters, against about 196 for the base model on the same
  prompts. The training signal rewards committing an answer, and the model has specialised
  toward short, submittable answers. **Do not use it for long-form patient-facing explanation.**
- **Retrieval is not the source of the gain.** The gain here should be read as agent-loop
  behaviour (commitment, format, turn use), not as evidence of learned retrieval-augmented
  reasoning.
- **English, medical/biomedical domain only.** Behaviour outside that distribution is the base
  model's.
- **Not a clinical tool.** This is a research checkpoint. It must not be used for diagnosis,
  treatment decisions, or any patient-facing purpose.

## Usage

```python
from transformers import AutoModelForCausalLM, AutoProcessor

model = AutoModelForCausalLM.from_pretrained(
    "Minbyul/TT-OPD-Qwen3.5-9B",
    torch_dtype="bfloat16", device_map="auto", trust_remote_code=True,
)
processor = AutoProcessor.from_pretrained(
    "Minbyul/TT-OPD-Qwen3.5-9B", trust_remote_code=True,
)
```

Serving (SGLang, matching how it was evaluated):

```bash
python3 -m sglang.launch_server \
  --model-path Minbyul/TT-OPD-Qwen3.5-9B \
  --served-model-name ttopd-9b \
  --host 0.0.0.0 --port 30000 --tp-size 1 \
  --context-length 131072 --mem-fraction-static 0.85 \
  --reasoning-parser qwen3 --tool-call-parser qwen3_coder \
  --trust-remote-code
```

The checkpoint keeps the base model's **vision tower** (`model.visual.*`, 333 tensors), so the
composite `Qwen3_5ForConditionalGeneration` config loads as-is. RL training was text-only; the
vision path is inherited from the base and was not trained here.

## Citation

```bibtex
@misc{ttopd_qwen35_9b,
  title  = {TT-OPD-Qwen3.5-9B: turn-level on-policy distillation for multi-turn medical agents},
  author = {Jeong, Minbyul},
  year   = {2026},
  howpublished = {\url{https://huggingface.co/Minbyul/TT-OPD-Qwen3.5-9B}}
}
```

## License

Apache-2.0, inherited from the base model (see `LICENSE`).