---
tags:
- ml-intern
- reinforcement-learning
- negotiation
- qwen3.5
base_model: Qwen/Qwen3.5-9B
---

# Qwen3.5 9B Negotiation SDPO+GRPO — 2-iteration smoke

This repository contains the final checkpoint and metrics from a 2-iteration production-shape smoke test of buyer-only negotiation RLVR with SDPO token shaping.

- HF Job: https://huggingface.co/jobs/ZeterMordio/6a0fbffbb33ece92698bfe69
- Source/script repo: https://huggingface.co/ZeterMordio/anchor-negotiation-sdpo-qwen35-smoke
- Metrics: https://huggingface.co/ZeterMordio/anchor-negotiation-sdpo-qwen35-2iter-gen96/blob/main/metrics.json

## Configuration

- Buyer model: `Qwen/Qwen3.5-9B`
- Seller/environment model: `Qwen/Qwen3.5-9B` frozen regulated seller
- `NUM_ITERS=2`, `BATCH_SIZE=16`, `GROUP_SIZE=8`, 128 episodes/iteration
- `MAX_TURNS=6`, `MAX_NEW_TOKENS=300`
- Reasoning: option-B native Qwen thinking, `NATIVE_THINK_TOKENS=300`, `NATIVE_FINAL_TOKENS=96`
- Objective: ref-free on-policy GRPO + SDPO, strict feedback, no frozen reference-policy model
- `SDPO_LAMBDA=0.9` at iter0 and `0.88` at iter1
- Optimizer: CPU-state AdamW, full-parameter update, `LR=3e-6`, `WARMUP_STEPS=10`, `WEIGHT_DECAY=0.01`, `GRAD_CLIP_NORM=1.0`
- Update optimizations enabled: length-bucketed update microbatches and `torch.inference_mode()` self-teacher forward
- Runtime stack: torch 2.6/cu124 on single A100 80GB; Qwen3.5 fast-path wheels were intentionally disabled after ABI/driver failures

## Results

| Iteration | Mean reward | Deal rate | Buyer format errors | Loss | First-offer ratio | Peak reserved VRAM |
|---:|---:|---:|---:|---:|---:|---:|
| 0 | -0.0744 | 46.9% | 12/128 | 0.1787 | 0.820 | 79.8 GB |
| 1 | -0.1196 | 38.3% | 19/128 | 0.1669 | 0.706 | 80.3 GB |

The run completed and pushed `iter-1`, `iter-2`, and final `main`. It early-stopped after two consecutive buyer-format-warning iterations.

## Runtime analysis

Total runtime was 49.2 minutes. Average per iteration:

- Rollout: ~661s
- Update: ~595s
- Total: ~20.9 min/iteration

Update bottlenecks:

- Backward: ~56% of update time
- Policy + teacher forwards: ~29%
- CPU AdamW optimizer: ~13%
- Pretokenize/collate: <1%

Conclusion: the implemented safe optimizations helped the code path but do not change the dominant bottleneck: full-parameter 9B backward plus CPU-state AdamW on a nearly saturated single A100.

## Caveats / next run

This is a smoke checkpoint, not a recommended final policy. Format errors rose from 12/128 to 19/128, and reward/deal rate degraded in the second iteration. For the next objective-preserving run, use a more conservative stability config: lower LR (e.g. `1e-6`) or longer warmup, slower SDPO handoff / more GRPO-heavy early iterations, and consider shorter native-thinking budgets only after testing whether they preserve reward.

## Usage

```python
from transformers import AutoProcessor, AutoModelForImageTextToText

model_id = "ZeterMordio/anchor-negotiation-sdpo-qwen35-2iter-gen96"
processor = AutoProcessor.from_pretrained(model_id, trust_remote_code=True)
model = AutoModelForImageTextToText.from_pretrained(model_id, torch_dtype="auto", device_map="auto")
```

For text-only negotiation use, call the processor/tokenizer with chat-template messages as done in `train_negotiation_sdpo.py` included in this repository.

<!-- ml-intern-provenance -->
## Generated by ML Intern

This model repository was generated by [ML Intern](https://github.com/huggingface/ml-intern), an agent for machine learning research and development on the Hugging Face Hub.

- Try ML Intern: https://smolagents-ml-intern.hf.space
- Source code: https://github.com/huggingface/ml-intern