Qwen3.5 9B Negotiation SDPO+GRPO — 2-iteration smoke

This repository contains the final checkpoint and metrics from a 2-iteration production-shape smoke test of buyer-only negotiation RLVR with SDPO token shaping.

Configuration

Buyer model: Qwen/Qwen3.5-9B
Seller/environment model: Qwen/Qwen3.5-9B frozen regulated seller
NUM_ITERS=2, BATCH_SIZE=16, GROUP_SIZE=8, 128 episodes/iteration
MAX_TURNS=6, MAX_NEW_TOKENS=300
Reasoning: option-B native Qwen thinking, NATIVE_THINK_TOKENS=300, NATIVE_FINAL_TOKENS=96
Objective: ref-free on-policy GRPO + SDPO, strict feedback, no frozen reference-policy model
SDPO_LAMBDA=0.9 at iter0 and 0.88 at iter1
Optimizer: CPU-state AdamW, full-parameter update, LR=3e-6, WARMUP_STEPS=10, WEIGHT_DECAY=0.01, GRAD_CLIP_NORM=1.0
Update optimizations enabled: length-bucketed update microbatches and torch.inference_mode() self-teacher forward
Runtime stack: torch 2.6/cu124 on single A100 80GB; Qwen3.5 fast-path wheels were intentionally disabled after ABI/driver failures

Results

Iteration	Mean reward	Deal rate	Buyer format errors	Loss	First-offer ratio	Peak reserved VRAM
0	-0.0744	46.9%	12/128	0.1787	0.820	79.8 GB
1	-0.1196	38.3%	19/128	0.1669	0.706	80.3 GB

The run completed and pushed iter-1, iter-2, and final main. It early-stopped after two consecutive buyer-format-warning iterations.

Runtime analysis

Total runtime was 49.2 minutes. Average per iteration:

Rollout: ~661s
Update: ~595s
Total: ~20.9 min/iteration

Update bottlenecks:

Backward: ~56% of update time
Policy + teacher forwards: ~29%
CPU AdamW optimizer: ~13%
Pretokenize/collate: <1%

Conclusion: the implemented safe optimizations helped the code path but do not change the dominant bottleneck: full-parameter 9B backward plus CPU-state AdamW on a nearly saturated single A100.

Caveats / next run

This is a smoke checkpoint, not a recommended final policy. Format errors rose from 12/128 to 19/128, and reward/deal rate degraded in the second iteration. For the next objective-preserving run, use a more conservative stability config: lower LR (e.g. 1e-6) or longer warmup, slower SDPO handoff / more GRPO-heavy early iterations, and consider shorter native-thinking budgets only after testing whether they preserve reward.

Usage

from transformers import AutoProcessor, AutoModelForImageTextToText

model_id = "ZeterMordio/anchor-negotiation-sdpo-qwen35-2iter-gen96"
processor = AutoProcessor.from_pretrained(model_id, trust_remote_code=True)
model = AutoModelForImageTextToText.from_pretrained(model_id, torch_dtype="auto", device_map="auto")

For text-only negotiation use, call the processor/tokenizer with chat-template messages as done in train_negotiation_sdpo.py included in this repository.

Generated by ML Intern

This model repository was generated by ML Intern, an agent for machine learning research and development on the Hugging Face Hub.

Try ML Intern: https://smolagents-ml-intern.hf.space
Source code: https://github.com/huggingface/ml-intern

Downloads last month: 2

Safetensors

Model size

9B params

Tensor type

BF16

Video Preview

Reinforcement Learning

Model tree for ZeterMordio/anchor-negotiation-sdpo-qwen35-2iter-gen96

Base model

Qwen/Qwen3.5-9B-Base

Finetuned

Qwen/Qwen3.5-9B

Finetuned

(467)

this model