Hugging Face's logo

ZeterMordio
/
anchor-negotiation-sdpo-qwen35-2iter-gen96

Reinforcement Learning
Safetensors
qwen3_5
ml-intern
negotiation
qwen3.5
Model card Files
xet
 Community
anchor-negotiation-sdpo-qwen35-2iter-gen96 / README.md
ZeterMordio's picture
ZeterMordio
Update ML Intern artifact metadata
8272daa verified
preview code
|
Raw
History Blame Contribute Delete
3.65 kB
metadata 
tags:
  - ml-intern
  - reinforcement-learning
  - negotiation
  - qwen3.5
base_model: Qwen/Qwen3.5-9B

Qwen3.5 9B Negotiation SDPO+GRPO — 2-iteration smoke

This repository contains the final checkpoint and metrics from a 2-iteration production-shape smoke test of buyer-only negotiation RLVR with SDPO token shaping.

Configuration

  • Buyer model: Qwen/Qwen3.5-9B
  • Seller/environment model: Qwen/Qwen3.5-9B frozen regulated seller
  • NUM_ITERS=2, BATCH_SIZE=16, GROUP_SIZE=8, 128 episodes/iteration
  • MAX_TURNS=6, MAX_NEW_TOKENS=300
  • Reasoning: option-B native Qwen thinking, NATIVE_THINK_TOKENS=300, NATIVE_FINAL_TOKENS=96
  • Objective: ref-free on-policy GRPO + SDPO, strict feedback, no frozen reference-policy model
  • SDPO_LAMBDA=0.9 at iter0 and 0.88 at iter1
  • Optimizer: CPU-state AdamW, full-parameter update, LR=3e-6, WARMUP_STEPS=10, WEIGHT_DECAY=0.01, GRAD_CLIP_NORM=1.0
  • Update optimizations enabled: length-bucketed update microbatches and torch.inference_mode() self-teacher forward
  • Runtime stack: torch 2.6/cu124 on single A100 80GB; Qwen3.5 fast-path wheels were intentionally disabled after ABI/driver failures

Results

Iteration Mean reward Deal rate Buyer format errors Loss First-offer ratio Peak reserved VRAM
0 -0.0744 46.9% 12/128 0.1787 0.820 79.8 GB
1 -0.1196 38.3% 19/128 0.1669 0.706 80.3 GB

The run completed and pushed iter-1, iter-2, and final main. It early-stopped after two consecutive buyer-format-warning iterations.

Runtime analysis

Total runtime was 49.2 minutes. Average per iteration:

  • Rollout: ~661s
  • Update: ~595s
  • Total: ~20.9 min/iteration

Update bottlenecks:

  • Backward: ~56% of update time
  • Policy + teacher forwards: ~29%
  • CPU AdamW optimizer: ~13%
  • Pretokenize/collate: <1%

Conclusion: the implemented safe optimizations helped the code path but do not change the dominant bottleneck: full-parameter 9B backward plus CPU-state AdamW on a nearly saturated single A100.

Caveats / next run

This is a smoke checkpoint, not a recommended final policy. Format errors rose from 12/128 to 19/128, and reward/deal rate degraded in the second iteration. For the next objective-preserving run, use a more conservative stability config: lower LR (e.g. 1e-6) or longer warmup, slower SDPO handoff / more GRPO-heavy early iterations, and consider shorter native-thinking budgets only after testing whether they preserve reward.

Usage 

from transformers import AutoProcessor, AutoModelForImageTextToText

model_id = "ZeterMordio/anchor-negotiation-sdpo-qwen35-2iter-gen96"
processor = AutoProcessor.from_pretrained(model_id, trust_remote_code=True)
model = AutoModelForImageTextToText.from_pretrained(model_id, torch_dtype="auto", device_map="auto")

For text-only negotiation use, call the processor/tokenizer with chat-template messages as done in train_negotiation_sdpo.py included in this repository.

Generated by ML Intern

This model repository was generated by ML Intern, an agent for machine learning research and development on the Hugging Face Hub.