Add DRM-Multi-8B and DRM-Pref-8B checkpoints

Adds RewardDiT release documentation, configs, and the DRM-Multi-8B / DRM-Pref-8B checkpoint files. Paper and code links are left as TODO placeholders for the release update.

DRM-Multi-8B/config.json

@@ -0,0 +1,28 @@
+{
+  "model_name": "DRM-Multi-8B",
+  "model_type": "RewardDiT",
+  "base_encoder": "sfairXC/FsfairX-LLaMA3-RM-v0.1",
+  "checkpoint_file": "model.pth",
+  "reward_dim": 19,
+  "text_emb_dim": 4096,
+  "hidden_size": 384,
+  "depth": 3,
+  "num_heads": 6,
+  "mlp_ratio": 4.0,
+  "dropout": 0.2,
+  "class_dropout_prob": 0.15,
+  "num_train_timesteps": 1000,
+  "beta_schedule": "squaredcos_cap_v2",
+  "prediction_type": "epsilon",
+  "training_data": "RLHFlow/ArmoRM-Multi-Objective-Data-v0.1",
+  "recommended_inference": {
+    "mask_split": false,
+    "num_steps": 10,
+    "guidance_scale": 7.0,
+    "num_samples": 32,
+    "gate": false,
+    "debias": false
+  },
+  "paper": "TODO: paste arXiv or project page link here",
+  "code": "TODO: paste GitHub repository link here"
+}

DRM-Multi-8B/model.pth

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:2a1554bca6662f2e37d813ccf1c11d07325fa890cf247843ffc78dceecd49485
+size 47966754

DRM-Pref-8B/config.json

@@ -0,0 +1,33 @@
+{
+  "model_name": "DRM-Pref-8B",
+  "model_type": "RewardDiT",
+  "base_encoder": "sfairXC/FsfairX-LLaMA3-RM-v0.1",
+  "checkpoint_file": "model.pth",
+  "reward_dim": 1,
+  "text_emb_dim": 4096,
+  "hidden_size": 384,
+  "depth": 3,
+  "num_heads": 6,
+  "mlp_ratio": 4.0,
+  "dropout": 0.2,
+  "class_dropout_prob": 0.15,
+  "num_train_timesteps": 1000,
+  "beta_schedule": "squaredcos_cap_v2",
+  "prediction_type": "epsilon",
+  "training_data": "allenai/llama-3.1-tulu-3-8b-preference-mixture",
+  "loss": {
+    "denoise_loss_weight": 1.0,
+    "bt_alpha": 0.5,
+    "reward_reg_weight": 0.001
+  },
+  "recommended_inference": {
+    "mask_split": false,
+    "num_steps": 10,
+    "guidance_scale": 7.0,
+    "num_samples": 32,
+    "gate": false,
+    "debias": false
+  },
+  "paper": "TODO: paste arXiv or project page link here",
+  "code": "TODO: paste GitHub repository link here"
+}

DRM-Pref-8B/model.pth

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:a30ff191bd14fe1cd6d40a93f09217b883911a03c1c4bbd3798ea082856e7de8
+size 143655819

README.md

@@ -1,3 +1,109 @@
 ---
-license: cc-by-nc-4.0
+license: apache-2.0
+library_name: pytorch
+tags:
+  - reward-model
+  - diffusion
+  - rewarddit
+  - preference-modeling
+  - text-generation-evaluation
+base_model:
+  - sfairXC/FsfairX-LLaMA3-RM-v0.1
 ---
+
+# DRM Checkpoints
+
+This repository hosts two RewardDiT checkpoints released with our paper.
+
+Paper: TODO: paste arXiv or project page link here.
+
+Code: TODO: paste GitHub repository link here.
+
+## Checkpoints
+
+| Name | File | Reward Dim | Training Data | Intended Use |
+| --- | --- | ---: | --- | --- |
+| DRM-Multi-8B | `DRM-Multi-8B/model.pth` | 19 | `RLHFlow/ArmoRM-Multi-Objective-Data-v0.1` | Multi-objective reward generation and scalar scoring by mean aggregation |
+| DRM-Pref-8B | `DRM-Pref-8B/model.pth` | 1 | `allenai/llama-3.1-tulu-3-8b-preference-mixture` | Pair-preference reward scoring |
+
+Both checkpoints use FsfairX-LLaMA3-RM-v0.1 embeddings as the text condition.
+
+## Recommended Inference Settings
+
+Use the following settings for both checkpoints:
+
+```text
+mask_split = false
+num_steps = 10
+guidance_scale = 7.0
+num_samples = 32
+gate = off
+debias = off
+```
+
+The public `ScoreGenerator` does not include gate or debias modules. Scores are computed by sampling RewardDiT rewards, averaging over samples, and then averaging over reward dimensions.
+
+## Usage
+
+After downloading the code repository and this model repository, run:
+
+```bash
+python src_upload/score_generator.py \
+  --ckpt path/to/DRM-Multi-8B/model.pth \
+  --prompt "User prompt" \
+  --response "Assistant response"
+```
+
+For the preference checkpoint:
+
+```bash
+python src_upload/score_generator.py \
+  --ckpt path/to/DRM-Pref-8B/model.pth \
+  --prompt "User prompt" \
+  --response "Assistant response"
+```
+
+The default command-line inference settings in `score_generator.py` are already set to `num_steps=10`, `guidance_scale=7.0`, and `num_samples=32`.
+
+## Model Details
+
+### DRM-Multi-8B
+
+- Reward dimension: 19
+- Text embedding dimension: 4096
+- Hidden size: 384
+- Depth: 3
+- Attention heads: 6
+- Dropout: 0.2
+- Beta schedule: `squaredcos_cap_v2`
+- Prediction type: `epsilon`
+
+### DRM-Pref-8B
+
+- Reward dimension: 1
+- Text embedding dimension: 4096
+- Hidden size: 384
+- Depth: 3
+- Attention heads: 6
+- Dropout: 0.2
+- Beta schedule: `squaredcos_cap_v2`
+- Prediction type: `epsilon`
+- Pair loss: denoising loss + Bradley-Terry loss + reward L2 regularization
+- `reward_reg_weight`: 0.001
+
+## Citation
+
+TODO: paste BibTeX citation here after the paper is available.
+
+```bibtex
+@article{TODO,
+  title = {TODO},
+  author = {TODO},
+  journal = {arXiv preprint},
+  year = {TODO}
+}
+```
+
+## Limitations
+
+These checkpoints are research artifacts intended for reward-modeling experiments. They inherit the limitations of their training data and text encoder, and should not be treated as calibrated absolute measures of response quality.

USAGE.md

@@ -0,0 +1,29 @@
+# Usage
+
+Download the checkpoints from this repository and use them with the released code.
+
+## DRM-Multi-8B
+
+```bash
+python src_upload/score_generator.py \
+  --ckpt path/to/DRM-Multi-8B/model.pth \
+  --prompt "User prompt" \
+  --response "Assistant response" \
+  --num_steps 10 \
+  --guidance_scale 7.0 \
+  --num_samples 32
+```
+
+## DRM-Pref-8B
+
+```bash
+python src_upload/score_generator.py \
+  --ckpt path/to/DRM-Pref-8B/model.pth \
+  --prompt "User prompt" \
+  --response "Assistant response" \
+  --num_steps 10 \
+  --guidance_scale 7.0 \
+  --num_samples 32
+```
+
+The scorer uses `mask_split=False`, gate off, and debias off.