---
license: apache-2.0
language:
  - en
library_name: transformers
tags:
  - reward-model
  - modernbert
  - text-ranking
  - style-scoring
  - faithfulness
  - literary-style
base_model: answerdotai/ModernBERT-base
---

# OracleRM ModernBERT Base v2

OracleRM ModernBERT Base v2 is a lightweight reward model for ranking written text. It scores a candidate using two heads:

- `style`: how strongly the response matches the target literary/stylistic preference.
- `faith`: how well the response preserves the meaning of the source prompt.

The model is built on top of `answerdotai/ModernBERT-base` with two scalar classification heads. It is intended for reranking multiple candidate rewrites, not for text generation.

## Files expected in this repo

This repo should contain the exported files from the training zip:

```text
config.json
metadata.json
model_state_dict.pt
tokenizer.json / tokenizer files
special_tokens_map.json
tokenizer_config.json
vocab or tokenizer model files, depending on tokenizer export
````

The model is not saved as a standard `AutoModelForSequenceClassification` checkpoint. Load it by reconstructing the wrapper class, then loading `model_state_dict.pt`.

## Input format

The model was trained with this text format:

```text
### Source:
{prompt}

### Rewrite:
{response}
```

Use an empty source when scoring style only:

```text
### Source:


### Rewrite:
{response}
```

## Raw scores

The model returns two sigmoid scores:

```python
style = sigmoid(style_logit)
faith = sigmoid(faith_logit)
```

A simple score is:

```python
score = style * faith
```

For practical reranking, a softer faith-weighted score often works better:

```python
score = style * (faith ** 0.65)
```

You may also apply a control penalty for candidates that are too short, too verbose, or unnecessarily hard to read.

## Example usage

```python
import json
from pathlib import Path

import torch
import torch.nn as nn
from huggingface_hub import snapshot_download
from transformers import AutoConfig, AutoTokenizer, AutoModel

HF_REPO_ID = "YOUR_USERNAME/oracle-rm-modernbert-base-v2"

local_dir = Path(snapshot_download(HF_REPO_ID))

device = "cuda" if torch.cuda.is_available() else "cpu"

metadata = json.loads((local_dir / "metadata.json").read_text())
MAX_LENGTH = int(metadata.get("max_length", 1024))

tokenizer = AutoTokenizer.from_pretrained(local_dir)
config = AutoConfig.from_pretrained(local_dir)

class TwoHeadRM(nn.Module):
    def __init__(self, config):
        super().__init__()
        try:
            self.backbone = AutoModel.from_config(config, attn_implementation="sdpa")
        except TypeError:
            self.backbone = AutoModel.from_config(config)

        hidden = self.backbone.config.hidden_size
        self.dropout = nn.Dropout(0.1)
        self.style_head = nn.Linear(hidden, 1)
        self.faith_head = nn.Linear(hidden, 1)

    def forward(self, input_ids, attention_mask):
        out = self.backbone(input_ids=input_ids, attention_mask=attention_mask)
        pooled = out.last_hidden_state[:, 0]
        pooled = self.dropout(pooled)
        style_logit = self.style_head(pooled).squeeze(-1)
        faith_logit = self.faith_head(pooled).squeeze(-1)
        return style_logit, faith_logit

def safe_torch_load(path):
    try:
        return torch.load(path, map_location="cpu", weights_only=True)
    except TypeError:
        return torch.load(path, map_location="cpu")

model = TwoHeadRM(config)
model.load_state_dict(safe_torch_load(local_dir / "model_state_dict.pt"), strict=True)
model.to(device)
model.eval()

def format_input(prompt, response):
    return f"### Source:\n{prompt}\n\n### Rewrite:\n{response}"

@torch.inference_mode()
def score_one(prompt, response):
    text = format_input(prompt, response)

    enc = tokenizer(
        text,
        max_length=MAX_LENGTH,
        padding=True,
        truncation=True,
        return_tensors="pt",
    ).to(device)

    style_logit, faith_logit = model(enc["input_ids"], enc["attention_mask"])

    style = torch.sigmoid(style_logit.float()).item()
    faith = torch.sigmoid(faith_logit.float()).item()

    return {
        "style": style,
        "faith": faith,
        "score": style * (faith ** 0.65),
    }

prompt = "The room was silent and time seemed to move slowly."

candidate = "In that room, time sank to the bottom like sediment."

print(score_one(prompt, candidate))
```

## Intended use

Use this model to:

* rerank candidate rewrites;
* select more literary or stylistically strong generations;
* filter outputs that drift too far from the source meaning;
* compare rewrite candidates during dataset curation.

This model is best used as a reranker after a generator has already produced several candidates.

## Not intended for

This model should not be used as a general factuality verifier, safety classifier, plagiarism detector, or universal writing-quality metric. It reflects the style preferences and contrastive examples used during training.

## Scoring guidance

For rewrite ranking, sort candidates by:

```python
adjusted_score = style * (faith ** 0.65) * control_penalty
```

Where `control_penalty` may include:

* prompt-relative length limits;
* verbosity penalty only outside the allowed length band;
* readability penalty for overly dense or hard-to-read prose.

For style-only ranking, use:

```python
style
```

For meaning preservation checks, inspect:

```python
faith
```

Do not rely only on the final composite score. For debugging, always print `style`, `faith`, and the final adjusted score separately.

## Limitations

* The model may over-reward ornate or elaborate prose if verbosity is not controlled.
* Very short inputs may produce unstable style judgments.
* Faithfulness is a learned similarity/preference signal, not a factual guarantee.
* The model was trained for English prose-style rewriting and may not transfer well to other languages or technical domains.
* Scores are relative and are most useful when comparing multiple candidates for the same prompt.

## Suggested implementation pattern

Generate several rewrite candidates, score each candidate with this RM, then choose the highest adjusted score.

```python
candidates = [
    "The room was quiet and time felt slow.",
    "In that room, time sank to the bottom like sediment.",
]

rows = []

for response in candidates:
    scores = score_one(prompt, response)
    rows.append((scores["score"], scores["style"], scores["faith"], response))

rows.sort(reverse=True)

for score, style, faith, response in rows:
    print(f"score={score:.3f} style={style:.3f} faith={faith:.3f} | {response}")
```