mikebrave/dinov3-vitb-aesthetic-scorer

DINOv3 ViT-B Aesthetic Scorer

A lightweight aesthetic/quality scoring model trained on human image preferences using frozen DINOv2 ViT-B/14 embeddings with a learned MLP head.

Model Description

This model uses a frozen DINOv2 ViT-Base/14 backbone (vit_base_patch16_dinov3.lvd1689m from timm) to extract 768-dimensional normalized visual features, then passes them through a small AestheticHead MLP:

LayerNorm → Linear(768, 256) → GELU → Dropout(0.1) → Linear(256, 1)

The output is a scalar score where higher values indicate higher aesthetic quality.

Training Data

The model was trained on pairwise preference data from two sources:

1. Human preference pairs — Collected via the image-preference-labeler tool. Labelers were shown two images generated from the same prompt (but different models/variations) and asked which they preferred. These are stored in a SQLite database (comparisons + decisions tables organized by case_id/prompt).

2. Pseudo-pairs from scored images — A collection of ~1,100+ curated AI-generated and digital artwork images with Elo-style scores (derived from multiple comparisons). Pseudo-pairs were sampled from this scored set with a minimum score gap of 0.15 to ensure meaningful distinctions.

Training Split

Pairs were split 80/10/10 train/val/test using deterministic hash-based partitioning on case_id/prompt to avoid data leakage.

Dataset Summary

Split	Pairs	Source
Train	17,061	Human preferences + pseudo-pairs
Val	2,182	Human preferences + pseudo-pairs
Test	2,150	Human preferences + pseudo-pairs
Total	21,393

Training Procedure

Backbone extraction (offline)

Images were pre-processed through the frozen DINOv2 ViT-B/14 backbone (timm) to extract L2-normalized 768-dimensional embeddings. This was done once and cached (embeddings_dinov3_timm_vitb.npz) so the head could be trained without keeping the vision backbone in GPU memory.

Head training

• Loss: Bradley-Terry pairwise loss — -log(sigmoid(score_winner - score_loser)), weighted by preference strength
• Optimizer: AdamW (lr=1e-3, weight_decay=1e-4)
• Epochs: 20
• Batch size: 512
• Hidden dim: 256
• Dropout: 0.1
• Best checkpoint selected by: validation accuracy

Performance

Split	Loss	Accuracy	Pairs
Train	0.018	99.5%	17,061
Val	0.157	94.3%	2,182
Test	0.144	94.5%	2,150

Usage

from pathlib import Path
import torch
import torch.nn as nn
from PIL import Image
import timm
from timm.data import create_transform, resolve_model_data_config

# --- AestheticHead definition (matches training) ---
class AestheticHead(nn.Module):
    def __init__(self, input_dim: int = 768, hidden_dim: int = 256, dropout: float = 0.1):
        super().__init__()
        self.net = nn.Sequential(
            nn.LayerNorm(input_dim),
            nn.Linear(input_dim, hidden_dim),
            nn.GELU(),
            nn.Dropout(dropout),
            nn.Linear(hidden_dim, 1),
        )

    def forward(self, features):
        return self.net(features).squeeze(-1)

# --- Load backbone ---
device = "cuda" if torch.cuda.is_available() else "cpu"
backbone = timm.create_model("vit_base_patch16_dinov3.lvd1689m", pretrained=True, num_classes=0)
config = resolve_model_data_config(backbone)
transform = create_transform(**config, is_training=False)
backbone.to(device)
backbone.eval()

# --- Load trained head ---
checkpoint = torch.load("v1_dinov3_vitb.pt", map_location=device)
head = AestheticHead(
    input_dim=checkpoint["input_dim"],
    hidden_dim=checkpoint["hidden_dim"],
    dropout=checkpoint["dropout"],
)
head.load_state_dict(checkpoint["state_dict"])
head.to(device)
head.eval()

# --- Score an image ---
def score_image(image_path: str) -> float:
    image = Image.open(image_path).convert("RGB")
    tensor = transform(image).unsqueeze(0).to(device)
    with torch.inference_mode():
        features = backbone(tensor)
        features = nn.functional.normalize(features.float(), dim=-1)
        score = head(features).item()
    return score

print(score_image("example.png"))  # Higher = more aesthetically pleasing

Checkpoint Contents

The v1_dinov3_vitb.pt checkpoint is a dict with:

Key	Description
input_dim	Backbone embedding dimension (768)
hidden_dim	MLP hidden layer size (256)
dropout	Dropout rate (0.1)
state_dict	AestheticHead weights
metrics	dict with train/val/test loss & accuracy
history	Per-epoch training metrics

Limitations

• The scoring head is trained on a specific distribution of images (AI-generated art, digital artwork). Performance may degrade on out-of-distribution images (photography, memes, medical images, etc.).
• The backbone is frozen and not fine-tuned, so the model cannot adapt to novel visual domains beyond DINOv2's feature space.
• The model outputs an unbounded scalar — not a calibrated probability or percentile. Scores are relative, not absolute.

Acknowledgements

• DINOv2 by Meta Research for the visual backbone
• timm for model loading and transforms
• The image-preference-labeler tool for collecting human preference data