---
license: cc-by-nc-4.0
base_model: stabilityai/stable-diffusion-3-medium-diffusers
library_name: peft
pipeline_tag: image-to-image
language: en
tags:
  - super-resolution
  - image-super-resolution
  - extreme-zoom
  - chain-of-zoom
  - diffusion
  - lora
  - peft
  - privileged-distillation
  - faithfulness
---

# OracleZoom

**Privileged-Latent Distillation for faithful extreme super-resolution.**
A tiny (7.1M-parameter) LoRA adapter that makes Chain-of-Zoom's recursive super-resolution add *faithful* detail instead of hallucinating, all the way to 256x. Shipped both as the LoRA adapter and as a **fully-merged transformer you can download and use directly**.

[![GitHub](https://img.shields.io/badge/Code-OPD--Zoom-black?logo=github)](https://github.com/dipta007/OPD-Zoom)
[![Base](https://img.shields.io/badge/Backbone-OSEDiff%20/%20SD3-blue)](https://huggingface.co/stabilityai/stable-diffusion-3-medium-diffusers)
[![Method](https://img.shields.io/badge/Method-Chain--of--Zoom-orange)](https://github.com/dipta007/OPD-Zoom)
[![Paper](https://img.shields.io/badge/Paper-WACV%202027%20(in%20submission)-red)](https://github.com/dipta007/OPD-Zoom)
[![License](https://img.shields.io/badge/License-CC--BY--NC--4.0-lightgrey)](https://creativecommons.org/licenses/by-nc/4.0/)

## Highlights
- **Faithful, not just sharp.** At extreme zoom the backbone must *invent* detail; the question is whether it is faithful or hallucinated. This model teaches faithfulness.
- **Holds where baselines collapse.** CLIPIQA **0.71 at 256x** while Chain-of-Zoom (CoZ) and five SOTA SR backbones fall to <=0.58; most faithful of all methods at 4x (LPIPS **0.20** vs CoZ 0.22).
- **Judged more faithful.** Two cross-family vision-language judges (InternVL + Gemini) prefer this zoom **68-78%** of the time at 64-256x and flag the strongest baseline hallucinating **2-5x more**.
- **Tiny to train, easy to use.** A rank-16 LoRA (**7.1M** trainable params) trained on only **1,000** curated 4K images; shipped as both the adapter and a merged, drop-in transformer.

## Files in this repo
| File | What it is |
|---|---|
| `merged_transformer.safetensors` | **The complete OracleZoom SR transformer** (SD3 + OSEDiff's SR-LoRA + our PLD adapter, all baked in), fp32, ~8.35 GB. Download-and-use: drop it in as the transformer, no LoRA step. |
| `adapter_model.safetensors` + `adapter_config.json` | The rank-16 PLD LoRA **alone** (~28 MB), if you prefer to apply it onto your own OSEDiff transformer. |
| `train_meta.json` | Training recipe / provenance. |

## Model Overview
| Property | Value |
|---|---|
| Model type | LoRA adapter (PEFT) for a one-step SR backbone, + merged transformer |
| Backbone | OSEDiff on Stable Diffusion 3-medium |
| Prompt extractor (frozen) | Qwen2.5-VL-3B-Instruct |
| Trainable params | 7.1M |
| LoRA | r = 16, alpha = 32, dropout = 0.0 |
| LoRA targets | `to_q, to_k, to_v, add_q_proj, add_k_proj, add_v_proj` (SD3 transformer) |
| Training data | 1,000 curated 4K photographs (supervised at 4x only) |
| Objective | decode-space LPIPS + anchored cycle-consistency - beta_reward * TOPIQ-NR + beta_kl * KL-to-base + EMA |
| Key weights | beta_reward 0.4, beta_kl 8.0, w_cyc 1.0, lambda_ema 0.1 (EMA decay 0.95) |
| Recursion at test | 4 steps (4x / 16x / 64x / 256x), 512x512 center crop |

## Method
Recursive SR (Chain-of-Zoom) reuses a 4x backbone step after step to reach 16x-256x. Each step is **blind**: it sees only a blurred crop of its own previous output and must invent the missing detail, so errors compound and the invention may be hallucinated.

**Privileged-latent distillation (the idea).** A *privileged teacher* is shown the ground-truth high-resolution patch **at training time only** and distills its real detail into the blind student, in **decode space** (a perceptual loss between the student's decoded image and the real patch). Only a small LoRA adapter is trained; the backbone, VAE, and prompter stay frozen.

**A KL leash keeps the deep reward faithful (the safeguard).** Ground truth exists only at 4x. To carry the distilled prior into the deeper recursion, the student chases a differentiable detail reward (TOPIQ-NR) through the real zoom. Left free, such a reward games the metric with a repetitive crosshatch; **leashed** to the deployed backbone by a KL trust region (a latent distance for a one-step map), it sharpens detail without drifting into hallucination.

## Quickstart
Both paths run inside the [OPD-Zoom](https://github.com/dipta007/OPD-Zoom) / Chain-of-Zoom pipeline.

```bash
git clone https://github.com/dipta007/OPD-Zoom && cd OPD-Zoom
huggingface-cli download dipta007/OracleZoom --local-dir ckpt/OracleZoom
```

**Option A - merged transformer (recommended, no LoRA step).** `merged_transformer.safetensors` already contains SD3 + OSEDiff's SR-LoRA + our PLD adapter, so you just load it as the SR transformer's weights:
```python
from safetensors.torch import load_file
sd = load_file("ckpt/OracleZoom/merged_transformer.safetensors")
# `transformer` = the OSEDiff SD3Transformer2DModel built by the pipeline (build_sr)
transformer.load_state_dict(sd, strict=False)
```

**Option B - LoRA adapter.** Apply the rank-16 adapter onto the OSEDiff transformer with PEFT:
```python
from peft import PeftModel
transformer = PeftModel.from_pretrained(transformer, "dipta007/OracleZoom")
```
Or point the repo's inference at the adapter directly:
```bash
python -m opd_zoom.teacher.oracle_infer \
  --mode student --pld_lora ckpt/OracleZoom \
  --gt_dir <your_images> --out <out_dir> --rec_num 4
```
Either way the VLM prompter is unchanged, so per-image inference cost equals Chain-of-Zoom's.

## Results
Under Chain-of-Zoom's exact protocol on a curated 4K benchmark and six test sets (in-domain 4K, DIV8K, DRealSR, RealSR, FFHQ, Flickr2K):

| Axis | Metric | Ours | CoZ / best baseline |
|---|---|---|---|
| Sharpness (no-reference) | CLIPIQA @256x | **0.71** | <= 0.58 |
| Fidelity @4x (ground truth exists) | LPIPS | **0.20** | 0.22 (CoZ) |
| Deep faithfulness (MLLM judge, 64-256x) | preferred vs CoZ | **68-78%** | - |
| Deep faithfulness | hallucination rate vs CoZ | **2-5x lower** | - |

Sharpness is the axis prior methods are built for; the decisive gap is **faithfulness**, verified by full-reference metrics at 4x and by two cross-family MLLM judges plus a blinded human study past 4x.

## Intended Use
- **In-scope:** research on faithful extreme (recursive) super-resolution; as the SR-backbone inside the Chain-of-Zoom recursion on natural photographs.
- **Out-of-scope:** a standalone single-shot SR model (it is the SR backbone for the CoZ loop); forensic/evidentiary use (detail past 4x is generated, not recovered); real-camera-zoom claims (the benchmark uses synthetic center-crop zoom).

## Training
Early-stopped on held-out validation at ~epoch 37 (step 9300); best val 0.216. Trained on one 8xH200 node (single GPU trains the adapter). Full config in `train_meta.json` and the [repo](https://github.com/dipta007/OPD-Zoom).

## Citation
```bibtex
@inproceedings{dipta2027oraclezoom,
  title={OracleZoom: Privileged-Latent Distillation for Faithful Extreme Super-Resolution},
  author={Shubhashis Roy Dipta},
  year={2027},
  note={In submission, WACV 2027},
  url={https://github.com/dipta007/OPD-Zoom}
}
```
Please also cite Chain-of-Zoom and OSEDiff, whose components this builds on.

## License
Released for **research, non-commercial** use (CC-BY-NC-4.0). This model is built on top of OSEDiff / Stable Diffusion 3 and used with a Qwen2.5-VL prompter inside Chain-of-Zoom; the respective upstream licenses apply to those components.