README.md

---
license: apache-2.0
pipeline_tag: text-to-image
library_name: diffusers
---

# PatchDPO: Patch-level DPO for Finetuning-free Personalized Image Generation

This repository contains the official implementation of the paper "[PatchDPO: Patch-level DPO for Finetuning-free Personalized Image Generation](https://huggingface.co/papers/2412.03177)".

The code and full project details are available on GitHub: [https://github.com/hqhQAQ/PatchDPO](https://github.com/hqhQAQ/PatchDPO)

### Overview

Finetuning-free personalized image generation can synthesize customized images without test-time finetuning, attracting wide research interest owing to its high efficiency. Current finetuning-free methods simply adopt a single training stage with a simple image reconstruction task, and they typically generate low-quality images inconsistent with the reference images during test-time. To mitigate this problem, inspired by the recent DPO (i.e., direct preference optimization) technique, this work proposes an additional training stage to improve the pre-trained personalized generation models. However, traditional DPO only determines the overall superiority or inferiority of two samples, which is not suitable for personalized image generation because the generated images are commonly inconsistent with the reference images only in some local image patches. To tackle this problem, this work proposes **PatchDPO** that estimates the quality of image patches within each generated image and accordingly trains the model. To this end, PatchDPO first leverages the pre-trained vision model with a proposed self-supervised training method to estimate the patch quality. Next, PatchDPO adopts a weighted training approach to train the model with the estimated patch quality, which rewards the image patches with high quality while penalizing the image patches with low quality.

With PatchDPO, our model achieves **state-of-the-art** performance on personalized image generation, with **only 4 hours** of training time on 8 GPUs.

![framework](./assets/framework.svg)

### 🔥🔥🔥 News!!

- 📰 [2024.12.05] Our paper is available at [arXiv](https://arxiv.org/abs/2412.03177).
- 🤗 [2024.12.05] Our model weights are available at [Hugging Face](https://huggingface.co/hqhQAQ/PatchDPO).
- 🚀 [2024.12.05] Training code is available [here](https://github.com/hqhQAQ/PatchDPO).
- 🚀 [2024.12.05] Inference code is available [here](https://github.com/hqhQAQ/PatchDPO).
- 🚀 [2024.12.05] Evaluation code is available [here](https://github.com/hqhQAQ/PatchDPO).
- 💬 [2024.12.05] Our preliminary work **MIP-Adapter** for multi-object personalized generation is available at [MIP-Adapter](https://github.com/hqhQAQ/MIP-Adapter).
- 💬 [2024.12.05] Our preliminary work **MS-Diffusion** for multi-object personalized generation is available at [MS-Diffusion](https://github.com/MS-Diffusion/MS-Diffusion).
- 💬 [2025.02.27] Our paper is accepted by CVPR 2025!

### Performance

#### Quantitative Comparison

We compare PatchDPO with other personalized image generation methods on the famous *DreamBench*.
Three metrics are used for evaluation on this benchmark: DINO, CLIP-I, and CLIP-T.
Note that CLIP-T evaluates the text alignment, and DINO, CLIP-I evaluate the image alignment.

The comparison results are demonstrated in Table 1 & 2 (the results of other methods are from their paper):

<div style="display: flex; justify-content: center;">
   <img src="./assets/dreambench_performance.png" alt="DreamBench Performance" width="45%">
</div>

Detailedly, two evaluation settings are adopted in Table 1 & 2, respectively.

*   **Table 1** uses the original setting following most existing methods. In this setting, DINO, CLIP-I are calculated by comparing the generated image and **all images of the same object**.

*   **Table 2** uses the evaluation setting following [Kosmos-G](https://github.com/xichenpan/Kosmos-G). In this setting, only one image is preserved for each object, and DINO, CLIP-I are calculated by comparing the generated image and **only this image**.

#### Qualitative Comparison

Examples of generated images from PatchDPO are demonstrated below:

![imgs](./assets/qualitative_imgs.png)

### Requirements

The Python packages required for this project are listed below:

```
torch==1.13.1
torchvision==0.14.1
diffusers==0.23.1
einops
modelscope
numpy==1.24.4
oss2
Pillow==10.1.0
PyYAML
safetensors
tqdm
imgviz
transformers==4.35.2
tensorboard
accelerate==0.23.0
opencv-python
openai-clip
setuptools==69.5.1
```

### Dataset

*   **Training dataset.** The release of training dataset is in preparation.

*   **Test dataset.** We evaluate our model on the famous **DreamBench**.

    Prepare this dataset by downloading the `dataset` folder ([DreamBooth dataset](https://github.com/google/dreambooth/tree/main/dataset)), and placing it in the `dreambench` folder of this project.
    Finally, the file structure of the `dreambench` folder is as below:

    ```
    --dreambench
        |--dataset
            |--backpack
            |--backpack_dog
            |--bear_plushie
            |--berry_bowl
            |...
        |--json_data
    ```

### Pre-trained models

*   **Base model.** PatchDPO is based on the SDXL model, which is required for both training and inference. Prepare this model by downloading the pre-trained weights from Hugging Face:

    *   [SG161222/RealVisXL_V1.0](https://huggingface.co/SG161222/RealVisXL_V1.0)

*   **Training.** PatchDPO is trained based on the IP-Adapter-Plus model. Prepare this model by downloading the pre-trained weights from Hugging Face:

    *   [h94/IP-Adapter](https://huggingface.co/h94/IP-Adapter)

*   **Inference and evaluation.** Our trained PatchDPO model can be downloaded from:

    *   [hqhQAQ/PatchDPO](https://huggingface.co/hqhQAQ/PatchDPO)

The path of these models will be `/PATH/TO/RealVisXL_V1.0/`, `/PATH/TO/IP-Adapter/`, and `/PATH/TO/PatchDPO/` respectively.

### Training

Run the following script for training the PatchDPO model based on the IP-Adapter-Plus model using the patchdpo dataset. (note that `/PATH/TO` in `--pretrained_model_name_or_path`, `--image_encoder_path`, `--pretrained_ip_adapter_path`, `--data_root_path`, and `patch_quality_file` should be changed to your own path, and 8 GPUs are used here).

```bash
accelerate launch --num_processes 8 --multi_gpu --mixed_precision "fp16" train_patchdpo.py \
    --pretrained_model_name_or_path /PATH/TO/RealVisXL_V1.0/ \
    --image_encoder_path /PATH/TO/IP-Adapter/models/image_encoder/ \
    --pretrained_ip_adapter_path /PATH/TO/IP-Adapter/sdxl_models/ip-adapter-plus_sdxl_vit-h.bin \
    --data_root_path /PATH/TO/patchdpo_dataset/ \
    --patch_quality_file /PATH/TO/patchdpo_dataset/patch_quality.pkl \
    --mixed_precision fp16 \
    --resolution 512 \
    --train_batch_size 4 \
    --dataloader_num_workers 4 \
    --learning_rate 3e-05 \
    --weight_decay 0.01 \
    --save_steps 10000 \
    --stop_step 30000 \
    --output_dir output/exp1/ \
    --use_dpo_loss True
```

### Inference

Run the following scripts to conduct inference on *DreamBench*.
Note that `/PATH/TO/model.bin` is the path of the trained model to be evaluated.

*   Inference using the **original setting** using 2 GPUs:

    ```bash
    accelerate launch --num_processes 2 --multi_gpu --mixed_precision "fp16" inference_dreambooth.py \
        --base_model_path /PATH/TO/RealVisXL_V1.0/ \
        --image_encoder_path /PATH/TO/IP-Adapter/models/image_encoder/ \
        --ip_ckpt /PATH/TO/PatchDPO/model.bin \
        --data_root dreambench \
        --output_dir output/exp1_eval/ \
        --scale 0.78 \
        --is_kosmosg False
    ```

*   Inference using the **Kosmos-G setting** using 2 GPUs:

    ```bash
    accelerate launch --num_processes 2 --multi_gpu --mixed_precision "fp16" inference_dreambooth.py \
        --base_model_path /PATH/TO/RealVisXL_V1.0/ \
        --image_encoder_path /PATH/TO/IP-Adapter/models/image_encoder/ \
        --ip_ckpt /PATH/TO/PatchDPO/model.bin \
        --data_root dreambench \
        --output_dir output/exp1_eval_kosmosg/ \
        --scale 0.65 \
        --is_kosmosg True
    ```

### Evaluation

We concisely merge the original evaluation setting and the Kosmos-G setting into a single script (`evaluate_dreambooth.py`) for DreamBench evaluation, making it convenient for future researchers to use.

Two steps for running this script:

*   First, generate the images into a folder `$output_dir` in the way you like, **as long as** it is in the following format:

    ```
    --$output_dir
        |--backpack
            |--a backpack floating in an ocean of milk.png
            |--a backpack floating on top of water.png
            |--a backpack in the jungle.png
            |...
        |--backpack_dog
        |--bear_plushie
        |--berry_bowl
        |...
    ```

    Detailedly, `$output_dir` contains 30 subfolders (corresponding to 30 objects), and each subfolder saves the generated images for each object, which is also named with this object (*i.e.*, the folder names are consistent with those in [dreambench/dataset](https://github.com/google/dreambooth/tree/main/dataset)).

    Each subfolder contains 25 images (corresponding to 25 prompts for each object), and each image is named with the corresponding prompt.

*   Next, run the following scripts for evaluation in two settings:

    *   **Original setting:**

    ```bash
    python evaluate_dreambooth.py \
        --output_dir $output_dir \
        --data_root dreambench \
        --is_kosmosg False
    ```

    *   **Kosmos-G setting:**

    ```bash
    python evaluate_dreambooth.py \
        --output_dir $output_dir \
        --data_root dreambench \
        --is_kosmosg True
    ```

## Citation

If you find our work helpful or inspiring, please feel free to cite it.

```bibtex
@article{zhou2024patchdpo,
  title={PatchDPO: Patch-level DPO for Finetuning-free Personalized Image Generation},
  author={Zhou, Zijian and Liu, Shikun and Han, Xiao and Liu, Haozhe and Ng, Kam Woh and Xie, Tian and Cong, Yuren and Li, Hang and Xu, Mengmeng and P{\'e}rez-R{\'u}a, Juan-Manuel and Patel, Aditya and Xiang, Tao and Shi, Miaojing and He, Sen},
  journal={arXiv preprint arXiv:2412.03177},
  year={2024},
}
```