| --- |
| license: mit |
| pipeline_tag: text-to-image |
| tags: |
| - diffusion |
| - efficient |
| - quantization |
| - StableDiffusionXLPipeline |
| - Diffusers |
| base_model: |
| - stabilityai/sdxl-turbo |
| --- |
| |
| # MixDQ Model Card |
|
|
| ## Model Description |
|
|
| MixDQ is a mixed precision quantization methods that compress the memory and computational usage of text-to-image diffusion models while preserving genration quality. |
| It supports few-step diffusion models (e.g., SDXL-turbo, LCM-lora) to construct both fast and tiny diffusion models. Efficient CUDA kernel implemention is provided for practical resource savings. |
|
|
| <img src="https://github.com/A-suozhang/MyPicBed/raw/master/img/mixdq_model_card_0.jpg" width="600"> |
|
|
|
|
| ## Model Sources |
|
|
| for more information, please refer to: |
|
|
| - Project Page: [https://a-suozhang.xyz/mixdq.github.io/](https://a-suozhang.xyz/mixdq.github.io/). |
| - Arxiv paper: [https://arxiv.org/abs/2405.17873](https://arxiv.org/abs/2405.17873) |
| - Github Repository: [https://github.com/A-suozhang/MixDQ](https://github.com/A-suozhang/MixDQ) |
|
|
| ## Evaluation |
|
|
| We evaluate the MixDQ model using various metrics, including FID (fidelity), CLIPScore (image-text alignment), and ImageReward (human preference). MixDQ can achieve W8A8 quantization without performance loss. The differences between images generated by MixDQ and those generated by FP16 models are negligible. |
|
|
| | Method | FID (↓) | ClipScore | ImageReward | |
| |------------|---------|-----------|-------------| |
| | FP16 | 17.15 | 0.2722 | 0.8631 | |
| | MixDQ-W8A8 | 17.03 | 0.2703 | 0.8415 | |
| | MixDQ-W5A8 | 17.23 | 0.2697 | 0.8307 | |
|
|
| ## Usage |
|
|
|
|
| install the prerequisite for Mixdq: |
| ```shell |
| # The Python versions required to run mixdq: 3.8, 3.9, 3.10 |
| pip install -i https://pypi.org/simple/ mixdq-extension |
| ``` |
|
|
| run the pipeline: |
| ```python |
| pipe = DiffusionPipeline.from_pretrained( |
| "stabilityai/sdxl-turbo", custom_pipeline="nics-efc/MixDQ", |
| torch_dtype=torch.float16, variant="fp16" |
| ) |
| |
| # quant the UNet |
| pipe.quantize_unet( |
| w_bit = 8, |
| a_bit = 8, |
| bos=True, |
| ) |
| |
| # The set_cuda_graph func is optional and used for acceleration |
| pipe.set_cuda_graph( |
| run_pipeline = True, |
| ) |
| |
| # test the memory and the lantency of the pipeline or the UNet |
| pipe.run_for_test( |
| device="cuda", |
| output_type="pil", |
| run_pipeline=True, |
| path="pipeline_test.png", |
| profile=True |
| ) |
| ''' |
| After execution is finished, there will be a report under log/sdxl folder in formats of json. |
| This report can be opened by tensorboard for users to examine profiling results: |
| tensorboard --logdir=./log |
| ''' |
| |
| # run the pipeline |
| pipe = pipe.to("cuda") |
| prompts = "A black Honda motorcycle parked in front of a garage." |
| image = pipe(prompts, num_inference_steps=1, guidance_scale=0.0).images[0] |
| image.save('mixdq_pipeline.png') |
| ``` |
|
|
|
|
|
|
| Performance tested on NVIDIA 4080: |
|
|
| | UNet Latency (ms) | No CUDA Graph | With CUDA Graph | |
| |-------------------|---------------|-----------------| |
| | FP16 version | 44.6 | 36.1 | |
| | Quantized version | 59.1 | 24.9 | |
| | Speedup | 0.75 | 1.45 | |
