| ---
|
| license: mit
|
| library_name: diffusers
|
| pipeline_tag: text-to-image
|
| language:
|
| - en
|
| tags:
|
| - diffusers
|
| - matfuse
|
| - pbr
|
| - material-generation
|
| - svbrdf
|
| - text-to-image
|
| ---
|
|
|
|
|
| # MatFuse — Controllable Material Generation with Diffusion Models
|
|
|
| MatFuse generates tileable PBR material maps (diffuse, normal, roughness,
|
| specular) from text, reference images, sketches, and/or color palettes.
|
|
|
| > **Paper:** [MatFuse: Controllable Material Generation with Diffusion Models](https://arxiv.org/abs/2308.11408) — CVPR 2024
|
| > **Project page:** <https://gvecchio.com/matfuse/>
|
|
|
| ## Quick Start
|
|
|
| ```python
|
| import torch
|
| from diffusers import DiffusionPipeline
|
|
|
| pipe = DiffusionPipeline.from_pretrained(
|
| "gvecchio/MatFuse",
|
| trust_remote_code=True,
|
| torch_dtype=torch.float16,
|
| )
|
| pipe = pipe.to("cuda")
|
|
|
| result = pipe(
|
| text="red brick wall",
|
| num_inference_steps=50,
|
| guidance_scale=4.0,
|
| generator=torch.Generator("cuda").manual_seed(42),
|
| )
|
|
|
| result["diffuse"][0].save("diffuse.png")
|
| result["normal"][0].save("normal.png")
|
| result["roughness"][0].save("roughness.png")
|
| result["specular"][0].save("specular.png")
|
| ```
|
|
|
| ## Conditioning Inputs
|
|
|
| All conditions are **optional** and freely composable:
|
|
|
| | Input | Type | Description |
|
| |-------|------|-------------|
|
| | `text` | `str` | Text description of the material |
|
| | `image` | `PIL.Image` | Reference image for style/appearance |
|
| | `sketch` | `PIL.Image` (grayscale) | Binary edge map for structure |
|
| | `palette` | `list[tuple]` | Up to 5 RGB colour tuples (0–255) |
|
|
|
| ```python
|
| from PIL import Image
|
|
|
| result = pipe(
|
| image=Image.open("reference.png"),
|
| text="rough stone texture",
|
| palette=[(120, 80, 60), (90, 60, 40), (150, 110, 80), (70, 50, 30), (180, 140, 100)],
|
| num_inference_steps=50,
|
| guidance_scale=4.0,
|
| )
|
| ```
|
|
|
| ## Architecture
|
|
|
| | Component | Class | Key parameters |
|
| |-----------|-------|----------------|
|
| | **UNet** | `UNet2DConditionModel` | in=16, out=12, blocks=[256,512,1024], cross_attn=512 |
|
| | **VAE** | `MatFuseVQModel` (custom) | 4 encoders + 4 VQ codebooks (4096×3), shared decoder, f=8 |
|
| | **Scheduler** | `DDIMScheduler` | β 0.0015–0.0195, scaled_linear, ε-prediction |
|
| | **Conditioning** | `MultiConditionEncoder` (custom) | CLIP ViT-B/16 · sentence-transformers · palette MLP · sketch CNN |
|
|
|
| ## 📜 Citation
|
|
|
| ```bibtex
|
| @inproceedings{vecchio2024matfuse,
|
| author = {Vecchio, Giuseppe and Sortino, Renato and Palazzo, Simone and Spampinato, Concetto},
|
| title = {MatFuse: Controllable Material Generation with Diffusion Models},
|
| booktitle = {Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)},
|
| month = {June},
|
| year = {2024},
|
| pages = {4429-4438}
|
| }
|
| ```
|
|
|
| ## License
|
|
|
| This project is licensed under the MIT License. |
