IDEOGRAM-4 for inpainting with Modular Diffusers and Differential Diffusion

Published August 5, 2026

I tested this model for inpainting, and I was really surprised by how well it performs. In my opinion, it's the best available at the moment. This is different from the edit models that replaced inpainting because the JSON bounding boxes give the user a level of control that edit models simply don't have, and the quality is amazing when paired with different diffusion models.

As an example, let's take this image where I added 3 new objects and modified another one. For me, this is the best inpainting quality I have seen so far. You can unfold the original image to compare the results.

Photo by roam in color on Unsplash

Unfold to see original

What differentiates Ideogram-4 from other models, aside from the quality, is the fine-grained control we have over the inpainting process. For example, it is trivial to move objects or change their order when you have a UI to manipulate the bounding boxes and masks.

With Diffusers, we can achieve this very easily. I'll go over the process in this blog post.

The steps to achieve this result are:

Caption the image and add or remove the bounding boxes for the regions you want to inpaint.
Draw a mask over the areas you want to modify while preserving the rest of the image. A soft mask ensures a smooth transition between the original image and the generated content.
Use the Ideogram-4 custom differential diffusion blocks to generate the final image.

Modular Diffusers

Modular Diffusers is a framework for quickly building flexible and customizable pipelines. These pipelines can go beyond what standard DiffusionPipelines can do. If you want to learn more, you can read the documentation here.

Ideogram-4 has native Modular Diffusers integration, which makes it really convenient to extend the base pipeline with additional functionality. For example, I added image-to-image and differential diffusion support. Since it's modular and uses auto blocks, it can automatically switch modes depending on the inputs.

I uploaded the custom blocks here, and the repository includes example code. I'll also walk through some practical examples in this post. My main goal was to enable inpainting with differential diffusion, but since the blocks are modular, the same pipeline also supports text-to-image and image-to-image, so there's no need to maintain separate pipelines for each task.

Important: The custom blocks set the nf4 quantized model as the default model, so you need to have bitsandbytes installed in your venv for this code to work.

Text to image

For text-to-image we can use this code which uses the official bitsandbytes nf4 model:

import torch

from diffusers import ModularPipeline


pipe = ModularPipeline.from_pretrained("OzzyGT/ideogram4_custom_blocks", trust_remote_code=True)
pipe.load_components(
    names=["text_encoder", "tokenizer", "transformer", "unconditional_transformer", "vae", "scheduler"],
    torch_dtype=torch.bfloat16,
)
pipe.to("cuda")

prompt = """
{"high_level_description":"A detailed photograph of a snowy owl perched on a weathered wooden sign in a
misty pine forest at sunrise.","compositional_deconstruction":{"background":"A dense pine forest shrouded
in low-lying, pale grey mist, with the ground covered in patches of snow and moss-covered rocks, illuminated
by the soft, diffused light of a rising sun filtering through the canopy.",
"elements":[{"type":"obj","desc":"A large snowy owl with meticulously detailed white and grey plumage,
perched upright on the wooden sign, its amber eyes catching the faint morning light."},{"type":"obj","desc":"A
weathered, dark brown wooden signpost, showing signs of age and moisture, positioned slightly off-center,
supporting the owl."},{"type":"text","text":"NORTH RIDGE TRAIL","desc":"Perfectly readable, carved text in
dark brown lettering displayed across the face of the weathered wooden sign."},{"type":"obj","desc":"Several
moss-covered rocks scattered on the snowy ground beneath the sign, providing a natural base element in the
lower portion of the frame."}]}}
"""

image = pipe(prompt=prompt, height=1024, width=1024, output="images")[0]
image.save("ideogram4_output.png")

But we have the issue that this code will still OOM on a 24GB consumer GPU because this model uses two DiTs (conditional and unconditional) at the same time, so model offlading won't help, we need to use group offloading, also I will change the model to use the SDNQ one here that in my opinion gives a little better quality.

We also need to install the sdnq library:

pip install sdnq

Loading sdnq models is now natively supported by diffusers so no extra code is needed, we just need to replace the models and apply group offload to them:

import torch

from diffusers import ModularPipeline
from diffusers.hooks import apply_group_offloading


pipe = ModularPipeline.from_pretrained("OzzyGT/ideogram4_custom_blocks", trust_remote_code=True)
pipe.load_components(
    names=["text_encoder", "tokenizer", "transformer", "unconditional_transformer", "vae", "scheduler"],
    pretrained_model_name_or_path="Disty0/Ideogram-4-SDNQ-4bit-dynamic-hadamard",
    torch_dtype=torch.bfloat16,
)


onload_device = torch.device("cuda")
for name in ("text_encoder", "transformer", "unconditional_transformer", "vae"):
    apply_group_offloading(
        pipe.components[name],
        onload_device=onload_device,
        offload_type="leaf_level",
        use_stream=True,
        low_cpu_mem_usage=True,  # Set to False if you have ~40GB of free RAM
    )

prompt = """
{"high_level_description":"A detailed photograph of a snowy owl perched on a weathered wooden sign in a
misty pine forest at sunrise.","compositional_deconstruction":{"background":"A dense pine forest shrouded
in low-lying, pale grey mist, with the ground covered in patches of snow and moss-covered rocks, illuminated
by the soft, diffused light of a rising sun filtering through the canopy.",
"elements":[{"type":"obj","desc":"A large snowy owl with meticulously detailed white and grey plumage,
perched upright on the wooden sign, its amber eyes catching the faint morning light."},{"type":"obj","desc":"A
weathered, dark brown wooden signpost, showing signs of age and moisture, positioned slightly off-center,
supporting the owl."},{"type":"text","text":"NORTH RIDGE TRAIL","desc":"Perfectly readable, carved text in
dark brown lettering displayed across the face of the weathered wooden sign."},{"type":"obj","desc":"Several
moss-covered rocks scattered on the snowy ground beneath the sign, providing a natural base element in the
lower portion of the frame."}]}}
"""

image = pipe(prompt=prompt, height=1024, width=1024, output="images")[0]
image.save("ideogram4_sdnq_output.png")

The result image is this one:

If you have Triton installed I recommend you add this lines to get a speed up:

from sdnq.common import use_torch_compile as triton_is_available
from sdnq.loader import apply_sdnq_options_to_model

...
# after loading the pipeline

if triton_is_available and torch.cuda.is_available():
    for name in ("transformer", "unconditional_transformer", "text_encoder"):
        apply_sdnq_options_to_model(pipe.components[name], use_quantized_matmul=True)

Image to Image

I haven't found a real use case for this yet. With the JSON prompting being so precise, I get pretty much the same results whether I use an input image or just the prompt. That said, image-to-image support is needed for differential diffusion, so we don't lose anything by having it. If you want to use it, just load an image and pass it to the pipeline:

from diffusers.utils import load_image

image = load_image("https://huggingface.co/datasets/OzzyGT/testing-resources/resolve/main/ideo4/20260720144615.png")

prompt 

result = pipe(
    prompt="a photo of a snowy mountain landscape at sunset, dramatic clouds",
    image=image,
    strength=0.6,
    output="images",
)[0]
result.save("img2img.png")

Inpainting with differential diffusion

For differential diffusion, we need to pass an image and a soft mask, but we need to have a json prompt that is coherent with what we want to do:

from diffusers.utils import load_image

image = load_image("https://huggingface.co/datasets/OzzyGT/testing-resources/resolve/main/ideo4/20260720135413.png")
diffdiff_map = load_image(
    "https://huggingface.co/datasets/OzzyGT/testing-resources/resolve/main/ideo4/20260720141824_source_mask.png"
)

prompt = """
{"high_level_description":"a fruit in a centered composition with mirror reflection and realistic water droplets
from condensation","style_description":{"aesthetics":"Commercial studio product photo, premium advertising photography,
premium food photography, luxury branding aesthetic","lighting":"dramatic low-key lighting, softbox key light, strip rim
light, controlled highlights, deep shadows","photo":"full-frame camera, 100mm macro lens, f/11, ISO 100, 1/160 sec,
RAW capture, polarized lighting, ultra sharp, professional color grading","medium":"photograph"},
"compositional_deconstruction":{"background":"seamless black backdrop, glossy black acrylic surface, flawless surface",
"elements":[{"type":"obj","bbox":[210,194,775,768],"desc":"full red apple with a vibrant natural red color"},{"type":"obj",
"bbox":[645,56,864,926],"desc":"a glass base under the apple"}]}}
"""

image = pipe(prompt=prompt, height=1024, width=1024, image=image, diffdiff_map=diffdiff_map, output="images")[0]

Captioning and prompt enhancer

Creating JSON prompts by hand can be tedious, so I added another modular block that can generate them automatically. If you provide an image, it captions it and produces the corresponding JSON prompt. If you only provide a text prompt, it works as a prompt enhancer instead.

Prompt Enhancer

Using it as a prompt enhancer:

import torch

from diffusers import ModularPipeline


pipe = ModularPipeline.from_pretrained("OzzyGT/ideogram4_caption_blocks", trust_remote_code=True)
pipe.load_components(torch_dtype=torch.bfloat16)
pipe.to("cuda")

prompt = """
Photograph of a snowy owl perched on a weathered wooden sign in a misty pine forest at sunrise,
captured with a 135mm telephoto lens at f/2.8, ISO 200, 1/1000s shutter speed. Every feather
individually visible, amber eyes reflecting the morning light, moss-covered rocks below,
gentle fog drifting between trees, the sign displaying perfectly readable text "NORTH RIDGE TRAIL",
balanced composition, shallow depth of field.
"""

caption = pipe(prompt=prompt, output="caption")
print(caption)

It gives this json formatted output which can be used with ideogram-4, this is the prompt for the image on the previous section.

{
  "high_level_description": "A detailed photograph of a snowy owl perched on a weathered wooden sign in a misty pine forest at sunrise.",
  "compositional_deconstruction": {
    "background": "A dense pine forest shrouded in low-lying, pale grey mist, with the ground covered in patches of snow and moss-covered rocks, illuminated by the soft, diffused light of a rising sun filtering through the canopy.",
    "elements": [
      {
        "type": "obj",
        "desc": "A large snowy owl with meticulously detailed white and grey plumage, perched upright on the wooden sign, its amber eyes catching the faint morning light."
      },
      {
        "type": "obj",
        "desc": "A weathered, dark brown wooden signpost, showing signs of age and moisture, positioned slightly off-center, supporting the owl."
      },
      {
        "type": "text",
        "text": "NORTH RIDGE TRAIL",
        "desc": "Perfectly readable, carved text in dark brown lettering displayed across the face of the weathered wooden sign."
      },
      {
        "type": "obj",
        "desc": "Several moss-covered rocks scattered on the snowy ground beneath the sign, providing a natural base element in the lower portion of the frame."
      }
    ]
  }
}

Captioning and labeling images

If you pass an image instead of a prompt it will caption and do the bounding boxes labeling, but same as before, I will use a 8-bit SDNQ model to save VRAM:

import torch
from sdnq import SDNQConfig  # noqa: F401  # importing sdnq registers its quantizer into transformers/diffusers

from diffusers import ModularPipeline
from diffusers.utils import load_image


pipe = ModularPipeline.from_pretrained("OzzyGT/ideogram4_caption_blocks", trust_remote_code=True)
pipe.load_components(
    pretrained_model_name_or_path="OzzyGT/gemma_4_E4B_it_sdnq_dynamic_8bit",
    torch_dtype=torch.bfloat16,
)
pipe.to("cuda")

image = load_image(
    "https://huggingface.co/datasets/OzzyGT/testing-resources/resolve/main/ideo4/roam-in-color-small.jpg"
)
caption = pipe(image=image, output="caption")  # caption the image -> JSON string
print(caption)

The output is a long json that I won't post here since it will make unnecesary long this post.

Space (local and on Hugging Face)

Now that we have the custom Modular Diffusers blocks, it's really easy to use them in any application built on top of Diffusers. However, a JSON caption by itself is not very convenient when you're working on an inpainting task or even just building a prompt. What you really need is a visual UI to inspect and edit it.

To demonstrate how flexible Modular Diffusers is, I created an application that not only generates the JSON caption, but also lets you visualize the bounding boxes, add new ones, edit them, and reposition them.

The Hugging Face Space is available here:

https://huggingface.co/spaces/OzzyGT/ideogram4-caption

If you prefer to run it locally or want to look at the code, just clone the repository. The application lets you choose between the BF16 and SDNQ models, and it also includes options to load and unload the model, making it easy to switch between this tool and other applications when needed.

git clone https://huggingface.co/spaces/OzzyGT/ideogram4-caption
pip install -r requirements.txt
python app.py

Tying it all together

Now that we have all the pieces in place, we can simply connect the custom blocks to build the inpainting pipeline. I didn't build an application to draw the mask, but that would be straightforward to implement in any Photoshop-like application or even as a Hugging Face Space.

This is the complete code used to generate the image shown at the beginning of the post:

import torch
from sdnq.common import use_torch_compile as triton_is_available
from sdnq.loader import apply_sdnq_options_to_model

from diffusers import ModularPipeline
from diffusers.hooks import apply_group_offloading
from diffusers.utils import load_image


image = load_image(
    "https://huggingface.co/datasets/OzzyGT/testing-resources/resolve/main/ideo4/roam-in-color-small.jpg"
)
diffdiff_map = load_image(
    "https://huggingface.co/datasets/OzzyGT/testing-resources/resolve/main/ideo4/20260719233150_source_mask.png"
)


pipe = ModularPipeline.from_pretrained("OzzyGT/ideogram4_custom_blocks", trust_remote_code=True)
pipe.load_components(
    names=["text_encoder", "tokenizer", "transformer", "unconditional_transformer", "vae", "scheduler"],
    pretrained_model_name_or_path="Disty0/Ideogram-4-SDNQ-4bit-dynamic-hadamard",
    torch_dtype=torch.bfloat16,
)

if triton_is_available and torch.cuda.is_available():
    for name in ("transformer", "unconditional_transformer", "text_encoder"):
        apply_sdnq_options_to_model(pipe.components[name], use_quantized_matmul=True)

onload_device = torch.device("cuda")
for name in ("text_encoder", "transformer", "unconditional_transformer", "vae"):
    apply_group_offloading(
        pipe.components[name],
        onload_device=onload_device,
        offload_type="leaf_level",
        use_stream=True,
        low_cpu_mem_usage=True,  # Set to False if you have ~40GB of free RAM
    )

prompt = """
{"high_level_description":"A bright, modern kitchen featuring white cabinetry, a hexagonal tile backsplash, 
and a large central island with wooden paneling.","style_description":{"aesthetics":"modern, clean, bright",
"lighting":"bright daylight, diffused","photo":"high resolution, sharp focus, eye-level","medium":"photograph",
"color_palette":["#FFFFFF","#F5F5DC","#A0522D","#D2B48C","#8B4513"]},"compositional_deconstruction":
{"background":"A kitchen interior with white upper cabinets, a white countertop, and a white hexagonal tile backsplash.",
"elements":[{"type":"obj","bbox":[14,148,415,345],"desc":"White glass-fronted upper kitchen cabinet with interior shelving.",
"color_palette":["#FFFFFF"]},{"type":"obj","bbox":[14,645,415,945],"desc":"White glass-fronted upper kitchen cabinet
with interior shelving.","color_palette":["#FFFFFF"]},{"type":"obj","bbox":[150,383,445,615],"desc":"Window with a woven
bamboo blind covering the top.","color_palette":["#D2B48C"]},{"type":"obj","bbox":[505,405,632,515],"desc":"Silver and
grey cooking pot on the kitchen island.","color_palette":["#A9A9A9"]},{"type":"obj","bbox":[508,690,596,789],
"desc":"White ceramic bowl on the kitchen island.","color_palette":["#FFFFFF"]},{"type":"obj","bbox":[632,0,1000,1000],
"desc":"Large kitchen island with a white marble countertop and dark brown wooden base.","color_palette":["#FFFFFF","#8B4513"]},
{"type":"obj","bbox":[815,65,1000,270],"desc":"Wooden bar stool with a round wooden seat.","color_palette":["#A0522D"]},
{"type":"obj","bbox":[815,295,1000,485],"desc":"Wooden bar stool with a round wooden seat.","color_palette":["#A0522D"]},
{"type":"obj","bbox":[815,550,1000,740],"desc":"Wooden bar stool with a round wooden seat.","color_palette":["#A0522D"]},
{"type":"obj","bbox":[815,745,1000,935],"desc":"Wooden bar stool with a round wooden seat.","color_palette":["#A0522D"]},
{"type":"obj","bbox":[144,0,644,112],"desc":"plant"},{"type":"obj","bbox":[541,218,643,255],"desc":"glass of juice"},
{"type":"obj","bbox":[511,815,644,876],"desc":"a small potted plant"},{"type":"obj","bbox":[417,141,552,215],"desc":"bottles"},
{"type":"obj","bbox":[554,301,639,385],"desc":"a basket with oranges"}]}}
"""

image = pipe(prompt=prompt, height=720, width=1280, image=image, diffdiff_map=diffdiff_map, output="images")[0]
image.save("ideogram4_final_output.png")

This is another image generated with a different seed than the first one:

Final thoughts

After spending some time with Ideogram-4, I think it offers the best inpainting experience I have used so far. The combination of pixel-precise masks, structured JSON prompts, and bounding boxes gives you a level of control that edit models simply don't provide. The tradeoff is that it requires more setup and much more structured prompting, but once everything is in place, iterating becomes very fast. Most changes only require adjusting a few bounding boxes instead of rewriting the entire prompt.

Just remember that this is a non-commercial model. If you want to use it professionally, you will need to negotiate a license with the model authors.

A couple of ideas that could make this workflow even better:

Everything could be unified into a single application. The mask could probably be generated automatically from the bounding boxes, with a simple option to enable or disable masking for each object.
I used 48 inference steps to match the original model configuration, but in most cases I use 20 steps and the results are still pretty good while cutting the generation time in half.

That's all for now. If you have questions, feedback, or just want to share what you're building with Diffusers, come join us on Discord: https://discord.com/invite/G7tWnz98XR I'd love to see what you're working on and help where I can.

Models mentioned in this article 2

Datasets mentioned in this article 1

Spaces mentioned in this article 1

Image compositing with diffusers

July 17, 2025

Recoloring photos with diffusers

October 9, 2024

Community

Upload images, audio, and videos by dragging in the text input, pasting, or clicking here.

Tap or paste here to upload images

· Sign up or log in to comment

Upvote

IDEOGRAM-4 for inpainting with Modular Diffusers and Differential Diffusion

Modular Diffusers

Text to image

Image to Image

Inpainting with differential diffusion

Captioning and prompt enhancer

Prompt Enhancer

Captioning and labeling images

Space (local and on Hugging Face)

Tying it all together

Final thoughts

Models mentioned in this article 2

Datasets mentioned in this article 1

Spaces mentioned in this article 1

Ideogram4 Caption

Image compositing with diffusers

Recoloring photos with diffusers

Community

Models mentioned in this article 2

Datasets mentioned in this article 1

Spaces mentioned in this article 1

Ideogram4 Caption