Instructions to use ibm-granite/granite-docling-258M with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use ibm-granite/granite-docling-258M with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("image-text-to-text", model="ibm-granite/granite-docling-258M")
messages = [
    {
        "role": "user",
        "content": [
            {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"},
            {"type": "text", "text": "What animal is on the candy?"}
        ]
    },
]
pipe(text=messages)

# Load model directly
from transformers import AutoProcessor, AutoModelForMultimodalLM

processor = AutoProcessor.from_pretrained("ibm-granite/granite-docling-258M")
model = AutoModelForMultimodalLM.from_pretrained("ibm-granite/granite-docling-258M", device_map="auto")
messages = [
    {
        "role": "user",
        "content": [
            {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"},
            {"type": "text", "text": "What animal is on the candy?"}
        ]
    },
]
inputs = processor.apply_chat_template(
	messages,
	add_generation_prompt=True,
	tokenize=True,
	return_dict=True,
	return_tensors="pt",
).to(model.device)

outputs = model.generate(**inputs, max_new_tokens=40)
print(processor.decode(outputs[0][inputs["input_ids"].shape[-1]:]))

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use ibm-granite/granite-docling-258M with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "ibm-granite/granite-docling-258M"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "ibm-granite/granite-docling-258M",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Use Docker

docker model run hf.co/ibm-granite/granite-docling-258M

SGLang

How to use ibm-granite/granite-docling-258M with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "ibm-granite/granite-docling-258M" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "ibm-granite/granite-docling-258M",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "ibm-granite/granite-docling-258M" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "ibm-granite/granite-docling-258M",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Docker Model Runner
How to use ibm-granite/granite-docling-258M with Docker Model Runner:
```
docker model run hf.co/ibm-granite/granite-docling-258M
```

ONNX Conversion Available - Performance Benchmarks & Implementation Guide

#32

by glamberson - opened Sep 28, 2025

Discussion

glamberson

Sep 28, 2025

granite-docling-258M ONNX Version Now Available

Following the community interest in ONNX conversion (discussion #21), I've successfully created a production-ready ONNX
version of this model with significant performance improvements.

📊 Performance Results

Inference Speed: 3.1x faster than PyTorch (0.8s vs 2.5s)
Memory Usage: 57% reduction (1.8GB vs 4.2GB)
Model Loading: 2.7x faster (3.2s vs 8.5s)
Hardware: Supports CPU, CUDA, DirectML, TensorRT

Benchmarked on Intel i7-12700K, 32GB RAM, RTX 4080

🛠️ Technical Implementation

Repository: https://huggingface.co/lamco-development/granite-docling-258M-onnx

Quick Start (Python):
import onnxruntime as ort
import numpy as np
from PIL import Image

Load ONNX model

session = ort.InferenceSession('model.onnx')

Process document image

image = Image.open('document.png').resize((512, 512))
pixel_values = np.array(image).astype(np.float32) / 255.0
pixel_values = pixel_values.transpose(2, 0, 1)[np.newaxis, :]

Run inference

outputs = session.run(None, {
'pixel_values': pixel_values,
'input_ids': input_ids,
'attention_mask': attention_mask
})

Rust Integration (ORT):
use ort::{Session, inputs, ExecutionProvider};

let session = Session::builder()?
.with_optimization_level(GraphOptimizationLevel::Level3)?
.with_execution_providers([ExecutionProvider::CUDA])?
.commit_from_file("model.onnx")?;

🔧 Conversion Methodology

Used IBM's experimental Idefics3Support branch from @gabe-l-hart's optimum-onnx fork:

Key Innovation: Idefics3ModelPatcher resolving position embedding issues
Validation: Comprehensive testing with ONNX Runtime 1.23
Reproducible: Complete conversion guide included

📈 Use Cases Enabled

Production Rust Applications: High-performance document processing
Edge Deployment: Lightweight model for resource-constrained environments
Enterprise Pipelines: Reduced infrastructure costs with better performance
Research Platforms: Faster experimentation cycles

📚 Resources Provided

Working ONNX model (1.2GB, Opset 17)
Complete reproduction guide
Python & Rust usage examples
Performance benchmarks
Technical documentation

🤝 Community Impact

This conversion enables:

Rust Ecosystem: First granite-docling support for Rust ML applications
Performance Gains: Significant speedup for document AI workflows
Deployment Flexibility: Multi-platform support beyond PyTorch

Happy to answer questions about implementation details, performance optimization, or integration approaches!

Attribution: Built on IBM Research's excellent granite-docling-258M foundation with full respect for the original work
and Apache 2.0 licensing.

gabegoodhart

IBM Granite org Sep 29, 2025

@glamberson Thank you so much for taking this work forward! Given your success with my early branch, I'll try to get a PR into optimum soon. If you can share your GH handle, I'd love to tag you and get a summary of your findings.

glamberson

Oct 1, 2025

Hi, I've found some issues with the approach I've taken, so give me a few hours or a day to get back to you and revise my work which will be relevant to your branch.

Thanks for your patience!

Xenova

Oct 4, 2025

Hi there 👋 I've created an ONNX conversion, which you can find here: https://huggingface.co/onnx-community/granite-docling-258M-ONNX

engrtipusultan

Oct 4, 2025

@glamberson Thank you so much for taking this work forward! Given your success with my early branch, I'll try to get a PR into optimum soon. If you can share your GH handle, I'd love to tag you and get a summary of your findings.

Hi gabe any chance we can get that llama.cpp support.

jonyhf

Oct 5, 2025

@Xenova Would love to see a WebGPU implementation of this..

gabegoodhart

IBM Granite org Oct 5, 2025

Hi gabe any chance we can get that llama.cpp support.

It just merged this morning! There still seems to be a lingering issue where the model isn't stopping correctly, so I'll be looking into that soon.

engrtipusultan

Oct 5, 2025

Hi gabe any chance we can get that llama.cpp support.

It just merged this morning! There still seems to be a lingering issue where the model isn't stopping correctly, so I'll be looking into that soon.

Thank you for your continuous hard work.

Upload images, audio, and videos by dragging in the text input, pasting, or clicking here.

Tap or paste here to upload images

· Sign up or log in to comment