Instructions to use allenai/Molmo-7B-O-0924 with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use allenai/Molmo-7B-O-0924 with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("image-text-to-text", model="allenai/Molmo-7B-O-0924", trust_remote_code=True)
messages = [
    {
        "role": "user",
        "content": [
            {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"},
            {"type": "text", "text": "What animal is on the candy?"}
        ]
    },
]
pipe(text=messages)

# Load model directly
from transformers import AutoModelForCausalLM
model = AutoModelForCausalLM.from_pretrained("allenai/Molmo-7B-O-0924", trust_remote_code=True, dtype="auto")

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use allenai/Molmo-7B-O-0924 with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "allenai/Molmo-7B-O-0924"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "allenai/Molmo-7B-O-0924",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Use Docker

docker model run hf.co/allenai/Molmo-7B-O-0924

SGLang

How to use allenai/Molmo-7B-O-0924 with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "allenai/Molmo-7B-O-0924" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "allenai/Molmo-7B-O-0924",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "allenai/Molmo-7B-O-0924" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "allenai/Molmo-7B-O-0924",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Docker Model Runner
How to use allenai/Molmo-7B-O-0924 with Docker Model Runner:
```
docker model run hf.co/allenai/Molmo-7B-O-0924
```

chrisc36 commited on Sep 26, 2024

Commit

ffdda66

verified ·

1 Parent(s): 5779628

Upload preprocessing_molmo.py with huggingface_hub

Browse files

Files changed (1) hide show

preprocessing_molmo.py +19 -3

preprocessing_molmo.py CHANGED Viewed

@@ -4,6 +4,10 @@ Processor class for Molmo.
 from typing import Optional
 try:
     from typing import Unpack
 except ImportError:
@@ -23,7 +27,7 @@ from transformers.tokenization_utils_base import TextInput
 from transformers.utils import logging
 from transformers import AutoTokenizer
-from .image_preprocessing_molmo import MolmoImagesKwargs, make_batched_images, MolmoImageProcessor
 logger = logging.get_logger(__name__)
@@ -129,8 +133,20 @@ class MolmoProcessor(ProcessorMixin):
         image_token_id = self.special_token_ids[IMAGE_PROMPT]
         if images is not None:
-            images = make_batched_images(images)
-            images = [np.array(image).astype(np.uint8) for image in images]
             # For now only support inserting images at the start
             image_idx = [-1]*len(images)
         else:

 from typing import Optional
+import PIL
+from PIL import ImageOps
+from PIL.Image import Image
 try:
     from typing import Unpack
 except ImportError:
 from transformers.utils import logging
 from transformers import AutoTokenizer
+from .image_preprocessing_molmo import MolmoImagesKwargs, MolmoImageProcessor
 logger = logging.get_logger(__name__)
         image_token_id = self.special_token_ids[IMAGE_PROMPT]
         if images is not None:
+            if not isinstance(images, (list, tuple)):
+                images = [images]
+            image_arrays = []
+            for image in images:
+                if isinstance(image, Image):
+                    image = image.convert("RGB")
+                    # Handle images with EXIF orientation tags, which PIL will ignore by default
+                    # https://github.com/python-pillow/Pillow/issues/4703
+                    img = ImageOps.exif_transpose(image)
+                    image_arrays.append(np.array(image))
+                else:
+                    assert len(image.shape) == 3 and image.shape[-1] == 3
+                    image_arrays.append(image.astype(np.uint8))
+            images = image_arrays
             # For now only support inserting images at the start
             image_idx = [-1]*len(images)
         else: