Instructions to use google/paligemma-3b-pt-224 with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use google/paligemma-3b-pt-224 with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("image-text-to-text", model="google/paligemma-3b-pt-224")

# Load model directly
from transformers import AutoProcessor, AutoModelForMultimodalLM

processor = AutoProcessor.from_pretrained("google/paligemma-3b-pt-224")
model = AutoModelForMultimodalLM.from_pretrained("google/paligemma-3b-pt-224")

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use google/paligemma-3b-pt-224 with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "google/paligemma-3b-pt-224"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "google/paligemma-3b-pt-224",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Use Docker

docker model run hf.co/google/paligemma-3b-pt-224

SGLang

How to use google/paligemma-3b-pt-224 with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "google/paligemma-3b-pt-224" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "google/paligemma-3b-pt-224",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "google/paligemma-3b-pt-224" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "google/paligemma-3b-pt-224",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Docker Model Runner
How to use google/paligemma-3b-pt-224 with Docker Model Runner:
```
docker model run hf.co/google/paligemma-3b-pt-224
```

Using `low_cpu_mem_usage=True` or a `device_map` requires Accelerate: `pip install accelerate`

#19

by mdeniz1 - opened Jul 6, 2024

Discussion

mdeniz1

Jul 6, 2024

Hi,

I am using the code in the model car in colab with a100 gpu.
I have run pip install accelerate successfully but still I get the error message in the subject of this discussion:

Using low_cpu_mem_usage=True or a device_map requires Accelerate: pip install accelerate

This is an error that I keep getting for other models too. I am short of gpu in my laptop so I can not try it in my local set up.

Somebody help me please.

from transformers import AutoProcessor, PaliGemmaForConditionalGeneration
from PIL import Image
import requests
import torch

model_id = "google/paligemma-3b-mix-224"
device = "cuda:0"
dtype = torch.bfloat16

url = "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/transformers/tasks/car.jpg?download=true"
image = Image.open(requests.get(url, stream=True).raw)

model = PaliGemmaForConditionalGeneration.from_pretrained(
model_id,
torch_dtype=dtype,
device_map=device,
revision="bfloat16",
).eval()
processor = AutoProcessor.from_pretrained(model_id)

Instruct the model to create a caption in Spanish

prompt = "caption es"
model_inputs = processor(text=prompt, images=image, return_tensors="pt").to(model.device)
input_len = model_inputs["input_ids"].shape[-1]

with torch.inference_mode():
generation = model.generate(**model_inputs, max_new_tokens=100, do_sample=False)
generation = generation[0][input_len:]
decoded = processor.decode(generation, skip_special_tokens=True)
print(decoded)

Renu11

Google org Sep 5, 2024

•

edited Sep 5, 2024

The issue might be related to the environment setup or the specific version of accelerate you are using. I have tried replicating the above code on Google Colab with T4 GPU and found no issues, even without installing accelerate. Please make sure that you have connected your notebook to the GPU and try updating accelerate to the latest version using !pip install -U accelerate. Let us know if the issue still persists.

You can find the replicated gist here for your reference. Thank you.

mdeniz1

Sep 20, 2024

thank you for your response. I remember that I resolved the problem by re-starting the kernel after pip install accelerate

mdeniz1 changed discussion status to closed Sep 20, 2024

Upload images, audio, and videos by dragging in the text input, pasting, or clicking here.

Tap or paste here to upload images

· Sign up or log in to comment