from transformers import Qwen2VLForConditionalGeneration, AutoTokenizer, AutoProcessor
from qwen_vl_utils import process_vision_info
import torch

default: Load the model on the available device(s)

model = Qwen2VLForConditionalGeneration.from_pretrained(
"Qwen/Qwen2-VL-2B-Instruct", torch_dtype="auto"
)

Move the model to the desired device

DEVICE = torch.device("cpu")
model.to(DEVICE)

Enable CPU offloading if desired

USE_CPU_OFFLOAD = True

... rest of the code ...

processor = AutoProcessor.from_pretrained("Qwen/Qwen2-VL-2B-Instruct")

messages = [
{
"role": "user",
"content": [
{
"type": "image",
"image": "https://qianwen-res.oss-cn-beijing.aliyuncs.com/Qwen-VL/assets/demo.jpeg",
},
{"type": "text", "text": "Describe this image."},
],
}
]

text = processor.apply_chat_template(
messages, tokenize=False, add_generation_prompt=True
)
image_inputs, video_inputs = process_vision_info(messages)
inputs = processor(
text=[text],
images=image_inputs,
videos=video_inputs,
padding=True,
return_tensors="pt",
)
inputs = inputs.to("cpu")

generated_ids = model.generate(**inputs, max_new_tokens=128)
generated_ids_trimmed = [
out_ids[len(in_ids) :] for in_ids, out_ids in zip(inputs.input_ids, generated_ids)
]
output_text = processor.batch_decode(
generated_ids_trimmed, skip_special_tokens=True, clean_up_tokenization_spaces=False
)
print(output_text)

The image depicts a serene beach scene with a woman and her dog. The woman is sitting on the sand, wearing a plaid shirt and black pants, and appears to be smiling. She is holding the dog's paw in a high-five gesture. The dog, which is a large breed, is wearing a harness and is sitting on the sand with its front paws raised, possibly in a playful or training pose. The background shows the ocean with gentle waves, and the sky is clear with a soft light, suggesting it might be either sunrise or sunset. The overall atmosphere is calm and joyful, capturing a moment of connection between the woman"]

https://colab.research.google.com/drive/1SBQox9f1tXpPRFNSPYXh0HnwAgGSD1VK#scrollTo=IFnZSer2Mp0A

Qwen2-VL-2B-Instruct_cpu.ipynb

!pip install -U diffusers

!pip3 install torch torchvision torchaudio --index-url https://download.pytorch.org/whl/cpu

!pip uninstall -y tensorflow && pip install tensorflow-cpu

pip install qwen-vl-utils

pip install git+https://github.com/huggingface/transformers

colab tpu for big ram only but code work on cpu