Instructions to use bartowski/google_medgemma-27b-it-GGUF with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use bartowski/google_medgemma-27b-it-GGUF with llama-cpp-python:

# !pip install llama-cpp-python

from llama_cpp import Llama

llm = Llama.from_pretrained(
	repo_id="bartowski/google_medgemma-27b-it-GGUF",
	filename="google_medgemma-27b-it-IQ2_M.gguf",
)

llm.create_chat_completion(
	messages = [
		{
			"role": "user",
			"content": [
				{
					"type": "text",
					"text": "Describe this image in one sentence."
				},
				{
					"type": "image_url",
					"image_url": {
						"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
					}
				}
			]
		}
	]
)

Notebooks
Google Colab
Kaggle
Local Apps Settings

llama.cpp

How to use bartowski/google_medgemma-27b-it-GGUF with llama.cpp:

Install from brew

brew install llama.cpp
# Start a local OpenAI-compatible server with a web UI:
llama-server -hf bartowski/google_medgemma-27b-it-GGUF:Q4_K_M
# Run inference directly in the terminal:
llama-cli -hf bartowski/google_medgemma-27b-it-GGUF:Q4_K_M

Install from WinGet (Windows)

winget install llama.cpp
# Start a local OpenAI-compatible server with a web UI:
llama-server -hf bartowski/google_medgemma-27b-it-GGUF:Q4_K_M
# Run inference directly in the terminal:
llama-cli -hf bartowski/google_medgemma-27b-it-GGUF:Q4_K_M

Use pre-built binary

# Download pre-built binary from:
# https://github.com/ggerganov/llama.cpp/releases
# Start a local OpenAI-compatible server with a web UI:
./llama-server -hf bartowski/google_medgemma-27b-it-GGUF:Q4_K_M
# Run inference directly in the terminal:
./llama-cli -hf bartowski/google_medgemma-27b-it-GGUF:Q4_K_M

Build from source code

git clone https://github.com/ggerganov/llama.cpp.git
cd llama.cpp
cmake -B build
cmake --build build -j --target llama-server llama-cli
# Start a local OpenAI-compatible server with a web UI:
./build/bin/llama-server -hf bartowski/google_medgemma-27b-it-GGUF:Q4_K_M
# Run inference directly in the terminal:
./build/bin/llama-cli -hf bartowski/google_medgemma-27b-it-GGUF:Q4_K_M

Use Docker

docker model run hf.co/bartowski/google_medgemma-27b-it-GGUF:Q4_K_M

LM Studio
Jan

vLLM

How to use bartowski/google_medgemma-27b-it-GGUF with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "bartowski/google_medgemma-27b-it-GGUF"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "bartowski/google_medgemma-27b-it-GGUF",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Use Docker

docker model run hf.co/bartowski/google_medgemma-27b-it-GGUF:Q4_K_M

Ollama
How to use bartowski/google_medgemma-27b-it-GGUF with Ollama:
```
ollama run hf.co/bartowski/google_medgemma-27b-it-GGUF:Q4_K_M
```

Unsloth Studio

How to use bartowski/google_medgemma-27b-it-GGUF with Unsloth Studio:

Install Unsloth Studio (macOS, Linux, WSL)

curl -fsSL https://unsloth.ai/install.sh | sh
# Run unsloth studio
unsloth studio -H 0.0.0.0 -p 8888
# Then open http://localhost:8888 in your browser
# Search for bartowski/google_medgemma-27b-it-GGUF to start chatting

Install Unsloth Studio (Windows)

irm https://unsloth.ai/install.ps1 | iex
# Run unsloth studio
unsloth studio -H 0.0.0.0 -p 8888
# Then open http://localhost:8888 in your browser
# Search for bartowski/google_medgemma-27b-it-GGUF to start chatting

Using HuggingFace Spaces for Unsloth

# No setup required
# Open https://huggingface.co/spaces/unsloth/studio in your browser
# Search for bartowski/google_medgemma-27b-it-GGUF to start chatting

Atomic Chat new
Docker Model Runner
How to use bartowski/google_medgemma-27b-it-GGUF with Docker Model Runner:
```
docker model run hf.co/bartowski/google_medgemma-27b-it-GGUF:Q4_K_M
```

Lemonade

How to use bartowski/google_medgemma-27b-it-GGUF with Lemonade:

Pull the model

# Download Lemonade from https://lemonade-server.ai/
lemonade pull bartowski/google_medgemma-27b-it-GGUF:Q4_K_M

Run and chat with the model

lemonade run user.google_medgemma-27b-it-GGUF-Q4_K_M

List all available models

lemonade list

no mmproj to run vision?

by alx-d - opened 7 days ago

Discussion

alx-d

7 days ago

I am trying to analyze an image file with medgemma 4q and I get this response:
"I am unable to access local files or specific URLs, including the one you
provided (/home/alex/img/path/diab.jpg). Therefore, I cannot
analyze the retinal fundus image and provide a clinical grading for
diabetic retinopathy or other pathologies."
To get an analysis, you would need to:

Upload the image: Share the image file directly with me (if
possible through the platform) or describe its features
in detail^C>>> /exit"

With the help of Claude, I found out that a mmproj file is missing?

Claude:
Bartowski hasn't included the mmproj in his 27b GGUF repo. The 27B multimodal variant provides all the capabilities of the 4B multimodal variant but with significantly improved language capabilities as well as improved EHR understanding and anatomy localization. arxiv
So the situation is:

✅ Google released medgemma-27b multimodal (vision capable)
✅ Bartowski quantized it to GGUF
❌ Bartowski did not include the mmproj file in the repo — so you can't use its vision capabilities via llama.cpp/Ollama

bartowski

Owner 7 days ago

They are in the repo though?

https://huggingface.co/bartowski/google_medgemma-27b-it-GGUF/blob/main/mmproj-google_medgemma-27b-it-bf16.gguf

alx-d

7 days ago

Thank you! I have found them.
Sorry, I didn't click on the files, I only looked on the right side.
Perhaps it would help to list them in the description so that it shows up in a search of "mmproj" which I have done and probably also Claude and Gemini who have advised wrongly

Upload images, audio, and videos by dragging in the text input, pasting, or clicking here.

Tap or paste here to upload images

· Sign up or log in to comment