Hi @zeeshanfarooq6 ,

I noticed you're having an issue with getting the image to work with gemma-4-E4B-it-Q4_K_S.gguf in llama_cpp – the model can't retrieve the provided image for inference. I've actually solved the problem of loading images and performing image understanding/inference with llama_cpp for Gemma 4 Vision.

You can refer to my code and detailed implementation here:
https://github.com/askxiaozhang/Gemma4-Vision-Server

It should help you resolve the image retrieval and visual reasoning issue you're facing!

Hi @zeeshanfarooq6 ,

I ran into the same issue about a year ago when Gemma 3 was released. I was trying to use the llama.cpp GGUF quants through llama-cpp-python, but Gemma couldn’t process images properly.

What worked much better for me was using llama-server directly.

First, download the llama.cpp binaries from:
llama.cpp releases

I’m on Windows and only use CPU inference, so I downloaded the Windows x64 (CPU) version.

You can start the server in two ways:

1. Using -hf (easiest method)

.\path-to-llama-server.exe -hf unsloth/gemma-4-E4B-it-GGUF:Q4_K_S --port 8080 --alias "gemma-4-E4B-it"

This hosts:

• an OpenAI-compatible API on localhost:8080
• and a built-in web UI

If you only want the API, you can add:

--no-webui

Then you can use it in Python like this:

from openai import OpenAI
import base64

client = OpenAI(
    api_key="no-api-key-needed-here",
    base_url="http://localhost:8080/v1"
)

# Read image and convert to base64
with open("image.png", "rb") as f:
    image_b64 = base64.b64encode(f.read()).decode("utf-8")

response = client.chat.completions.create(
    model="gemma-4-E4B-it",  # use the alias here
    messages=[
        {
            "role": "user",
            "content": [
                {
                    "type": "text",
                    "text": "Describe this image."
                },
                {
                    "type": "image_url",
                    "image_url": {
                        "url": f"data:image/png;base64,{image_b64}"
                    }
                }
            ]
        }
    ]
)

print(response.choices[0].message.content)

2. Download the model + mmproj manually

Download:

• the model .gguf
• and the mmproj.gguf file
  (you can use the BF16 version for the mmproj)

The mmproj file enables multimodal support (images/audio).

Then start the server like this:

.\path-to-llama-server.exe -m model.gguf -mmproj mmproj.gguf --port 8080 --alias "gemma-4-E4B-it"

After that, you can access it through the OpenAI-compatible API on localhost:8080.

Hope this helps :)