Instructions to use yuxinlu1/gemma-4-12B-agentic-fable5-composer2.5-v2-3.5x-tau2 with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use yuxinlu1/gemma-4-12B-agentic-fable5-composer2.5-v2-3.5x-tau2 with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="yuxinlu1/gemma-4-12B-agentic-fable5-composer2.5-v2-3.5x-tau2")
messages = [
    {
        "role": "user",
        "content": [
            {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"},
            {"type": "text", "text": "What animal is on the candy?"}
        ]
    },
]
pipe(text=messages)

# Load model directly
from transformers import AutoProcessor, AutoModelForMultimodalLM

processor = AutoProcessor.from_pretrained("yuxinlu1/gemma-4-12B-agentic-fable5-composer2.5-v2-3.5x-tau2")
model = AutoModelForMultimodalLM.from_pretrained("yuxinlu1/gemma-4-12B-agentic-fable5-composer2.5-v2-3.5x-tau2", device_map="auto")
messages = [
    {
        "role": "user",
        "content": [
            {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"},
            {"type": "text", "text": "What animal is on the candy?"}
        ]
    },
]
inputs = processor.apply_chat_template(
	messages,
	add_generation_prompt=True,
	tokenize=True,
	return_dict=True,
	return_tensors="pt",
).to(model.device)

outputs = model.generate(**inputs, max_new_tokens=40)
print(processor.decode(outputs[0][inputs["input_ids"].shape[-1]:]))

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use yuxinlu1/gemma-4-12B-agentic-fable5-composer2.5-v2-3.5x-tau2 with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "yuxinlu1/gemma-4-12B-agentic-fable5-composer2.5-v2-3.5x-tau2"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "yuxinlu1/gemma-4-12B-agentic-fable5-composer2.5-v2-3.5x-tau2",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker

docker model run hf.co/yuxinlu1/gemma-4-12B-agentic-fable5-composer2.5-v2-3.5x-tau2

SGLang

How to use yuxinlu1/gemma-4-12B-agentic-fable5-composer2.5-v2-3.5x-tau2 with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "yuxinlu1/gemma-4-12B-agentic-fable5-composer2.5-v2-3.5x-tau2" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "yuxinlu1/gemma-4-12B-agentic-fable5-composer2.5-v2-3.5x-tau2",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "yuxinlu1/gemma-4-12B-agentic-fable5-composer2.5-v2-3.5x-tau2" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "yuxinlu1/gemma-4-12B-agentic-fable5-composer2.5-v2-3.5x-tau2",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Docker Model Runner
How to use yuxinlu1/gemma-4-12B-agentic-fable5-composer2.5-v2-3.5x-tau2 with Docker Model Runner:
```
docker model run hf.co/yuxinlu1/gemma-4-12B-agentic-fable5-composer2.5-v2-3.5x-tau2
```

Did You Forget to Upload the Model?

by llmfan46 - opened Jun 28

Discussion

llmfan46

Jun 28

Just wondering because it's been 12 hours since you created this repository but the model files have never been uploaded, so I was just wondering if you forgot to upload them or something?

Also congratulation on landing cooperative work with an AI Lab and good luck with the v3!

yuxinlu1

Owner Jun 28

Haha no, I didn't forget — the files exist, the upload is the problem. My home network (Verizon) is painfully slow for large files: I left the ~22GB safetensors master uploading all of last night and it still hasn't gone through. I'm heading to a coffee shop with a faster connection later today to push it up from there. Really sorry for the wait — it'll be up soon.

And thank you, genuinely — much appreciated on the AI Lab collaboration and the v3 wishes. More coming.

Constantinehw

Jun 29

不支持Apple的MLX格式？

timaidoitac

Jun 29

Haha no, I didn't forget — the files exist, the upload is the problem. My home network (Verizon) is painfully slow for large files: I left the ~22GB safetensors master uploading all of last night and it still hasn't gone through. I'm heading to a coffee shop with a faster connection later today to push it up from there. Really sorry for the wait — it'll be up soon.

And thank you, genuinely — much appreciated on the AI Lab collaboration and the v3 wishes. More coming.

we are appreciate thanks and waiting NVFP4 support at V3 comming soon

yuxinlu1

Owner Jul 5

NVFP4 (@timaidoitac ): On the roadmap and genuinely feasible — NVFP4 weight-only runs on Blackwell via vLLM. But to be straight so nobody's disappointed: it'll most likely land after v3, not bundled into the v3 release. I need to validate vLLM compatibility for this checkpoint before I quantize it, and I'd rather ship v3 (and the 27B) first, then do the vLLM/NVFP4 path properly. Not forgotten — just sequenced after.

yuxinlu1 changed discussion status to closed Jul 5

Upload images, audio, and videos by dragging in the text input, pasting, or clicking here.

Tap or paste here to upload images

· Sign up or log in to comment