ilia-dybal/Qwen-2.5-VL-3b-Instruct-4WA16-GPTQ-Culinary

Model Card for Model ID

The repository contains a finetuned version of Qwen-2.5-VL-3B-Instruct model. The model was finetuned on custom vision dataset specialized for culinary tasks: structured receipe and engredients extraction.

The model was initially finetuned in a A100 GPU with 4-bit QLoRA and post-quantized with llm-compressor using W6A16 scheme and GPTQ algorithm. The vision encoder retained untouched to preserce dynamic resolution capabilities.

Model Details

Model Description

• Developed by: Ilia Dybal
• Funded by: Ilia Dybal
• Shared by: Ilia Dybal
• Model type: Multimodal Large Visual Language Model (LVLM)
• Language(s) (NLP): Russian
• License: Apache 2.0
• Finetuned from model: Qwen/Qwen2.5-VL-3B-Instruct

Model Sources [optional]

• Repository: Qwen/Qwen2.5-VL-3B-Instruct
• Paper: https://arxiv.org/pdf/2502.13923

Uses

Direct Use

• Structured receipe and ingredients extraction from an image.

Out-of-Scope Use

• Medical or clinical dietary diagnostics.
• General non-culinary visual analysis (for general VLM tasks, use the base Qwen2.5-VL-3B-Instruct).

Bias, Risks, and Limitations

[More Information Needed]

Recommendations

Users (both direct and downstream) should be made aware of the risks, biases and limitations of the model. More information needed for further recommendations.

How to Get Started with the Model

Use the code below to get started with the model.

Option 1: Fast Serving via vLLM (Recommended for Production)

Launch the vLLM server from your terminal:

vllm serve ilia-dybal/Qwen-2.5-VL-3b-Instruct-4bit-LoRA-culinary \
    --trust-remote-code \
    --max-model-len 4096 \
    --limit-mm-per-prompt '{"image": 1}'

Query the server using the OpenAI Python Client:

import base64
from openai import OpenAI

client = OpenAI(base_url="http://localhost:8000/v1", api_key="EMPTY")

def encode_image(image_path):
    with open(image_path, "rb") as f:
        return f"data:image/jpeg;base64,{base64.b64encode(f.read()).decode('utf-8')}"

response = client.chat.completions.create(
    model="ilia-dybal/Qwen-2.5-VL-3b-Instruct-4bit-LoRA-culinary",
    messages=[
        {
            "role": "user",
            "content": [
                {"type": "text", "text": "Analyze this dish and provide the recipe."},
                {"type": "image_url", "image_url": {"url": encode_image("dish.jpg")}}
            ]
        }
    ],
    max_tokens=512
)

print(response.choices[0].message.content)

Option 2: Native Hugging Face Transformers

import torch
from transformers import AutoProcessor, Qwen2_5_VLForConditionalGeneration

model_id = "ilia-dybal/Qwen-2.5-VL-3b-Instruct-4bit-LoRA-culinary"

processor = AutoProcessor.from_pretrained(model_id)
model = Qwen2_5_VLForConditionalGeneration.from_pretrained(
    model_id,
    torch_dtype=torch.bfloat16,
    device_map="auto"
)

# Prepare multimodal input
messages = [
    {
        "role": "user",
        "content": [
            {"type": "image", "image": "[https://example.com/dish.jpg](https://example.com/dish.jpg)"},
            {"type": "text", "text": "List the key ingredients in this photo."}
        ]
    }
]

text = processor.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
inputs = processor(text=[text], images=None, return_tensors="pt").to("cuda")

output_ids = model.generate(**inputs, max_new_tokens=512)
print(processor.decode(output_ids[0], skip_special_tokens=True))

Training Hyperparameters

• Training regime:
  • per_device_train_batch_size=16,
  • per_device_eval_batch_size=16,
  • gradient_accumulation_steps=4,
  • num_train_epochs=15,
  • gradient_checkpointing=True,
  • gradient_checkpointing_kwargs={"use_reentrant": False},
  • logging_strategy="steps",
  • logging_steps=3,
  • eval_strategy="epoch",
  • bf16=True,
  • logging_first_step=True,
  • max_length=2048,
  • learning_rate=5e-4,

Carbon emissions can be estimated using the Machine Learning Impact calculator presented in Lacoste et al. (2019).

• Hardware Type: 1xNVIDIA A100(40Gb)
• Hours used: 1:21:17 (15 epochs)
• Cloud Provider: Google Colab