fatsam13/beforeat-food-nutrition-vision-lora

Beforeat Food Nutrition Vision LoRA

This repository contains one LoRA adapter for Qwen/Qwen3-VL-4B-Instruct in PEFT and llama.cpp GGUF formats. Version 2 prioritizes Food-101 dish recognition and adds experimental visible-ingredient and rough-gram output.

Nutrition totals are not predicted directly by this adapter. Beforeat uses the recognized ingredients and estimated portions as inputs to a separate USDA FoodData Central lookup and rules layer for calories and macronutrients.

Current Files

• adapter_model.safetensors: v2 PEFT weights
• adapter_config.json: rank 8, alpha 16 PEFT configuration
• beforeat-food-nutrition-vision-lora-v2-f16.gguf: v2 F16 llama.cpp adapter
• blend_metadata.json: reproducible v2 weight-interpolation metadata
• beforeat-food-nutrition-vision-lora-f16.gguf: historical v1 GGUF release

Only one adapter is loaded at runtime. The v1 file remains available for rollback; new integrations should use the versioned v2 file.

Prompts

Dish recognition:

Identify the dish in this image. Reply with only the exact Food-101 class name.

Experimental visible ingredients and portions:

Estimate the visible ingredients and their portions in grams. Reply only with a JSON array of objects using keys "name" and "grams".

Evaluation

Task	Evaluation	Result
Food-101 dish recognition	500-image held-out set, strict	438/500 (87.60%)
Food-101 dish recognition	Alias/near matching	440/500 (88.00%)
Visible ingredients	Nutrition5k, 100 images	44.51% mean recall
Matched ingredient portions	Nutrition5k, 100 images	52.93% mean recall
Matched ingredient grams	Nutrition5k, 85 parsed rows	43.34 g MAE

The previous food-only release scored 434/500 (86.80%) strict and 438/500 (87.60%) with aliases on the same Food-101 evaluation.

The ingredient/portion results are experimental and should not be interpreted as direct nutrition accuracy. Hidden ingredients, cooking oils, recipe details, and true serving mass cannot be reliably recovered from one image.

Adapter Construction

The v2 weights are a linear interpolation between the 2,500-step Food-101 LoRA and the continued 4,000-step ingredient/portion LoRA. The nutrition checkpoint weight is 0.25. Both inputs use the same rank, alpha, tensor names, and target modules, producing a single 504-tensor LoRA adapter.

• Rank: 8
• Alpha: 16
• Target modules: q_proj, k_proj, v_proj, o_proj, gate_proj, up_proj, down_proj

llama.cpp GGUF Adapter

• Base model: Qwen/Qwen3-VL-4B-Instruct
• Required base GGUF: Qwen3-VL-4B-Instruct-UD-Q4_K_XL.gguf
• Base GGUF source: unsloth/Qwen3-VL-4B-Instruct-GGUF
• Required projector: mmproj-F16.gguf from the same base GGUF repository
• Recommended adapter scale: 1.0
• llama.cpp commit: 00fa7cb284cbf133fc426733bd64238a3588a33e
• GGUF tensor count: 504
• GGUF file size: 33065152 bytes
• SHA-256: ac31fb420d91fd8a4149e694c4da540db6208889c53f86b383069a6422a01f62

Conversion command:

PYTHONPATH=tools/llama.cpp \
python tools/llama.cpp/convert_lora_to_gguf.py \
  adapters/qwen3vl4b_food_nutrition_blend_w025 \
  --base models/Qwen3-VL-4B-Instruct-bf16-remapped \
  --outfile adapters/qwen3vl4b_food_nutrition_blend_w025/beforeat-food-nutrition-vision-lora-v2-f16.gguf \
  --outtype f16 \
  --verbose

Runtime command:

llama-mtmd-cli \
  -m Qwen3-VL-4B-Instruct-UD-Q4_K_XL.gguf \
  --mmproj mmproj-F16.gguf \
  --image food.jpg \
  --lora-scaled beforeat-food-nutrition-vision-lora-v2-f16.gguf:1.0 \
  -p "Identify the dish in this image. Reply with only the exact Food-101 class name." \
  --jinja --temp 0 -n 16 -c 4096 --image-min-tokens 1024

Validation used the same command with --device none -ngl 0 --no-mmproj-offload --no-op-offload --no-kv-offload --fit off after the local Metal command queue was unavailable. llama.cpp accepted all adapter tensors with no missing, incompatible, or unexpected tensor errors. The Unsloth UD-Q4_K_XL base and its existing mmproj-F16.gguf are compatible and reusable.

Runtime Spot Checks

Expected	PyTorch/PEFT v2	Base GGUF	GGUF + v2 LoRA
prime rib	prime rib	roast beef	prime rib
bread pudding	panna cotta	dessert	panna cotta
chocolate cake	chocolate mousse	Chocolate Cake	chocolate mousse

The matching PEFT and GGUF+LoRA predictions demonstrate conversion fidelity, including the same errors.

Limitations and Use

• Food recognition is specialized around the 101 Food-101 classes and may not generalize to arbitrary dishes, brands, mixed plates, or regional variants.
• Ingredient names and grams are rough visual estimates, not measured portions.
• Use a separate food-composition database and calculation layer for nutrition.
• Food-101 and Nutrition5k were used for research and prototyping; review their terms before commercial training or redistribution workflows.
• This adapter is not medical or dietary advice.