---
language:
- en
- fr
- de
- es
- it
- pl
license: apache-2.0
tags:
- vision-language-model
- multimodal
- visual-question-answering
- image-captioning
- vlm

base_model:
- OpenGVLab/InternViT-300M-448px-V2_5
- PleIAs/Baguettotron
---

# Baguettotron-VLM — Stage 1 (Projector Warmup)

**Stage 1 checkpoint.** Only the MLP projector has been trained (on LLaVA-CC3M-Pretrain-595K); the ViT and the Baguettotron LLM are the unmodified base weights. Published for reproducibility — for actual use prefer the Stage 2 or Stage 3 checkpoints.

Part of the [Baguettotron-VLM](https://github.com/andreagemelli/baguettotron-vlm)
project: an open, reproducible, multilingual Vision-Language Model built by
extending [PleIAs/Baguettotron](https://huggingface.co/PleIAs/Baguettotron)
(321M reasoning SLM) with visual capabilities via
[InternViT-300M-448px-V2.5](https://huggingface.co/OpenGVLab/InternViT-300M-448px-V2_5).

Related checkpoints:
- Stage 1 (projector warmup): [andreagemelli/Baguettotron-VLM-Stage1](https://huggingface.co/andreagemelli/Baguettotron-VLM-Stage1)
- Stage 2 (instruction tuning): [andreagemelli/Baguettotron-VLM-Stage2](https://huggingface.co/andreagemelli/Baguettotron-VLM-Stage2)
- Stage 3 (reasoning SFT, flagship): [andreagemelli/Baguettotron-VLM](https://huggingface.co/andreagemelli/Baguettotron-VLM)

## Architecture

```
Image (448×448)
  → InternViT-300M-448px-V2.5 (304M, frozen) → 1024 tokens × 1024d
  → Pixel unshuffle (factor=2)                → 256 tokens × 4096d
  → MLP projector (2-layer, ~2.7M)            → 256 tokens × 576d
  → Interleave with text tokens
  → Baguettotron (321M, Llama arch, 80L, h=576)
  → Text output with <think> reasoning traces

Total: ~628M parameters
```

## Usage

```python
import torch
from transformers import AutoModelForImageTextToText, AutoProcessor
from PIL import Image

model = AutoModelForImageTextToText.from_pretrained(
    "andreagemelli/Baguettotron-VLM-Stage1",
    trust_remote_code=True,
    torch_dtype=torch.bfloat16,
    device_map="auto",
)
processor = AutoProcessor.from_pretrained(
    "andreagemelli/Baguettotron-VLM-Stage1",
    trust_remote_code=True,
)

image = Image.open("photo.jpg").convert("RGB")
inputs = processor(
    messages=[{"role": "user", "content": "<image>\nDescribe the image"}],
    image=image,
)
inputs = {k: v.to(model.device) for k, v in inputs.items() if v is not None}

print(model.chat(**inputs))
```

### Chat template

Stage 1 was trained on short image captions with no `<think>` traces. The processor emits a bare assistant prefix (`<|im_start|>assistant\n`) and the model completes the caption directly. Keep prompts simple ("Describe the image").

## Training details

| | Stage 1 |
|---|---|
| Data | LLaVA-CC3M-Pretrain-595K (595K image-caption pairs) |
| Trainable params | ~2.7M (projector only) |
| Frozen | ViT + LLM |
| Effective batch size | 256 |
| Learning rate | 1e-3, cosine, 250-step warmup |
| Precision | bf16 |
| Hardware | 1× H100 SXM (RunPod) |
| Duration | ~5h |

## License

Apache 2.0 — see the [GitHub repo](https://github.com/andreagemelli/baguettotron-vlm).