---
language:
- en
- fr
- de
- es
- it
- pl
license: apache-2.0
tags:
- vision-language-model
- multimodal
- visual-question-answering
- image-captioning
- vlm
base_model:
- OpenGVLab/InternViT-300M-448px-V2_5
- PleIAs/Baguettotron
---

# baguettotron-internvit-alignment

**Baguettotron-VLM** is an open, fully-reproducible, multilingual Vision-Language
Model in the **sub-1B parameter** class. It extends
[PleIAs/Baguettotron](https://huggingface.co/PleIAs/Baguettotron) — a 321M
text-only reasoning SLM — with visual capabilities via the
[InternViT-300M-448px-V2.5](https://huggingface.co/OpenGVLab/InternViT-300M-448px-V2_5)
vision encoder and a lightweight MLP projector, for a total of ~628M parameters.

Trained end-to-end in **~72h on a single H100 (~€219)**. It inherits six European
languages (EN, FR, DE, ES, IT, PL) from the Baguettotron backbone, and — unusually —
its entire LM pretraining corpus is auditable: Baguettotron was trained on
[SYNTH](https://huggingface.co/datasets/PleIAs/SYNTH), ~58K Wikipedia articles expanded
synthetically, with no scraped web.

Two checkpoints are published:

- [**baguettotron-vision-vqa**](https://huggingface.co/andreagemelli/baguettotron-vision-vqa)
  — instruction-tuned on The Cauldron. **This is the model you want.**
- [baguettotron-internvit-alignment](https://huggingface.co/andreagemelli/baguettotron-internvit-alignment)
  — projector-only warmup, published so the alignment stage can be reproduced.

Apache 2.0. This is a proof of concept and a costed recipe, not a SOTA attempt.
Source: [github.com/andreagemelli/baguettotron-vlm](https://github.com/andreagemelli/baguettotron-vlm).


> **Alignment checkpoint.** Only the MLP projector was trained (on LLaVA-CC3M-Pretrain-595K) — the ViT and the Baguettotron LLM are unmodified base weights. It writes short captions and nothing more. Published so the alignment stage can be reproduced; for actual use take [baguettotron-vision-vqa](https://huggingface.co/andreagemelli/baguettotron-vision-vqa).

## Architecture

```
Image (448×448)
  → InternViT-300M-448px-V2.5 (304M, frozen) → 1024 tokens × 1024d
  → Pixel unshuffle (factor=2)                → 256 tokens × 4096d
  → MLP projector (2-layer, ~2.7M)            → 256 tokens × 576d
  → Interleave with text tokens
  → Baguettotron (321M, Llama arch, 80L, h=576)

Total: ~628M parameters
```

## Usage

```bash
pip install "transformers>=4.56,<5" torch pillow timm einops accelerate
```

```python
import torch
from transformers import AutoModelForImageTextToText, AutoProcessor
from PIL import Image

model = AutoModelForImageTextToText.from_pretrained(
    "andreagemelli/baguettotron-internvit-alignment",
    trust_remote_code=True,
    dtype=torch.bfloat16,
    device_map="auto",
)
processor = AutoProcessor.from_pretrained(
    "andreagemelli/baguettotron-internvit-alignment",
    trust_remote_code=True,
)

image = Image.open("photo.jpg").convert("RGB")
inputs = processor(
    messages=[{"role": "user", "content": "<image>\nDescribe the image concisely."}],
    image=image,
)
inputs = {k: v.to(model.device) for k, v in inputs.items() if v is not None}

print(model.chat(**inputs))
```

`chat()` returns the answer as a plain string, already trimmed at the end of the
assistant turn. `max_new_tokens=...` bounds the length.

### Apple Silicon and CPU

The snippet above runs unchanged on `mps` and `cpu` — keep `device_map="auto"`, or drop
it and call `.to("mps")` yourself. `dtype=torch.float32` works too. A short answer takes
~1-3s on `mps` against ~20-25s on CPU, so prefer `mps` on a Mac.

One caveat handled for you: transformers' repetition-penalty processor corrupts the
first decoding step on MPS, which produced a garbage first token. `chat()` disables the
penalty on that backend automatically.

### Example output

Four COCO images, greedy, `Describe the image concisely.` — verbatim:

| image | output |
|---|---|
| two cats on a couch | `a cat is sleeping on the couch` |
| close-up of a brown bear | `the bear is a good friend.` |
| an upside-down STOP sign | `a sign for a stop sign` |
| a red double-decker bus | `the bus is a red double - decoration` |

That is the honest range of this checkpoint: it locates the subject and then drifts, which is what 2.7M trained parameters buys. It does not read the text on the sign or the bus.

### Chat template

Trained on short image captions with no `<think>` traces. The processor emits a bare assistant prefix (`<|im_start|>assistant\n`) and the model completes the caption directly. Keep prompts simple ("Describe the image").

## What the evaluation found

Measured on 6 held-out [RealWorldQA](https://huggingface.co/datasets/xai-org/RealworldQA)
images, judged 1–5 by a vision judge, 244 of 288 cells completed. Thin, and reported as
such — full tables and caveats in
[RESULTS.md](https://github.com/andreagemelli/baguettotron-vlm/blob/main/.planning/RESULTS.md).

- **Short factual questions are its strength** (2.5–3.5), open description its weakest
  (~1.5). That inverted our own assumption that this would be a describer.
- **A third checkpoint made things worse.** A reasoning-SFT stage on top of the
  instruction-tuned model scored 1.25 against its 2.17, roughly 3 SE. The reasoning
  traces are undertrained for imagery and mostly add fluent hallucination, so that
  checkpoint is **not published**. It is kept as a recorded negative result.
- **Decoding presets are inside noise** — greedy 2.17 vs 1.88 for the other three.

### Decoding

Use **greedy**, which is the default. It is recommended for determinism, not because
it beat anything. Pass `do_sample=True, temperature=...` to `chat()` if you want
sampling anyway.


## Training

| | |
|---|---|
| Data | LLaVA-CC3M-Pretrain-595K (595K image-caption pairs) |
| Trainable params | ~2.7M (projector only) |
| Frozen | ViT + LLM |
| Effective batch size | 256 |
| Learning rate | 1e-3, cosine, 250-step warmup |
| Precision | bf16 |
| Hardware | 1× H100 SXM (RunPod) |
| Duration | ~5h (~€15) |

## Limitations

- **No stop token.** `<|im_end|>` was masked out of the training loss, so the model
  never learned to emit one — it spells the turn marker out as ordinary text instead.
  `chat()` works around this by stopping on that text (a 512-token call drops from
  ~21s to ~1.5s), but the defect is in the weights and needs a retrain to fix.
- **Resolution ceiling.** One 448×448 crop → 256 visual tokens puts document text at
  roughly 2–4 px/char. OCR, charts and documents are out of reach by architecture, not
  by budget. Neither published checkpoint reads text in an image.
- **Hallucinations**, especially on fine-grained or text-heavy questions.
- **Undertrained for VQA alignment** — ~60h of instruction tuning on one H100 is small
  relative to the data mixture.
- **Multilingual capability is inherited, not verified.** The backbone covers six
  languages; the VLM was never evaluated on non-English benchmarks.
- **Evaluation is thin**: 6 held-out images, one judge, one rubric, 244 judged cells.
  Every number here is directional.
- **`transformers` 5.x cannot load this model.** The blocker is upstream in InternViT's
  remote code, which predates v5's tied-weight API. Pin `transformers<5`.

**Contributions and suggestions are very welcome** — issues, PRs, and ideas for
better data mixes, training recipes, or evaluation setups are all appreciated.
Open an issue or PR on the
[GitHub repo](https://github.com/andreagemelli/baguettotron-vlm).


## Citation

If you use or extend Baguettotron-VLM in your research, please cite it:

```bibtex
@misc{gemelli2026baguettotronvlm,
  title        = {Baguettotron-VLM: An Open, Reproducible, Multilingual Sub-1B Vision-Language Model},
  author       = {Gemelli, Andrea},
  year         = {2026},
  howpublished = {\url{https://huggingface.co/andreagemelli/baguettotron-vision-vqa}},
  note         = {Code: \url{https://github.com/andreagemelli/baguettotron-vlm}}
}
```


## License

Apache 2.0 — see the [GitHub repo](https://github.com/andreagemelli/baguettotron-vlm).