--- language: - en - fr - de - es - it - pl license: apache-2.0 tags: - vision-language-model - multimodal - visual-question-answering - image-captioning - vlm base_model: - OpenGVLab/InternViT-300M-448px-V2_5 - PleIAs/Baguettotron --- # baguettotron-internvit-alignment **Baguettotron-VLM** is an open, fully-reproducible, multilingual Vision-Language Model in the **sub-1B parameter** class. It extends [PleIAs/Baguettotron](https://huggingface.co/PleIAs/Baguettotron) — a 321M text-only reasoning SLM — with visual capabilities via the [InternViT-300M-448px-V2.5](https://huggingface.co/OpenGVLab/InternViT-300M-448px-V2_5) vision encoder and a lightweight MLP projector, for a total of ~628M parameters. It inherits six European languages (EN, FR, DE, ES, IT, PL) from the Baguettotron backbone, and — unusually — its entire LM pretraining corpus is auditable: Baguettotron was trained on [SYNTH](https://huggingface.co/datasets/PleIAs/SYNTH), ~58K Wikipedia articles expanded synthetically, with no scraped web. Two checkpoints are published: - [**baguettotron-vision-vqa**](https://huggingface.co/andreagemelli/baguettotron-vision-vqa) — instruction-tuned on The Cauldron. **This is the model you want.** - [baguettotron-internvit-alignment](https://huggingface.co/andreagemelli/baguettotron-internvit-alignment) — projector-only warmup, published so the alignment stage can be reproduced. Apache 2.0. This is a proof of concept and a costed recipe, not a SOTA attempt. Source: [github.com/andreagemelli/baguettotron-vlm](https://github.com/andreagemelli/baguettotron-vlm). > **Alignment checkpoint.** Only the MLP projector was trained (on LLaVA-CC3M-Pretrain-595K) — the ViT and the Baguettotron LLM are unmodified base weights. It writes short captions and nothing more. Published so the alignment stage can be reproduced; for actual use take [baguettotron-vision-vqa](https://huggingface.co/andreagemelli/baguettotron-vision-vqa). ## Architecture ``` Image (448×448) → InternViT-300M-448px-V2.5 (304M, frozen) → 1024 tokens × 1024d → Pixel unshuffle (factor=2) → 256 tokens × 4096d → MLP projector (2-layer, ~2.7M) → 256 tokens × 576d → Interleave with text tokens → Baguettotron (321M, Llama arch, 80L, h=576) Total: ~628M parameters ``` ## Usage ```bash pip install "transformers>=4.56,<5" torch pillow timm einops accelerate ``` ```python import torch from transformers import AutoModelForImageTextToText, AutoProcessor from PIL import Image model = AutoModelForImageTextToText.from_pretrained( "andreagemelli/baguettotron-internvit-alignment", trust_remote_code=True, dtype=torch.bfloat16, device_map="auto", # also tested on Apple Silicon (mps) and CPU ) processor = AutoProcessor.from_pretrained( "andreagemelli/baguettotron-internvit-alignment", trust_remote_code=True, ) image = Image.open("photo.jpg").convert("RGB") inputs = processor( messages=[{"role": "user", "content": "\nDescribe the image concisely."}], image=image, ) inputs = {k: v.to(model.device) for k, v in inputs.items() if v is not None} print(model.chat(**inputs)) ``` ### Example output Greedy, prompt `Describe the image concisely.` — verbatim output: | | output | |---|---| |

| `a cat is sleeping on the couch` | |

| `the bear is a good friend.` | |

| `a sign for a stop sign` | |

| `the bus is a red double - decoration` | That is the honest range of this checkpoint: it locates the subject and then drifts, which is what 2.7M trained parameters buys. It does not read the text on the sign or the bus. ### Chat template Trained on short image captions with no `` traces. The processor emits a bare assistant prefix (`<|im_start|>assistant\n`) and the model completes the caption directly. Keep prompts simple ("Describe the image"). ## Recommended settings Greedy decoding, which is what `chat()` does by default — just call it. Four decoding presets were compared on held-out images and the differences sat inside noise, so greedy is preferred for being deterministic rather than for scoring better. Two things that *do* move the needle: - **Ask specific questions.** `"How many cats are there?"` answers `2.`; the vaguer `"What is in the picture?"` answers `Yes.` Phrasing matters far more at this scale than it does with a large model. - **Describe first, then ask.** A follow-up question in a second turn is answered better than the same question asked cold, because the model conditions on its own previous answer. Multi-turn is in-distribution — The Cauldron contains multi-turn conversations. ## Limitations - **Resolution ceiling.** One 448×448 crop → 256 visual tokens puts document text at roughly 2–4 px/char. OCR, charts and documents are out of reach by architecture, not by budget. Neither published checkpoint reads text in an image. - **Hallucinations**, especially on fine-grained or text-heavy questions. - **Undertrained for VQA alignment** relative to the size of the data mixture. - **Multilingual capability is inherited, not verified.** The backbone covers six languages; the VLM was never evaluated on non-English benchmarks. > Tested against `transformers` 4.57. Newer major versions may need adjustments. **Contributions and suggestions are very welcome** — issues, PRs, and ideas for better data mixes, training recipes, or evaluation setups are all appreciated. Open an issue or PR on the [GitHub repo](https://github.com/andreagemelli/baguettotron-vlm). ## Citation If you use or extend Baguettotron-VLM in your research, please cite it: ```bibtex @misc{gemelli2026baguettotronvlm, title = {Baguettotron-VLM: An Open, Reproducible, Multilingual Sub-1B Vision-Language Model}, author = {Gemelli, Andrea}, year = {2026}, howpublished = {\url{https://huggingface.co/andreagemelli/baguettotron-vision-vqa}}, note = {Code: \url{https://github.com/andreagemelli/baguettotron-vlm}} } ``` ## License Apache 2.0 — see the [GitHub repo](https://github.com/andreagemelli/baguettotron-vlm).