Stage 1 checkpoint (projector warmup on LLaVA-CC3M-Pretrain-595K)

README.md

@@ -21,18 +21,33 @@ base_model:
 
 # Baguettotron-VLM — Stage 1 (Projector Warmup)
 
-**Stage 1 checkpoint.** Only the MLP projector has been trained (on LLaVA-CC3M-Pretrain-595K); the ViT and the Baguettotron LLM are the unmodified base weights. Published for reproducibility — for actual use prefer the Stage 2 or Stage 3 checkpoints.
+**Baguettotron-VLM** is an open, fully-reproducible, multilingual Vision-Language
+Model in the **sub-1B parameter** class. It extends
+[PleIAs/Baguettotron](https://huggingface.co/PleIAs/Baguettotron) — a 321M
+text-only reasoning SLM — with visual capabilities via the
+[InternViT-300M-448px-V2.5](https://huggingface.co/OpenGVLab/InternViT-300M-448px-V2_5)
+vision encoder and a lightweight MLP projector, for a total of ~628M parameters.
 
-Part of the [Baguettotron-VLM](https://github.com/andreagemelli/baguettotron-vlm)
-project: an open, reproducible, multilingual Vision-Language Model built by
-extending [PleIAs/Baguettotron](https://huggingface.co/PleIAs/Baguettotron)
-(321M reasoning SLM) with visual capabilities via
-[InternViT-300M-448px-V2.5](https://huggingface.co/OpenGVLab/InternViT-300M-448px-V2_5).
+It supports **six European languages** (EN, FR, DE, ES, IT, PL, inherited from
+the Baguettotron backbone), natively emits `<think>` reasoning traces, and was
+trained end-to-end on a single H100 SXM — code, data pipeline, weights, and
+training logs are all public. Everything is released under **Apache 2.0** with
+a proof-of-concept goal: exploring what an open sub-1B multilingual VLM can do,
+not chasing SOTA.
 
-Related checkpoints:
-- Stage 1 (projector warmup): [andreagemelli/Baguettotron-VLM-Stage1](https://huggingface.co/andreagemelli/Baguettotron-VLM-Stage1)
-- Stage 2 (instruction tuning): [andreagemelli/Baguettotron-VLM-Stage2](https://huggingface.co/andreagemelli/Baguettotron-VLM-Stage2)
-- Stage 3 (reasoning SFT, flagship): [andreagemelli/Baguettotron-VLM](https://huggingface.co/andreagemelli/Baguettotron-VLM)
+The project ships three checkpoints from a staged training curriculum:
+
+- **Stage 1 — projector warmup**: [andreagemelli/Baguettotron-VLM-Stage1](https://huggingface.co/andreagemelli/Baguettotron-VLM-Stage1)
+  (MLP projector only, LLaVA-CC3M-Pretrain-595K — published for reproducibility).
+- **Stage 2 — instruction tuning**: [andreagemelli/Baguettotron-VLM-Stage2](https://huggingface.co/andreagemelli/Baguettotron-VLM-Stage2)
+  (full fine-tune on The Cauldron + SYNTH, direct-answer mode).
+- **Stage 3 — reasoning SFT (flagship)**: [andreagemelli/Baguettotron-VLM](https://huggingface.co/andreagemelli/Baguettotron-VLM)
+  (reasoning traces via R1-Vision-Reasoning-Instructions, dynamic `<think>` toggle).
+
+Source & training pipeline: [github.com/andreagemelli/baguettotron-vlm](https://github.com/andreagemelli/baguettotron-vlm).
+
+
+> **Stage 1 checkpoint.** Only the MLP projector has been trained (on LLaVA-CC3M-Pretrain-595K); the ViT and the Baguettotron LLM are the unmodified base weights. Published for reproducibility — for actual use prefer the Stage 2 or Stage 3 checkpoints.
 
 ## Architecture
 
@@ -93,6 +108,49 @@ Stage 1 was trained on short image captions with no `<think>` traces. The proces
 | Hardware | 1× H100 SXM (RunPod) |
 | Duration | ~5h |
 
+## Project scope & limitations
+
+Baguettotron-VLM is a **self-educational, open-source proof of concept** trained
+on a limited compute budget. The goal has always been to explore what an open,
+fully-reproducible, multilingual sub-1B VLM can do when every design choice has
+to justify itself against a hard budget cap.
+
+Concretely, this means the model may have limitations:
+
+- **Likely undertrained for VQA alignment.** The Stage 2 instruction-tuning
+  budget (~60h on a single H100) is small relative to the data mixture; the
+  model has seen far fewer tokens than typical VLMs of comparable size.
+- **Hallucinations.** The Baguettotron backbone is pretrained on synthetic
+  corpora and reasoning traces, which gives it strong `<think>` behaviour but
+  also a tendency to confabulate details that aren't grounded in the image,
+  especially on fine-grained or text-heavy questions.
+- **Best on simple, scene-level prompts.** In informal testing the model is
+  most reliable at open-ended description and scene understanding
+  ("what is happening in this image?") and less reliable at OCR, counting,
+  chart reading, or multi-hop visual reasoning. Quantitative benchmark
+  numbers will be published once the evaluation harness lands.
+
+**Contributions and suggestions are very welcome** — issues, PRs, and ideas for
+better data mixes, training recipes, or evaluation setups are all appreciated.
+Open an issue or PR on the
+[GitHub repo](https://github.com/andreagemelli/baguettotron-vlm).
+
+
+## Citation
+
+If you use or extend Baguettotron-VLM in your research, please cite it:
+
+```bibtex
+@misc{gemelli2026baguettotronvlm,
+  title        = {Baguettotron-VLM: An Open, Reproducible, Multilingual Sub-1B Vision-Language Model},
+  author       = {Gemelli, Andrea},
+  year         = {2026},
+  howpublished = {\url{https://huggingface.co/andreagemelli/Baguettotron-VLM}},
+  note         = {Code: \url{https://github.com/andreagemelli/baguettotron-vlm}}
+}
+```
+
+
 ## License
 
 Apache 2.0 — see the [GitHub repo](https://github.com/andreagemelli/baguettotron-vlm).

modeling_baguettotron_vlm.py

@@ -83,7 +83,7 @@ class BaguettotronVLMForConditionalGeneration(PreTrainedModel):
         self.llm = AutoModelForCausalLM.from_pretrained(
             config.llm_model_id, dtype=torch.bfloat16
         )
-        self.llm.resize_token_embeddings(len(tokenizer))
+        self.llm.resize_token_embeddings(len(tokenizer), mean_resizing=False)
         # Break weight tying — safetensors rejects shared-storage tensors
         self.llm.lm_head.weight = nn.Parameter(self.llm.lm_head.weight.data.clone())
 
@@ -109,12 +109,13 @@ class BaguettotronVLMForConditionalGeneration(PreTrainedModel):
         inputs_embeds = self.llm.get_input_embeddings()(input_ids)
 
         if pixel_values is not None:
+            pixel_values = pixel_values.to(dtype=self.vit.dtype, device=self.vit.device)
             with torch.no_grad():
                 vit_out = self.vit(pixel_values)
             image_features = vit_out.last_hidden_state
             if image_features.shape[1] == self.config.vit_tokens + 1:
                 image_features = image_features[:, 1:, :]
-            visual_tokens = self.projector(image_features.float())
+            visual_tokens = self.projector(image_features.to(inputs_embeds.dtype))
             image_mask = input_ids == self.image_token_id
             inputs_embeds[image_mask] = visual_tokens.reshape(
                 -1, self.config.llm_hidden
@@ -146,11 +147,12 @@ class BaguettotronVLMForConditionalGeneration(PreTrainedModel):
         inputs_embeds = self.llm.get_input_embeddings()(input_ids)
 
         if pixel_values is not None:
+            pixel_values = pixel_values.to(dtype=self.vit.dtype, device=self.vit.device)
             vit_out = self.vit(pixel_values)
             image_features = vit_out.last_hidden_state
             if image_features.shape[1] == self.config.vit_tokens + 1:
                 image_features = image_features[:, 1:, :]
-            visual_tokens = self.projector(image_features.float())
+            visual_tokens = self.projector(image_features.to(inputs_embeds.dtype))
             image_mask = input_ids == self.image_token_id
             inputs_embeds[image_mask] = visual_tokens.reshape(
                 -1, self.config.llm_hidden