andreagemelli commited on
Commit
bdd4301
Β·
verified Β·
1 Parent(s): d651d1c

Fix generation from the published artifact; rewrite model card

Browse files
.gitattributes CHANGED
@@ -33,3 +33,7 @@ saved_model/**/* filter=lfs diff=lfs merge=lfs -text
33
  *.zip filter=lfs diff=lfs merge=lfs -text
34
  *.zst filter=lfs diff=lfs merge=lfs -text
35
  *tfevents* filter=lfs diff=lfs merge=lfs -text
 
 
 
 
 
33
  *.zip filter=lfs diff=lfs merge=lfs -text
34
  *.zst filter=lfs diff=lfs merge=lfs -text
35
  *tfevents* filter=lfs diff=lfs merge=lfs -text
36
+ examples/bear.jpg filter=lfs diff=lfs merge=lfs -text
37
+ examples/bus.jpg filter=lfs diff=lfs merge=lfs -text
38
+ examples/cats.jpg filter=lfs diff=lfs merge=lfs -text
39
+ examples/stop.jpg filter=lfs diff=lfs merge=lfs -text
README.md CHANGED
@@ -25,22 +25,20 @@ Model in the **sub-1B parameter** class. It extends
25
  [PleIAs/Baguettotron](https://huggingface.co/PleIAs/Baguettotron) β€” a 321M
26
  text-only reasoning SLM β€” with visual capabilities via the
27
  [InternViT-300M-448px-V2.5](https://huggingface.co/OpenGVLab/InternViT-300M-448px-V2_5)
28
- vision encoder and a lightweight MLP projector, for a total of ~628M parameters.
29
-
30
- It inherits six European languages (EN, FR, DE, ES, IT, PL) from the Baguettotron
31
- backbone, and β€” unusually β€” its entire LM pretraining corpus is auditable: Baguettotron
32
- was trained on [SYNTH](https://huggingface.co/datasets/PleIAs/SYNTH), ~58K Wikipedia
33
- articles expanded synthetically, with no scraped web.
34
 
35
  Two checkpoints are published:
36
 
37
- - [**baguettotron-vision-vqa**](https://huggingface.co/andreagemelli/baguettotron-vision-vqa)
38
- β€” instruction-tuned on The Cauldron. **This is the model you want.**
39
  - [baguettotron-internvit-alignment](https://huggingface.co/andreagemelli/baguettotron-internvit-alignment)
40
- β€” projector-only warmup, published so the alignment stage can be reproduced.
 
 
 
41
 
42
- Apache 2.0. This is a proof of concept and a costed recipe, not a SOTA attempt.
43
- Source: [github.com/andreagemelli/baguettotron-vlm](https://github.com/andreagemelli/baguettotron-vlm).
 
44
 
45
 
46
  > **Alignment checkpoint.** Only the MLP projector was trained (on LLaVA-CC3M-Pretrain-595K) β€” the ViT and the Baguettotron LLM are unmodified base weights. It writes short captions and nothing more. Published so the alignment stage can be reproduced; for actual use take [baguettotron-vision-vqa](https://huggingface.co/andreagemelli/baguettotron-vision-vqa).
@@ -92,43 +90,25 @@ print(model.chat(**inputs))
92
 
93
  ### Example output
94
 
95
- Greedy, prompt `Describe the image concisely.` β€” verbatim output:
96
 
97
  | | output |
98
  |---|---|
99
- | <img src="http://images.cocodataset.org/val2017/000000039769.jpg" width="180"> | `a cat is sleeping on the couch` |
100
- | <img src="http://images.cocodataset.org/val2017/000000000285.jpg" width="180"> | `the bear is a good friend.` |
101
- | <img src="http://images.cocodataset.org/val2017/000000000724.jpg" width="180"> | `a sign for a stop sign` |
102
- | <img src="http://images.cocodataset.org/val2017/000000001584.jpg" width="180"> | `the bus is a red double - decoration` |
103
-
104
- That is the honest range of this checkpoint: it locates the subject and then drifts, which is what 2.7M trained parameters buys. It does not read the text on the sign or the bus.
105
 
106
  ### Chat template
107
 
108
  Trained on short image captions with no `<think>` traces. The processor emits a bare assistant prefix (`<|im_start|>assistant\n`) and the model completes the caption directly. Keep prompts simple ("Describe the image").
109
 
110
- ## Recommended settings
111
-
112
- Greedy decoding, which is what `chat()` does by default β€” just call it. Four decoding
113
- presets were compared on held-out images and the differences sat inside noise, so greedy
114
- is preferred for being deterministic rather than for scoring better.
115
-
116
- Two things that *do* move the needle:
117
-
118
- - **Ask specific questions.** `"How many cats are there?"` answers `2.`; the vaguer
119
- `"What is in the picture?"` answers `Yes.` Phrasing matters far more at this scale
120
- than it does with a large model.
121
- - **Describe first, then ask.** A follow-up question in a second turn is answered better
122
- than the same question asked cold, because the model conditions on its own previous
123
- answer. Multi-turn is in-distribution β€” The Cauldron contains multi-turn conversations.
124
-
125
  ## Limitations
126
 
127
  - **Resolution ceiling.** One 448Γ—448 crop β†’ 256 visual tokens puts document text at
128
  roughly 2–4 px/char. OCR, charts and documents are out of reach by architecture, not
129
  by budget. Neither published checkpoint reads text in an image.
130
  - **Hallucinations**, especially on fine-grained or text-heavy questions.
131
- - **Undertrained for VQA alignment** relative to the size of the data mixture.
132
  - **Multilingual capability is inherited, not verified.** The backbone covers six
133
  languages; the VLM was never evaluated on non-English benchmarks.
134
 
 
25
  [PleIAs/Baguettotron](https://huggingface.co/PleIAs/Baguettotron) β€” a 321M
26
  text-only reasoning SLM β€” with visual capabilities via the
27
  [InternViT-300M-448px-V2.5](https://huggingface.co/OpenGVLab/InternViT-300M-448px-V2_5)
28
+ vision encoder and a lightweight MLP projector, for a total of ~628M parameters. It
29
+ inherits six European languages (EN, FR, DE, ES, IT, PL) from the Baguettotron backbone.
 
 
 
 
30
 
31
  Two checkpoints are published:
32
 
 
 
33
  - [baguettotron-internvit-alignment](https://huggingface.co/andreagemelli/baguettotron-internvit-alignment)
34
+ β€” the projector-only warmup. It describes images, and nothing more.
35
+ - [baguettotron-vision-vqa](https://huggingface.co/andreagemelli/baguettotron-vision-vqa)
36
+ β€” instruction-tuned on top of it. It goes past plain description and follows visual
37
+ instructions, so prefer it for a richer chat experience.
38
 
39
+ Apache 2.0.
40
+ Source: [github.com/andreagemelli/baguettotron-vlm](https://github.com/andreagemelli/baguettotron-vlm) Β·
41
+ Write-up: [andreagemelli.me/posts/baguettotron-vlm](https://andreagemelli.me/posts/baguettotron-vlm/)
42
 
43
 
44
  > **Alignment checkpoint.** Only the MLP projector was trained (on LLaVA-CC3M-Pretrain-595K) β€” the ViT and the Baguettotron LLM are unmodified base weights. It writes short captions and nothing more. Published so the alignment stage can be reproduced; for actual use take [baguettotron-vision-vqa](https://huggingface.co/andreagemelli/baguettotron-vision-vqa).
 
90
 
91
  ### Example output
92
 
93
+ Greedy, prompt `Describe the image concisely.` β€” verbatim output. Images from [COCO](https://cocodataset.org) val2017.
94
 
95
  | | output |
96
  |---|---|
97
+ | <img src="examples/cats.jpg" width="180"> | `a cat is sleeping on the couch` |
98
+ | <img src="examples/bear.jpg" width="180"> | `the bear is a good friend.` |
99
+ | <img src="examples/stop.jpg" width="180"> | `a sign for a stop sign` |
100
+ | <img src="examples/bus.jpg" width="180"> | `the bus is a red double - decoration` |
 
 
101
 
102
  ### Chat template
103
 
104
  Trained on short image captions with no `<think>` traces. The processor emits a bare assistant prefix (`<|im_start|>assistant\n`) and the model completes the caption directly. Keep prompts simple ("Describe the image").
105
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
106
  ## Limitations
107
 
108
  - **Resolution ceiling.** One 448Γ—448 crop β†’ 256 visual tokens puts document text at
109
  roughly 2–4 px/char. OCR, charts and documents are out of reach by architecture, not
110
  by budget. Neither published checkpoint reads text in an image.
111
  - **Hallucinations**, especially on fine-grained or text-heavy questions.
 
112
  - **Multilingual capability is inherited, not verified.** The backbone covers six
113
  languages; the VLM was never evaluated on non-English benchmarks.
114
 
examples/bear.jpg ADDED

Git LFS Details

  • SHA256: f3a2974ce3686332609124c70e3e6a2e3aca43fccf1cd1bd7c5c03820977f57d
  • Pointer size: 131 Bytes
  • Size of remote file: 336 kB
examples/bus.jpg ADDED

Git LFS Details

  • SHA256: 77131e2f74aa7e374f951a69157cf7f51e00201f81bacfa8a9ab7309b245553f
  • Pointer size: 131 Bytes
  • Size of remote file: 138 kB
examples/cats.jpg ADDED

Git LFS Details

  • SHA256: dea9e7ef97386345f7cff32f9055da4982da5471c48d575146c796ab4563b04e
  • Pointer size: 131 Bytes
  • Size of remote file: 173 kB
examples/stop.jpg ADDED

Git LFS Details

  • SHA256: 5c0e559c75d3969c8e3e297b61f61063f78045c9d4802b526ba616361f3823fd
  • Pointer size: 131 Bytes
  • Size of remote file: 130 kB