Fix generation from the published artifact; rewrite model card
Browse files- .gitattributes +4 -0
- README.md +14 -34
- examples/bear.jpg +3 -0
- examples/bus.jpg +3 -0
- examples/cats.jpg +3 -0
- examples/stop.jpg +3 -0
.gitattributes
CHANGED
|
@@ -33,3 +33,7 @@ saved_model/**/* filter=lfs diff=lfs merge=lfs -text
|
|
| 33 |
*.zip filter=lfs diff=lfs merge=lfs -text
|
| 34 |
*.zst filter=lfs diff=lfs merge=lfs -text
|
| 35 |
*tfevents* filter=lfs diff=lfs merge=lfs -text
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 33 |
*.zip filter=lfs diff=lfs merge=lfs -text
|
| 34 |
*.zst filter=lfs diff=lfs merge=lfs -text
|
| 35 |
*tfevents* filter=lfs diff=lfs merge=lfs -text
|
| 36 |
+
examples/bear.jpg filter=lfs diff=lfs merge=lfs -text
|
| 37 |
+
examples/bus.jpg filter=lfs diff=lfs merge=lfs -text
|
| 38 |
+
examples/cats.jpg filter=lfs diff=lfs merge=lfs -text
|
| 39 |
+
examples/stop.jpg filter=lfs diff=lfs merge=lfs -text
|
README.md
CHANGED
|
@@ -25,22 +25,20 @@ Model in the **sub-1B parameter** class. It extends
|
|
| 25 |
[PleIAs/Baguettotron](https://huggingface.co/PleIAs/Baguettotron) β a 321M
|
| 26 |
text-only reasoning SLM β with visual capabilities via the
|
| 27 |
[InternViT-300M-448px-V2.5](https://huggingface.co/OpenGVLab/InternViT-300M-448px-V2_5)
|
| 28 |
-
vision encoder and a lightweight MLP projector, for a total of ~628M parameters.
|
| 29 |
-
|
| 30 |
-
It inherits six European languages (EN, FR, DE, ES, IT, PL) from the Baguettotron
|
| 31 |
-
backbone, and β unusually β its entire LM pretraining corpus is auditable: Baguettotron
|
| 32 |
-
was trained on [SYNTH](https://huggingface.co/datasets/PleIAs/SYNTH), ~58K Wikipedia
|
| 33 |
-
articles expanded synthetically, with no scraped web.
|
| 34 |
|
| 35 |
Two checkpoints are published:
|
| 36 |
|
| 37 |
-
- [**baguettotron-vision-vqa**](https://huggingface.co/andreagemelli/baguettotron-vision-vqa)
|
| 38 |
-
β instruction-tuned on The Cauldron. **This is the model you want.**
|
| 39 |
- [baguettotron-internvit-alignment](https://huggingface.co/andreagemelli/baguettotron-internvit-alignment)
|
| 40 |
-
β projector-only warmup
|
|
|
|
|
|
|
|
|
|
| 41 |
|
| 42 |
-
Apache 2.0.
|
| 43 |
-
Source: [github.com/andreagemelli/baguettotron-vlm](https://github.com/andreagemelli/baguettotron-vlm)
|
|
|
|
| 44 |
|
| 45 |
|
| 46 |
> **Alignment checkpoint.** Only the MLP projector was trained (on LLaVA-CC3M-Pretrain-595K) β the ViT and the Baguettotron LLM are unmodified base weights. It writes short captions and nothing more. Published so the alignment stage can be reproduced; for actual use take [baguettotron-vision-vqa](https://huggingface.co/andreagemelli/baguettotron-vision-vqa).
|
|
@@ -92,43 +90,25 @@ print(model.chat(**inputs))
|
|
| 92 |
|
| 93 |
### Example output
|
| 94 |
|
| 95 |
-
Greedy, prompt `Describe the image concisely.` β verbatim output:
|
| 96 |
|
| 97 |
| | output |
|
| 98 |
|---|---|
|
| 99 |
-
| <img src="
|
| 100 |
-
| <img src="
|
| 101 |
-
| <img src="
|
| 102 |
-
| <img src="
|
| 103 |
-
|
| 104 |
-
That is the honest range of this checkpoint: it locates the subject and then drifts, which is what 2.7M trained parameters buys. It does not read the text on the sign or the bus.
|
| 105 |
|
| 106 |
### Chat template
|
| 107 |
|
| 108 |
Trained on short image captions with no `<think>` traces. The processor emits a bare assistant prefix (`<|im_start|>assistant\n`) and the model completes the caption directly. Keep prompts simple ("Describe the image").
|
| 109 |
|
| 110 |
-
## Recommended settings
|
| 111 |
-
|
| 112 |
-
Greedy decoding, which is what `chat()` does by default β just call it. Four decoding
|
| 113 |
-
presets were compared on held-out images and the differences sat inside noise, so greedy
|
| 114 |
-
is preferred for being deterministic rather than for scoring better.
|
| 115 |
-
|
| 116 |
-
Two things that *do* move the needle:
|
| 117 |
-
|
| 118 |
-
- **Ask specific questions.** `"How many cats are there?"` answers `2.`; the vaguer
|
| 119 |
-
`"What is in the picture?"` answers `Yes.` Phrasing matters far more at this scale
|
| 120 |
-
than it does with a large model.
|
| 121 |
-
- **Describe first, then ask.** A follow-up question in a second turn is answered better
|
| 122 |
-
than the same question asked cold, because the model conditions on its own previous
|
| 123 |
-
answer. Multi-turn is in-distribution β The Cauldron contains multi-turn conversations.
|
| 124 |
-
|
| 125 |
## Limitations
|
| 126 |
|
| 127 |
- **Resolution ceiling.** One 448Γ448 crop β 256 visual tokens puts document text at
|
| 128 |
roughly 2β4 px/char. OCR, charts and documents are out of reach by architecture, not
|
| 129 |
by budget. Neither published checkpoint reads text in an image.
|
| 130 |
- **Hallucinations**, especially on fine-grained or text-heavy questions.
|
| 131 |
-
- **Undertrained for VQA alignment** relative to the size of the data mixture.
|
| 132 |
- **Multilingual capability is inherited, not verified.** The backbone covers six
|
| 133 |
languages; the VLM was never evaluated on non-English benchmarks.
|
| 134 |
|
|
|
|
| 25 |
[PleIAs/Baguettotron](https://huggingface.co/PleIAs/Baguettotron) β a 321M
|
| 26 |
text-only reasoning SLM β with visual capabilities via the
|
| 27 |
[InternViT-300M-448px-V2.5](https://huggingface.co/OpenGVLab/InternViT-300M-448px-V2_5)
|
| 28 |
+
vision encoder and a lightweight MLP projector, for a total of ~628M parameters. It
|
| 29 |
+
inherits six European languages (EN, FR, DE, ES, IT, PL) from the Baguettotron backbone.
|
|
|
|
|
|
|
|
|
|
|
|
|
| 30 |
|
| 31 |
Two checkpoints are published:
|
| 32 |
|
|
|
|
|
|
|
| 33 |
- [baguettotron-internvit-alignment](https://huggingface.co/andreagemelli/baguettotron-internvit-alignment)
|
| 34 |
+
β the projector-only warmup. It describes images, and nothing more.
|
| 35 |
+
- [baguettotron-vision-vqa](https://huggingface.co/andreagemelli/baguettotron-vision-vqa)
|
| 36 |
+
β instruction-tuned on top of it. It goes past plain description and follows visual
|
| 37 |
+
instructions, so prefer it for a richer chat experience.
|
| 38 |
|
| 39 |
+
Apache 2.0.
|
| 40 |
+
Source: [github.com/andreagemelli/baguettotron-vlm](https://github.com/andreagemelli/baguettotron-vlm) Β·
|
| 41 |
+
Write-up: [andreagemelli.me/posts/baguettotron-vlm](https://andreagemelli.me/posts/baguettotron-vlm/)
|
| 42 |
|
| 43 |
|
| 44 |
> **Alignment checkpoint.** Only the MLP projector was trained (on LLaVA-CC3M-Pretrain-595K) β the ViT and the Baguettotron LLM are unmodified base weights. It writes short captions and nothing more. Published so the alignment stage can be reproduced; for actual use take [baguettotron-vision-vqa](https://huggingface.co/andreagemelli/baguettotron-vision-vqa).
|
|
|
|
| 90 |
|
| 91 |
### Example output
|
| 92 |
|
| 93 |
+
Greedy, prompt `Describe the image concisely.` β verbatim output. Images from [COCO](https://cocodataset.org) val2017.
|
| 94 |
|
| 95 |
| | output |
|
| 96 |
|---|---|
|
| 97 |
+
| <img src="examples/cats.jpg" width="180"> | `a cat is sleeping on the couch` |
|
| 98 |
+
| <img src="examples/bear.jpg" width="180"> | `the bear is a good friend.` |
|
| 99 |
+
| <img src="examples/stop.jpg" width="180"> | `a sign for a stop sign` |
|
| 100 |
+
| <img src="examples/bus.jpg" width="180"> | `the bus is a red double - decoration` |
|
|
|
|
|
|
|
| 101 |
|
| 102 |
### Chat template
|
| 103 |
|
| 104 |
Trained on short image captions with no `<think>` traces. The processor emits a bare assistant prefix (`<|im_start|>assistant\n`) and the model completes the caption directly. Keep prompts simple ("Describe the image").
|
| 105 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 106 |
## Limitations
|
| 107 |
|
| 108 |
- **Resolution ceiling.** One 448Γ448 crop β 256 visual tokens puts document text at
|
| 109 |
roughly 2β4 px/char. OCR, charts and documents are out of reach by architecture, not
|
| 110 |
by budget. Neither published checkpoint reads text in an image.
|
| 111 |
- **Hallucinations**, especially on fine-grained or text-heavy questions.
|
|
|
|
| 112 |
- **Multilingual capability is inherited, not verified.** The backbone covers six
|
| 113 |
languages; the VLM was never evaluated on non-English benchmarks.
|
| 114 |
|
examples/bear.jpg
ADDED
|
Git LFS Details
|
examples/bus.jpg
ADDED
|
Git LFS Details
|
examples/cats.jpg
ADDED
|
Git LFS Details
|
examples/stop.jpg
ADDED
|
Git LFS Details
|