Update README.md
Browse files
README.md
CHANGED
|
@@ -16,6 +16,8 @@ tags:
|
|
| 16 |
base_model:
|
| 17 |
- OpenGVLab/InternViT-300M-448px-V2_5
|
| 18 |
- PleIAs/Baguettotron
|
|
|
|
|
|
|
| 19 |
---
|
| 20 |
|
| 21 |
# baguettotron-internvit-alignment
|
|
@@ -36,9 +38,9 @@ Two checkpoints are published:
|
|
| 36 |
— instruction-tuned on top of it. It goes past plain description and follows visual
|
| 37 |
instructions, so prefer it for a richer chat experience.
|
| 38 |
|
| 39 |
-
|
| 40 |
-
|
| 41 |
-
|
| 42 |
|
| 43 |
|
| 44 |
> **Alignment checkpoint.** Only the MLP projector was trained (on LLaVA-CC3M-Pretrain-595K) — the ViT and the Baguettotron LLM are unmodified base weights. It writes short captions and nothing more. Published so the alignment stage can be reproduced; for actual use take [baguettotron-vision-vqa](https://huggingface.co/andreagemelli/baguettotron-vision-vqa).
|
|
@@ -103,15 +105,6 @@ Greedy, prompt `Describe the image concisely.` — verbatim output. Images from
|
|
| 103 |
|
| 104 |
Trained on short image captions with no `<think>` traces. The processor emits a bare assistant prefix (`<|im_start|>assistant\n`) and the model completes the caption directly. Keep prompts simple ("Describe the image").
|
| 105 |
|
| 106 |
-
## Limitations
|
| 107 |
-
|
| 108 |
-
- **Resolution ceiling.** One 448×448 crop → 256 visual tokens puts document text at
|
| 109 |
-
roughly 2–4 px/char. OCR, charts and documents are out of reach by architecture, not
|
| 110 |
-
by budget. Neither published checkpoint reads text in an image.
|
| 111 |
-
- **Hallucinations**, especially on fine-grained or text-heavy questions.
|
| 112 |
-
- **Multilingual capability is inherited, not verified.** The backbone covers six
|
| 113 |
-
languages; the VLM was never evaluated on non-English benchmarks.
|
| 114 |
-
|
| 115 |
> Tested against `transformers` 4.57. Newer major versions may need adjustments.
|
| 116 |
|
| 117 |
**Contributions and suggestions are very welcome** — issues, PRs, and ideas for
|
|
@@ -137,4 +130,4 @@ If you use or extend Baguettotron-VLM in your research, please cite it:
|
|
| 137 |
|
| 138 |
## License
|
| 139 |
|
| 140 |
-
Apache 2.0 — see the [GitHub repo](https://github.com/andreagemelli/baguettotron-vlm).
|
|
|
|
| 16 |
base_model:
|
| 17 |
- OpenGVLab/InternViT-300M-448px-V2_5
|
| 18 |
- PleIAs/Baguettotron
|
| 19 |
+
datasets:
|
| 20 |
+
- liuhaotian/LLaVA-CC3M-Pretrain-595K
|
| 21 |
---
|
| 22 |
|
| 23 |
# baguettotron-internvit-alignment
|
|
|
|
| 38 |
— instruction-tuned on top of it. It goes past plain description and follows visual
|
| 39 |
instructions, so prefer it for a richer chat experience.
|
| 40 |
|
| 41 |
+
|
| 42 |
+
- Code: [github.com/andreagemelli/baguettotron-vlm](https://github.com/andreagemelli/baguettotron-vlm) ·
|
| 43 |
+
- Blogpost: [andreagemelli.me/posts/baguettotron-vlm](https://andreagemelli.me/posts/baguettotron-vlm/)
|
| 44 |
|
| 45 |
|
| 46 |
> **Alignment checkpoint.** Only the MLP projector was trained (on LLaVA-CC3M-Pretrain-595K) — the ViT and the Baguettotron LLM are unmodified base weights. It writes short captions and nothing more. Published so the alignment stage can be reproduced; for actual use take [baguettotron-vision-vqa](https://huggingface.co/andreagemelli/baguettotron-vision-vqa).
|
|
|
|
| 105 |
|
| 106 |
Trained on short image captions with no `<think>` traces. The processor emits a bare assistant prefix (`<|im_start|>assistant\n`) and the model completes the caption directly. Keep prompts simple ("Describe the image").
|
| 107 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 108 |
> Tested against `transformers` 4.57. Newer major versions may need adjustments.
|
| 109 |
|
| 110 |
**Contributions and suggestions are very welcome** — issues, PRs, and ideas for
|
|
|
|
| 130 |
|
| 131 |
## License
|
| 132 |
|
| 133 |
+
Apache 2.0 — see the [GitHub repo](https://github.com/andreagemelli/baguettotron-vlm).
|