Duplicate from PaddlePaddle/PaddleOCR-VL-1.5-GGUF
Browse filesCo-authored-by: Yue Zhang <xiaohei66@users.noreply.huggingface.co>
- .gitattributes +37 -0
- PaddleOCR-VL-1.5-mmproj.gguf +3 -0
- PaddleOCR-VL-1.5.gguf +3 -0
- README.md +134 -0
- chat_template.jinja +55 -0
.gitattributes
ADDED
|
@@ -0,0 +1,37 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
*.7z filter=lfs diff=lfs merge=lfs -text
|
| 2 |
+
*.arrow filter=lfs diff=lfs merge=lfs -text
|
| 3 |
+
*.bin filter=lfs diff=lfs merge=lfs -text
|
| 4 |
+
*.bz2 filter=lfs diff=lfs merge=lfs -text
|
| 5 |
+
*.ckpt filter=lfs diff=lfs merge=lfs -text
|
| 6 |
+
*.ftz filter=lfs diff=lfs merge=lfs -text
|
| 7 |
+
*.gz filter=lfs diff=lfs merge=lfs -text
|
| 8 |
+
*.h5 filter=lfs diff=lfs merge=lfs -text
|
| 9 |
+
*.joblib filter=lfs diff=lfs merge=lfs -text
|
| 10 |
+
*.lfs.* filter=lfs diff=lfs merge=lfs -text
|
| 11 |
+
*.mlmodel filter=lfs diff=lfs merge=lfs -text
|
| 12 |
+
*.model filter=lfs diff=lfs merge=lfs -text
|
| 13 |
+
*.msgpack filter=lfs diff=lfs merge=lfs -text
|
| 14 |
+
*.npy filter=lfs diff=lfs merge=lfs -text
|
| 15 |
+
*.npz filter=lfs diff=lfs merge=lfs -text
|
| 16 |
+
*.onnx filter=lfs diff=lfs merge=lfs -text
|
| 17 |
+
*.ot filter=lfs diff=lfs merge=lfs -text
|
| 18 |
+
*.parquet filter=lfs diff=lfs merge=lfs -text
|
| 19 |
+
*.pb filter=lfs diff=lfs merge=lfs -text
|
| 20 |
+
*.pickle filter=lfs diff=lfs merge=lfs -text
|
| 21 |
+
*.pkl filter=lfs diff=lfs merge=lfs -text
|
| 22 |
+
*.pt filter=lfs diff=lfs merge=lfs -text
|
| 23 |
+
*.pth filter=lfs diff=lfs merge=lfs -text
|
| 24 |
+
*.rar filter=lfs diff=lfs merge=lfs -text
|
| 25 |
+
*.safetensors filter=lfs diff=lfs merge=lfs -text
|
| 26 |
+
saved_model/**/* filter=lfs diff=lfs merge=lfs -text
|
| 27 |
+
*.tar.* filter=lfs diff=lfs merge=lfs -text
|
| 28 |
+
*.tar filter=lfs diff=lfs merge=lfs -text
|
| 29 |
+
*.tflite filter=lfs diff=lfs merge=lfs -text
|
| 30 |
+
*.tgz filter=lfs diff=lfs merge=lfs -text
|
| 31 |
+
*.wasm filter=lfs diff=lfs merge=lfs -text
|
| 32 |
+
*.xz filter=lfs diff=lfs merge=lfs -text
|
| 33 |
+
*.zip filter=lfs diff=lfs merge=lfs -text
|
| 34 |
+
*.zst filter=lfs diff=lfs merge=lfs -text
|
| 35 |
+
*tfevents* filter=lfs diff=lfs merge=lfs -text
|
| 36 |
+
PaddleOCR-VL-1.5-mmproj.gguf filter=lfs diff=lfs merge=lfs -text
|
| 37 |
+
PaddleOCR-VL-1.5.gguf filter=lfs diff=lfs merge=lfs -text
|
PaddleOCR-VL-1.5-mmproj.gguf
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:e7f1a72400fba517046f90d964e2fa0f4dac7781ee3b1bc5d2022f5f8cecbd87
|
| 3 |
+
size 881770496
|
PaddleOCR-VL-1.5.gguf
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:299051d54faa065abc505cc39b8383ea338fd3020c775ea3e0ba514a7022328c
|
| 3 |
+
size 935768992
|
README.md
ADDED
|
@@ -0,0 +1,134 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
---
|
| 2 |
+
license: apache-2.0
|
| 3 |
+
---
|
| 4 |
+
|
| 5 |
+
<div align="center">
|
| 6 |
+
|
| 7 |
+
|
| 8 |
+
<h1 align="center">
|
| 9 |
+
|
| 10 |
+
PaddleOCR-VL-1.5: Towards a Multi-Task 0.9B VLM for Robust In-the-Wild Document Parsing
|
| 11 |
+
|
| 12 |
+
</h1>
|
| 13 |
+
|
| 14 |
+
[](https://github.com/PaddlePaddle/PaddleOCR)
|
| 15 |
+
[](https://huggingface.co/PaddlePaddle/PaddleOCR-VL-1.5)
|
| 16 |
+
[](https://modelscope.cn/models/PaddlePaddle/PaddleOCR-VL-1.5)
|
| 17 |
+
[](https://huggingface.co/spaces/PaddlePaddle/PaddleOCR-VL-1.5_Online_Demo)
|
| 18 |
+
[](https://modelscope.cn/studios/PaddlePaddle/PaddleOCR-VL-1.5_Online_Demo/summary)
|
| 19 |
+
[](https://discord.gg/JPmZXDsEEK)
|
| 20 |
+
[](https://x.com/PaddlePaddle)
|
| 21 |
+
[](./LICENSE)
|
| 22 |
+
|
| 23 |
+
**🔥 [Official Website](https://www.paddleocr.com)**
|
| 24 |
+
**📝 [Technical Report](https://arxiv.org/abs/2601.21957)**
|
| 25 |
+
|
| 26 |
+
</div>
|
| 27 |
+
|
| 28 |
+
<div align="center">
|
| 29 |
+
<img src="https://raw.githubusercontent.com/cuicheng01/PaddleX_doc_images/refs/heads/main/images/paddleocr_vl_1_5/paddleocr-vl-1.5_metrics.png" width="800"/>
|
| 30 |
+
</div>
|
| 31 |
+
|
| 32 |
+
## Introduction
|
| 33 |
+
|
| 34 |
+
**PaddleOCR-VL-1.5 is an advanced next-generation model of PaddleOCR-VL, achieving a new state-of-the-art accuracy of 94.5% on OmniDocBench v1.5.** To rigorously evaluate robustness against real-world physical distortions—including scanning artifacts, skew, warping, screen photography, and illumination—we propose the Real5-OmniDocBench benchmark. Experimental results demonstrate that this enhanced model attains SOTA performance on the newly curated benchmark. Furthermore, we extend the model’s capabilities by incorporating seal recognition and text spotting tasks, while remaining a 0.9B ultra-compact VLM with high efficiency.
|
| 35 |
+
|
| 36 |
+
### **Model Architecture**
|
| 37 |
+
|
| 38 |
+
<div align="center">
|
| 39 |
+
<img src="https://raw.githubusercontent.com/cuicheng01/PaddleX_doc_images/refs/heads/main/images/paddleocr_vl_1_5/PaddleOCR-VL-1.5.png" width="800"/>
|
| 40 |
+
</div>
|
| 41 |
+
|
| 42 |
+
## PaddleOCR-VL-1.5 Usage with llama.cpp
|
| 43 |
+
|
| 44 |
+
### Install Dependencies
|
| 45 |
+
|
| 46 |
+
Install [PaddlePaddle](https://www.paddlepaddle.org.cn/install/quick) and [PaddleOCR](https://github.com/PaddlePaddle/PaddleOCR):
|
| 47 |
+
|
| 48 |
+
```bash
|
| 49 |
+
# The following command installs the PaddlePaddle version for CUDA 12.6. For other CUDA versions and the CPU version, please refer to https://www.paddlepaddle.org.cn/en/install/quick?docurl=/documentation/docs/en/develop/install/pip/linux-pip_en.html
|
| 50 |
+
python -m pip install paddlepaddle-gpu==3.2.1 -i https://www.paddlepaddle.org.cn/packages/stable/cu126/
|
| 51 |
+
python -m pip install -U "paddleocr[doc-parser]"
|
| 52 |
+
```
|
| 53 |
+
|
| 54 |
+
> **Please ensure that you install PaddlePaddle framework version 3.2.1 or above, along with the special version of safetensors.** For macOS users, please use Docker to set up the environment.
|
| 55 |
+
|
| 56 |
+
|
| 57 |
+
### Basic Usage
|
| 58 |
+
|
| 59 |
+
1. Start the VLM inference server:
|
| 60 |
+
|
| 61 |
+
```
|
| 62 |
+
llama-server \
|
| 63 |
+
-m /path/to/PaddleOCR-VL-1.5-GGUF.gguf \
|
| 64 |
+
--mmproj /path/to/PaddleOCR-VL-1.5-GGUF-mmproj.gguf \
|
| 65 |
+
--port 8080 \
|
| 66 |
+
--host 0.0.0.0 \
|
| 67 |
+
--temp 0
|
| 68 |
+
```
|
| 69 |
+
|
| 70 |
+
2. Call the PaddleOCR CLI or Python API:
|
| 71 |
+
|
| 72 |
+
```bash
|
| 73 |
+
paddleocr doc_parser \
|
| 74 |
+
-i https://paddle-model-ecology.bj.bcebos.com/paddlex/imgs/demo_image/paddleocr_vl_demo.png \
|
| 75 |
+
--vl_rec_backend llama-cpp-server \
|
| 76 |
+
--vl_rec_server_url http://127.0.0.1:8080/v1
|
| 77 |
+
```
|
| 78 |
+
|
| 79 |
+
```python
|
| 80 |
+
from paddleocr import PaddleOCRVL
|
| 81 |
+
pipeline = PaddleOCRVL(vl_rec_backend="llama-cpp-server", vl_rec_server_url="http://127.0.0.1:8080/v1")
|
| 82 |
+
output = pipeline.predict("https://paddle-model-ecology.bj.bcebos.com/paddlex/imgs/demo_image/paddleocr_vl_demo.png")
|
| 83 |
+
for res in output:
|
| 84 |
+
res.print()
|
| 85 |
+
res.save_to_json(save_path="output")
|
| 86 |
+
res.save_to_markdown(save_path="output")
|
| 87 |
+
```
|
| 88 |
+
|
| 89 |
+
**For more usage details and parameter explanations, see the [documentation](https://www.paddleocr.ai/latest/en/version3.x/pipeline_usage/PaddleOCR-VL.html).**
|
| 90 |
+
|
| 91 |
+
## PaddleOCR-VL-1.5-0.9B Usage with llama.cpp
|
| 92 |
+
|
| 93 |
+
Currently, the PaddleOCR-VL-1.5-0.9B model facilitates seamless inference via the transformers library, supporting comprehensive text spotting and the recognition of complex elements including formulas, tables, charts, and seals. Below is a simple script we provide to support inference using the PaddleOCR-VL-1.5-0.9B model with llama.cpp.
|
| 94 |
+
|
| 95 |
+
#### Usage Tips
|
| 96 |
+
|
| 97 |
+
We have six types of element-level recognition:
|
| 98 |
+
|
| 99 |
+
- Text recognition, indicated by the prompt `OCR:`.
|
| 100 |
+
- Formula recognition, indicated by the prompt `Formula Recognition:`.
|
| 101 |
+
- Table recognition, indicated by the prompt `Table Recognition:`.
|
| 102 |
+
- Chart recognition, indicated by the prompt `Chart Recognition:`.
|
| 103 |
+
- Seal recognition, indicated by the prompt `Seal Recognition:`.
|
| 104 |
+
- Spotting, indicated by the prompt `Spotting:`, and need to set `image_max_pixels` to `1605632`:
|
| 105 |
+
|
| 106 |
+
```
|
| 107 |
+
git clone https://github.com/ggml-org/llama.cpp.git
|
| 108 |
+
cd llama.cpp
|
| 109 |
+
python -m pip install gguf
|
| 110 |
+
python ./gguf-py/gguf/scripts/gguf_set_metadata.py /path/to/PaddleOCR-VL-1.5-GGUF/PaddleOCR-VL-1.5-mmproj.gguf clip.vision.image_max_pixels 1605632 --force
|
| 111 |
+
|
| 112 |
+
# back to default value (1003520):
|
| 113 |
+
# python ./gguf-py/gguf/scripts/gguf_set_metadata.py /path/to/PaddleOCR-VL-1.5-GGUF/PaddleOCR-VL-1.5-mmproj.gguf clip.vision.image_max_pixels 1003520 --force
|
| 114 |
+
```
|
| 115 |
+
|
| 116 |
+
#### llama-cli Usage
|
| 117 |
+
|
| 118 |
+
```shell
|
| 119 |
+
llama-cli \
|
| 120 |
+
-m /path/to/PaddleOCR-VL-1.5-GGUF/PaddleOCR-VL-1.5.gguf \
|
| 121 |
+
--mmproj /path/to/PaddleOCR-VL-1.5-GGUF/PaddleOCR-VL-1.5-mmproj.gguf \
|
| 122 |
+
-p 'OCR:' \
|
| 123 |
+
--image 'test_image.jpg'
|
| 124 |
+
```
|
| 125 |
+
|
| 126 |
+
#### llama-server Usage
|
| 127 |
+
|
| 128 |
+
```
|
| 129 |
+
llama-server -m /path/to/PaddleOCR-VL-1.5.gguf --mmproj /path/to/PaddleOCR-VL-1.5-GGUF/PaddleOCR-VL-1.5-mmproj.gguf --temp 0
|
| 130 |
+
```
|
| 131 |
+
|
| 132 |
+
# Acknowledgements
|
| 133 |
+
|
| 134 |
+
We extend our gratitude to [@megemini](https://github.com/megemini) for their significant [pull request](https://github.com/ggml-org/llama.cpp/pull/18825), which adds support for [PaddleOCR-VL-1.5-0.9B](https://huggingface.co/PaddlePaddle/PaddleOCR-VL-1.5) and [PaddleOCR-VL-0.9B](https://huggingface.co/PaddlePaddle/PaddleOCR-VL) models to llama.cpp.
|
chat_template.jinja
ADDED
|
@@ -0,0 +1,55 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
{%- if not add_generation_prompt is defined -%}
|
| 2 |
+
{%- set add_generation_prompt = true -%}
|
| 3 |
+
{%- endif -%}
|
| 4 |
+
{%- if not cls_token is defined -%}
|
| 5 |
+
{%- set cls_token = "<|begin_of_sentence|>" -%}
|
| 6 |
+
{%- endif -%}
|
| 7 |
+
{%- if not eos_token is defined -%}
|
| 8 |
+
{%- set eos_token = "</s>" -%}
|
| 9 |
+
{%- endif -%}
|
| 10 |
+
{{- cls_token -}}
|
| 11 |
+
{%- for message in messages -%}
|
| 12 |
+
{%- if message["role"] == "user" -%}
|
| 13 |
+
{{- "User: " -}}
|
| 14 |
+
{%- if message["content"] is string -%}
|
| 15 |
+
{{- message["content"] }}
|
| 16 |
+
{%- else -%}
|
| 17 |
+
{%- for content in message["content"] -%}
|
| 18 |
+
{%- if content["type"] == "image" -%}
|
| 19 |
+
{{ "<|IMAGE_START|><|IMAGE_PLACEHOLDER|><|IMAGE_END|>" }}
|
| 20 |
+
{%- endif -%}
|
| 21 |
+
{%- endfor -%}
|
| 22 |
+
{%- for content in message["content"] -%}
|
| 23 |
+
{%- if content["type"] == "text" -%}
|
| 24 |
+
{{ content["text"] }}
|
| 25 |
+
{%- endif -%}
|
| 26 |
+
{%- endfor -%}
|
| 27 |
+
{%- endif -%}
|
| 28 |
+
{{ "\n" -}}
|
| 29 |
+
{%- elif message["role"] == "assistant" -%}
|
| 30 |
+
{{- "Assistant:\n" -}}
|
| 31 |
+
{%- if message["content"] is string -%}
|
| 32 |
+
{{- message["content"] }}
|
| 33 |
+
{%- else -%}
|
| 34 |
+
{%- for content in message["content"] -%}
|
| 35 |
+
{%- if content["type"] == "text" -%}
|
| 36 |
+
{{ content["text"] }}
|
| 37 |
+
{%- endif -%}
|
| 38 |
+
{%- endfor -%}
|
| 39 |
+
{%- endif -%}
|
| 40 |
+
{{ eos_token -}}
|
| 41 |
+
{%- elif message["role"] == "system" -%}
|
| 42 |
+
{%- if message["content"] is string -%}
|
| 43 |
+
{{- message["content"] }}
|
| 44 |
+
{%- else -%}
|
| 45 |
+
{%- for content in message["content"] -%}
|
| 46 |
+
{%- if content["type"] == "text" -%}
|
| 47 |
+
{{ content["text"] + "\n" }}
|
| 48 |
+
{%- endif -%}
|
| 49 |
+
{%- endfor -%}
|
| 50 |
+
{%- endif -%}
|
| 51 |
+
{%- endif -%}
|
| 52 |
+
{%- endfor -%}
|
| 53 |
+
{%- if add_generation_prompt -%}
|
| 54 |
+
{{- "Assistant:\n" -}}
|
| 55 |
+
{%- endif -%}
|