Upload folder using huggingface_hub

.gitattributes

@@ -33,3 +33,5 @@ saved_model/**/* filter=lfs diff=lfs merge=lfs -text
 *.zip filter=lfs diff=lfs merge=lfs -text
 *.zst filter=lfs diff=lfs merge=lfs -text
 *tfevents* filter=lfs diff=lfs merge=lfs -text
+HPD_Parsing.pdf filter=lfs diff=lfs merge=lfs -text
+tokenizer.json filter=lfs diff=lfs merge=lfs -text

HPD_Parsing.pdf

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:5ab70bc3e7a8a494dbf18e1b6e43e8c5efc2716d968c939351aa0538b914e2ea
+size 10594788

P-MTP/config.json

@@ -0,0 +1,6 @@
+{
+  "architectures": [
+    "MedusaModel"
+  ],
+  "model_type": "medusa"
+}

README.md

@@ -1,3 +1,232 @@
 ---
 license: apache-2.0
+pipeline_tag: image-text-to-text
+tags:
+- image-text-to-text
+- transformers
+- safetensors
+- english
+- chinese
+- visual-language
+- paddleocr
+- document-parse
+- HPD-Parsing
+- speculative-decoding
+- P-MTP
+- custom-code
+- eval results
+- arxiv
+- license
+base_model: OpenGVLab/InternVL3_5-1B
+language:
+- en
+- zh
+library_name: transformers
 ---
+<div align="center">
+
+<h1 align="center">
+
+HPD-Parsing: Hierarchical Parallel Document Parsing
+
+</h1>
+
+[![PaddleOCR](https://img.shields.io/github/stars/PaddlePaddle/PaddleOCR?color=ccf)](https://github.com/PaddlePaddle/PaddleOCR)
+[![vLLM](https://img.shields.io/badge/Serving-vLLM-6080F0)](https://github.com/vllm-project/vllm)
+[![arxiv](https://img.shields.io/badge/arXiv-b31b1b?logo=arxiv&logoColor=white)](./HPD_Parsing.pdf)
+[![X](https://img.shields.io/badge/X-PaddlePaddle-6080F0)](https://x.com/PaddlePaddle)
+[![License](https://img.shields.io/badge/license-Apache_2.0-green)](https://www.apache.org/licenses/LICENSE-2.0)
+
+</div>
+
+<div align="center">
+
+<img src="https://raw.githubusercontent.com/weishu20/HPD-Parsing-images/master/images/teaser.svg" width="800"/>
+
+</div>
+
+
+
+## News
+
+* ```2026.07``` 🚀 We release **HPD-Parsing**, a 1B hierarchical parallel document parser. It achieves 94.91% overall on OmniDocBench v1.6 and 4,752 TPS peak throughput.
+
+
+## Introduction
+
+<div align="center">
+<img src="https://raw.githubusercontent.com/weishu20/HPD-Parsing-images/master/images/hierarchical_decoding.gif " width="800"/>
+</div>
+
+We introduce **HPD-Parsing**, a lightweight (1B) and high-throughput document parsing model built on a Hierarchical Parallel Decoding paradigm. Unified VLM-based parsers process an entire page jointly but generate the output through a single token-by-token autoregressive trajectory, creating a sequential bottleneck that grows with document length. HPD-Parsing is motivated by a key property of document parsing: **page structure requires global coordination, whereas content generation is largely localized within individual regions.** Based on this observation, a main layout branch coordinates the global document structure and dynamically dispatches localized content generation to concurrent branches, while Progressive Multi-Token Prediction (P-MTP) further reduces the decoding steps within each branch. **HPD-Parsing achieves an overall score of 94.91% on OmniDocBench v1.6 — a new state of the art among end-to-end unified parsers — while reaching a peak throughput of 4,752 TPS, 2.62× the fastest existing document parser and 3.06× its own autoregressive baseline.**
+
+
+
+
+
+### **Key Capabilities of HPD-Parsing**
+
+**🚀 Hierarchical Parallel Decoding for High-Throughput Document Parsing**: We introduce Hierarchical Parallel Decoding (HPD), a new decoding paradigm that restructures full-page autoregressive generation into globally coordinated, localized parallel decoding. A main layout branch performs global coordination and dynamically decomposes the conventional single decoding trajectory into concurrent content branches, each responsible for a localized document region. Within each branch, P-MTP further reduces the number of decoding steps by predicting multiple future tokens at each iteration. Together with shared-prefix KV cache reuse, HPD substantially shortens the effective sequential decoding path along both branch and token dimensions.
+
+**🔄 Staged Adaptation with Automated Difficulty-Aware Data Curation**: We develop a staged adaptation strategy that transfers conventional autoregressive document parsing capabilities to the proposed hierarchical parallel decoding paradigm while preserving parsing accuracy. The strategy is supported by an automated difficulty-aware data curation pipeline that integrates large-scale data collection, model-assisted annotation, difficulty estimation, and balanced sampling. By progressively adapting the model and emphasizing challenging samples, the training framework mitigates the accuracy degradation caused by the transition to parallel decoding with minimal manual annotation effort.
+
+
+**⚡ State-of-the-Art Throughput with Competitive Parsing Accuracy**: HPD-Parsing achieves state-of-the-art inference efficiency on OmniDocBench v1.6, reaching a peak throughput of 4,752 Tokens Per Second (TPS). It delivers $1.62\times$ the throughput of the fastest existing document parsing model and more than $3.06\times$ that of its autoregressive baseline, while maintaining competitive parsing accuracy. These results demonstrate that document parsing can be effectively executed through global layout coordination and localized parallel decoding rather than a single sequential generation trajectory.
+
+
+### **HPD-Parsing Architecture**
+
+HPD-Parsing adopts **InternVL3.5-1B** as its backbone, applies dynamic tile-based cropping (up to 24 tiles of 448×448) to preserve high-resolution details. Its primary departure from conventional unified parsers is the decoding paradigm: instead of generating the entire page along a single autoregressive trajectory, HPD-Parsing employs a main layout branch to coordinate the global structure and spawns localized content branches for concurrent region-level decoding, with P-MTP integrated into each branch.
+
+<div align="center">
+<img src="https://raw.githubusercontent.com/weishu20/HPD-Parsing-images/master/images/method.svg" width="800"/>
+</div>
+
+
+
+
+## Usage
+
+HPD-Parsing runs on a **customized build of vLLM** (based on vLLM v0.17.1) that implements the dynamic request forking required by hierarchical parallel decoding and adapts P-MTP speculative decoding.
+
+
+### 1. Prepare the Runtime Environment
+
+Either option works; **the Docker image is strongly recommended** to minimize environment issues.
+
+**Option A — Docker image** (ships the customized vLLM build and all dependencies):
+
+```text
+ccr-2vdh3abv-pub.cnc.bj.baidubce.com/paddlepaddle/hpd-parsing-vllm:latest-nvidia-gpu
+```
+
+The default entrypoint starts the inference server automatically (see Serving below). It is pulled automatically on the first `docker run`. For machines without internet access, use the offline image `...:latest-nvidia-gpu-offline` (~24.5 GB; it has the model weights built in). Pull it on a networked machine and transfer it:
+
+```shell
+# On a machine with internet access
+docker pull ccr-2vdh3abv-pub.cnc.bj.baidubce.com/paddlepaddle/hpd-parsing-vllm:latest-nvidia-gpu-offline
+docker save ccr-2vdh3abv-pub.cnc.bj.baidubce.com/paddlepaddle/hpd-parsing-vllm:latest-nvidia-gpu-offline -o hpd-parsing-vllm-offline.tar
+# Transfer the tar to the offline machine, then:
+docker load -i hpd-parsing-vllm-offline.tar
+```
+
+**Option B — Prebuilt package** (Python 3.10–3.13, NVIDIA driver with CUDA 12.8+). Install in a virtual environment to avoid dependency conflicts:
+
+```shell
+python -m venv .venv_hpd_parsing
+source .venv_hpd_parsing/bin/activate
+python -m pip install https://paddle-model-ecology.bj.bcebos.com/paddlex/PaddleX3.0/deploy/hpd_parsing/vllm-0.17.1+hpdparsing-cp38-abi3-manylinux_2_31_x86_64.whl
+```
+
+### 2. vLLM Python API
+
+```python
+# Set the environment variable before running: export MAX_PATCHES_WITH_RESIZE=true
+import base64
+from vllm import LLM, SamplingParams
+
+llm = LLM(
+    model="PaddlePaddle/HPD-Parsing",
+    trust_remote_code=True,
+    max_model_len=16384,
+    limit_mm_per_prompt={"image": 1},
+    gpu_memory_utilization=0.9,
+    attention_backend="FLASHINFER",
+    enable_prefix_caching=True,
+    speculative_config={
+        "method": "medusa",
+        "model": "PaddlePaddle/HPD-Parsing/P-MTP",
+        "num_speculative_tokens": 6,
+    },
+)
+sampling_params = SamplingParams(temperature=0, max_tokens=8000)
+
+with open("demo.png", "rb") as f:
+    image_base64 = base64.b64encode(f.read()).decode("utf-8")
+
+messages = [
+    {
+        "role": "user",
+        "content": [
+            {"type": "image_url", "image_url": {"url": f"data:image/png;base64,{image_base64}"}},
+            {"type": "text", "text": "document parsing with fork."},
+        ],
+    }
+]
+
+outputs = llm.chat(messages=messages, sampling_params=sampling_params)
+print(outputs[0].outputs[0].text)
+```
+
+
+
+## Inference with transformers
+
+`model.generate_hpd(...)` reproduces the vLLM decoding paradigm in plain `transformers` (single image, batch size 1): the parent **layout** branch is decoded greedily and every `<FORK>` token spawns a **content** child branch that inherits the parent KV cache (shared-prefix reuse); the branches are then spliced back into one sequence. The `P-MTP/` head is used for speculative decoding when `use_mtp=True`.
+
+The `P-MTP` weights ship inside the main checkpoint (keys `language_model.mtp.*`) and are loaded automatically by `from_pretrained`; `load_mtp_weights()` simply enables them. To load them from the standalone `P-MTP/` directory instead, pass its path.
+
+Image preprocessing (dynamic tiling that mirrors vLLM's InternVL path with `MAX_PATCHES_WITH_RESIZE=true`) lives in [`image_preprocess.py`](./image_preprocess.py); `load_image` returns the stacked `pixel_values` tensor consumed by `generate_hpd`.
+
+```python
+import torch
+from transformers import AutoModel, AutoTokenizer
+
+from image_preprocess import load_image
+
+MODEL = "PaddlePaddle/HPD-Parsing"
+
+DEVICE = "cuda" if torch.cuda.is_available() else "cpu"
+model = AutoModel.from_pretrained(
+    MODEL, torch_dtype=torch.bfloat16, trust_remote_code=True,
+).eval().to(DEVICE)
+tokenizer = AutoTokenizer.from_pretrained(MODEL, trust_remote_code=True, use_fast=False)
+
+model.load_mtp_weights()                       # or model.load_mtp_weights(f"{MODEL}/P-MTP")
+
+pixel_values = load_image("test.png").to(torch.bfloat16).to(DEVICE)
+
+response = model.generate_hpd(
+    tokenizer,
+    pixel_values,
+    "document parsing with fork.",             # `document parsing.` for non-fork layout
+    dict(max_new_tokens=8000),
+    use_mtp=True,                              # P-MTP speculative decoding (parent branch)
+    num_speculative_tokens=6,
+    batch_children=False,                      # see note below
+)
+print(response)
+```
+
+Key arguments:
+
+- `use_mtp` (default `False`) / `num_speculative_tokens` (default `6`): enable P-MTP speculative decoding with the given draft length; requires `load_mtp_weights(...)`. Greedy verification keeps output identical to plain autoregressive decoding.
+- `batch_children` (default `False`): serial per-child decoding that reuses the parent KV cache in place — lowest memory and fastest in practice. Set `True` to decode all children concurrently in one left-padded batch (same output, but heavier since `transformers` duplicates the shared prefix KV per child).
+- `return_token_ids` (default `False`): also return the final token id list.
+
+> Reference implementation only. Use the vLLM path for production throughput — its paged KV cache enables true concurrent branches with zero-copy prefix sharing.
+
+## Evaluation & Benchmark
+
+Scripts under [`eval/`](./eval) reproduce both the throughput (TPS) and the OmniDocBench v1.6 accuracy numbers via a decoupled **infer -> convert -> evaluate** pipeline:
+
+- [`eval/benchmark_tps.py`](./eval/benchmark_tps.py) — batched vLLM inference that reports TPS metrics and dumps the raw predictions in one run.
+- [`eval/hpd_to_markdown.py`](./eval/hpd_to_markdown.py) — converts the `<BLOCK>...<CHILD>...` predictions into per-page markdown for OmniDocBench's end2end evaluation.
+
+See [`eval/README.md`](./eval/README.md) for the full workflow, including how to run the official [OmniDocBench](https://github.com/opendatalab/OmniDocBench) evaluation on the generated markdown.
+
+## Acknowledgments
+
+We would like to thank [InternVL](https://github.com/OpenGVLab/InternVL) and [Qwen3](https://github.com/QwenLM/Qwen3) for the backbone, [vLLM](https://github.com/vllm-project/vllm) for the serving framework, and [PaddleOCR-VL](https://github.com/PaddlePaddle/PaddleOCR), [MinerU](https://github.com/opendatalab/MinerU) and [OmniDocBench](https://github.com/opendatalab/OmniDocBench) for providing valuable data, model weights and benchmarks. We also appreciate everyone's contribution to this open-source project!
+
+## Citation
+
+If you find HPD-Parsing helpful, feel free to give us a star and citation.
+
+```bibtex
+@misc{wei2026hpdparsing,
+      title={HPD-Parsing: Hierarchical Parallel Document Parsing},
+      author={Shu Wei and Jingjing Wu and Lingshu Zhang and Qunyi Xie and Hao Zou and Le Xiang and Xu Fan and Yangliu Xu and Manhui Lin and Xiaolong Ma and Cheng Cui and Tengyu Du and YY},
+      year={2026},
+}
+```

added_tokens.json

@@ -0,0 +1,40 @@
+{
+  "</box>": 151677,
+  "</img>": 151670,
+  "</quad>": 151673,
+  "</ref>": 151675,
+  "</think>": 151668,
+  "</tool_call>": 151658,
+  "</tool_response>": 151666,
+  "<BLOCK>": 151678,
+  "<CHILD>": 151680,
+  "<FORK>": 151679,
+  "<IMG_CONTEXT>": 151671,
+  "<box>": 151676,
+  "<img>": 151669,
+  "<quad>": 151672,
+  "<ref>": 151674,
+  "<think>": 151667,
+  "<tool_call>": 151657,
+  "<tool_response>": 151665,
+  "<|box_end|>": 151649,
+  "<|box_start|>": 151648,
+  "<|endoftext|>": 151643,
+  "<|file_sep|>": 151664,
+  "<|fim_middle|>": 151660,
+  "<|fim_pad|>": 151662,
+  "<|fim_prefix|>": 151659,
+  "<|fim_suffix|>": 151661,
+  "<|im_end|>": 151645,
+  "<|im_start|>": 151644,
+  "<|image_pad|>": 151655,
+  "<|object_ref_end|>": 151647,
+  "<|object_ref_start|>": 151646,
+  "<|quad_end|>": 151651,
+  "<|quad_start|>": 151650,
+  "<|repo_name|>": 151663,
+  "<|video_pad|>": 151656,
+  "<|vision_end|>": 151653,
+  "<|vision_pad|>": 151654,
+  "<|vision_start|>": 151652
+}

chat_template.jinja

@@ -0,0 +1,6 @@
+{% for message in messages %}{{'<|im_start|>' + message['role'] + '
+'}}{% if message['content'] is string %}{{ message['content'] }}{% else %}{% for content in message['content'] %}{% if content['type'] == 'image' %}{{ '<image>
+' }}{% elif content['type'] == 'video' %}{{ '<video>
+' }}{% elif content['type'] == 'text' %}{{ content['text'] }}{% endif %}{% endfor %}{% endif %}{{'<|im_end|>
+'}}{% endfor %}{% if add_generation_prompt %}{{'<|im_start|>assistant
+' }}{% endif %}

config.json

@@ -0,0 +1,131 @@
+{
+  "architectures": [
+    "InternVLChatModel"
+  ],
+  "auto_map": {
+    "AutoConfig": "configuration_internvl_chat.InternVLChatConfig",
+    "AutoModel": "modeling_internvl_chat.InternVLChatModel",
+    "AutoModelForCausalLM": "modeling_internvl_chat.InternVLChatModel"
+  },
+  "downsample_ratio": 0.5,
+  "dtype": "bfloat16",
+  "dynamic_image_size": true,
+  "eos_token_id": 151645,
+  "force_image_size": 448,
+  "hidden_size": 1024,
+  "keys_to_ignore_at_inference": [
+    "past_key_values"
+  ],
+  "fork_token_id": 151679, "child_token_id": 151680,
+  "llm_config": {
+    "_name_or_path": "/root/codespace/checkpoints/Qwen3-0.6B",
+    "architectures": [
+      "Qwen3ForCausalLM"
+    ],
+    "attention_bias": false,
+    "attention_dropout": 0.0,
+    "bos_token_id": 151643,
+    "debug": false,
+    "dtype": "bfloat16",
+    "eos_token_id": 151645,
+    "ep_size": 1,
+    "head_dim": 128,
+    "hidden_act": "silu",
+    "hidden_size": 1024,
+    "initializer_range": 0.02,
+    "intermediate_size": 3072,
+    "layer_types": [
+      "full_attention",
+      "full_attention",
+      "full_attention",
+      "full_attention",
+      "full_attention",
+      "full_attention",
+      "full_attention",
+      "full_attention",
+      "full_attention",
+      "full_attention",
+      "full_attention",
+      "full_attention",
+      "full_attention",
+      "full_attention",
+      "full_attention",
+      "full_attention",
+      "full_attention",
+      "full_attention",
+      "full_attention",
+      "full_attention",
+      "full_attention",
+      "full_attention",
+      "full_attention",
+      "full_attention",
+      "full_attention",
+      "full_attention",
+      "full_attention",
+      "full_attention"
+    ],
+    "max_position_embeddings": 40960,
+    "max_window_layers": 28,
+    "micro_forward": false,
+    "model_type": "qwen3",
+    "num_attention_heads": 16,
+    "num_hidden_layers": 28,
+    "num_key_value_heads": 8,
+    "pad_token_id": 151643,
+    "rms_norm_eps": 1e-06,
+    "rope_scaling": null,
+    "rope_theta": 1000000,
+    "skip_checkpoint": false,
+    "sliding_window": null,
+    "use_cache": false,
+    "use_deepep": false,
+    "use_sliding_window": false,
+    "vocab_size": 151681
+  },
+  "max_dynamic_patch": 24,
+  "min_dynamic_patch": 1,
+  "model_type": "internvl_chat",
+  "output_attentions": false,
+  "pad2square": false,
+  "pad_token_id": 151643,
+  "ps_version": "v2",
+  "select_layer": -1,
+  "template": "internvl2_5",
+  "tie_word_embeddings": false,
+  "transformers_version": null,
+  "use_backbone_lora": 0,
+  "use_llm_lora": 0,
+  "use_thumbnail": true,
+  "vision_config": {
+    "architectures": [
+      "InternVisionModel"
+    ],
+    "attention_dropout": 0.0,
+    "auto_map": {
+      "AutoConfig": "configuration_intern_vit.InternVisionConfig",
+      "AutoModel": "modeling_intern_vit.InternVisionModel"
+    },
+    "drop_path_rate": 0.1,
+    "dropout": 0.0,
+    "dtype": "bfloat16",
+    "hidden_act": "gelu",
+    "hidden_size": 1024,
+    "image_size": 448,
+    "initializer_factor": 1.0,
+    "initializer_range": 0.02,
+    "intermediate_size": 4096,
+    "layer_norm_eps": 1e-06,
+    "model_type": "intern_vit_6b",
+    "norm_type": "layer_norm",
+    "num_attention_heads": 16,
+    "num_channels": 3,
+    "num_hidden_layers": 24,
+    "pad_token_id": 151643,
+    "patch_size": 14,
+    "qk_normalization": false,
+    "qkv_bias": true,
+    "use_fa3": false,
+    "use_flash_attn": true
+  },
+  "vocab_size": 151681
+}

configuration_intern_vit.py

@@ -0,0 +1,119 @@
+# --------------------------------------------------------
+# InternVL
+# Copyright (c) 2024 OpenGVLab
+# Licensed under The MIT License [see LICENSE for details]
+# --------------------------------------------------------
+import os
+from typing import Union
+
+from transformers.configuration_utils import PretrainedConfig
+from transformers.utils import logging
+
+logger = logging.get_logger(__name__)
+
+
+class InternVisionConfig(PretrainedConfig):
+    r"""
+    This is the configuration class to store the configuration of a [`InternVisionModel`]. It is used to
+    instantiate a vision encoder according to the specified arguments, defining the model architecture.
+
+    Configuration objects inherit from [`PretrainedConfig`] and can be used to control the model outputs. Read the
+    documentation from [`PretrainedConfig`] for more information.
+
+    Args:
+        num_channels (`int`, *optional*, defaults to 3):
+            Number of color channels in the input images (e.g., 3 for RGB).
+        patch_size (`int`, *optional*, defaults to 14):
+            The size (resolution) of each patch.
+        image_size (`int`, *optional*, defaults to 224):
+            The size (resolution) of each image.
+        qkv_bias (`bool`, *optional*, defaults to `False`):
+            Whether to add a bias to the queries and values in the self-attention layers.
+        hidden_size (`int`, *optional*, defaults to 3200):
+            Dimensionality of the encoder layers and the pooler layer.
+        num_attention_heads (`int`, *optional*, defaults to 25):
+            Number of attention heads for each attention layer in the Transformer encoder.
+        intermediate_size (`int`, *optional*, defaults to 12800):
+            Dimensionality of the "intermediate" (i.e., feed-forward) layer in the Transformer encoder.
+        qk_normalization (`bool`, *optional*, defaults to `True`):
+            Whether to normalize the queries and keys in the self-attention layers.
+        num_hidden_layers (`int`, *optional*, defaults to 48):
+            Number of hidden layers in the Transformer encoder.
+        use_flash_attn (`bool`, *optional*, defaults to `True`):
+            Whether to use flash attention mechanism.
+        hidden_act (`str` or `function`, *optional*, defaults to `"gelu"`):
+            The non-linear activation function (function or string) in the encoder and pooler. If string, `"gelu"`,
+            `"relu"`, `"selu"` and `"gelu_new"` ``"gelu"` are supported.
+        layer_norm_eps (`float`, *optional*, defaults to 1e-6):
+            The epsilon used by the layer normalization layers.
+        dropout (`float`, *optional*, defaults to 0.0):
+            The dropout probability for all fully connected layers in the embeddings, encoder, and pooler.
+        drop_path_rate (`float`, *optional*, defaults to 0.0):
+            Dropout rate for stochastic depth.
+        attention_dropout (`float`, *optional*, defaults to 0.0):
+            The dropout ratio for the attention probabilities.
+        initializer_range (`float`, *optional*, defaults to 0.02):
+            The standard deviation of the truncated_normal_initializer for initializing all weight matrices.
+        initializer_factor (`float`, *optional*, defaults to 0.1):
+            A factor for layer scale.
+    """
+
+    model_type = 'intern_vit_6b'
+
+    def __init__(
+            self,
+            num_channels=3,
+            patch_size=14,
+            image_size=224,
+            qkv_bias=False,
+            hidden_size=3200,
+            num_attention_heads=25,
+            intermediate_size=12800,
+            qk_normalization=True,
+            num_hidden_layers=48,
+            use_flash_attn=True,
+            hidden_act='gelu',
+            norm_type='rms_norm',
+            layer_norm_eps=1e-6,
+            dropout=0.0,
+            drop_path_rate=0.0,
+            attention_dropout=0.0,
+            initializer_range=0.02,
+            initializer_factor=0.1,
+            **kwargs,
+    ):
+        super().__init__(**kwargs)
+
+        self.hidden_size = hidden_size
+        self.intermediate_size = intermediate_size
+        self.dropout = dropout
+        self.drop_path_rate = drop_path_rate
+        self.num_hidden_layers = num_hidden_layers
+        self.num_attention_heads = num_attention_heads
+        self.num_channels = num_channels
+        self.patch_size = patch_size
+        self.image_size = image_size
+        self.initializer_range = initializer_range
+        self.initializer_factor = initializer_factor
+        self.attention_dropout = attention_dropout
+        self.layer_norm_eps = layer_norm_eps
+        self.hidden_act = hidden_act
+        self.norm_type = norm_type
+        self.qkv_bias = qkv_bias
+        self.qk_normalization = qk_normalization
+        self.use_flash_attn = use_flash_attn
+
+    @classmethod
+    def from_pretrained(cls, pretrained_model_name_or_path: Union[str, os.PathLike], **kwargs) -> 'PretrainedConfig':
+        config_dict, kwargs = cls.get_config_dict(pretrained_model_name_or_path, **kwargs)
+
+        if 'vision_config' in config_dict:
+            config_dict = config_dict['vision_config']
+
+        if 'model_type' in config_dict and hasattr(cls, 'model_type') and config_dict['model_type'] != cls.model_type:
+            logger.warning(
+                f"You are using a model of type {config_dict['model_type']} to instantiate a model of type "
+                f'{cls.model_type}. This is not supported for all configurations of models and can yield errors.'
+            )
+
+        return cls.from_dict(config_dict, **kwargs)

configuration_internvl_chat.py

@@ -0,0 +1,115 @@
+# --------------------------------------------------------
+# InternVL
+# Copyright (c) 2024 OpenGVLab
+# Licensed under The MIT License [see LICENSE for details]
+# --------------------------------------------------------
+
+import copy
+from typing import Dict, Any, Optional
+
+from transformers.configuration_utils import PretrainedConfig
+from transformers.utils import logging
+
+from .configuration_intern_vit import InternVisionConfig
+
+logger = logging.get_logger(__name__)
+
+
+class InternVLChatConfig(PretrainedConfig):
+    model_type = 'internvl_chat'
+    is_composition = True
+
+    def __init__(
+        self,
+        vision_config: Optional[Dict[str, Any]] = None,
+        llm_config: Optional[Dict[str, Any]] = None,
+        use_backbone_lora=0,
+        use_llm_lora=0,
+        select_layer=-1,
+        force_image_size=None,
+        downsample_ratio=0.5,
+        template=None,
+        dynamic_image_size=False,
+        use_thumbnail=False,
+        ps_version="v1",
+        min_dynamic_patch=1,
+        max_dynamic_patch=6,
+        **kwargs,
+    ):
+        super().__init__(**kwargs)
+
+        if vision_config is None:
+            vision_config = {'architectures': ['InternVisionModel']}
+            logger.info('vision_config is None. Initializing the InternVisionConfig with default values.')
+
+        if llm_config is None:
+            llm_config = {'architectures': ['Qwen2ForCausalLM']}
+            logger.info('llm_config is None. Initializing the LlamaConfig config with default values (`LlamaConfig`).')
+        assert 'architectures' in llm_config, "Should specify architecture in llm_config"
+
+        if isinstance(vision_config, dict):
+            self.vision_config = InternVisionConfig(**vision_config)
+        else:
+            self.vision_config = vision_config
+
+        if isinstance(llm_config, dict):
+            architecture: str = llm_config['architectures'][0]
+            if architecture == 'LlamaForCausalLM':
+                from transformers import LlamaConfig
+                self.llm_config = LlamaConfig(**llm_config)
+            elif architecture == 'Qwen2ForCausalLM':
+                from transformers import Qwen2Config
+                self.llm_config = Qwen2Config(**llm_config)
+            elif architecture == 'Qwen3MoeForCausalLM':
+                from transformers import Qwen3MoeConfig
+                self.llm_config = Qwen3MoeConfig(**llm_config)
+            elif architecture == 'Qwen3ForCausalLM':
+                from transformers import Qwen3Config
+                self.llm_config = Qwen3Config(**llm_config)
+            else:
+                raise ValueError('Unsupported architecture: {}'.format(architecture))
+        else:
+            self.llm_config = llm_config
+
+        self.use_backbone_lora = use_backbone_lora
+        self.use_llm_lora = use_llm_lora
+        self.select_layer = select_layer
+        self.force_image_size = force_image_size
+        self.downsample_ratio = downsample_ratio
+        self.template = template
+        self.dynamic_image_size = dynamic_image_size
+        self.use_thumbnail = use_thumbnail
+        self.ps_version = ps_version  # pixel shuffle version
+        self.min_dynamic_patch = min_dynamic_patch
+        self.max_dynamic_patch = max_dynamic_patch
+        self.tie_word_embeddings = self.llm_config.tie_word_embeddings
+
+        logger.info(f'vision_select_layer: {self.select_layer}')
+        logger.info(f'ps_version: {self.ps_version}')
+        logger.info(f'min_dynamic_patch: {self.min_dynamic_patch}')
+        logger.info(f'max_dynamic_patch: {self.max_dynamic_patch}')
+
+    def to_dict(self):
+        """
+        Serializes this instance to a Python dictionary. Override the default [`~PretrainedConfig.to_dict`].
+
+        Returns:
+            `Dict[str, any]`: Dictionary of all the attributes that make up this configuration instance,
+        """
+        output = copy.deepcopy(self.__dict__)
+        output['vision_config'] = self.vision_config.to_dict()
+        output['llm_config'] = self.llm_config.to_dict()
+        output['model_type'] = self.__class__.model_type
+        output['use_backbone_lora'] = self.use_backbone_lora
+        output['use_llm_lora'] = self.use_llm_lora
+        output['select_layer'] = self.select_layer
+        output['force_image_size'] = self.force_image_size
+        output['downsample_ratio'] = self.downsample_ratio
+        output['template'] = self.template
+        output['dynamic_image_size'] = self.dynamic_image_size
+        output['use_thumbnail'] = self.use_thumbnail
+        output['ps_version'] = self.ps_version
+        output['min_dynamic_patch'] = self.min_dynamic_patch
+        output['max_dynamic_patch'] = self.max_dynamic_patch
+
+        return output

conversation.py

@@ -0,0 +1,391 @@
+"""
+Conversation prompt templates.
+
+We kindly request that you import fastchat instead of copying this file if you wish to use it.
+If you have changes in mind, please contribute back so the community can benefit collectively and continue to maintain these valuable templates.
+
+Modified from https://github.com/lm-sys/FastChat/blob/main/fastchat/conversation.py
+"""
+
+import dataclasses
+from enum import IntEnum, auto
+from typing import Dict, List, Tuple, Union
+
+
+class SeparatorStyle(IntEnum):
+    """Separator styles."""
+
+    ADD_COLON_SINGLE = auto()
+    ADD_COLON_TWO = auto()
+    ADD_COLON_SPACE_SINGLE = auto()
+    NO_COLON_SINGLE = auto()
+    NO_COLON_TWO = auto()
+    ADD_NEW_LINE_SINGLE = auto()
+    LLAMA2 = auto()
+    CHATGLM = auto()
+    CHATML = auto()
+    CHATINTERN = auto()
+    DOLLY = auto()
+    RWKV = auto()
+    PHOENIX = auto()
+    ROBIN = auto()
+    FALCON_CHAT = auto()
+    CHATGLM3 = auto()
+    INTERNVL_ZH = auto()
+    MPT = auto()
+
+
+@dataclasses.dataclass
+class Conversation:
+    """A class that manages prompt templates and keeps all conversation history."""
+
+    # The name of this template
+    name: str
+    # The template of the system prompt
+    system_template: str = '{system_message}'
+    # The system message
+    system_message: str = ''
+    # The names of two roles
+    roles: Tuple[str] = ('USER', 'ASSISTANT')
+    # All messages. Each item is (role, message).
+    messages: List[List[str]] = ()
+    # The number of few shot examples
+    offset: int = 0
+    # The separator style and configurations
+    sep_style: SeparatorStyle = SeparatorStyle.ADD_COLON_SINGLE
+    sep: str = '\n'
+    sep2: str = None
+    # Stop criteria (the default one is EOS token)
+    stop_str: Union[str, List[str]] = None
+    # Stops generation if meeting any token in this list
+    stop_token_ids: List[int] = None
+
+    def get_prompt(self) -> str:
+        """Get the prompt for generation."""
+        system_prompt = self.system_template.format(system_message=self.system_message)
+        if self.sep_style == SeparatorStyle.ADD_COLON_SINGLE:
+            ret = system_prompt + self.sep
+            for role, message in self.messages:
+                if message:
+                    ret += role + ': ' + message + self.sep
+                else:
+                    ret += role + ':'
+            return ret
+        elif self.sep_style == SeparatorStyle.ADD_COLON_TWO:
+            seps = [self.sep, self.sep2]
+            ret = system_prompt + seps[0]
+            for i, (role, message) in enumerate(self.messages):
+                if message:
+                    ret += role + ': ' + message + seps[i % 2]
+                else:
+                    ret += role + ':'
+            return ret
+        elif self.sep_style == SeparatorStyle.ADD_COLON_SPACE_SINGLE:
+            ret = system_prompt + self.sep
+            for role, message in self.messages:
+                if message:
+                    ret += role + ': ' + message + self.sep
+                else:
+                    ret += role + ': '  # must be end with a space
+            return ret
+        elif self.sep_style == SeparatorStyle.ADD_NEW_LINE_SINGLE:
+            ret = '' if system_prompt == '' else system_prompt + self.sep
+            for role, message in self.messages:
+                if message:
+                    ret += role + '\n' + message + self.sep
+                else:
+                    ret += role + '\n'
+            return ret
+        elif self.sep_style == SeparatorStyle.NO_COLON_SINGLE:
+            ret = system_prompt
+            for role, message in self.messages:
+                if message:
+                    ret += role + message + self.sep
+                else:
+                    ret += role
+            return ret
+        elif self.sep_style == SeparatorStyle.NO_COLON_TWO:
+            seps = [self.sep, self.sep2]
+            ret = system_prompt
+            for i, (role, message) in enumerate(self.messages):
+                if message:
+                    ret += role + message + seps[i % 2]
+                else:
+                    ret += role
+            return ret
+        elif self.sep_style == SeparatorStyle.RWKV:
+            ret = system_prompt
+            for i, (role, message) in enumerate(self.messages):
+                if message:
+                    ret += (
+                        role
+                        + ': '
+                        + message.replace('\r\n', '\n').replace('\n\n', '\n')
+                    )
+                    ret += '\n\n'
+                else:
+                    ret += role + ':'
+            return ret
+        elif self.sep_style == SeparatorStyle.LLAMA2:
+            seps = [self.sep, self.sep2]
+            if self.system_message:
+                ret = system_prompt
+            else:
+                ret = '[INST] '
+            for i, (role, message) in enumerate(self.messages):
+                tag = self.roles[i % 2]
+                if message:
+                    if i == 0:
+                        ret += message + ' '
+                    else:
+                        ret += tag + ' ' + message + seps[i % 2]
+                else:
+                    ret += tag
+            return ret
+        elif self.sep_style == SeparatorStyle.CHATGLM:
+            # source: https://huggingface.co/THUDM/chatglm-6b/blob/1d240ba371910e9282298d4592532d7f0f3e9f3e/modeling_chatglm.py#L1302-L1308
+            # source2: https://huggingface.co/THUDM/chatglm2-6b/blob/e186c891cf64310ac66ef10a87e6635fa6c2a579/modeling_chatglm.py#L926
+            round_add_n = 1 if self.name == 'chatglm2' else 0
+            if system_prompt:
+                ret = system_prompt + self.sep
+            else:
+                ret = ''
+
+            for i, (role, message) in enumerate(self.messages):
+                if i % 2 == 0:
+                    ret += f'[Round {i//2 + round_add_n}]{self.sep}'
+
+                if message:
+                    ret += f'{role}：{message}{self.sep}'
+                else:
+                    ret += f'{role}：'
+            return ret
+        elif self.sep_style == SeparatorStyle.CHATML:
+            ret = '' if system_prompt == '' else system_prompt + self.sep + '\n'
+            for role, message in self.messages:
+                if message:
+                    ret += role + '\n' + message + self.sep + '\n'
+                else:
+                    ret += role + '\n'
+            return ret
+        elif self.sep_style == SeparatorStyle.CHATGLM3:
+            ret = ''
+            if self.system_message:
+                ret += system_prompt
+            for role, message in self.messages:
+                if message:
+                    ret += role + '\n' + ' ' + message
+                else:
+                    ret += role
+            return ret
+        elif self.sep_style == SeparatorStyle.CHATINTERN:
+            # source: https://huggingface.co/internlm/internlm-chat-7b-8k/blob/bd546fa984b4b0b86958f56bf37f94aa75ab8831/modeling_internlm.py#L771
+            seps = [self.sep, self.sep2]
+            ret = system_prompt
+            for i, (role, message) in enumerate(self.messages):
+                # if i % 2 == 0:
+                #     ret += "<s>"
+                if message:
+                    ret += role + ':' + message + seps[i % 2] + '\n'
+                else:
+                    ret += role + ':'
+            return ret
+        elif self.sep_style == SeparatorStyle.DOLLY:
+            seps = [self.sep, self.sep2]
+            ret = system_prompt
+            for i, (role, message) in enumerate(self.messages):
+                if message:
+                    ret += role + ':\n' + message + seps[i % 2]
+                    if i % 2 == 1:
+                        ret += '\n\n'
+                else:
+                    ret += role + ':\n'
+            return ret
+        elif self.sep_style == SeparatorStyle.PHOENIX:
+            ret = system_prompt
+            for role, message in self.messages:
+                if message:
+                    ret += role + ': ' + '<s>' + message + '</s>'
+                else:
+                    ret += role + ': ' + '<s>'
+            return ret
+        elif self.sep_style == SeparatorStyle.ROBIN:
+            ret = system_prompt + self.sep
+            for role, message in self.messages:
+                if message:
+                    ret += role + ':\n' + message + self.sep
+                else:
+                    ret += role + ':\n'
+            return ret
+        elif self.sep_style == SeparatorStyle.FALCON_CHAT:
+            ret = ''
+            if self.system_message:
+                ret += system_prompt + self.sep
+            for role, message in self.messages:
+                if message:
+                    ret += role + ': ' + message + self.sep
+                else:
+                    ret += role + ':'
+
+            return ret
+        elif self.sep_style == SeparatorStyle.INTERNVL_ZH:
+            seps = [self.sep, self.sep2]
+            ret = self.system_message + seps[0]
+            for i, (role, message) in enumerate(self.messages):
+                if message:
+                    ret += role + ': ' + message + seps[i % 2]
+                else:
+                    ret += role + ':'
+            return ret
+        elif self.sep_style == SeparatorStyle.MPT:
+            ret = system_prompt + self.sep
+            for role, message in self.messages:
+                if message:
+                    if type(message) is tuple:
+                        message, _, _ = message
+                    ret += role + message + self.sep
+                else:
+                    ret += role
+            return ret
+        else:
+            raise ValueError(f'Invalid style: {self.sep_style}')
+
+    def set_system_message(self, system_message: str):
+        """Set the system message."""
+        self.system_message = system_message
+
+    def append_message(self, role: str, message: str):
+        """Append a new message."""
+        self.messages.append([role, message])
+
+    def update_last_message(self, message: str):
+        """Update the last output.
+
+        The last message is typically set to be None when constructing the prompt,
+        so we need to update it in-place after getting the response from a model.
+        """
+        self.messages[-1][1] = message
+
+    def to_gradio_chatbot(self):
+        """Convert the conversation to gradio chatbot format."""
+        ret = []
+        for i, (role, msg) in enumerate(self.messages[self.offset :]):
+            if i % 2 == 0:
+                ret.append([msg, None])
+            else:
+                ret[-1][-1] = msg
+        return ret
+
+    def to_openai_api_messages(self):
+        """Convert the conversation to OpenAI chat completion format."""
+        ret = [{'role': 'system', 'content': self.system_message}]
+
+        for i, (_, msg) in enumerate(self.messages[self.offset :]):
+            if i % 2 == 0:
+                ret.append({'role': 'user', 'content': msg})
+            else:
+                if msg is not None:
+                    ret.append({'role': 'assistant', 'content': msg})
+        return ret
+
+    def copy(self):
+        return Conversation(
+            name=self.name,
+            system_template=self.system_template,
+            system_message=self.system_message,
+            roles=self.roles,
+            messages=[[x, y] for x, y in self.messages],
+            offset=self.offset,
+            sep_style=self.sep_style,
+            sep=self.sep,
+            sep2=self.sep2,
+            stop_str=self.stop_str,
+            stop_token_ids=self.stop_token_ids,
+        )
+
+    def dict(self):
+        return {
+            'template_name': self.name,
+            'system_message': self.system_message,
+            'roles': self.roles,
+            'messages': self.messages,
+            'offset': self.offset,
+        }
+
+
+# A global registry for all conversation templates
+conv_templates: Dict[str, Conversation] = {}
+
+
+def register_conv_template(template: Conversation, override: bool = False):
+    """Register a new conversation template."""
+    if not override:
+        assert (
+            template.name not in conv_templates
+        ), f'{template.name} has been registered.'
+
+    conv_templates[template.name] = template
+
+
+def get_conv_template(name: str) -> Conversation:
+    """Get a conversation template."""
+    return conv_templates[name].copy()
+
+
+# Both Hermes-2 and internlm2-chat are chatml-format conversation templates. The difference
+# is that during training, the preprocessing function for the Hermes-2 template doesn't add
+# <s> at the beginning of the tokenized sequence, while the internlm2-chat template does.
+# Therefore, they are completely equivalent during inference.
+register_conv_template(
+    Conversation(
+        name='Hermes-2',
+        system_template='<|im_start|>system\n{system_message}',
+        # note: The new system prompt was not used here to avoid changes in benchmark performance.
+        # system_message='我是书生·万象，英文名是InternVL，是由上海人工智能实验室、清华大学及多家合作单位联合开发的多模态大语言模型。',
+        system_message='你是由上海人工智能实验室联合商汤科技开发的书生多模态大模型，英文名叫InternVL, 是一个有用无害的人工智能助手。',
+        roles=('<|im_start|>user\n', '<|im_start|>assistant\n'),
+        sep_style=SeparatorStyle.MPT,
+        sep='<|im_end|>',
+        stop_str='<|endoftext|>',
+    )
+)
+
+
+register_conv_template(
+    Conversation(
+        name='internlm2-chat',
+        system_template='<|im_start|>system\n{system_message}',
+        # note: The new system prompt was not used here to avoid changes in benchmark performance.
+        # system_message='我是书生·万象，英文名是InternVL，是由上海人工智能实验室、清华大学及多家合作单位联合开发的多模态大语言模型。',
+        system_message='你是由上海人工智能实验室联合商汤科技开发的书生多模态大模型，英文名叫InternVL, 是一个有用无害的人工智能助手。',
+        roles=('<|im_start|>user\n', '<|im_start|>assistant\n'),
+        sep_style=SeparatorStyle.MPT,
+        sep='<|im_end|>',
+    )
+)
+
+
+register_conv_template(
+    Conversation(
+        name='phi3-chat',
+        system_template='<|system|>\n{system_message}',
+        # note: The new system prompt was not used here to avoid changes in benchmark performance.
+        # system_message='我是书生·万象，英文名是InternVL，是由上海人工智能实验室、清华大学及多家合作单位联合开发的多模态大语言模型。',
+        system_message='你是由上海人工智能实验室联合商汤科技开发的书生多模态大模型，英文名叫InternVL, 是一个有用无害的人工智能助手。',
+        roles=('<|user|>\n', '<|assistant|>\n'),
+        sep_style=SeparatorStyle.MPT,
+        sep='<|end|>',
+    )
+)
+
+
+register_conv_template(
+    Conversation(
+        name='internvl2_5',
+        system_template='<|im_start|>system\n{system_message}',
+        system_message='你是书生·万象，英文名是InternVL，是由上海人工智能实验室、清华大学及多家合作单位联合开发的多模态大语言模型。',
+        roles=('<|im_start|>user\n', '<|im_start|>assistant\n'),
+        sep_style=SeparatorStyle.MPT,
+        sep='<|im_end|>\n',
+    )
+)

eval/README.md

@@ -0,0 +1,59 @@
+# HPD-Parsing Evaluation & Benchmark
+
+Scripts to reproduce the HPD-Parsing **throughput (TPS)** numbers and the **OmniDocBench v1.6** accuracy, using the customized vLLM build (`<FORK>`/`<CHILD>` hierarchical parallel decoding + P-MTP speculative decoding).
+
+```
+eval/
+├── benchmark_tps.py     # vLLM batch inference: measures TPS and dumps raw predictions
+└── hpd_to_markdown.py   # converts <BLOCK>...<CHILD>... predictions to per-page markdown
+```
+
+The pipeline is decoupled into three steps: **infer → convert → evaluate**. A single inference run produces both the speed metrics and the prediction file that feeds the accuracy evaluation.
+
+## Prerequisites
+
+- The customized vLLM build that supports `<FORK>`/`<CHILD>` decoding and `RepetitionDetectionParams` (see the main [README](../README.md) install section).
+- The model weights `PaddlePaddle/HPD-Parsing` (including the `P-MTP/` speculative head).
+- The [OmniDocBench](https://github.com/opendatalab/OmniDocBench) evaluation dataset (`OmniDocBench.json` + `images/`) and its evaluation suite (for step 3).
+
+## Step 1 — Throughput benchmark + dump predictions
+
+`benchmark_tps.py` runs batched inference over an image folder, times the whole batch loop with `time.perf_counter()`, and reports TPS metrics. It also writes the raw predictions used by the accuracy evaluation, so you only run the model once.
+
+```bash
+MAX_PATCHES_WITH_RESIZE=true python eval/benchmark_tps.py
+```
+
+Edit the variables at the top of `__main__` to match your setup:
+
+- `model_path` / `model_path_medusa` — model dir and its `P-MTP/` head (default `PaddlePaddle/HPD-Parsing/`).
+- `root` — image folder (default `OmniDocBench_1_6/images/`).
+- `prompt` — `document parsing with fork.` enables hierarchical parallel decoding; use `document parsing.` for standard full-page parsing.
+- `batch_size` (default `512`), `max_model_len`, `max_num_seqs`, and the `speculative_config` (`num_speculative_tokens=6` for P-MTP). To measure the autoregressive baseline, remove `speculative_config`.
+
+Outputs:
+
+- Predictions -> `batch_512_pred_HPD-Parsing.json` (a list of `{index, img_path, pred}`), consumed by step 2.
+- Metrics -> `records/<ckpt>.txt`: Total Time, Throughput (Requests/s), Input/Output/Total Tokens/s, and average tokens per request.
+
+## Step 2 — Convert predictions to markdown
+
+`hpd_to_markdown.py` parses each `<BLOCK> <type> [bbox] <CHILD> <content>` prediction into a reading-order markdown file named after the source image (`<image_stem>.md`), which is the input format OmniDocBench's end2end evaluation expects.
+
+```bash
+python eval/hpd_to_markdown.py \
+    --input batch_512_pred_HPD-Parsing.json \
+    --out-md pred_md/HPD-Parsing/
+```
+
+## Step 3 — Run OmniDocBench end2end evaluation
+
+Use the official [OmniDocBench](https://github.com/opendatalab/OmniDocBench) suite. Point its end2end config at the markdown folder from step 2 and the ground-truth `OmniDocBench.json`, then run its evaluator to get the overall score and per-metric breakdown (text / formula / table / reading order).
+
+```bash
+git clone https://github.com/opendatalab/OmniDocBench.git
+# set the prediction folder to pred_md/HPD-Parsing/ and gt to OmniDocBench.json
+# in the end2end config, then run the OmniDocBench evaluation entry point
+```
+
+> The OmniDocBench evaluation code is a separate project with its own license and is intentionally not vendored here.

eval/benchmark_tps.py

@@ -0,0 +1,143 @@
+"""HPD-Parsing throughput (TPS) benchmark on vLLM.
+
+Batched offline inference over an image folder; reports TPS metrics and dumps
+the raw predictions consumed by hpd_to_markdown.py (model is run once).
+
+    MAX_PATCHES_WITH_RESIZE=true python eval/benchmark_tps.py
+"""
+import os
+
+os.environ.setdefault("MAX_PATCHES_WITH_RESIZE", "true")  # before importing vLLM
+
+import base64
+import json
+import time
+from typing import Any, Dict
+
+from vllm import LLM, SamplingParams
+from vllm.sampling_params import RepetitionDetectionParams
+
+PRED_OUT = "batch_512_pred_HPD-Parsing.json"
+
+llm = None
+sampling_params = None
+
+
+def encode_image_to_base64(image_path: str) -> str:
+    with open(image_path, "rb") as f:
+        return base64.b64encode(f.read()).decode("utf-8")
+
+
+def batch_predict_vllm(tasks, batch_size=512):
+    all_outputs = []
+    records = []
+    for start in range(0, len(tasks), batch_size):
+        batch_tasks = tasks[start:start + batch_size]
+        messages_list = [
+            [{
+                "role": "user",
+                "content": [
+                    {"type": "image_url", "image_url": {"url": f"data:image/jpeg;base64,{img_b64}"}},
+                    {"type": "text", "text": question},
+                ],
+            }]
+            for question, img_b64, _ in batch_tasks
+        ]
+        outputs = llm.chat(messages=messages_list, sampling_params=sampling_params)
+        all_outputs.extend(outputs)
+        for j, out in enumerate(outputs):
+            records.append({
+                "index": start + j,
+                "img_path": batch_tasks[j][-1],
+                "pred": out.outputs[0].text,
+            })
+    return all_outputs, records
+
+
+def parse_ckpt_name(model_path: str) -> str:
+    parts = model_path.rstrip('/').split('/')
+    return '_'.join(p for p in parts if p and p not in ['checkpoints', 'output', 'rank_0'])
+
+
+def record_metrics(model_path: str, metrics: Dict[str, Any], output_dir: str = "./records"):
+    os.makedirs(output_dir, exist_ok=True)
+    output_file = os.path.join(output_dir, f"{parse_ckpt_name(model_path)}.txt")
+    with open(output_file, 'w', encoding='utf-8') as f:
+        f.write(f"Model Path: {model_path}\n")
+        f.write(f"Checkpoint: {parse_ckpt_name(model_path)}\n")
+        f.write("=" * 50 + "\n")
+        for key, value in metrics.items():
+            f.write(f"{key}: {value}\n")
+    print(f"Metrics saved to: {output_file}")
+
+
+if __name__ == '__main__':
+    model_path = "PaddlePaddle/HPD-Parsing/"
+    model_path_medusa = "PaddlePaddle/HPD-Parsing/P-MTP/"
+    root = "OmniDocBench_1_6/images/"
+    prompt = "document parsing with fork."
+    batch_size = 512
+
+    llm = LLM(
+        model=model_path,
+        tensor_parallel_size=1,
+        gpu_memory_utilization=0.9,
+        enable_prefix_caching=True,
+        attention_backend="FLASHINFER",
+        trust_remote_code=True,
+        max_model_len=24000,
+        max_num_seqs=512,
+        speculative_config={"method": "medusa", "model": model_path_medusa, "num_speculative_tokens": 6},
+    )
+    sampling_params = SamplingParams(
+        temperature=0,
+        max_tokens=8000,
+        stop=None,
+        repetition_detection=RepetitionDetectionParams(
+            min_pattern_size=64,
+            max_pattern_size=128,
+            min_count=10,
+        ),
+    )
+
+    valid_extensions = {'.jpg', '.jpeg', '.png', '.bmp', '.gif', '.webp'}
+    image_paths = [
+        os.path.join(root, p) for p in os.listdir(root)
+        if os.path.splitext(p)[1].lower() in valid_extensions
+    ]
+    tasks = [(prompt, encode_image_to_base64(p), p) for p in image_paths]
+    print(f"Total images: {len(tasks)}")
+
+    start_time = time.perf_counter()
+    outputs, records = batch_predict_vllm(tasks, batch_size=batch_size)
+    end_time = time.perf_counter()
+
+    with open(PRED_OUT, "w", encoding="utf-8") as fp:
+        json.dump(records, fp, indent=4, ensure_ascii=False)
+    print(f"Predictions saved to: {PRED_OUT}")
+
+    total_time = end_time - start_time
+    total_input_tokens = sum(len(out.prompt_token_ids) for out in outputs)
+    total_output_tokens = sum(len(out.outputs[0].token_ids) for out in outputs)
+
+    print(f"Total Time: {total_time:.2f} s")
+    print(f"Throughput (Requests/s): {len(tasks) / total_time:.2f}")
+    print(f"Input Tokens/s: {total_input_tokens / total_time:.2f}")
+    print(f"Output Tokens/s: {total_output_tokens / total_time:.2f}")
+    print(f"Total Tokens/s: {(total_input_tokens + total_output_tokens) / total_time:.2f}")
+    print(f"Avg Input Tokens per Request: {total_input_tokens / len(tasks):.2f}")
+    print(f"Avg Output Tokens per Request: {total_output_tokens / len(tasks):.2f}")
+
+    metrics = {
+        "Total Time (s)": f"{total_time:.2f}",
+        "Throughput (Requests/s)": f"{len(tasks) / total_time:.2f}",
+        "Input Tokens/s": f"{total_input_tokens / total_time:.2f}",
+        "Output Tokens/s": f"{total_output_tokens / total_time:.2f}",
+        "Total Tokens/s": f"{(total_input_tokens + total_output_tokens) / total_time:.2f}",
+        "Avg Input Tokens per Request": f"{total_input_tokens / len(tasks):.2f}",
+        "Avg Output Tokens per Request": f"{total_output_tokens / len(tasks):.2f}",
+        "Total Requests": len(tasks),
+        "Total Input Tokens": total_input_tokens,
+        "Total Output Tokens": total_output_tokens,
+    }
+    record_metrics(model_path, metrics)

eval/hpd_to_markdown.py

@@ -0,0 +1,206 @@
+"""Convert HPD-Parsing predictions (JSON) into per-page markdown for OmniDocBench.
+
+Input : JSON, a list of ``{img_path, pred}`` (pred is the ``<BLOCK> <type> [bbox]
+        <CHILD> <content>`` stream from ``document parsing with fork.``).
+Output: a folder of ``<image_stem>.md`` files matching the OmniDocBench GT paths.
+
+    python hpd_to_markdown.py --input preds.json --out-md pred_md/ \
+        --simplify-left-paren --clean-formula-tail --norm-formula-flag --wrap-cjk-arith
+"""
+
+import argparse
+import json
+import os
+import re
+from pathlib import Path
+
+
+_TALL = re.compile(
+    r'\\d?frac|\\tfrac|\\cfrac|\\binom|\\sqrt'
+    r'|\\sum|\\prod|\\coprod|\\int|\\iint|\\iiint|\\oint'
+    r'|\\bigcup|\\bigcap|\\bigoplus|\\bigotimes|\\bigsqcup'
+    r'|\\begin\{'
+    r'|\\overbrace|\\underbrace|\\overset|\\underset|\\stackrel'
+    r'|\\substack|\\atop|\\\\'
+)
+
+
+def _scan_delims(s):
+    out = []
+    for m in re.finditer(r'\\(left|right)\s*', s):
+        dm = re.match(r'\\[a-zA-Z]+|\\.|.', s[m.end():])
+        if not dm:
+            continue
+        out.append({'kind': m.group(1), 'delim': dm.group(0),
+                    'start': m.start(), 'end': m.end() + dm.end()})
+    return out
+
+
+def simplify_left_right(s: str) -> str:
+    """Downgrade `\\left( ... \\right)` with no tall inner structure to plain `( )`."""
+    if '\\left' not in s:
+        return s
+    stack, pairs = [], []
+    for d in _scan_delims(s):
+        if d['kind'] == 'left':
+            stack.append(d)
+        elif stack:
+            pairs.append((stack.pop(), d))
+    edits = []
+    for L, R in pairs:
+        if L['delim'] == '(' and R['delim'] == ')' and not _TALL.search(s[L['end']:R['start']]):
+            edits.append((L['start'], L['end'], '('))
+            edits.append((R['start'], R['end'], ')'))
+    for st, en, rep in sorted(edits, key=lambda x: x[0], reverse=True):
+        s = s[:st] + rep + s[en:]
+    return s
+
+
+_ELLIPSIS = r'(?:\\dots|\\cdots|\\ldots|\\dotsb|\\dotsc)'
+_CLOSER = r'(?:\\right\s*[.\}\]\)]|\\end\s*\{(?:array|matrix|cases|bmatrix|pmatrix|vmatrix|smallmatrix)\})'
+_TAIL_WRAP = re.compile(r'^(?P<core>.*?)(?P<wrap>\s*(?:\\\]|\\\)|\$\$))?\s*$', re.DOTALL)
+
+
+def clean_formula_tail(s: str) -> str:
+    """Strip degenerate formula tails (repeated/dangling ellipses, stray `\\quad`)."""
+    if not s:
+        return s
+    m = _TAIL_WRAP.match(s)
+    core, wrap = m.group('core'), m.group('wrap') or ''
+    prev = None
+    while prev != core:
+        prev = core
+        core = re.sub(r'(' + _ELLIPSIS + r')(?:\s*' + _ELLIPSIS + r')+', r'\1', core)
+        core = re.sub(r'(?P<keep>' + _CLOSER + r')\s*(?:\\q?quad\s*)*' + _ELLIPSIS + r'\s*$',
+                      lambda mm: mm.group('keep'), core)
+        core = re.sub(r'(?:\s*\\q?quad)+\s*' + _ELLIPSIS + r'\s*$', '', core)
+        core = re.sub(r'(?:\s*\\q?quad)+\s*$', '', core)
+        core = core.rstrip()
+    return core + wrap
+
+
+_OP_MAP = {
+    '≈': r'\approx', '≠': r'\neq', '≤': r'\leq', '≥': r'\geq', '×': r'\times',
+    '÷': r'\div', '±': r'\pm', '∓': r'\mp', '·': r'\cdot', '∙': r'\cdot',
+    '⋅': r'\cdot', '∗': '*', '−': '-', '≡': r'\equiv', '∝': r'\propto',
+    '∞': r'\infty', '√': r'\sqrt', '→': r'\to', '≪': r'\ll', '≫': r'\gg',
+}
+_ARITH_ALLOWED = re.compile(r'^[0-9A-Za-z\s=+\-*/^_().,:;<>|%!\u4e00-\u9fff' + ''.join(_OP_MAP.keys()) + r']+$')
+_ARITH_HASOP = re.compile(r'[=+\-*/' + ''.join(_OP_MAP.keys()) + r']')
+_KNOWN_FUNCS = {'sin', 'cos', 'tan', 'cot', 'sec', 'csc', 'log', 'ln', 'exp',
+                'lim', 'max', 'min', 'det', 'mod', 'arcsin', 'arccos', 'arctan', 'sqrt'}
+_CJK_RUN = re.compile(r'[\u4e00-\u9fff]+')
+_MATH_SPAN = re.compile(r'(\\\[.*?\\\]|\$\$.*?\$\$|\\\(.*?\\\)|\$.*?\$)', re.DOTALL)
+
+WRAP_CJK_IN_ARITH = True
+
+
+def _convert_unicode_ops(s: str) -> str:
+    for k, v in _OP_MAP.items():
+        s = s.replace(k, (v + ' ') if v.startswith('\\') else v)
+    if WRAP_CJK_IN_ARITH:
+        s = _CJK_RUN.sub(lambda m: r'\text{' + m.group(0) + '}', s)
+    return re.sub(r'[ \t]{2,}', ' ', s)
+
+
+def _is_pure_arith_line(line: str) -> bool:
+    t = line.strip()
+    if not t or '\\(' in t or '\\[' in t or '$' in t or '<' in t:
+        return False
+    if not WRAP_CJK_IN_ARITH and re.search(r'[\u4e00-\u9fff]', t):
+        return False
+    if not _ARITH_ALLOWED.match(t) or not _ARITH_HASOP.search(t):
+        return False
+    return all(w.lower() in _KNOWN_FUNCS for w in re.findall(r'[A-Za-z]{2,}', t))
+
+
+def normalize_arith(text: str) -> str:
+    """Normalize Unicode operators to LaTeX and wrap pure-arithmetic lines as `\\( .. \\)`."""
+    if not text:
+        return text
+    text = _MATH_SPAN.sub(lambda m: _convert_unicode_ops(m.group(0)), text)
+    out = []
+    for line in text.split('\n'):
+        if _is_pure_arith_line(line):
+            out.append('\\( ' + _convert_unicode_ops(line.strip()) + ' \\)')
+        else:
+            out.append(line)
+    return '\n'.join(out)
+
+
+def remove_block_fork_tags(result, simplify_left_paren=True, clean_formula_tail_flag=True,
+                           norm_formula_flag=True):
+    """Split on `<BLOCK>`, keep the text after each `<CHILD>`, and join in reading order."""
+    seg_pattern = re.compile(r'[^<]*<CHILD>(.*)', re.DOTALL)
+    lines = []
+    for seg in result.split('<BLOCK>')[1:]:
+        cat_m = re.match(r'\s*([a-zA-Z_]+)', seg)
+        if cat_m and cat_m.group(1).lower() in ['chart', 'seal']:
+            continue
+        m = seg_pattern.match(seg)
+        if not m:
+            continue
+        text = m.group(1).strip()
+        text = re.sub(r'\b\w+\s*\[\s*[-\d.,\s]+\]\s*<(?:FORK|CHILD|BLOCK)>', '', text)
+        text = re.sub(r'<(?:FORK|CHILD|BLOCK)>', '', text).strip()
+        text = text.replace('The image is too blurry to recognize any text content.', '').strip()
+        text = text.replace("The image contains no text or characters. It is a graphical element (a horizontal line with a vertical line) and does not contain any chart, graph, or data points that can be extracted. Therefore, the correct OCR output is an empty string.", "").strip()
+        if not text or text == '[Non-Text]':
+            continue
+        if text.startswith('\\[') and not text.endswith('\n\\]'):
+            text += '\n\\]'
+        if text.startswith('<table>') and not text.endswith('</table>'):
+            text += '</table>'
+        if '\\[\n' in text and '\\\\' not in text:
+            text = text.replace('\\[\n', '\\(').replace('\n\\]', '\\)')
+        text = text.replace('\\) \\(', '\\)\n\n\\(')
+        if '÷' in text and '\\(' not in text:
+            text = '\\( ' + text + ' \\)'
+        text = re.sub(r'\\tag\s*\{[^{}]*\}', '', text)
+        text = text.replace('\\supset', '\\sqsupset')
+        if simplify_left_paren:
+            text = simplify_left_right(text)
+        if clean_formula_tail_flag:
+            text = clean_formula_tail(text)
+        if norm_formula_flag:
+            text = normalize_arith(text)
+        lines.append(text)
+    return '\n\n'.join(lines).strip()
+
+
+def basename_to_md_name(img_path: str) -> str:
+    return os.path.splitext(os.path.basename(img_path))[0] + ".md"
+
+
+def convert_json(in_path, out_md_dir, simplify_left_paren=True,
+                 clean_formula_tail_flag=True, norm_formula_flag=True) -> int:
+    with open(in_path, "r", encoding="utf-8") as f:
+        rows = json.load(f)
+    os.makedirs(out_md_dir, exist_ok=True)
+    n = 0
+    for row in rows:
+        img_path = row.get("img_path") or row.get("image_path")
+        pred = row.get("pred") or row.get("prediction") or ""
+        if not img_path:
+            continue
+        md = remove_block_fork_tags(pred, simplify_left_paren, clean_formula_tail_flag, norm_formula_flag)
+        with open(os.path.join(out_md_dir, basename_to_md_name(img_path)), "w", encoding="utf-8") as f:
+            f.write(md)
+        n += 1
+    return n
+
+
+def main():
+    ap = argparse.ArgumentParser()
+    ap.add_argument("--input", required=True, help="json path (list of {img_path, pred})")
+    ap.add_argument("--out-md", required=True, help="output markdown folder")
+    args = ap.parse_args()
+
+    if Path(args.input).suffix.lower() != ".json":
+        raise SystemExit(f"unsupported extension: {Path(args.input).suffix} (expects .json)")
+    n = convert_json(args.input, args.out_md)
+    print(f"[ok] wrote {n} markdown files -> {args.out_md}")
+
+
+if __name__ == "__main__":
+    main()

generation_config.json

@@ -0,0 +1,5 @@
+{
+  "_from_model_config": true,
+  "eos_token_id": 151645,
+  "transformers_version": "4.56.2"
+}

image_preprocess.py

@@ -0,0 +1,84 @@
+"""Image preprocessing utilities for HPD-Parsing (transformers path).
+
+Mirrors vLLM's InternVL dynamic tiling path with ``MAX_PATCHES_WITH_RESIZE=true``:
+resize to the closest aspect ratio in a ``(min_num, max_num)`` grid, split into
+``448x448`` tiles, and optionally append a thumbnail.
+"""
+
+import torch
+import torchvision.transforms as T
+from torchvision.transforms.functional import InterpolationMode
+from PIL import Image
+
+IMAGENET_MEAN, IMAGENET_STD = (0.485, 0.456, 0.406), (0.229, 0.224, 0.225)
+
+IMAGE_SIZE = 448
+MIN_DYNAMIC_PATCH = 1
+MAX_DYNAMIC_PATCH = 24
+USE_THUMBNAIL = True
+
+
+def build_transform(input_size=IMAGE_SIZE):
+    return T.Compose([
+        T.Lambda(lambda img: img.convert("RGB")),
+        T.Resize((input_size, input_size), interpolation=InterpolationMode.BICUBIC),
+        T.ToTensor(),
+        T.Normalize(IMAGENET_MEAN, IMAGENET_STD),
+    ])
+
+
+def get_target_ratios(min_num, max_num):
+    ratios = {(i, j)
+              for n in range(min_num, max_num + 1)
+              for i in range(1, n + 1) for j in range(1, n + 1)
+              if min_num <= i * j <= max_num}
+    return sorted(ratios, key=lambda x: x[0] * x[1])
+
+
+def find_closest_aspect_ratio_optim(aspect_ratio, target_ratios, width, height,
+                                    image_size, top_k=3, ar_threshold=0.2):
+    area = width * height
+    candidates = []
+    for ratio in target_ratios:
+        ar_diff = abs(aspect_ratio - ratio[0] / ratio[1])
+        if ar_threshold is not None and ar_diff > ar_threshold:
+            continue
+        area_diff = abs(area - image_size * image_size * ratio[0] * ratio[1])
+        candidates.append((ratio, area_diff, ar_diff))
+    if not candidates:  # fall back to no aspect-ratio filtering
+        for ratio in target_ratios:
+            ar_diff = abs(aspect_ratio - ratio[0] / ratio[1])
+            area_diff = abs(area - image_size * image_size * ratio[0] * ratio[1])
+            candidates.append((ratio, area_diff, ar_diff))
+    candidates.sort(key=lambda x: x[1])
+    top = candidates[:top_k]
+    top.sort(key=lambda x: x[2])
+    return top[0][0]
+
+
+def dynamic_preprocess(image, target_ratios, image_size=IMAGE_SIZE, use_thumbnail=USE_THUMBNAIL):
+    w, h = image.size
+    ratio = find_closest_aspect_ratio_optim(w / h, target_ratios, w, h, image_size)
+    tw, th = image_size * ratio[0], image_size * ratio[1]
+    blocks = ratio[0] * ratio[1]
+    resized = image.resize((tw, th))
+    cols = tw // image_size
+    tiles = []
+    for i in range(blocks):
+        box = ((i % cols) * image_size, (i // cols) * image_size,
+               ((i % cols) + 1) * image_size, ((i // cols) + 1) * image_size)
+        tiles.append(resized.crop(box))
+    if use_thumbnail and blocks != 1:
+        tiles.append(image.resize((image_size, image_size)))
+    return tiles
+
+
+def load_image(path):
+    image = Image.open(path).convert("RGB")
+    min_num, max_num = MIN_DYNAMIC_PATCH, MAX_DYNAMIC_PATCH
+    if USE_THUMBNAIL and max_num != 1:
+        max_num += 1
+    target_ratios = get_target_ratios(min_num, max_num)
+    transform = build_transform(IMAGE_SIZE)
+    tiles = dynamic_preprocess(image, target_ratios, IMAGE_SIZE, USE_THUMBNAIL)
+    return torch.stack([transform(t) for t in tiles])

modeling_intern_vit.py

@@ -0,0 +1,433 @@
+# --------------------------------------------------------
+# InternVL
+# Copyright (c) 2024 OpenGVLab
+# Licensed under The MIT License [see LICENSE for details]
+# --------------------------------------------------------
+
+from typing import Optional, Tuple, Union
+
+import torch
+import torch.nn.functional as F
+import torch.utils.checkpoint
+from einops import rearrange
+from timm.layers import DropPath
+from torch import nn
+from transformers.activations import ACT2FN
+from transformers.modeling_outputs import (BaseModelOutput,
+                                           BaseModelOutputWithPooling)
+from transformers.modeling_utils import PreTrainedModel
+from transformers.utils import logging
+
+from .configuration_intern_vit import InternVisionConfig
+
+try:
+    from flash_attn.bert_padding import pad_input, unpad_input
+    from flash_attn.flash_attn_interface import \
+        flash_attn_varlen_qkvpacked_func
+    has_flash_attn = True
+except:
+    print('FlashAttention2 is not installed.')
+    has_flash_attn = False
+
+logger = logging.get_logger(__name__)
+
+
+class FlashAttention(nn.Module):
+    """Implement the scaled dot product attention with softmax.
+    Arguments
+    ---------
+        softmax_scale: The temperature to use for the softmax attention.
+                      (default: 1/sqrt(d_keys) where d_keys is computed at
+                      runtime)
+        attention_dropout: The dropout rate to apply to the attention
+                           (default: 0.0)
+    """
+
+    def __init__(self, softmax_scale=None, attention_dropout=0.0, device=None, dtype=None):
+        super().__init__()
+        self.softmax_scale = softmax_scale
+        self.dropout_p = attention_dropout
+
+    def forward(self, qkv, key_padding_mask=None, causal=False, cu_seqlens=None,
+                max_s=None, need_weights=False):
+        """Implements the multihead softmax attention.
+        Arguments
+        ---------
+            qkv: The tensor containing the query, key, and value. (B, S, 3, H, D) if key_padding_mask is None
+                if unpadded: (nnz, 3, h, d)
+            key_padding_mask: a bool tensor of shape (B, S)
+        """
+        assert not need_weights
+        assert qkv.dtype in [torch.float16, torch.bfloat16]
+        assert qkv.is_cuda
+
+        if cu_seqlens is None:
+            batch_size = qkv.shape[0]
+            seqlen = qkv.shape[1]
+            if key_padding_mask is None:
+                qkv = rearrange(qkv, 'b s ... -> (b s) ...')
+                max_s = seqlen
+                cu_seqlens = torch.arange(0, (batch_size + 1) * seqlen, step=seqlen, dtype=torch.int32,
+                                          device=qkv.device)
+                output = flash_attn_varlen_qkvpacked_func(
+                    qkv, cu_seqlens, max_s, self.dropout_p if self.training else 0.0,
+                    softmax_scale=self.softmax_scale, causal=causal
+                )
+                output = rearrange(output, '(b s) ... -> b s ...', b=batch_size)
+            else:
+                nheads = qkv.shape[-2]
+                x = rearrange(qkv, 'b s three h d -> b s (three h d)')
+                x_unpad, indices, cu_seqlens, max_s = unpad_input(x, key_padding_mask)
+                x_unpad = rearrange(x_unpad, 'nnz (three h d) -> nnz three h d', three=3, h=nheads)
+                output_unpad = flash_attn_varlen_qkvpacked_func(
+                    x_unpad, cu_seqlens, max_s, self.dropout_p if self.training else 0.0,
+                    softmax_scale=self.softmax_scale, causal=causal
+                )
+                output = rearrange(pad_input(rearrange(output_unpad, 'nnz h d -> nnz (h d)'),
+                                             indices, batch_size, seqlen),
+                                   'b s (h d) -> b s h d', h=nheads)
+        else:
+            assert max_s is not None
+            output = flash_attn_varlen_qkvpacked_func(
+                qkv, cu_seqlens, max_s, self.dropout_p if self.training else 0.0,
+                softmax_scale=self.softmax_scale, causal=causal
+            )
+
+        return output, None
+
+
+class InternRMSNorm(nn.Module):
+    def __init__(self, hidden_size, eps=1e-6):
+        super().__init__()
+        self.weight = nn.Parameter(torch.ones(hidden_size))
+        self.variance_epsilon = eps
+
+    def forward(self, hidden_states):
+        input_dtype = hidden_states.dtype
+        hidden_states = hidden_states.to(torch.float32)
+        variance = hidden_states.pow(2).mean(-1, keepdim=True)
+        hidden_states = hidden_states * torch.rsqrt(variance + self.variance_epsilon)
+        return self.weight * hidden_states.to(input_dtype)
+
+
+try:
+    from apex.normalization import FusedRMSNorm
+
+    InternRMSNorm = FusedRMSNorm  # noqa
+
+    logger.info('Discovered apex.normalization.FusedRMSNorm - will use it instead of InternRMSNorm')
+except ImportError:
+    # using the normal InternRMSNorm
+    pass
+except Exception:
+    logger.warning('discovered apex but it failed to load, falling back to InternRMSNorm')
+    pass
+
+
+NORM2FN = {
+    'rms_norm': InternRMSNorm,
+    'layer_norm': nn.LayerNorm,
+}
+
+
+class InternVisionEmbeddings(nn.Module):
+    def __init__(self, config: InternVisionConfig):
+        super().__init__()
+        self.config = config
+        self.embed_dim = config.hidden_size
+        self.image_size = config.image_size
+        self.patch_size = config.patch_size
+
+        self.class_embedding = nn.Parameter(
+            torch.randn(1, 1, self.embed_dim),
+        )
+
+        self.patch_embedding = nn.Conv2d(
+            in_channels=3, out_channels=self.embed_dim, kernel_size=self.patch_size, stride=self.patch_size
+        )
+
+        self.num_patches = (self.image_size // self.patch_size) ** 2
+        self.num_positions = self.num_patches + 1
+
+        self.position_embedding = nn.Parameter(torch.randn(1, self.num_positions, self.embed_dim))
+
+    def _get_pos_embed(self, pos_embed, H, W):
+        target_dtype = pos_embed.dtype
+        pos_embed = pos_embed.float().reshape(
+            1, self.image_size // self.patch_size, self.image_size // self.patch_size, -1).permute(0, 3, 1, 2)
+        pos_embed = F.interpolate(pos_embed, size=(H, W), mode='bicubic', align_corners=False). \
+            reshape(1, -1, H * W).permute(0, 2, 1).to(target_dtype)
+        return pos_embed
+
+    def forward(self, pixel_values: torch.FloatTensor) -> torch.Tensor:
+        target_dtype = self.patch_embedding.weight.dtype
+        patch_embeds = self.patch_embedding(pixel_values)  # shape = [*, channel, width, height]
+        batch_size, _, height, width = patch_embeds.shape
+        patch_embeds = patch_embeds.flatten(2).transpose(1, 2)
+        class_embeds = self.class_embedding.expand(batch_size, 1, -1).to(target_dtype)
+        embeddings = torch.cat([class_embeds, patch_embeds], dim=1)
+        position_embedding = torch.cat([
+            self.position_embedding[:, :1, :],
+            self._get_pos_embed(self.position_embedding[:, 1:, :], height, width)
+        ], dim=1)
+        embeddings = embeddings + position_embedding.to(target_dtype)
+        return embeddings
+
+
+class InternAttention(nn.Module):
+    """Multi-headed attention from 'Attention Is All You Need' paper"""
+
+    def __init__(self, config: InternVisionConfig):
+        super().__init__()
+        self.config = config
+        self.embed_dim = config.hidden_size
+        self.num_heads = config.num_attention_heads
+        self.use_flash_attn = config.use_flash_attn and has_flash_attn
+        if config.use_flash_attn and not has_flash_attn:
+            print('Warning: Flash Attention is not available, use_flash_attn is set to False.')
+        self.head_dim = self.embed_dim // self.num_heads
+        if self.head_dim * self.num_heads != self.embed_dim:
+            raise ValueError(
+                f'embed_dim must be divisible by num_heads (got `embed_dim`: {self.embed_dim} and `num_heads`:'
+                f' {self.num_heads}).'
+            )
+
+        self.scale = self.head_dim ** -0.5
+        self.qkv = nn.Linear(self.embed_dim, 3 * self.embed_dim, bias=config.qkv_bias)
+        self.attn_drop = nn.Dropout(config.attention_dropout)
+        self.proj_drop = nn.Dropout(config.dropout)
+
+        self.qk_normalization = config.qk_normalization
+
+        if self.qk_normalization:
+            self.q_norm = InternRMSNorm(self.embed_dim, eps=config.layer_norm_eps)
+            self.k_norm = InternRMSNorm(self.embed_dim, eps=config.layer_norm_eps)
+
+        if self.use_flash_attn:
+            self.inner_attn = FlashAttention(attention_dropout=config.attention_dropout)
+        self.proj = nn.Linear(self.embed_dim, self.embed_dim)
+
+    def _naive_attn(self, x):
+        B, N, C = x.shape
+        qkv = self.qkv(x).reshape(B, N, 3, self.num_heads, C // self.num_heads).permute(2, 0, 3, 1, 4)
+        q, k, v = qkv.unbind(0)  # make torchscript happy (cannot use tensor as tuple)
+
+        if self.qk_normalization:
+            B_, H_, N_, D_ = q.shape
+            q = self.q_norm(q.transpose(1, 2).flatten(-2, -1)).view(B_, N_, H_, D_).transpose(1, 2)
+            k = self.k_norm(k.transpose(1, 2).flatten(-2, -1)).view(B_, N_, H_, D_).transpose(1, 2)
+
+        attn = ((q * self.scale) @ k.transpose(-2, -1))
+        attn = attn.softmax(dim=-1)
+        attn = self.attn_drop(attn)
+
+        x = (attn @ v).transpose(1, 2).reshape(B, N, C)
+        x = self.proj(x)
+        x = self.proj_drop(x)
+        return x
+
+    def _flash_attn(self, x, key_padding_mask=None, need_weights=False):
+        qkv = self.qkv(x)
+        qkv = rearrange(qkv, 'b s (three h d) -> b s three h d', three=3, h=self.num_heads)
+
+        if self.qk_normalization:
+            q, k, v = qkv.unbind(2)
+            q = self.q_norm(q.flatten(-2, -1)).view(q.shape)
+            k = self.k_norm(k.flatten(-2, -1)).view(k.shape)
+            qkv = torch.stack([q, k, v], dim=2)
+
+        context, _ = self.inner_attn(
+            qkv, key_padding_mask=key_padding_mask, need_weights=need_weights, causal=False
+        )
+        outs = self.proj(rearrange(context, 'b s h d -> b s (h d)'))
+        outs = self.proj_drop(outs)
+        return outs
+
+    def forward(self, hidden_states: torch.Tensor) -> torch.Tensor:
+        x = self._naive_attn(hidden_states) if not self.use_flash_attn else self._flash_attn(hidden_states)
+        return x
+
+
+class InternMLP(nn.Module):
+    def __init__(self, config: InternVisionConfig):
+        super().__init__()
+        self.config = config
+        self.act = ACT2FN[config.hidden_act]
+        self.fc1 = nn.Linear(config.hidden_size, config.intermediate_size)
+        self.fc2 = nn.Linear(config.intermediate_size, config.hidden_size)
+
+    def forward(self, hidden_states: torch.Tensor) -> torch.Tensor:
+        hidden_states = self.fc1(hidden_states)
+        hidden_states = self.act(hidden_states)
+        hidden_states = self.fc2(hidden_states)
+        return hidden_states
+
+
+class InternVisionEncoderLayer(nn.Module):
+    def __init__(self, config: InternVisionConfig, drop_path_rate: float):
+        super().__init__()
+        self.embed_dim = config.hidden_size
+        self.intermediate_size = config.intermediate_size
+        self.norm_type = config.norm_type
+
+        self.attn = InternAttention(config)
+        self.mlp = InternMLP(config)
+        self.norm1 = NORM2FN[self.norm_type](self.embed_dim, eps=config.layer_norm_eps)
+        self.norm2 = NORM2FN[self.norm_type](self.embed_dim, eps=config.layer_norm_eps)
+
+        self.ls1 = nn.Parameter(config.initializer_factor * torch.ones(self.embed_dim))
+        self.ls2 = nn.Parameter(config.initializer_factor * torch.ones(self.embed_dim))
+        self.drop_path1 = DropPath(drop_path_rate) if drop_path_rate > 0. else nn.Identity()
+        self.drop_path2 = DropPath(drop_path_rate) if drop_path_rate > 0. else nn.Identity()
+
+    def forward(
+            self,
+            hidden_states: torch.Tensor,
+    ) -> Tuple[torch.FloatTensor, Optional[torch.FloatTensor], Optional[Tuple[torch.FloatTensor]]]:
+        """
+        Args:
+            hidden_states (`Tuple[torch.FloatTensor, Optional[torch.FloatTensor]]`): input to the layer of shape `(batch, seq_len, embed_dim)`
+        """
+        hidden_states = hidden_states + self.drop_path1(self.attn(self.norm1(hidden_states).to(hidden_states.dtype)) * self.ls1)
+
+        hidden_states = hidden_states + self.drop_path2(self.mlp(self.norm2(hidden_states).to(hidden_states.dtype)) * self.ls2)
+
+        return hidden_states
+
+
+class InternVisionEncoder(nn.Module):
+    """
+    Transformer encoder consisting of `config.num_hidden_layers` self attention layers. Each layer is a
+    [`InternEncoderLayer`].
+
+    Args:
+        config (`InternConfig`):
+            The corresponding vision configuration for the `InternEncoder`.
+    """
+
+    def __init__(self, config: InternVisionConfig):
+        super().__init__()
+        self.config = config
+        # stochastic depth decay rule
+        dpr = [x.item() for x in torch.linspace(0, config.drop_path_rate, config.num_hidden_layers)]
+        self.layers = nn.ModuleList([
+            InternVisionEncoderLayer(config, dpr[idx]) for idx in range(config.num_hidden_layers)])
+        self.gradient_checkpointing = True
+
+    def forward(
+            self,
+            inputs_embeds,
+            output_hidden_states: Optional[bool] = None,
+            return_dict: Optional[bool] = None,
+    ) -> Union[Tuple, BaseModelOutput]:
+        r"""
+        Args:
+            inputs_embeds (`torch.FloatTensor` of shape `(batch_size, sequence_length, hidden_size)`):
+                Embedded representation of the inputs. Should be float, not int tokens.
+            output_hidden_states (`bool`, *optional*):
+                Whether or not to return the hidden states of all layers. See `hidden_states` under returned tensors
+                for more detail.
+            return_dict (`bool`, *optional*):
+                Whether or not to return a [`~utils.ModelOutput`] instead of a plain tuple.
+        """
+        output_hidden_states = (
+            output_hidden_states if output_hidden_states is not None else self.config.output_hidden_states
+        )
+        return_dict = return_dict if return_dict is not None else self.config.use_return_dict
+
+        encoder_states = () if output_hidden_states else None
+        hidden_states = inputs_embeds
+
+        for idx, encoder_layer in enumerate(self.layers):
+            if output_hidden_states:
+                encoder_states = encoder_states + (hidden_states,)
+            if self.gradient_checkpointing and self.training:
+                layer_outputs = torch.utils.checkpoint.checkpoint(
+                    encoder_layer,
+                    hidden_states)
+            else:
+                layer_outputs = encoder_layer(
+                    hidden_states,
+                )
+            hidden_states = layer_outputs
+
+        if output_hidden_states:
+            encoder_states = encoder_states + (hidden_states,)
+
+        if not return_dict:
+            return tuple(v for v in [hidden_states, encoder_states] if v is not None)
+        return BaseModelOutput(
+            last_hidden_state=hidden_states, hidden_states=encoder_states
+        )
+
+
+class InternVisionModel(PreTrainedModel):
+    main_input_name = 'pixel_values'
+    _supports_flash_attn_2 = True
+    supports_gradient_checkpointing = True
+    config_class = InternVisionConfig
+    _no_split_modules = ['InternVisionEncoderLayer']
+    # support transformers 4.51.+
+    _tp_plan = ''
+
+    def __init__(self, config: InternVisionConfig):
+        super().__init__(config)
+        self.config = config
+
+        self.embeddings = InternVisionEmbeddings(config)
+        self.encoder = InternVisionEncoder(config)
+
+    def resize_pos_embeddings(self, old_size, new_size, patch_size):
+        pos_emb = self.embeddings.position_embedding
+        _, num_positions, embed_dim = pos_emb.shape
+        cls_emb = pos_emb[:, :1, :]
+        pos_emb = pos_emb[:, 1:, :].reshape(1, old_size // patch_size, old_size // patch_size, -1).permute(0, 3, 1, 2)
+        pos_emb = F.interpolate(pos_emb.float(), size=new_size // patch_size, mode='bicubic', align_corners=False)
+        pos_emb = pos_emb.to(cls_emb.dtype).reshape(1, embed_dim, -1).permute(0, 2, 1)
+        pos_emb = torch.cat([cls_emb, pos_emb], dim=1)
+        self.embeddings.position_embedding = nn.Parameter(pos_emb)
+        self.embeddings.image_size = new_size
+        logger.info('Resized position embeddings from {} to {}'.format(old_size, new_size))
+
+    def get_input_embeddings(self):
+        return self.embeddings
+
+    def forward(
+            self,
+            pixel_values: Optional[torch.FloatTensor] = None,
+            output_hidden_states: Optional[bool] = None,
+            return_dict: Optional[bool] = None,
+            pixel_embeds: Optional[torch.FloatTensor] = None,
+    ) -> Union[Tuple, BaseModelOutputWithPooling]:
+        output_hidden_states = (
+            output_hidden_states if output_hidden_states is not None else self.config.output_hidden_states
+        )
+        return_dict = return_dict if return_dict is not None else self.config.use_return_dict
+
+        if pixel_values is None and pixel_embeds is None:
+            raise ValueError('You have to specify pixel_values or pixel_embeds')
+
+        if pixel_embeds is not None:
+            hidden_states = pixel_embeds
+        else:
+            if len(pixel_values.shape) == 4:
+                hidden_states = self.embeddings(pixel_values)
+            else:
+                raise ValueError(f'wrong pixel_values size: {pixel_values.shape}')
+        encoder_outputs = self.encoder(
+            inputs_embeds=hidden_states,
+            output_hidden_states=output_hidden_states,
+            return_dict=return_dict,
+        )
+        last_hidden_state = encoder_outputs.last_hidden_state
+        pooled_output = last_hidden_state[:, 0, :]
+
+        if not return_dict:
+            return (last_hidden_state, pooled_output) + encoder_outputs[1:]
+
+        return BaseModelOutputWithPooling(
+            last_hidden_state=last_hidden_state,
+            pooler_output=pooled_output,
+            hidden_states=encoder_outputs.hidden_states,
+            attentions=encoder_outputs.attentions,
+        )

modeling_internvl_chat.py

@@ -0,0 +1,775 @@
+# --------------------------------------------------------
+# InternVL
+# Copyright (c) 2024 OpenGVLab
+# Licensed under The MIT License [see LICENSE for details]
+# --------------------------------------------------------
+
+import os
+import warnings
+from typing import List, Optional, Tuple, Union
+
+import torch.utils.checkpoint
+import torch.nn.functional as F
+import transformers
+from torch import nn
+from torch.nn import CrossEntropyLoss
+from transformers import GenerationConfig
+from transformers.cache_utils import DynamicCache
+from transformers.modeling_outputs import CausalLMOutputWithPast
+from transformers.modeling_utils import PreTrainedModel
+from transformers.utils import logging
+from transformers import LlamaForCausalLM, Qwen2ForCausalLM, Qwen3ForCausalLM, Qwen3MoeForCausalLM
+
+from .configuration_internvl_chat import InternVLChatConfig
+from .conversation import get_conv_template
+from .modeling_intern_vit import InternVisionModel, has_flash_attn
+
+logger = logging.get_logger(__name__)
+
+
+def version_cmp(v1, v2, op='eq'):
+    import operator
+
+    from packaging import version
+    op_func = getattr(operator, op)
+    return op_func(version.parse(v1), version.parse(v2))
+
+
+class HPDRMSNorm(nn.Module):
+    def __init__(self, hidden_size, eps=1e-6):
+        super().__init__()
+        self.weight = nn.Parameter(torch.ones(hidden_size))
+        self.variance_epsilon = eps
+
+    def forward(self, hidden_states):
+        input_dtype = hidden_states.dtype
+        hidden_states = hidden_states.to(torch.float32)
+        variance = hidden_states.pow(2).mean(-1, keepdim=True)
+        hidden_states = hidden_states * torch.rsqrt(variance + self.variance_epsilon)
+        return self.weight * hidden_states.to(input_dtype)
+
+
+class _HPDGatedMLP(nn.Module):
+    def __init__(self, hidden_size, intermediate_size):
+        super().__init__()
+        self.gate_proj = nn.Linear(hidden_size, intermediate_size, bias=False)
+        self.up_proj = nn.Linear(hidden_size, intermediate_size, bias=False)
+        self.down_proj = nn.Linear(intermediate_size, hidden_size, bias=False)
+
+    def forward(self, x):
+        return self.down_proj(F.silu(self.gate_proj(x)) * self.up_proj(x))
+
+
+class _HPDMTPLayer(nn.Module):
+    def __init__(self, hidden_size, intermediate_size):
+        super().__init__()
+        self.mlp = _HPDGatedMLP(hidden_size, intermediate_size)
+
+
+class HPDMultiTokenPredictor(nn.Module):
+    """Single-block Progressive Multi-Token Prediction head (P-MTP).
+
+    Mirrors the vLLM ``Qwen3_5MultiTokenPredictor`` module (loaded there as a
+    ``medusa`` speculative model) so its weights, stored under
+    ``language_model.mtp.*`` in the checkpoint, load without remapping. One MTP
+    block is applied autoregressively to draft several future tokens; each step
+    is conditioned on the previous step's predicted token. The token embedding
+    and the output head are shared with the backbone (the checkpoint contains no
+    ``mtp.embed_tokens``/``mtp.lm_head``), so ``step`` takes an already-embedded
+    ``prev_embeds`` tensor.
+    """
+
+    def __init__(self, hidden_size, intermediate_size, rms_norm_eps=1e-6):
+        super().__init__()
+        self.fc = nn.Linear(hidden_size * 2, hidden_size, bias=False)
+        self.pre_fc_norm_hidden = HPDRMSNorm(hidden_size, rms_norm_eps)
+        self.pre_fc_norm_embedding = HPDRMSNorm(hidden_size, rms_norm_eps)
+        self.layers = nn.ModuleList([_HPDMTPLayer(hidden_size, intermediate_size)])
+        self.norm = HPDRMSNorm(hidden_size, rms_norm_eps)
+
+    def step(self, hidden_states, prev_embeds):
+        # hidden_states: [B, H] backbone hidden that produced prev_token;
+        # prev_embeds: [B, H] backbone embedding of that token.
+        embeds = self.pre_fc_norm_embedding(prev_embeds)
+        hidden = self.pre_fc_norm_hidden(hidden_states)
+        x = self.fc(torch.cat([hidden, embeds], dim=-1))
+        mlp_out = self.layers[0].mlp(x)
+        return self.norm(mlp_out + x)
+
+
+class InternVLChatModel(PreTrainedModel):
+    config_class = InternVLChatConfig
+    main_input_name = 'pixel_values'
+    base_model_prefix = 'language_model'
+    _supports_flash_attn_2 = True
+    supports_gradient_checkpointing = True
+    _no_split_modules = [
+        "InternVisionModel",
+        "Qwen3DecoderLayer",
+    ]
+
+    # support transformers 4.51.+
+    _tp_plan = ''
+
+    def __init__(self, config: InternVLChatConfig, vision_model=None, language_model=None, use_flash_attn=True):
+        super().__init__(config)
+
+        assert version_cmp(transformers.__version__, '4.37.0', 'ge')
+        image_size = config.force_image_size or config.vision_config.image_size
+        patch_size = config.vision_config.patch_size
+        self.patch_size = patch_size
+        self.select_layer = config.select_layer
+        self.template = config.template
+        self.num_image_token = int((image_size // patch_size) ** 2 * (config.downsample_ratio ** 2))
+        self.downsample_ratio = config.downsample_ratio
+        self.ps_version = config.ps_version
+        use_flash_attn = use_flash_attn if has_flash_attn else False
+        config.vision_config.use_flash_attn = True if use_flash_attn else False
+        config.llm_config._attn_implementation = 'flash_attention_2' if use_flash_attn else 'eager'
+
+        logger.info(f'num_image_token: {self.num_image_token}')
+        logger.info(f'ps_version: {self.ps_version}')
+        if vision_model is not None:
+            self.vision_model = vision_model
+        else:
+            self.vision_model = InternVisionModel(config.vision_config)
+        if language_model is not None:
+            self.language_model = language_model
+        else:
+            architecture: str = config.llm_config.architectures[0]
+            if architecture == 'LlamaForCausalLM':
+                self.language_model = LlamaForCausalLM(config.llm_config)
+            elif architecture == 'Qwen2ForCausalLM':
+                self.language_model = Qwen2ForCausalLM(config.llm_config)
+            elif architecture == 'Qwen3MoeForCausalLM':
+                self.language_model = Qwen3MoeForCausalLM(config.llm_config)
+            elif architecture == 'Qwen3ForCausalLM':
+                self.language_model = Qwen3ForCausalLM(config.llm_config)
+            else:
+                raise NotImplementedError(f'{architecture} is not implemented.')
+
+        vit_hidden_size = config.vision_config.hidden_size
+        llm_hidden_size = config.llm_config.hidden_size
+
+        self.mlp1 = nn.Sequential(
+            nn.LayerNorm(vit_hidden_size * int(1 / self.downsample_ratio) ** 2),
+            nn.Linear(vit_hidden_size * int(1 / self.downsample_ratio) ** 2, llm_hidden_size),
+            nn.GELU(),
+            nn.Linear(llm_hidden_size, llm_hidden_size)
+        )
+
+        self.img_context_token_id = None
+        self.conv_template = get_conv_template(self.template)
+        self.system_message = self.conv_template.system_message
+
+        # P-MTP head + hierarchical fork decoding metadata (see generate_hpd).
+        self.fork_token_id = getattr(config, 'fork_token_id', None)
+        self.child_token_id = getattr(config, 'child_token_id', None)
+        self._mtp_weights_loaded = False
+        llm_cfg = config.llm_config
+        self.language_model.mtp = HPDMultiTokenPredictor(
+            hidden_size=llm_cfg.hidden_size,
+            intermediate_size=llm_cfg.intermediate_size,
+            rms_norm_eps=getattr(llm_cfg, 'rms_norm_eps', 1e-6),
+        )
+
+    def forward(
+            self,
+            pixel_values: torch.FloatTensor,
+            input_ids: torch.LongTensor = None,
+            attention_mask: Optional[torch.Tensor] = None,
+            position_ids: Optional[torch.LongTensor] = None,
+            image_flags: Optional[torch.LongTensor] = None,
+            past_key_values: Optional[List[torch.FloatTensor]] = None,
+            labels: Optional[torch.LongTensor] = None,
+            use_cache: Optional[bool] = None,
+            output_attentions: Optional[bool] = None,
+            output_hidden_states: Optional[bool] = None,
+            return_dict: Optional[bool] = None,
+    ) -> Union[Tuple, CausalLMOutputWithPast]:
+        return_dict = return_dict if return_dict is not None else self.config.use_return_dict
+
+        image_flags = image_flags.squeeze(-1)
+        input_embeds = self.language_model.get_input_embeddings()(input_ids).clone()
+
+        vit_embeds = self.extract_feature(pixel_values)
+        vit_embeds = vit_embeds[image_flags == 1]
+        vit_batch_size = pixel_values.shape[0]
+
+        B, N, C = input_embeds.shape
+        input_embeds = input_embeds.reshape(B * N, C)
+
+        # if torch.distributed.is_initialized() and torch.distributed.get_rank() == 0:
+        #     print(f'dynamic ViT batch size: {vit_batch_size}, images per sample: {vit_batch_size / B}, dynamic token length: {N}')
+
+        input_ids = input_ids.reshape(B * N)
+        selected = (input_ids == self.img_context_token_id)
+        try:
+            input_embeds[selected] = input_embeds[selected] * 0.0 + vit_embeds.reshape(-1, C)
+        except Exception as e:
+            vit_embeds = vit_embeds.reshape(-1, C)
+            print(f'warning: {e}, input_embeds[selected].shape={input_embeds[selected].shape}, '
+                  f'vit_embeds.shape={vit_embeds.shape}')
+            n_token = min(selected.sum(), vit_embeds.size(0))
+            input_embeds[selected][:n_token] = input_embeds[selected][:n_token] * 0.0 + vit_embeds[:n_token]
+
+        input_embeds = input_embeds.reshape(B, N, C)
+
+        outputs = self.language_model(
+            inputs_embeds=input_embeds,
+            attention_mask=attention_mask,
+            position_ids=position_ids,
+            past_key_values=past_key_values,
+            use_cache=use_cache,
+            output_attentions=output_attentions,
+            output_hidden_states=output_hidden_states,
+            return_dict=return_dict,
+        )
+        logits = outputs.logits
+
+        loss = None
+        if labels is not None:
+            # Shift so that tokens < n predict n
+            shift_logits = logits[..., :-1, :].contiguous()
+            shift_labels = labels[..., 1:].contiguous()
+            # Flatten the tokens
+            loss_fct = CrossEntropyLoss()
+            shift_logits = shift_logits.view(-1, self.language_model.config.vocab_size)
+            shift_labels = shift_labels.view(-1)
+            # Enable model parallelism
+            shift_labels = shift_labels.to(shift_logits.device)
+            loss = loss_fct(shift_logits, shift_labels)
+
+        if not return_dict:
+            output = (logits,) + outputs[1:]
+            return (loss,) + output if loss is not None else output
+
+        return CausalLMOutputWithPast(
+            loss=loss,
+            logits=logits,
+            past_key_values=outputs.past_key_values,
+            hidden_states=outputs.hidden_states,
+            attentions=outputs.attentions,
+        )
+
+    def pixel_shuffle(self, x, scale_factor=0.5):
+        n, w, h, c = x.size()
+        # N, W, H, C --> N, W, H * scale, C // scale
+        x = x.view(n, w, int(h * scale_factor), int(c / scale_factor))
+        # N, W, H * scale, C // scale --> N, H * scale, W, C // scale
+        x = x.permute(0, 2, 1, 3).contiguous()
+        # N, H * scale, W, C // scale --> N, H * scale, W * scale, C // (scale ** 2)
+        x = x.view(n, int(h * scale_factor), int(w * scale_factor),
+                   int(c / (scale_factor * scale_factor)))
+        if self.ps_version == 'v1':
+            warnings.warn("In ps_version 'v1', the height and width have not been swapped back, "
+                          'which results in a transposed image.')
+        else:
+            x = x.permute(0, 2, 1, 3).contiguous()
+        return x
+
+    def extract_feature(self, pixel_values):
+        if self.select_layer == -1:
+            vit_embeds = self.vision_model(
+                pixel_values=pixel_values,
+                output_hidden_states=False,
+                return_dict=True).last_hidden_state
+        else:
+            vit_embeds = self.vision_model(
+                pixel_values=pixel_values,
+                output_hidden_states=True,
+                return_dict=True).hidden_states[self.select_layer]
+        vit_embeds = vit_embeds[:, 1:, :]
+
+        h = w = int(vit_embeds.shape[1] ** 0.5)
+        vit_embeds = vit_embeds.reshape(vit_embeds.shape[0], h, w, -1)
+        vit_embeds = self.pixel_shuffle(vit_embeds, scale_factor=self.downsample_ratio)
+        vit_embeds = vit_embeds.reshape(vit_embeds.shape[0], -1, vit_embeds.shape[-1])
+        vit_embeds = self.mlp1(vit_embeds)
+        return vit_embeds
+
+    def batch_chat(self, tokenizer, pixel_values, questions, generation_config, num_patches_list=None,
+                   history=None, return_history=False, IMG_START_TOKEN='<img>', IMG_END_TOKEN='</img>',
+                   IMG_CONTEXT_TOKEN='<IMG_CONTEXT>', verbose=False, image_counts=None):
+        if history is not None or return_history:
+            print('Now multi-turn chat is not supported in batch_chat.')
+            raise NotImplementedError
+
+        if image_counts is not None:
+            num_patches_list = image_counts
+            print('Warning: `image_counts` is deprecated. Please use `num_patches_list` instead.')
+
+        img_context_token_id = tokenizer.convert_tokens_to_ids(IMG_CONTEXT_TOKEN)
+        self.img_context_token_id = img_context_token_id
+
+        if verbose and pixel_values is not None:
+            image_bs = pixel_values.shape[0]
+            print(f'dynamic ViT batch size: {image_bs}')
+
+        queries = []
+        for idx, num_patches in enumerate(num_patches_list):
+            question = questions[idx]
+            if pixel_values is not None and '<image>' not in question:
+                question = '<image>\n' + question
+            template = get_conv_template(self.template)
+            template.system_message = self.system_message
+            template.append_message(template.roles[0], question)
+            template.append_message(template.roles[1], None)
+            query = template.get_prompt()
+
+            image_tokens = IMG_START_TOKEN + IMG_CONTEXT_TOKEN * self.num_image_token * num_patches + IMG_END_TOKEN
+            query = query.replace('<image>', image_tokens, 1)
+            queries.append(query)
+
+        tokenizer.padding_side = 'left'
+        model_inputs = tokenizer(queries, return_tensors='pt', padding=True)
+        input_ids = model_inputs['input_ids'].to(self.device)
+        attention_mask = model_inputs['attention_mask'].to(self.device)
+        eos_token_id = tokenizer.convert_tokens_to_ids(template.sep.strip())
+        generation_config['eos_token_id'] = eos_token_id
+        generation_output = self.generate(
+            pixel_values=pixel_values,
+            input_ids=input_ids,
+            attention_mask=attention_mask,
+            **generation_config
+        )
+        responses = tokenizer.batch_decode(generation_output, skip_special_tokens=True)
+        responses = [response.split(template.sep.strip())[0].strip() for response in responses]
+        return responses
+
+    def chat(self, tokenizer, pixel_values, question, generation_config, history=None, return_history=False,
+             num_patches_list=None, IMG_START_TOKEN='<img>', IMG_END_TOKEN='</img>', IMG_CONTEXT_TOKEN='<IMG_CONTEXT>',
+             verbose=False):
+
+        if history is None and pixel_values is not None and '<image>' not in question:
+            question = '<image>\n' + question
+
+        if num_patches_list is None:
+            num_patches_list = [pixel_values.shape[0]] if pixel_values is not None else []
+        assert pixel_values is None or len(pixel_values) == sum(num_patches_list)
+
+        img_context_token_id = tokenizer.convert_tokens_to_ids(IMG_CONTEXT_TOKEN)
+        self.img_context_token_id = img_context_token_id
+
+        template = get_conv_template(self.template)
+        template.system_message = self.system_message
+        eos_token_id = tokenizer.convert_tokens_to_ids(template.sep.strip())
+
+        history = [] if history is None else history
+        for (old_question, old_answer) in history:
+            template.append_message(template.roles[0], old_question)
+            template.append_message(template.roles[1], old_answer)
+        template.append_message(template.roles[0], question)
+        template.append_message(template.roles[1], None)
+        query = template.get_prompt()
+
+        if verbose and pixel_values is not None:
+            image_bs = pixel_values.shape[0]
+            print(f'dynamic ViT batch size: {image_bs}')
+
+        for num_patches in num_patches_list:
+            image_tokens = IMG_START_TOKEN + IMG_CONTEXT_TOKEN * self.num_image_token * num_patches + IMG_END_TOKEN
+            query = query.replace('<image>', image_tokens, 1)
+
+        model_inputs = tokenizer(query, return_tensors='pt')
+        input_ids = model_inputs['input_ids'].to(self.device)
+        attention_mask = model_inputs['attention_mask'].to(self.device)
+        generation_config['eos_token_id'] = eos_token_id
+        generation_output = self.generate(
+            pixel_values=pixel_values,
+            input_ids=input_ids,
+            attention_mask=attention_mask,
+            **generation_config
+        )
+        response = tokenizer.batch_decode(generation_output, skip_special_tokens=True)[0]
+        response = response.split(template.sep.strip())[0].strip()
+        history.append((question, response))
+        if return_history:
+            return response, history
+        else:
+            query_to_print = query.replace(IMG_CONTEXT_TOKEN, '')
+            query_to_print = query_to_print.replace(f'{IMG_START_TOKEN}{IMG_END_TOKEN}', '<image>')
+            if verbose:
+                print(query_to_print, response)
+            return response
+
+    @torch.no_grad()
+    def generate(
+            self,
+            pixel_values: Optional[torch.FloatTensor] = None,
+            input_ids: Optional[torch.FloatTensor] = None,
+            attention_mask: Optional[torch.LongTensor] = None,
+            visual_features: Optional[torch.FloatTensor] = None,
+            generation_config: Optional[GenerationConfig] = None,
+            output_hidden_states: Optional[bool] = None,
+            **generate_kwargs,
+    ) -> torch.LongTensor:
+
+        assert self.img_context_token_id is not None
+        if pixel_values is not None:
+            if visual_features is not None:
+                vit_embeds = visual_features
+            else:
+                vit_embeds = self.extract_feature(pixel_values)
+            input_embeds = self.language_model.get_input_embeddings()(input_ids)
+            B, N, C = input_embeds.shape
+            input_embeds = input_embeds.reshape(B * N, C)
+
+            input_ids = input_ids.reshape(B * N)
+            selected = (input_ids == self.img_context_token_id)
+            assert selected.sum() != 0
+            input_embeds[selected] = vit_embeds.reshape(-1, C).to(input_embeds.device)
+
+            input_embeds = input_embeds.reshape(B, N, C)
+        else:
+            input_embeds = self.language_model.get_input_embeddings()(input_ids)
+
+        outputs = self.language_model.generate(
+            inputs_embeds=input_embeds,
+            attention_mask=attention_mask,
+            generation_config=generation_config,
+            output_hidden_states=output_hidden_states,
+            use_cache=True,
+            **generate_kwargs,
+        )
+
+        return outputs
+
+    def load_mtp_weights(self, path=None, prefix='language_model.mtp.'):
+        """Load P-MTP head weights.
+
+        If ``path`` is None the weights are assumed to already be present (e.g.
+        loaded from the main checkpoint via ``from_pretrained`` because the keys
+        ``language_model.mtp.*`` match this module tree) and this call only marks
+        MTP as usable. Otherwise ``path`` may be a ``.safetensors`` file or a
+        directory containing one; keys are stripped of ``prefix`` (or a leading
+        ``mtp.``) before loading.
+        """
+        if path is None:
+            self._mtp_weights_loaded = True
+            return
+        from safetensors.torch import load_file
+        if os.path.isdir(path):
+            files = [os.path.join(path, f) for f in sorted(os.listdir(path))
+                     if f.endswith('.safetensors')]
+        else:
+            files = [path]
+        remapped = {}
+        for fp in files:
+            for k, v in load_file(fp).items():
+                name = k
+                if name.startswith(prefix):
+                    name = name[len(prefix):]
+                elif name.startswith('mtp.'):
+                    name = name[len('mtp.'):]
+                else:
+                    continue
+                remapped[name] = v
+        if not remapped:
+            raise RuntimeError(f'No MTP tensors (prefix {prefix!r}) found under {path}')
+        missing, unexpected = self.language_model.mtp.load_state_dict(remapped, strict=False)
+        self.language_model.mtp.to(device=self.device, dtype=self.dtype)
+        self._mtp_weights_loaded = True
+        return missing, unexpected
+
+    def _embed_ids(self, input_ids, vit_embeds=None):
+        embeds = self.language_model.get_input_embeddings()(input_ids)
+        if vit_embeds is not None:
+            B, N, C = embeds.shape
+            flat = embeds.reshape(B * N, C)
+            selected = (input_ids.reshape(B * N) == self.img_context_token_id)
+            flat[selected] = vit_embeds.reshape(-1, C).to(flat.dtype)
+            embeds = flat.reshape(B, N, C)
+        return embeds
+
+    def _lm_forward(self, inputs_embeds, cache, need_hidden):
+        past_len = cache.get_seq_length()
+        seq_len = inputs_embeds.shape[1]
+        cache_position = torch.arange(past_len, past_len + seq_len, device=inputs_embeds.device)
+        out = self.language_model(
+            inputs_embeds=inputs_embeds,
+            past_key_values=cache,
+            use_cache=True,
+            cache_position=cache_position,
+            position_ids=cache_position.unsqueeze(0),
+            output_hidden_states=need_hidden,
+            return_dict=True,
+        )
+        hidden = out.hidden_states[-1] if need_hidden else None
+        return out.logits, hidden, out.past_key_values
+
+    @torch.no_grad()
+    def _decode_children_batched(self, parent_cache, prompt_len, fork_indices,
+                                 eos_ids, max_new_tokens):
+        """Decode all fork children concurrently as one batch (greedy AR).
+
+        Each child ``i`` inherits the parent prefix up to its ``<FORK>`` (real
+        length ``Li = prompt_len + fork_indices[i]``). The parent KV is copied
+        into a left-padded batched cache (row ``i`` right-aligned to hold its
+        ``Li`` prefix positions), then all children step in lockstep until every
+        row hits EOS. Per-row ``position_ids`` carry each child's true positions
+        so RoPE matches the parent context. Returns a list of token-id lists.
+
+        Children run without the P-MTP head (batched ragged acceptance would
+        desync row lengths); the greedy output is identical to the serial path.
+        """
+        device = self.device
+        B = len(fork_indices)
+        Ls = [prompt_len + f for f in fork_indices]
+        Lmax = max(Ls)
+
+        cache = DynamicCache()
+        for idx, layer in enumerate(parent_cache.layers):
+            K, V = layer.keys, layer.values           # [1, h, Pseq, d]
+            h, d = K.shape[1], K.shape[3]
+            Kb = K.new_zeros((B, h, Lmax, d))
+            Vb = V.new_zeros((B, h, Lmax, d))
+            for i, Li in enumerate(Ls):
+                Kb[i, :, Lmax - Li:, :] = K[0, :, :Li, :]
+                Vb[i, :, Lmax - Li:, :] = V[0, :, :Li, :]
+            cache.update(Kb, Vb, idx)
+
+        mask = torch.zeros((B, Lmax), dtype=torch.long, device=device)
+        for i, Li in enumerate(Ls):
+            mask[i, Lmax - Li:] = 1
+
+        real_pos = torch.tensor(Ls, device=device)    # <CHILD> sits at position Li
+        cur_embeds = self.language_model.get_input_embeddings()(
+            torch.tensor([[self.child_token_id]], device=device)).expand(B, 1, -1).contiguous()
+
+        outputs = [[] for _ in range(B)]
+        done = torch.zeros(B, dtype=torch.bool, device=device)
+        steps = 0
+        while steps < max_new_tokens and not bool(done.all()):
+            cache_len = cache.get_seq_length()
+            full_mask = torch.cat([mask, torch.ones((B, 1), dtype=torch.long, device=device)], dim=1)
+            out = self.language_model(
+                inputs_embeds=cur_embeds,
+                past_key_values=cache,
+                use_cache=True,
+                attention_mask=full_mask,
+                position_ids=real_pos.unsqueeze(1),
+                cache_position=torch.tensor([cache_len], device=device),
+                return_dict=True,
+            )
+            cache = out.past_key_values
+            nxt = out.logits[:, -1, :].argmax(-1)      # [B]
+            mask = full_mask
+            real_pos = real_pos + 1
+            for i in range(B):
+                if not done[i]:
+                    tid = int(nxt[i])
+                    outputs[i].append(tid)
+                    if tid in eos_ids or len(outputs[i]) >= max_new_tokens:
+                        done[i] = True
+            cur_embeds = self.language_model.get_input_embeddings()(nxt.unsqueeze(1))
+            steps += 1
+        return outputs
+
+    @torch.no_grad()
+    def _decode_from(self, cache, first_embeds, eos_ids, max_new_tokens, use_mtp, k, allow_fork):
+        """Greedily decode one branch (batch size 1) starting from ``cache``.
+
+        ``first_embeds`` primes the branch (the prompt for the parent, or the
+        single ``<CHILD>`` token embedding for a child that inherits the parent
+        KV cache). With ``use_mtp`` the P-MTP head drafts ``k`` tokens per step
+        and the backbone verifies them in a single forward (greedy accept: keep
+        the longest matching prefix, then one correction/bonus token), rolling
+        the cache back with ``crop``. Returns ``(real_tokens, fork_indices,
+        cache)``; ``fork_indices`` marks positions of ``<FORK>`` tokens (only
+        when ``allow_fork``).
+        """
+        device = first_embeds.device
+        logits, hidden, cache = self._lm_forward(first_embeds, cache, use_mtp)
+        q = cache.get_seq_length() - 1                # cache holds positions [0..q]
+        prod_hidden = hidden[:, -1, :] if use_mtp else None
+        t = int(logits[0, -1].argmax(-1))             # confirmed token at position q+1
+
+        real = []
+        fork_indices = []
+
+        def emit(tok):
+            real.append(tok)
+            if allow_fork and self.fork_token_id is not None and tok == self.fork_token_id:
+                fork_indices.append(len(real) - 1)
+            return tok in eos_ids
+
+        stopped = False
+        while len(real) < max_new_tokens and not stopped:
+            if use_mtp:
+                drafts = []
+                h = prod_hidden
+                prev = t
+                for _ in range(k):
+                    prev_embeds = self.language_model.get_input_embeddings()(
+                        torch.tensor([prev], device=device))
+                    h = self.language_model.mtp.step(h, prev_embeds)
+                    prev = int(self.language_model.lm_head(h).argmax(-1))
+                    drafts.append(prev)
+
+                verify_ids = torch.tensor([[t] + drafts], device=device)
+                vlogits, vhidden, cache = self._lm_forward(self._embed_ids(verify_ids), cache, True)
+                target = vlogits[0].argmax(-1).tolist()   # target[i] = argmax after verify_ids[i]
+
+                emitted = []
+                j = 0
+                all_match = True
+                for i in range(k):
+                    emitted.append(target[i])
+                    if drafts[i] != target[i]:
+                        all_match = False
+                        break
+                    j += 1
+                if all_match:
+                    emitted.append(target[k])             # bonus token
+
+                confirmed = [t] + emitted[:-1]
+                next_t = emitted[-1]
+                cache.crop((q + 1) + 1 + j)               # keep [0..q+1+j]
+                prod_hidden = vhidden[0, j:j + 1, :]
+                q = q + 1 + j
+                t = next_t
+                for tok in confirmed:
+                    if emit(tok):
+                        stopped = True
+                        break
+            else:
+                vlogits, _, cache = self._lm_forward(
+                    self._embed_ids(torch.tensor([[t]], device=device)), cache, False)
+                if emit(t):
+                    stopped = True
+                q = q + 1
+                t = int(vlogits[0, -1].argmax(-1))
+
+        if not stopped and len(real) < max_new_tokens:
+            emit(t)
+        return real, fork_indices, cache
+
+    @torch.no_grad()
+    def generate_hpd(self, tokenizer, pixel_values, question, generation_config=None,
+                     use_mtp=False, num_speculative_tokens=6, num_patches_list=None,
+                     IMG_START_TOKEN='<img>', IMG_END_TOKEN='</img>',
+                     IMG_CONTEXT_TOKEN='<IMG_CONTEXT>', return_token_ids=False,
+                     batch_children=False, verbose=False):
+        """Hierarchical parallel decoding (``<FORK>``/``<CHILD>``) with optional P-MTP.
+
+        Reproduces the vLLM decoding paradigm in plain transformers for a single
+        image / single prompt (batch size 1):
+
+        - The parent "layout" branch is decoded greedily; each ``<FORK>`` token
+          spawns a content branch. The parent keeps the real ``<FORK>`` in its
+          context and continues.
+        - Each child branch inherits the parent KV cache (shared-prefix reuse,
+          like vLLM). By default children are decoded serially, reusing the
+          parent cache in place (crop to the tokens before each ``<FORK>``);
+          this is the fastest, lowest-memory option in transformers. Set
+          ``batch_children=True`` to instead decode all children concurrently in
+          one left-padded batch (``_decode_children_batched``) -- identical
+          output, but for many short children over a long prefix it is slower
+          and much more memory-hungry (no paged prefix sharing like vLLM).
+        - Outputs are spliced: each ``<FORK>`` in the parent stream is rewritten
+          to ``<CHILD>`` and the matching child's tokens are inserted after it,
+          in fork order.
+
+        Use ``use_mtp=True`` only after ``load_mtp_weights(...)`` has been called.
+        """
+        if self.fork_token_id is None or self.child_token_id is None:
+            raise RuntimeError('fork_token_id/child_token_id missing from config; cannot fork-decode.')
+        if use_mtp and not self._mtp_weights_loaded:
+            raise RuntimeError('P-MTP weights not loaded. Call model.load_mtp_weights(path) '
+                               'or set use_mtp=False.')
+
+        generation_config = generation_config or {}
+        max_new_tokens = generation_config.get('max_new_tokens', 8000)
+        k = num_speculative_tokens
+
+        if pixel_values is not None and '<image>' not in question:
+            question = '<image>\n' + question
+        if num_patches_list is None:
+            num_patches_list = [pixel_values.shape[0]] if pixel_values is not None else []
+        self.img_context_token_id = tokenizer.convert_tokens_to_ids(IMG_CONTEXT_TOKEN)
+
+        template = get_conv_template(self.template)
+        template.system_message = self.system_message
+        template.append_message(template.roles[0], question)
+        template.append_message(template.roles[1], None)
+        query = template.get_prompt()
+        for num_patches in num_patches_list:
+            image_tokens = IMG_START_TOKEN + IMG_CONTEXT_TOKEN * self.num_image_token * num_patches + IMG_END_TOKEN
+            query = query.replace('<image>', image_tokens, 1)
+
+        prompt_ids = tokenizer(query, return_tensors='pt')['input_ids'].to(self.device)
+        prompt_len = prompt_ids.shape[1]
+
+        eos_ids = set()
+        sep_id = tokenizer.convert_tokens_to_ids(template.sep.strip())
+        if isinstance(sep_id, int) and sep_id >= 0:
+            eos_ids.add(sep_id)
+        if self.config.eos_token_id is not None:
+            eos_ids.add(self.config.eos_token_id)
+
+        vit_embeds = self.extract_feature(pixel_values) if pixel_values is not None else None
+        prompt_embeds = self._embed_ids(prompt_ids, vit_embeds)
+
+        parent_real, fork_indices, parent_cache = self._decode_from(
+            DynamicCache(), prompt_embeds, eos_ids, max_new_tokens, use_mtp, k, allow_fork=True)
+
+        # Once the parent finishes, every fork point and its shared prefix is
+        # fixed. Children can then be decoded either serially (reusing the parent
+        # cache in place, low memory) or concurrently in one padded batch
+        # (``batch_children=True``). NOTE: for many short children over a long
+        # shared prefix, batching is slower and far more memory-hungry in stock
+        # transformers (the prefix KV is duplicated per row and short children
+        # still step until the longest one finishes), so serial is the default.
+        children = []
+        if fork_indices and batch_children:
+            children = self._decode_children_batched(
+                parent_cache, prompt_len, fork_indices, eos_ids, max_new_tokens)
+        elif fork_indices:
+            child_embeds = self.language_model.get_input_embeddings()(
+                torch.tensor([[self.child_token_id]], device=self.device))
+            children = [None] * len(fork_indices)
+            # Reverse fork order: crop the shared parent cache to the tokens
+            # before each <FORK> and reuse it in place. A child only appends
+            # beyond its crop point, which the next (smaller) crop discards, so
+            # the parent prefix KV stays intact and no clone is needed.
+            for i in range(len(fork_indices) - 1, -1, -1):
+                parent_cache.crop(prompt_len + fork_indices[i])
+                child_real, _, _ = self._decode_from(
+                    parent_cache, child_embeds, eos_ids, max_new_tokens, use_mtp, k, allow_fork=False)
+                children[i] = child_real
+
+        final_ids = []
+        child_ptr = 0
+        for tok in parent_real:
+            if tok == self.fork_token_id:
+                final_ids.append(self.child_token_id)
+                if child_ptr < len(children):
+                    final_ids.extend(children[child_ptr])
+                    child_ptr += 1
+            else:
+                final_ids.append(tok)
+
+        text = tokenizer.decode(final_ids, skip_special_tokens=True)
+        if verbose:
+            print(f'[HPD] parent tokens={len(parent_real)}, forks={len(fork_indices)}, '
+                  f'children_tokens={[len(c) for c in children]}')
+        if return_token_ids:
+            return text, final_ids
+        return text
+
+    @property
+    def lm_head(self):
+        return self.language_model.get_output_embeddings()
+
+    def get_output_embeddings(self):
+        return self.language_model.get_output_embeddings()
+
+    def get_input_embeddings(self):
+        return self.language_model.get_input_embeddings()
+
+    def set_input_embeddings(self, value):
+        return self.language_model.set_input_embeddings(value)
+
+    def set_output_embeddings(self, value):
+        return self.language_model.set_output_embeddings(value)

preprocessor_config.json

@@ -0,0 +1,34 @@
+{
+  "crop_size": null,
+  "crop_to_patches": false,
+  "data_format": "channels_first",
+  "default_to_square": true,
+  "device": null,
+  "do_center_crop": null,
+  "do_convert_rgb": true,
+  "do_normalize": true,
+  "do_rescale": true,
+  "do_resize": true,
+  "image_mean": [
+    0.485,
+    0.456,
+    0.406
+  ],
+  "image_processor_type": "GotOcr2ImageProcessorFast",
+  "image_std": [
+    0.229,
+    0.224,
+    0.225
+  ],
+  "input_data_format": null,
+  "max_patches": 12,
+  "min_patches": 1,
+  "processor_class": "InternVLProcessor",
+  "resample": 3,
+  "rescale_factor": 0.00392156862745098,
+  "return_tensors": null,
+  "size": {
+    "height": 448,
+    "width": 448
+  }
+}

special_tokens_map.json

@@ -0,0 +1,40 @@
+{
+  "additional_special_tokens": [
+    "<|im_start|>",
+    "<|im_end|>",
+    "<|object_ref_start|>",
+    "<|object_ref_end|>",
+    "<|box_start|>",
+    "<|box_end|>",
+    "<|quad_start|>",
+    "<|quad_end|>",
+    "<|vision_start|>",
+    "<|vision_end|>",
+    "<|vision_pad|>",
+    "<|image_pad|>",
+    "<|video_pad|>",
+    "<img>",
+    "</img>",
+    "<IMG_CONTEXT>",
+    "<quad>",
+    "</quad>",
+    "<ref>",
+    "</ref>",
+    "<box>",
+    "</box>"
+  ],
+  "eos_token": {
+    "content": "<|im_end|>",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  },
+  "pad_token": {
+    "content": "<|endoftext|>",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  }
+}

tokenizer.json

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:2e9a5bd06115c607d2bda92cdefb7b21e915832ea58afd1688796a4db56e6d67
+size 11424851

tokenizer_config.json

@@ -0,0 +1,346 @@
+{
+  "add_bos_token": false,
+  "add_eos_token": false,
+  "add_prefix_space": false,
+  "added_tokens_decoder": {
+    "151643": {
+      "content": "<|endoftext|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151644": {
+      "content": "<|im_start|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151645": {
+      "content": "<|im_end|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151646": {
+      "content": "<|object_ref_start|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151647": {
+      "content": "<|object_ref_end|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151648": {
+      "content": "<|box_start|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151649": {
+      "content": "<|box_end|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151650": {
+      "content": "<|quad_start|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151651": {
+      "content": "<|quad_end|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151652": {
+      "content": "<|vision_start|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151653": {
+      "content": "<|vision_end|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151654": {
+      "content": "<|vision_pad|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151655": {
+      "content": "<|image_pad|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151656": {
+      "content": "<|video_pad|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151657": {
+      "content": "<tool_call>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "151658": {
+      "content": "</tool_call>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "151659": {
+      "content": "<|fim_prefix|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "151660": {
+      "content": "<|fim_middle|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "151661": {
+      "content": "<|fim_suffix|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "151662": {
+      "content": "<|fim_pad|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "151663": {
+      "content": "<|repo_name|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "151664": {
+      "content": "<|file_sep|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "151665": {
+      "content": "<tool_response>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "151666": {
+      "content": "</tool_response>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "151667": {
+      "content": "<think>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "151668": {
+      "content": "</think>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "151669": {
+      "content": "<img>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151670": {
+      "content": "</img>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151671": {
+      "content": "<IMG_CONTEXT>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151672": {
+      "content": "<quad>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151673": {
+      "content": "</quad>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151674": {
+      "content": "<ref>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151675": {
+      "content": "</ref>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151676": {
+      "content": "<box>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151677": {
+      "content": "</box>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151678": {
+      "content": "<BLOCK>",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "151679": {
+      "content": "<FORK>",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "151680": {
+      "content": "<CHILD>",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    }
+  },
+  "additional_special_tokens": [
+    "<|im_start|>",
+    "<|im_end|>",
+    "<|object_ref_start|>",
+    "<|object_ref_end|>",
+    "<|box_start|>",
+    "<|box_end|>",
+    "<|quad_start|>",
+    "<|quad_end|>",
+    "<|vision_start|>",
+    "<|vision_end|>",
+    "<|vision_pad|>",
+    "<|image_pad|>",
+    "<|video_pad|>",
+    "<img>",
+    "</img>",
+    "<IMG_CONTEXT>",
+    "<quad>",
+    "</quad>",
+    "<ref>",
+    "</ref>",
+    "<box>",
+    "</box>"
+  ],
+  "bos_token": null,
+  "clean_up_tokenization_spaces": false,
+  "eos_token": "<|im_end|>",
+  "errors": "replace",
+  "extra_special_tokens": {},
+  "model_max_length": 14588,
+  "pad_token": "<|endoftext|>",
+  "processor_class": "InternVLProcessor",
+  "split_special_tokens": false,
+  "tokenizer_class": "Qwen2Tokenizer",
+  "unk_token": null
+}