Upload granite-vision-4.1-4b-oQ8 via oMLX

README.md

@@ -0,0 +1,387 @@
+---
+language:
+- en
+license: apache-2.0
+pipeline_tag: image-text-to-text
+library_name: transformers
+---
+
+# Granite-Vision-4.1-4B
+
+**Model Summary:**
+Granite Vision 4.1 4B is a vision-language model (VLM) that delivers frontier-level
+performance on structured document extraction tasks — chart extraction, table extraction,
+and semantic key-value pair extraction — in a compact 4B parameter footprint, providing
+a lightweight alternative to much larger frontier models for these tasks:
+
+- **Chart extraction:** Converting charts into structured, machine-readable formats (Chart2CSV, Chart2Summary, and Chart2Code)
+- **Table extraction:** Accurately extracting tables with complex layouts from document images to JSON, HTML, or OTSL
+- **Semantic Key-Value Pair (KVP) extraction:** Extracting values based on key names and descriptions across diverse document layouts
+
+The model is finetuned on top of [Granite-4.1-3B](https://huggingface.co/ibm-granite/granite-4.1-3b), with a 3.4B LLM and 0.6B Vision Encoder and Projectors. See
+[Model Architecture](#model-architecture) for details.
+
+The methodology and data (ChartNet) used for this model are described in the paper [ChartNet: A Million-Scale, High-Quality Multimodal Dataset for Robust Chart Understanding](https://huggingface.co/papers/2603.27064).
+
+While our focus is on specialized document extraction tasks, the current model preserves and extends the 
+capabilities of Granite Vision 4.0 3B, ensuring that existing users can adopt it seamlessly with no changes to their workflow. 
+It continues to support vision‑language tasks such as producing detailed natural‑language descriptions from images (image‑to‑text).
+The model can be used standalone and integrates seamlessly
+with [Docling](https://github.com/docling-project/docling) to enhance document processing pipelines
+with deep visual understanding capabilities.
+
+- **Developer:** IBM Research
+- **GitHub Repository:** https://github.com/ibm-granite
+- **Release Date:** April 29th, 2026
+- **License:** [Apache 2.0](https://www.apache.org/licenses/LICENSE-2.0)
+
+## Supported Tasks
+
+The model supports specialized extraction tasks, each activated by a simple task tag in
+the user message. The chat template automatically expands tags into the full prompt — no
+need to write verbose instructions.
+
+| Tag | Task | Output |
+|-----|------|--------|
+| `<chart2csv>` | Chart to CSV | CSV table with headers and numeric values |
+| `<chart2code>` | Chart to Python code | Python code that recreates the chart |
+| `<chart2summary>` | Chart to summary | Natural-language description of the chart |
+| `<tables_json>` | Table extraction (JSON) | Structured JSON with dimensions and cells |
+| `<tables_html>` | Table extraction (HTML) | HTML `<table>` markup |
+| `<tables_otsl>` | Table extraction (OTSL) | OTSL markup with cell/merge tags |
+| KVP (see prompt instructions below) | Schema based Key-Value pairs extraction | JSON with nested dictionaries and arrays |
+
+
+## Model Performance
+
+### Benchmark Results
+
+Granite Vision 4.1 4B provides a lightweight alternative to frontier models on structured document extraction
+benchmarks, delivering comparable performance at a fraction of the parameter count.
+
+<img src="bench_all_benchmarks.png" width="80%">
+
+#### Chart Extraction
+
+We evaluate chart extraction using the human-verified test-set from [ChartNet](https://huggingface.co/datasets/ibm-granite/ChartNet).
+Models are scored by LLM-as-a-judge (GPT4o) comparing predictions against ground truth.
+We report average scores (0–100) on Chart2CSV and Chart2Summary tasks.
+
+<div style="display: flex; gap: 10px;">
+  <img src="bench_c2c.png" width="49%">
+  <img src="bench_c2s.png" width="49%">
+</div>
+
+#### Table Extraction
+
+To benchmark table extraction, we construct a unified evaluation suite spanning multiple datasets and settings to assess end-to-end table extraction capabilities of vision-language models:
+
+1. **[TableVQA-Extract](https://github.com/naver-ai/tablevqabench)** — Converts the original visual table QA benchmark into a cropped table extraction task.  
+2. **[OmniDocBench-tables](https://github.com/opendatalab/OmniDocBench)** — A document parsing benchmark over diverse PDF types with detailed annotations for layout, text, formulas, and tables. We use the subset of pages that contain one or more tables to evaluate table extraction in full-page settings.  
+3. **[PubTablesV2](https://huggingface.co/datasets/kensho/PubTables-v2)** — A large-scale table extraction benchmark evaluated in both cropped-table and full-page document settings.
+
+To unify evaluation, we replace each dataset’s original annotations (e.g., Q&A pairs) with a single instruction: *extract the table(s) from the image in HTML format*, using the corresponding HTML as ground truth. For full-page inputs, only tabular elements are considered; when multiple tables appear, they are aggregated into a Python list.
+
+We report results using **TEDS** ([Tree-Edit Distance-based Similarity](https://arxiv.org/abs/2208.00385)), which measures structural and content similarity between predicted and ground-truth HTML tables.
+
+Results are presented separately for cropped-table and full-page settings to highlight performance across controlled and realistic document scenarios.
+
+<div style="display: flex; gap: 10px;">
+  <img src="bench_tvqa.png" width="49%">
+  <img src="bench_omni.png" width="49%">
+</div>
+<div style="display: flex; gap: 10px;">
+  <img src="bench_pubcrop.png" width="49%">
+  <img src="bench_pubfull.png" width="49%">
+</div>
+
+#### Key-Value Pair (KVP) Extraction
+
+We evaluate on [VAREX](https://udibarzi.github.io/varex-bench/), a benchmark for
+multimodal structured extraction from documents. Granite Vision 4.1 4B achieves
+94.4% exact-match accuracy (zero-shot), competitive with much larger frontier models
+(view results [here](https://udibarzi.github.io/varex-bench/#leaderboard)).
+
+<div style="display: flex; gap: 10px;">
+  <img src="bench_varex.png" width="60%">
+</div>
+
+### Setup
+
+Tested with python=3.11
+```shell
+pip install torch==2.10.0 --index-url https://download.pytorch.org/whl/cu128
+pip install transformers>=5.6.2 peft>=0.19.1 tokenizers>=0.22.2 pillow>=12.2.0
+```
+
+## Usage with Transformers
+
+
+```python
+import re
+from io import StringIO
+
+import pandas as pd
+import torch
+from transformers import AutoProcessor, AutoModelForImageTextToText
+from PIL import Image
+from huggingface_hub import hf_hub_download
+
+model_id = "ibm-granite/granite-vision-4.1-4b"
+device = "cuda" if torch.cuda.is_available() else "cpu"
+
+processor = AutoProcessor.from_pretrained(model_id, trust_remote_code=True)
+model = AutoModelForImageTextToText.from_pretrained(
+    model_id,
+    trust_remote_code=True,
+    dtype=torch.bfloat16,
+    device_map=device
+).eval()
+
+
+def run_inference(model, processor, images, prompts):
+    """Run batched inference on image+prompt pairs (one image per prompt)."""
+    conversations = [
+        [{"role": "user", "content": [
+            {"type": "image"},
+            {"type": "text", "text": prompt},
+        ]}]
+        for prompt in prompts
+    ]
+    texts = [
+        processor.apply_chat_template(conv, tokenize=False, add_generation_prompt=True)
+        for conv in conversations
+    ]
+    inputs = processor(
+        text=texts, images=images, return_tensors="pt", padding=True, do_pad=True
+    ).to(model.device)
+    outputs = model.generate(
+        **inputs,
+        max_new_tokens=4096, 
+        use_cache=True
+    )
+    results = []
+    for i in range(len(prompts)):
+        gen = outputs[i, inputs["input_ids"].shape[1]:]
+        results.append(processor.decode(gen, skip_special_tokens=True))
+    return results
+
+
+def display_table(text):
+    """Pretty-print CSV (possibly wrapped in ```csv```) or HTML table content via pandas."""
+    m = re.search(r"```csv\s*
+(.*?)```", text, re.DOTALL)
+    if m:
+        df = pd.read_csv(StringIO(m.group(1)))
+        print(df.to_string(index=False))
+    elif "<table" in text.lower():
+        df = pd.read_html(StringIO(text))[0]
+        print(df.to_string(index=False))
+    else:
+        print(text)
+```
+
+### Chart and Table Tasks
+
+You can pass tags and the chat template handles the rest:
+
+```python
+chart_path = hf_hub_download(repo_id=model_id, filename="chart.jpg")
+table_path = hf_hub_download(repo_id=model_id, filename="table.png")
+chart_img = Image.open(chart_path).convert("RGB")
+table_img = Image.open(table_path).convert("RGB")
+
+# Batched chart tasks
+chart_prompts = ["<chart2csv>", "<chart2summary>", "<chart2code>"]
+chart_results = run_inference(model, processor, [chart_img] * len(chart_prompts), chart_prompts)
+for prompt, result in zip(chart_prompts, chart_results):
+    print(f"{prompt}:")
+    display_table(result)
+    print()
+
+# Batched table tasks
+table_prompts = ["<tables_html>", "<tables_otsl>"]
+table_results = run_inference(model, processor, [table_img] * len(table_prompts), table_prompts)
+for prompt, result in zip(table_prompts, table_results):
+    print(f"{prompt}:")
+    display_table(result)
+    print()
+```
+
+### Key-Value Pair Extraction (KVP)
+
+For KVP extraction use the [VAREX](https://github.com/udibarzi/varex-bench) prompt format. Provide a JSON Schema describing the fields to extract and the model will return a JSON object with the extracted values.
+
+```python
+import json
+
+invoice_path = hf_hub_download(repo_id=model_id, filename="invoice.png")
+invoice_img = Image.open(invoice_path).convert("RGB")
+schema = {
+    "type": "object",
+    "properties": {
+        "invoice_date": {"type": "string", "description": "The date the invoice was issued"},
+        "order_number": {"type": "string", "description": "The unique identifier for the order"},
+        "seller_tax_id": {"type": "string", "description": "The tax identification number of the seller"},
+    }
+}
+
+prompt = f"""Extract structured data from this document.
+Return a JSON object matching this schema:
+
+{json.dumps(schema, indent=2)}
+
+Return null for fields you cannot find.
+Return ONLY valid JSON.
+Return an instance of the JSON with extracted values, not the schema itself."""
+
+result = run_inference(model, processor, [invoice_img], [prompt])[0]
+print(result)
+```
+
+
+
+## Usage with vLLM
+
+Granite Vision 4.1 is supported natively in vLLM as of commit
+[bde0efd](https://github.com/vllm-project/vllm/commit/bde0efdbb78a57dc10375e8d0686cf862332192c).
+Until an official release ships, install vLLM from source:
+
+```bash
+git clone https://github.com/vllm-project/vllm.git
+cd vllm
+pip install -e ".[cuda]"
+```
+
+### Serving
+
+```bash
+vllm serve ibm-granite/granite-vision-4.1-4b \
+    --host 0.0.0.0 --port 8000
+```
+
+### Client example
+
+Query the running server using the OpenAI-compatible API:
+
+```python
+import base64
+from openai import OpenAI
+from huggingface_hub import hf_hub_download
+from PIL import Image
+
+model_id = "ibm-granite/granite-vision-4.1-4b"
+client = OpenAI(base_url="http://localhost:8000/v1", api_key="EMPTY")
+
+def run_inference(client, model_id, image_path, tag):
+    with open(image_path, "rb") as f:
+        image_b64 = base64.b64encode(f.read()).decode("utf-8")
+    messages = [
+        {"role": "user", "content": [
+            {"type": "image_url", "image_url": {"url": f"data:image/png;base64,{image_b64}"}},
+            {"type": "text", "text": tag},
+        ]}
+    ]
+    response = client.chat.completions.create(
+        model=model_id, messages=messages, max_tokens=4096, temperature=0,
+    )
+    return response.choices[0].message.content
+
+chart_path = hf_hub_download(repo_id=model_id, filename="chart.jpg")
+table_path = hf_hub_download(repo_id=model_id, filename="table.png")
+
+# Chart tasks
+for tag in ["<chart2csv>", "<chart2summary>", "<chart2code>"]:
+    result = run_inference(client, model_id, chart_path, tag)
+    print(f"{tag}:
+{result}
+")
+
+# Table tasks
+for tag in ["<tables_json>", "<tables_html>", "<tables_otsl>"]:
+    result = run_inference(client, model_id, table_path, tag)
+    print(f"{tag}:
+{result}
+")
+```
+
+
+## Usage with Docling
+
+[Docling](https://github.com/docling-project/docling) integrates Granite Vision for document conversion pipelines:
+
+- [Table extraction](https://github.com/docling-project/docling/blob/v2.90.0/docs/examples/granite_vision_table_structure.py) — uses Granite Vision to extract the layout and content of detected tables.
+- [Chart data extraction](https://github.com/docling-project/docling/blob/v2.90.0/docs/examples/chart_extraction.py) — uses Granite Vision to extract structured data from bar, pie, and line charts (`pip install docling[granite_vision]`)
+
+
+## Training Data
+
+The model was fine-tuned on a curated mixture of extraction-focused datasets spanning
+chart understanding, complex table parsing, and document KVP extraction, supplemented
+by the general-purpose [Granite Vision instruction-following dataset](https://arxiv.org/abs/2502.09927) for broad visual understanding.
+
+Chart understanding data was created through a novel code‑guided augmentation methodology that produces diverse, 
+semantically aligned chart samples containing rendering code, chart images, underlying data CSVs, and natural‑language summaries. 
+Using this pipeline, we are also releasing [ChartNet](https://huggingface.co/datasets/ibm-granite/ChartNet), a comprehensive million‑scale multimodal dataset enriched with real‑world, 
+human‑annotated, safety, and grounding subsets. The dataset and its methodology are detailed in the paper [ChartNet: A Million-Scale, High-Quality Multimodal Dataset for Robust Chart Understanding](https://huggingface.co/papers/2603.27064).
+
+## Model Architecture
+
+1. **SigLIP2 Vision encoder:** `google/siglip2-so400m-patch16-384`. Input images are tiled into 384×384 patches (with a base downscaled view always included), and each tile is encoded independently. The vision encoder is finetuned with LoRA adapters during training. The chechpoint provides the weights with merged adapters.
+2. **Window Q-Former projectors:** Visual features are compressed 4× using windowed Q-Former projectors: each 4×4 patch window is reduced to 2×2 tokens via cross-attention, where the queries are initialized from a downsampled version of the window features. This reduces the visual token count fed to the LLM.
+3. **Feature injection:** A variant of [Deepstack](https://arxiv.org/abs/2406.04334) where visual features are additively injected into the LLM hidden states at multiple layers through two complementary mechanisms:
+   - *LayerDeepstack:* Features from 4 vision encoder depths are each projected and injected into a different LLM layer. The Q-Former queries are initialized from downsampled features. The mapping is reversed — the deepest (most semantic) vision features feed the earliest LLM layers, providing strong semantic grounding from the start.
+   - *SpatialDeepstack:* The deepest vision features at full resolution are split into 4 complementary spatial groups. Each group's Q-Former queries are initialized from the corresponding spatial subset, and injected at a separate later LLM layer, providing fine-grained spatial detail.
+
+   In total, **8 vision-to-LLM injection points** distribute visual information across the network for stronger visual grounding.
+4. **Language model:** Granite-4.1 (3B) with LoRA (rank 256) across all self-attention projections and MLP layers. The chechpoint provides the weights with merged adapters.
+
+**Supported input:** English instructions and images (PNG, JPEG).
+
+
+## Infrastructure
+
+Granite 4.1 Vision was trained on IBM's Blue Vela supercomputing cluster, outfitted with
+NVIDIA H100 GPUs. The training was done on 32 GPUs for approximately 200 hours.
+
+
+## Ethical Considerations and Limitations
+
+The use of vision-language models involves certain risks that should be considered before
+deployment:
+- **Task scope:** The model is specifically designed for structured extraction tasks and
+  may not generalize well to open-ended vision-language tasks.
+- **Hallucination:** As with all generative models, outputs should be validated before
+  use in automated pipelines, particularly for high-stakes document processing.
+- **Language:** The model is trained on English instructions only and may produce
+  degraded results for documents in other languages.
+
+To enhance safety in enterprise deployments, we recommend using Granite 4.1 Vision
+alongside [Granite Guardian](https://huggingface.co/ibm-granite/granite-guardian-4.1-8b),
+a model designed to detect and flag risks in inputs and outputs across key dimensions
+outlined in the IBM AI Risk Atlas.
+
+
+## Resources
+- ⭐️ Learn about the latest updates with Granite: https://www.ibm.com/granite
+- 🚀 Get started with tutorials, best practices, and prompt engineering advice: https://www.ibm.com/granite/docs/
+- 💡 Granite learning resources: https://ibm.biz/granite-learning-resources
+
+
+## Citation
+
+```bibtex
+@misc{granite-vision-4.1-4b,
+  title={Granite 4.1 Vision},
+  author={IBM Granite Vision Team},
+  year={2026},
+  url={https://huggingface.co/ibm-granite/granite-vision-4.1-4b}
+}
+
+@article{kondic2026chartnet,
+  title={ChartNet: A Million-Scale, High-Quality Multimodal Dataset for Robust Chart Understanding},
+  author={Kondic, Jovana and Li, Pengyuan and Joshi, Dhiraj and Sanchez, Isaac and Wiesel, Ben and Abedin, Shafiq and Alfassy, Amit and Schwartz, Eli and Caraballo, Daniel and Cinar, Yagmur Gizem and Scheidegger, Florian and Ross, Steven I. and Weidele, Daniel Karl I. and Hua, Hang and Arutyunova, Ekaterina and Herzig, Roei and He, Zexue and Wang, Zihan and Yu, Xinyue and Zhao, Yunfei and Jiang, Sicong and Liu, Minghao and Lin, Qunshu and Staar, Peter and Lastras, Luis and Oliva, Aude and Feris, Rogerio},
+  journal={arXiv preprint arXiv:2603.27064},
+  year={2026}
+}
+```

added_tokens.json

@@ -0,0 +1,3 @@
+{
+  "<image>": 100352
+}

chat_template.jinja

@@ -0,0 +1,180 @@
+{#- ===== Task tag prompt constants ===== -#}
+{%- set chart2code_prompt = "Generate code that recreates the chart as best as possible." -%}
+{%- set chart2csv_prompt = "Please examine this chart image. Consider you are a data visualization expert, and extract the data into a CSV table.\n\nYour CSV should:\n- Include a header row with clear column names\n- Represent all data series/categories shown in the chart\n- Use numeric values that match the chart as closely as possible\n\nOutput only the CSV data, nothing else." -%}
+{%- set chart2summary_prompt = "Can you describe this chart image?" -%}
+{%- set tables_json_prompt = "Identify and extract the table schema\n Extract the schema of all the tables in the image sorted according to the reading order.\nThe output must be a valid JSON object containing a list of dictionaries with the following structure:\n\n                {\n                    \"dimensions\": {\n                        \"rows\": <number of data rows (excluding header rows)>,\n                        \"columns\": <number of columns>,\n                        \"header_rows\": <number of header rows>,\n                        \"total_rows\": <total number of rows including headers>\n                    },\n                    \"cells\": [\n                        {\n                        \"row\": <row index starting at 1>,\n                        \"col\": <column index starting at 1>,\n                        \"colspan\": <number of columns spanned>,\n                        \"rowspan\": <number of rows spanned>,\n                        \"type\": \"<'header' or 'data'>\",\n                        \"header_level\": <header nesting level if type=header, else omit or null>,\n                        \"content\": \"<string content of the cell>\"\n                        },\n                        ...\n                    ]\n                }" -%}
+{%- set tables_html_prompt = "Identify and extract the table schema\n Extract the schema of all the tables in the image sorted according to the reading order.\nThe output must be a list of valid HTML tables" -%}
+{%- set tables_otsl_prompt = "Identify and extract the table schema\n Extract the schema of all the tables in the image sorted according to the reading order.\nThe output must be a list of valid OTSL objects, each consists of the following fields: \n                        <fcel> - a cell with content in it\n                        <ecel> - an empty cell\n                        <lcel> - a cell that is merged with the cell to its left\n                        <ucel> - a cell that is merged with the cell above it\n                        <xcel> - a cell that is merged with both the cell above it and the cell to its left\n                        <nl> - a new line\n                        <ched> - a column header\n                        <otsl> - the beginning of the OTSL table\n                        </otsl> - the end of the OTSL table\n\n                        An example for an output:\n                        [\n                        <otsl><ched>first table header1<ched>first table header2<nl><fcel>data1<fcel>data2<nl><fcel>data with horizontal span<lcel><nl><fcel>data with vertical span<ecel><nl><ucel><fcel>data3<nl></otsl>,\n                        <otsl><ched>second table header1<ched>second table header2<nl><fcel>data1<fcel>data2<nl><fcel>data with horizontal span<lcel><nl><fcel>data with vertical span<ecel><nl><ucel><fcel>data3<nl></otsl>\n                        ]" -%}
+
+
+{#- ===== Tag expansion dispatcher ===== -#}
+{%- macro expand_tags(text) -%}
+{%- set has_image = "<image>" in text -%}
+{#- Determine image position: prefix if <image> appears before the tag, suffix if after -#}
+{%- if has_image -%}
+  {%- set img_idx = text.index("<image>") -%}
+  {%- if "<chart2code>" in text -%}{%- set tag_idx = text.index("<chart2code>") -%}
+  {%- elif "<chart2csv>" in text -%}{%- set tag_idx = text.index("<chart2csv>") -%}
+  {%- elif "<chart2summary>" in text -%}{%- set tag_idx = text.index("<chart2summary>") -%}
+  {%- elif "<tables_json>" in text -%}{%- set tag_idx = text.index("<tables_json>") -%}
+  {%- elif "<tables_html>" in text -%}{%- set tag_idx = text.index("<tables_html>") -%}
+  {%- elif "<tables_otsl>" in text -%}{%- set tag_idx = text.index("<tables_otsl>") -%}
+  {%- else -%}{%- set tag_idx = 999999 -%}
+  {%- endif -%}
+  {%- set img_prefix = "<image>\n" if img_idx < tag_idx else "" -%}
+  {%- set img_suffix = "<image>\n" if img_idx >= tag_idx else "" -%}
+{%- else -%}
+  {%- set img_prefix = "" -%}
+  {%- set img_suffix = "" -%}
+{%- endif -%}
+{%- if "<chart2code>" in text -%}
+  {{- img_prefix + chart2code_prompt + img_suffix -}}
+{%- elif "<chart2csv>" in text -%}
+  {{- img_prefix + chart2csv_prompt + img_suffix -}}
+{%- elif "<chart2summary>" in text -%}
+  {{- img_prefix + chart2summary_prompt + img_suffix -}}
+{%- elif "<tables_json>" in text -%}
+  {{- img_prefix + tables_json_prompt + img_suffix -}}
+{%- elif "<tables_html>" in text -%}
+  {{- img_prefix + tables_html_prompt + img_suffix -}}
+{%- elif "<tables_otsl>" in text -%}
+  {{- img_prefix + tables_otsl_prompt + img_suffix -}}
+{%- else -%}
+  {{- text -}}
+{%- endif -%}
+{%- endmacro -%}
+
+{#- ===== Original chat template ===== -#}
+{% macro render_content(x) %}
+    {%- if x is string %}
+        {{ x }}
+    {%- else %}
+        {%- for chunk in x %}
+            {%- if chunk['type'] == 'text' -%}
+                {{ chunk['text']}}
+            {%- elif chunk['type'] == 'image' -%}
+                {{- "<image>
+" }}
+            {%- endif -%}
+        {%- endfor -%}
+    {%- endif -%}
+    {% endmacro %}
+
+    {%- set tools_system_message_prefix = 'You are a helpful assistant with access to the following tools. You may call one or more tools to assist with the user query.\n\nYou are provided with function signatures within <tools></tools> XML tags:\n<tools>'  %}
+{%- set tools_system_message_suffix = '\n</tools>\n\nFor each tool call, return a json object with function name and arguments within <tool_call></tool_call> XML tags:\n<tool_call>\n{\"name\": <function-name>, \"arguments\": <args-json-object>}\n</tool_call>. If a tool does not exist in the provided list of tools, notify the user that you do not have the ability to fulfill the request.' %}
+{%- set documents_system_message_prefix = 'You are a helpful assistant with access to the following documents. You may use one or more documents to assist with the user query.\n\nYou are given a list of documents within <documents></documents> XML tags:\n<documents>' %}
+{%- set documents_system_message_suffix = '\n</documents>\n\nWrite the response to the user\'s input by strictly aligning with the facts in the provided documents. If the information needed to answer the question is not available in the documents, inform the user that the question cannot be answered based on the available data.' %}
+{%- set g4_default_system_message = 'You are a helpful assistant. Please ensure responses are professional, accurate, and safe.' %}
+{%- if available_tools is defined and available_tools %}
+    {%- set tools = available_tools %}
+{%- endif %}
+{%- set ns = namespace(tools_system_message=tools_system_message_prefix,
+                       documents_system_message=documents_system_message_prefix,
+                       default_system_message=g4_default_system_message,
+                       system_message=''
+                       ) %}
+{%- if tools %}
+    {%- for tool in tools %}
+        {%- set ns.tools_system_message = ns.tools_system_message + '\n' + (tool | tojson) %}
+    {%- endfor %}
+    {%- set ns.tools_system_message = ns.tools_system_message + tools_system_message_suffix %}
+{%- else %}
+    {%- set ns.tools_system_message = '' %}
+{%- endif %}
+{%- if documents %}
+    {%- for document in documents %}
+        {%- set ns.documents_system_message = ns.documents_system_message + '\n' + (document | tojson) %}
+    {%- endfor %}
+    {%- set ns.documents_system_message = ns.documents_system_message + documents_system_message_suffix %}
+{%- else %}
+    {%- set ns.documents_system_message = '' %}
+{%- endif %}
+{%- if messages[0].role == 'system' %}
+    {%- if messages[0].content is string %}
+        {%- set ns.system_message = messages[0].content %}
+    {%- elif messages[0].content is iterable %}
+        {%- for entry in messages[0].content %}
+            {%- if entry.type== 'text' %}
+                {%- if ns.system_message != '' %}
+                    {%- set ns.system_message = ns.system_message + '\n' %}
+                {%- endif %}
+                {%- set ns.system_message = ns.system_message + entry.text %}
+            {%- endif %}
+        {%- endfor %}
+    {%- endif %}
+    {%- if tools and documents %}
+        {%- set ns.system_message = ns.system_message + '\n\n' +  ns.tools_system_message + '\n\n' + ns.documents_system_message %}
+    {%- elif tools %}
+        {%- set ns.system_message = ns.system_message + '\n\n' + ns.tools_system_message %}
+    {%- elif documents %}
+        {%- set ns.system_message = ns.system_message + '\n\n' + ns.documents_system_message %}
+    {%- endif %}
+{%- else %}
+    {%- if tools and documents %}
+        {%- set ns.system_message = ns.tools_system_message + '\n\n' + ns.documents_system_message  %}
+    {%- elif tools %}
+        {%- set ns.system_message = ns.tools_system_message %}
+    {%- elif documents %}
+        {%- set ns.system_message = ns.documents_system_message %}
+    {%- endif %}
+{%- endif %}
+{%- if ns.system_message %}
+    {{- '<|start_of_role|>system<|end_of_role|>' + ns.system_message + '<|end_of_text|>\n' }}
+{%- else %}
+    {{- '<|start_of_role|>system<|end_of_role|>' + ns.default_system_message + '<|end_of_text|>\n' }}
+{%- endif %}
+{%- for message in messages %}
+    {%- set content = namespace(val='') %}
+    {%- if render_content(message['content']) is string %}
+        {%- set content.val = render_content(message['content']) %}
+    {%- else %}
+        {%- if render_content(message['content']) is iterable %}
+            {%- for entry in render_content(message['content']) %}
+                {%- if entry.type== 'text' %}
+                    {%- if content.val != '' %}
+                        {%- set content.val = content.val + '\n' %}
+                    {%- endif %}
+                    {%- set content.val = content.val + entry.text %}
+                {%- endif %}
+            {%- endfor %}
+        {%- endif %}
+    {%- endif %}
+    {%- if (message.role == 'user') or (message.role == 'system' and not loop.first) %}
+        {{- '<|start_of_role|>' + message.role + '<|end_of_role|>' + expand_tags(content.val) + '<|end_of_text|>\n' }}
+    {%- elif message.role == 'assistant' %}
+        {{- '<|start_of_role|>' + message.role + '<|end_of_role|>' + content.val }}
+        {%- if message.tool_calls %}
+            {%- for tool_call in message.tool_calls %}
+                {%- if (loop.first and content.val) or (not loop.first) %}
+                    {{- '\n' }}
+                {%- endif %}
+                {%- if tool_call.function %}
+                    {%- set tool_call = tool_call.function %}
+                {%- endif %}
+                {{- '<tool_call>\n{"name": "' }}
+                {{- tool_call.name }}
+                {{- '", "arguments": ' }}
+                {%- if tool_call.arguments is string %}
+                    {{- tool_call.arguments }}
+                {%- else %}
+                    {{- tool_call.arguments | tojson }}
+                {%- endif %}
+                {{- '}\n</tool_call>' }}
+            {%- endfor %}
+        {%- endif %}
+        {{- '<|end_of_text|>\n' }}
+    {%- elif message.role == 'tool' %}
+        {%- if loop.first or (messages[loop.index0 - 1].role != 'tool') %}
+            {{- '<|start_of_role|>user<|end_of_role|>' }}
+        {%- endif %}
+        {{- '\n<tool_response>\n' }}
+        {{- content.val }}
+        {{- '\n</tool_response>' }}
+        {%- if loop.last or (messages[loop.index0 + 1].role != 'tool') %}
+            {{- '<|end_of_text|>\n' }}
+        {%- endif %}
+    {%- endif %}
+{%- endfor %}
+{%- if add_generation_prompt %}
+    {{- '<|start_of_role|>assistant<|end_of_role|>' }}
+{%- endif %}

config.json

@@ -0,0 +1,217 @@
+{
+  "architectures": [
+    "Granite4VisionForConditionalGeneration"
+  ],
+  "auto_map": {
+    "AutoConfig": "configuration.Granite4VisionConfig",
+    "AutoModel": "modeling.Granite4VisionForConditionalGeneration",
+    "AutoModelForVision2Seq": "modeling.Granite4VisionForConditionalGeneration",
+    "AutoModelForImageTextToText": "modeling.Granite4VisionForConditionalGeneration",
+    "AutoProcessor": "processing.Granite4VisionProcessor"
+  },
+  "spatial_target_layers": [
+    12,
+    15,
+    18,
+    21
+  ],
+  "spatial_stride": 2,
+  "spatial_vision_layer": -1,
+  "downsample_rate": "4/8",
+  "dtype": "bfloat16",
+  "image_grid_pinpoints": [
+    [
+      384,
+      384
+    ],
+    [
+      384,
+      768
+    ],
+    [
+      384,
+      1152
+    ],
+    [
+      384,
+      1536
+    ],
+    [
+      384,
+      1920
+    ],
+    [
+      384,
+      2304
+    ],
+    [
+      384,
+      2688
+    ],
+    [
+      384,
+      3072
+    ],
+    [
+      384,
+      3456
+    ],
+    [
+      384,
+      3840
+    ],
+    [
+      768,
+      384
+    ],
+    [
+      768,
+      768
+    ],
+    [
+      768,
+      1152
+    ],
+    [
+      768,
+      1536
+    ],
+    [
+      768,
+      1920
+    ],
+    [
+      1152,
+      384
+    ],
+    [
+      1152,
+      768
+    ],
+    [
+      1152,
+      1152
+    ],
+    [
+      1536,
+      384
+    ],
+    [
+      1536,
+      768
+    ],
+    [
+      1920,
+      384
+    ],
+    [
+      1920,
+      768
+    ],
+    [
+      2304,
+      384
+    ],
+    [
+      2688,
+      384
+    ],
+    [
+      3072,
+      384
+    ],
+    [
+      3456,
+      384
+    ],
+    [
+      3840,
+      384
+    ]
+  ],
+  "image_seq_length": 576,
+  "image_token_index": 100352,
+  "initializer_range": 0.02,
+  "model_type": "granite4_vision",
+  "projector_dropout": 0.1,
+  "projector_hidden_act": "gelu",
+  "text_config": {
+    "architectures": [
+      "GraniteForCausalLM"
+    ],
+    "attention_bias": false,
+    "attention_dropout": 0.0,
+    "attention_multiplier": 0.015625,
+    "bos_token_id": 100257,
+    "embedding_multiplier": 12.0,
+    "eos_token_id": 100257,
+    "hidden_act": "silu",
+    "hidden_size": 2560,
+    "initializer_range": 0.1,
+    "intermediate_size": 8192,
+    "logits_scaling": 10.0,
+    "max_position_embeddings": 131072,
+    "mlp_bias": false,
+    "model_type": "granite",
+    "num_attention_heads": 40,
+    "num_hidden_layers": 40,
+    "num_key_value_heads": 8,
+    "pad_token_id": 100256,
+    "residual_multiplier": 0.22,
+    "rms_norm_eps": 1e-05,
+    "rope_scaling": null,
+    "rope_theta": 10000000,
+    "tie_word_embeddings": true,
+    "torch_dtype": "bfloat16",
+    "transformers_version": "4.53.3",
+    "use_cache": true,
+    "vocab_size": 100353
+  },
+  "tie_word_embeddings": true,
+  "transformers_version": "4.57.3",
+  "use_spatial_sampling": true,
+  "use_image_newline_parameter": true,
+  "vision_config": {
+    "attention_dropout": 0.0,
+    "hidden_act": "gelu_pytorch_tanh",
+    "hidden_size": 1152,
+    "image_size": 384,
+    "intermediate_size": 4304,
+    "layer_norm_eps": 1e-06,
+    "model_type": "siglip_vision_model",
+    "num_attention_heads": 16,
+    "num_channels": 3,
+    "num_hidden_layers": 27,
+    "patch_size": 16
+  },
+  "vision_feature_select_strategy": "full",
+  "deepstack_layer_map": [
+    [
+      -19,
+      9
+    ],
+    [
+      -13,
+      6
+    ],
+    [
+      -7,
+      3
+    ],
+    [
+      -1,
+      0
+    ]
+  ],
+  "eos_token_id": 100257,
+  "quantization": {
+    "group_size": 64,
+    "bits": 8,
+    "mode": "affine"
+  },
+  "quantization_config": {
+    "group_size": 64,
+    "bits": 8,
+    "mode": "affine"
+  }
+}

configuration.py

@@ -0,0 +1,44 @@
+from typing import Optional
+import logging
+from transformers import LlavaNextConfig
+
+logger = logging.getLogger(__name__)
+
+
+class Granite4VisionConfig(LlavaNextConfig):
+    model_type = "granite4_vision"
+    def __init__(
+        self,
+        downsample_rate=None,
+        use_image_newline_parameter=True,
+        deepstack_layer_map: Optional[list] = None,
+        use_spatial_sampling: bool = False,
+        spatial_stride: int = 2,
+        spatial_vision_layer: int = -1,
+        spatial_target_layers: Optional[list] = None,
+        projector_dropout=0.1,
+        **kwargs
+    ):
+        self.downsample_rate = downsample_rate
+        self.use_image_newline_parameter = use_image_newline_parameter
+        self.projector_dropout = projector_dropout
+
+        # Deepstack layer map: list of (vision_layer_idx, llm_layer_idx) tuples.
+        # Features from each vision layer are extracted, downsampled, and injected
+        # at the corresponding LLM layer during forward pass.
+        # e.g., [(-25, 12), (-17, 8), (-9, 4), (-1, 0)]
+        if deepstack_layer_map is not None:
+            self.deepstack_layer_map = [(int(v), int(l)) for v, l in deepstack_layer_map]
+            assert len(self.deepstack_layer_map) == len(set(self.deepstack_layer_map)), "expecting no duplicates"
+        else:
+            self.deepstack_layer_map = None
+
+        # Spatial sampling: extracts 4 groups from a single vision layer using
+        # spatial offset sampling (top-left, top-right, bottom-left, bottom-right
+        # of each 2x2 block), each injected at a different LLM layer.
+        self.use_spatial_sampling = use_spatial_sampling
+        self.spatial_stride = spatial_stride
+        self.spatial_vision_layer = spatial_vision_layer
+        self.spatial_target_layers = spatial_target_layers or [0, 10, 20, 30]
+
+        super().__init__(**kwargs)

downsampling.py

@@ -0,0 +1,151 @@
+import torch
+from torch import nn
+import math
+from fractions import Fraction
+from transformers.models.blip_2.configuration_blip_2 import Blip2QFormerConfig
+from transformers.models.blip_2.modeling_blip_2 import Blip2QFormerModel
+
+
+class InterpolateDownsampler:
+    """Spatial downsampling via area interpolation."""
+    def __init__(self, config, mode="area"):
+        self.orig_image_side = config.vision_config.image_size // config.vision_config.patch_size
+        self.new_image_side = int(self.orig_image_side * Fraction(config.downsample_rate))
+        self.mode = mode
+
+    def __call__(self, image_features):
+        batch_size, _, dim = image_features.size()
+        up_shape = [batch_size] + [self.orig_image_side] * 2 + [dim]
+        large_image_permuted = image_features.view(up_shape).permute(0,3,1,2)
+        small_image_permuted = torch.nn.functional.interpolate(
+                large_image_permuted, size=(self.new_image_side, self.new_image_side),
+                mode=self.mode,
+        )
+        final = small_image_permuted.permute(0,2,3,1).flatten(1,2)
+        return final
+
+
+class SpatialOffsetDownsampler:
+    """
+    Downsampler that samples one position from each 2x2 block across the image.
+    Maintains full spatial coverage while creating local continuity.
+    """
+    def __init__(self, config, offset=0):
+        """
+        Args:
+            config: Model configuration
+            offset: Integer offset (0, 1, 2, or 3) for position within each 2x2 block
+                   0: top-left, 1: top-right, 2: bottom-left, 3: bottom-right
+        """
+        self.orig_image_side = config.vision_config.image_size // config.vision_config.patch_size
+        self.new_image_side = self.orig_image_side // 2
+        self.offset = offset
+        self.offsets = [(0, 0), (0, 1), (1, 0), (1, 1)]
+        self.offset_h, self.offset_w = self.offsets[offset]
+
+    def __call__(self, image_features):
+        batch_size, seq_len, hidden_dim = image_features.shape
+        features_2d = image_features.reshape(batch_size, self.orig_image_side, self.orig_image_side, hidden_dim)
+
+        n_blocks = self.new_image_side
+        features_blocks = features_2d.reshape(
+            batch_size, n_blocks, 2, n_blocks, 2, hidden_dim
+        )
+
+        sampled = features_blocks[:, :, self.offset_h, :, self.offset_w, :]
+        sampled = sampled.reshape(batch_size, -1, hidden_dim)
+
+        return sampled
+
+
+class WindowQFormerDownsampler(nn.Module):
+    """Window-based QFormer downsampler that processes image patches in windows."""
+    def __init__(self, config, spatial_offset=None):
+        super().__init__()
+        llm_hidden_size = config.text_config.hidden_size
+        vision_hidden_size = config.vision_config.hidden_size
+
+        self.dropout = nn.Dropout(config.projector_dropout)
+
+        if spatial_offset is not None:
+            self.downsampler = SpatialOffsetDownsampler(config, offset=spatial_offset)
+        else:
+            self.downsampler = InterpolateDownsampler(config)
+
+        configuration = Blip2QFormerConfig(
+            hidden_size=vision_hidden_size,
+            num_attention_heads=vision_hidden_size // 64,
+            intermediate_size=3072,
+            num_hidden_layers=1,
+            encoder_hidden_size=vision_hidden_size,
+            cross_attention_frequency=1,
+            max_position_embeddings=2048,
+            use_qformer_text_input=False,
+        )
+        self.qformer = Blip2QFormerModel(configuration)
+
+        self.image_side = config.vision_config.image_size // config.vision_config.patch_size
+        q, w = config.downsample_rate.split("/")
+        self.query_side, self.window_side = int(q), int(w)
+        self.query_length = self.query_side ** 2
+        embed_std = 1 / math.sqrt(vision_hidden_size)
+        self.norm = nn.LayerNorm(vision_hidden_size, eps=1e-6)
+        self.query = nn.Parameter(torch.randn(1, self.query_length, vision_hidden_size) * embed_std)
+        self.image_positions = nn.Parameter(torch.randn(1, self.window_side ** 2, vision_hidden_size) * embed_std)
+        self.out_linear = nn.Linear(vision_hidden_size, llm_hidden_size, bias=True)
+
+    def _win(self, x, side, win):
+        """
+        (B, side*side, C) raster -> (B*n*n, win*win, C) where n=side//win
+        windows are raster-ordered, and tokens inside each window are raster-ordered.
+        """
+        B, _, C = x.shape
+        n = side // win
+        return (
+            x.view(B, side, side, C)
+            .view(B, n, win, n, win, C)
+            .transpose(2, 3)          # (B, n, n, win, win, C)
+            .flatten(0, 2)            # (B*n*n, win, win, C)
+            .flatten(1, 2)            # (B*n*n, win*win, C)
+        )
+
+    def _unwin(self, xw, n, win):
+        """
+        (B*n*n, win*win, C) -> (B, (n*win)^2, C) raster
+        """
+        Bnn, _, C = xw.shape
+        assert Bnn % (n * n) == 0
+        B = Bnn // (n * n)
+        side = n * win
+        return (
+            xw.view(B, n, n, win, win, C)
+            .transpose(2, 3)                 # (B, n, win, n, win, C)
+            .contiguous()
+            .view(B, side, side, C)
+            .flatten(1, 2)
+        )
+
+    def forward(self, image_features):
+        B, HW, C = image_features.shape
+        assert HW == self.image_side * self.image_side
+        n = self.image_side // self.window_side
+        image_features = self.norm(image_features)
+        enc = self._win(image_features, self.image_side, self.window_side)
+
+        downsampled = self.downsampler(image_features)
+
+        new_side = n * self.query_side
+        downsampled_w = self._win(downsampled, new_side, self.query_side)
+
+        query_embeds = self.query + downsampled_w
+        encoder_embeds = self.dropout(enc + self.image_positions)
+        out_w = self.qformer(
+            query_embeds=query_embeds,
+            encoder_hidden_states=encoder_embeds,
+            return_dict=True,
+        ).last_hidden_state
+
+        out = self._unwin(out_w, n=n, win=self.query_side)
+
+        out = self.dropout(out)
+        return self.out_linear(out)

generation_config.json

@@ -0,0 +1,7 @@
+{
+  "_from_model_config": true,
+  "bos_token_id": 100257,
+  "eos_token_id": 100257,
+  "pad_token_id": 100256,
+  "transformers_version": "4.57.3"
+}

model.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:d9d4a2d2bcbf82ea2e2ee68351b852a79051c8c3c715c7299ac637463741de0c
+size 4649181308

modeling.py

@@ -0,0 +1,550 @@
+import math
+from fractions import Fraction
+from typing import Optional, Union
+
+import numpy as np
+import torch
+from torch import nn
+import transformers
+from transformers import (
+    AutoModel,
+    LlavaNextForConditionalGeneration,
+)
+
+_V5 = int(transformers.__version__.split(".")[0]) >= 5
+from transformers.masking_utils import create_causal_mask
+from transformers.cache_utils import Cache, DynamicCache
+from transformers.modeling_flash_attention_utils import FlashAttentionKwargs
+from transformers.models.llava_next.modeling_llava_next import (
+    LlavaNextCausalLMOutputWithPast,
+    LlavaNextModelOutputWithPast,
+    LlavaNextPreTrainedModel,
+    get_anyres_image_grid_shape,
+    image_size_to_num_patches,
+    unpad_image,
+)
+from transformers.processing_utils import Unpack
+from transformers.utils import TransformersKwargs, can_return_tuple, logging
+
+from .configuration import Granite4VisionConfig
+from .downsampling import WindowQFormerDownsampler
+
+logger = logging.get_logger(__name__)
+
+
+class Granite4VisionForConditionalGeneration(LlavaNextForConditionalGeneration):
+    config_class = Granite4VisionConfig
+
+    def __init__(self, config: Granite4VisionConfig):
+        LlavaNextPreTrainedModel.__init__(self, config)
+
+        self.model = Granite4VisionModel(config)
+
+        self.lm_head = nn.Linear(
+            config.text_config.hidden_size, config.text_config.vocab_size, bias=False
+        )
+
+        self.post_init()
+
+    def generate(self, *args, **kwargs) -> torch.LongTensor:
+        # When loaded with a LoRA adapter, disable the adapter for text-only
+        # inputs (no pixel_values) so the base LLM runs standalone.
+        pixel_values = kwargs.get("pixel_values", None)
+        if hasattr(self, "_hf_peft_config_loaded") and self._hf_peft_config_loaded:
+            if pixel_values is not None:
+                self.enable_adapters()
+            else:
+                self.disable_adapters()
+        return super().generate(*args, **kwargs)
+
+    @can_return_tuple
+    def forward(
+        self,
+        input_ids: Optional[torch.LongTensor] = None,
+        pixel_values: Optional[torch.FloatTensor] = None,
+        image_sizes: Optional[torch.LongTensor] = None,
+        attention_mask: Optional[torch.Tensor] = None,
+        position_ids: Optional[torch.LongTensor] = None,
+        past_key_values: Optional[Cache] = None,
+        inputs_embeds: Optional[torch.FloatTensor] = None,
+        vision_feature_layer: Optional[Union[int, list[int]]] = None,
+        vision_feature_select_strategy: Optional[str] = None,
+        labels: Optional[torch.LongTensor] = None,
+        use_cache: Optional[bool] = None,
+        output_attentions: Optional[bool] = None,
+        output_hidden_states: Optional[bool] = None,
+        logits_to_keep: Union[int, torch.Tensor] = 0,
+        **kwargs: Unpack[TransformersKwargs],
+    ) -> Union[tuple, LlavaNextCausalLMOutputWithPast]:
+        cache_position = kwargs.pop("cache_position", None)
+
+        output_attentions = output_attentions if output_attentions is not None else self.config.output_attentions
+        output_hidden_states = (
+            output_hidden_states if output_hidden_states is not None else self.config.output_hidden_states
+        )
+        vision_feature_layer = (
+            vision_feature_layer if vision_feature_layer is not None else self.config.vision_feature_layer
+        )
+        vision_feature_select_strategy = (
+            vision_feature_select_strategy
+            if vision_feature_select_strategy is not None
+            else self.config.vision_feature_select_strategy
+        )
+
+        model_kwargs = dict(
+            pixel_values=pixel_values,
+            image_sizes=image_sizes,
+            vision_feature_layer=vision_feature_layer,
+            vision_feature_select_strategy=vision_feature_select_strategy,
+            attention_mask=attention_mask,
+            position_ids=position_ids,
+            past_key_values=past_key_values,
+            inputs_embeds=inputs_embeds,
+            use_cache=use_cache,
+            output_attentions=output_attentions,
+            output_hidden_states=output_hidden_states,
+            return_dict=True,
+        )
+        if not _V5:
+            model_kwargs["cache_position"] = cache_position
+        outputs = self.model(input_ids, **model_kwargs, **kwargs)
+
+        hidden_states = outputs.last_hidden_state
+
+        loss = None
+        logits = self.lm_head(hidden_states)
+        logits = logits / self.config.text_config.logits_scaling
+        if labels is not None:
+            loss = self.loss_function(
+                logits,
+                labels,
+                vocab_size=self.config.text_config.vocab_size,
+                **kwargs,
+            )
+
+        if isinstance(logits_to_keep, int) and logits_to_keep > 0:
+            logits = logits[:, -logits_to_keep:, :]
+
+        return LlavaNextCausalLMOutputWithPast(
+            loss=loss,
+            logits=logits,
+            past_key_values=outputs.past_key_values,
+            hidden_states=outputs.hidden_states,
+            attentions=outputs.attentions,
+            image_hidden_states=outputs.image_hidden_states,
+        )
+
+    def prepare_inputs_for_generation(
+        self,
+        input_ids,
+        past_key_values=None,
+        inputs_embeds=None,
+        pixel_values=None,
+        image_sizes=None,
+        attention_mask=None,
+        cache_position=None,
+        logits_to_keep=None,
+        **kwargs,
+    ):
+        if _V5:
+            is_first = kwargs.get("is_first_iteration", False)
+            model_inputs = super().prepare_inputs_for_generation(
+                input_ids,
+                past_key_values=past_key_values,
+                inputs_embeds=inputs_embeds,
+                attention_mask=attention_mask,
+                logits_to_keep=logits_to_keep,
+                **kwargs,
+            )
+        else:
+            is_first = cache_position[0] == 0 if cache_position is not None else True
+            model_inputs = super().prepare_inputs_for_generation(
+                input_ids,
+                past_key_values=past_key_values,
+                inputs_embeds=inputs_embeds,
+                attention_mask=attention_mask,
+                cache_position=cache_position,
+                logits_to_keep=logits_to_keep,
+                **kwargs,
+            )
+        if is_first:
+            model_inputs["pixel_values"] = pixel_values
+            model_inputs["image_sizes"] = image_sizes
+
+        return model_inputs
+
+
+class Granite4VisionModel(LlavaNextPreTrainedModel):
+    config_class = Granite4VisionConfig
+
+    def __init__(self, config: Granite4VisionConfig):
+        super().__init__(config)
+        self.vision_tower = AutoModel.from_config(config.vision_config)
+        self.spatial_projectors = None
+
+        assert config.deepstack_layer_map is not None
+        assert config.downsample_rate is not None
+
+        self.downsample_rate = config.downsample_rate
+
+        # Deepstack projectors: one per (vision_layer, llm_layer) pair
+        self.layerwise_projectors = nn.ModuleList([
+            WindowQFormerDownsampler(config)
+            for _ in range(len(config.deepstack_layer_map))
+        ])
+
+        # Spatial sampling projectors: 4 offset groups (TL, TR, BL, BR)
+        if config.use_spatial_sampling:
+            self.spatial_projectors = nn.ModuleList([
+                WindowQFormerDownsampler(config, spatial_offset=i)
+                for i in range(4)
+            ])
+
+        self.image_newline = None
+        if config.use_image_newline_parameter:
+            embed_std = 1 / math.sqrt(config.text_config.hidden_size)
+            self.image_newline = nn.Parameter(torch.randn(config.text_config.hidden_size, dtype=self.dtype) * embed_std)
+
+        self.vocab_size = config.text_config.vocab_size
+        self.language_model = AutoModel.from_config(config.text_config)
+        self.pad_token_id = getattr(self.config, "pad_token_id", None) or -1
+        self.post_init()
+
+    def get_input_embeddings(self):
+        return self.language_model.get_input_embeddings()
+
+    def set_input_embeddings(self, value):
+        self.language_model.set_input_embeddings(value)
+
+    def set_decoder(self, decoder):
+        self.language_model = decoder
+
+    def get_decoder(self):
+        return self.language_model
+
+    def pack_and_unpad_image_features(self, image_features, image_sizes, vision_feature_select_strategy, image_newline=None):
+        """
+        Reshape, unpad and then pack each image_feature into a single image_features tensor containing all visual vectors.
+
+        Args:
+            image_features (`list[torch.Tensor]` of length num_images, each of shape `(num_patches, image_length, embed_dim)`)
+                List of image feature tensor, each contains all the visual feature of all patches.
+            image_sizes (`torch.Tensor` of shape `(num_images, 2)`)
+                Actual image size of each images (H, W).
+            vision_feature_select_strategy (`str`)
+                The feature selection strategy used to select the vision feature from the vision backbone.
+            image_newline (`torch.Tensor` of shape `(embed_dim)`)
+                New line embedding vector.
+        Returns:
+            image_features (`torch.Tensor` of shape `(all_feat_len, embed_dim)`)
+            feature_lens (`list[int]`)
+                token length of each image in image_features
+        """
+        new_image_features = []
+        feature_lens = []
+        for image_idx, image_feature in enumerate(image_features):
+            if image_feature.shape[0] > 1:
+                base_image_feature = image_feature[0]
+                image_feature = image_feature[1:]
+                height = width = self.config.vision_config.image_size // self.config.vision_config.patch_size
+
+                num_patch_height, num_patch_width = get_anyres_image_grid_shape(
+                    image_sizes[image_idx],
+                    self.config.image_grid_pinpoints,
+                    self.config.vision_config.image_size,
+                )
+                if self.layerwise_projectors is not None:
+                    ds_rate = Fraction(self.downsample_rate)
+                    height = int(height  * ds_rate)
+                    width = int(width  * ds_rate)
+
+                if (
+                    np.prod(image_feature.shape) % (num_patch_height * num_patch_width * height * width) != 0
+                    and vision_feature_select_strategy == "default"
+                ):
+                    logger.warning_once(
+                        "Image feature shape does not line up with the provided patch size. "
+                        "You may be using the `default` vision_feature_select_strategy with a"
+                        " visual encoder that does not have CLS."
+                    )
+
+                image_feature = image_feature.view(num_patch_height, num_patch_width, height, width, -1)
+                image_feature = image_feature.permute(4, 0, 2, 1, 3).contiguous()
+                image_feature = image_feature.flatten(1, 2).flatten(2, 3)
+                image_feature = unpad_image(image_feature, image_sizes[image_idx])
+                if image_newline is not None:
+                    image_feature = torch.cat(
+                        (
+                            image_feature,
+                            image_newline[:, None, None]
+                            .expand(*image_feature.shape[:-1], 1)
+                            .to(image_feature.device, image_feature.dtype),
+                        ),
+                        dim=-1,
+                    )
+                image_feature = image_feature.flatten(1, 2).transpose(0, 1)
+                image_feature = torch.cat((base_image_feature, image_feature), dim=0)
+            else:
+                image_feature = image_feature[0]
+                if image_newline is not None:
+                    image_feature = torch.cat((image_feature, image_newline[None].to(image_feature)), dim=0)
+            new_image_features.append(image_feature)
+            feature_lens.append(image_feature.size(0))
+        feature_lens = torch.tensor(feature_lens, dtype=torch.long, device=image_features[0].device)
+        return new_image_features, feature_lens
+
+    def get_image_features(
+        self,
+        pixel_values: torch.FloatTensor,
+        image_sizes: torch.Tensor,
+        vision_feature_layer: Optional[Union[int, list[int]]] = None,
+        vision_feature_select_strategy: Optional[str] = None,
+    ):
+        """
+        Extract image features via deepstack (multi-layer) and spatial sampling projections.
+
+        Runs the vision tower once, then:
+        1. Deepstack: for each (vision_layer, llm_layer) in deepstack_layer_map,
+           extracts features from that vision layer, downsamples via interpolation + QFormer,
+           and pairs them with the target LLM layer.
+        2. Spatial: if enabled, extracts the spatial_vision_layer and creates 4 spatial
+           offset groups (TL, TR, BL, BR), each targeting a different LLM layer.
+
+        Args:
+            pixel_values: Image tensors of shape (batch, num_patches, C, H, W) or (N, C, H, W).
+            image_sizes: Actual image sizes (num_images, 2).
+            vision_feature_layer: Unused (kept for API compatibility).
+            vision_feature_select_strategy: "default" (remove CLS) or "full".
+        Returns:
+            List of (llm_layer_idx, packed_features) tuples for injection during forward pass.
+        """
+        vision_feature_select_strategy = (
+            vision_feature_select_strategy
+            if vision_feature_select_strategy is not None
+            else self.config.vision_feature_select_strategy
+        )
+
+        image_num_patches = [
+            image_size_to_num_patches(
+                image_size=imsize,
+                grid_pinpoints=self.config.image_grid_pinpoints,
+                patch_size=self.config.vision_config.image_size,
+            )
+            for imsize in image_sizes
+        ]
+
+        if pixel_values.dim() == 5:
+            _pixel_values_list = [pix_val[:num_patch] for pix_val, num_patch in zip(pixel_values, image_num_patches)]
+            pixel_values = torch.cat(_pixel_values_list, dim=0)
+        elif pixel_values.dim() != 4:
+            raise ValueError(f"pixel_values of shape {pixel_values.shape}, expect to be of 4 or 5 dimensions")
+
+        vision_outputs = self.vision_tower(pixel_values, output_hidden_states=True)
+
+        # Deepstack features: extract from multiple vision layers, downsample via interpolation
+        all_features = []
+        for projection_idx, (vision_layer, llm_layer) in enumerate(self.config.deepstack_layer_map):
+            selected_feature = vision_outputs.hidden_states[vision_layer]
+
+            if vision_feature_select_strategy == "default":
+                selected_feature = selected_feature[:, 1:]
+
+            projected_features = self.layerwise_projectors[projection_idx](selected_feature)
+            projected_features = torch.split(projected_features, image_num_patches, dim=0)
+
+            packed_features, _ = self.pack_and_unpad_image_features(
+                projected_features,
+                image_sizes,
+                vision_feature_select_strategy=vision_feature_select_strategy,
+                image_newline=self.image_newline,
+            )
+
+            all_features.append((llm_layer, packed_features))
+
+        # Spatial features: extract 4 offset groups from a single vision layer
+        if self.config.use_spatial_sampling:
+            spatial_feature = vision_outputs.hidden_states[self.config.spatial_vision_layer]
+
+            if vision_feature_select_strategy == "default":
+                spatial_feature = spatial_feature[:, 1:]
+
+            for group_idx, llm_layer in enumerate(self.config.spatial_target_layers):
+                projected_group = self.spatial_projectors[group_idx](spatial_feature)
+                projected_group_split = torch.split(projected_group, image_num_patches, dim=0)
+
+                packed_group, _ = self.pack_and_unpad_image_features(
+                    projected_group_split,
+                    image_sizes,
+                    vision_feature_select_strategy=vision_feature_select_strategy,
+                    image_newline=self.image_newline,
+                )
+
+                all_features.append((llm_layer, packed_group))
+
+        return all_features
+
+    def get_image_token_mask(
+        self, input_ids: torch.LongTensor, inputs_embeds: torch.FloatTensor, image_features: torch.FloatTensor
+    ):
+        """
+        Build a boolean mask over inputs_embeds marking positions of <image> tokens,
+        and verify that the count matches the number of image feature vectors.
+        """
+        if input_ids is None:
+            special_image_mask = inputs_embeds == self.get_input_embeddings()(
+                torch.tensor(self.config.image_token_id, dtype=torch.long, device=inputs_embeds.device)
+            )
+            special_image_mask = special_image_mask.all(-1)
+        else:
+            special_image_mask = input_ids == self.config.image_token_id
+
+        n_image_tokens = special_image_mask.sum()
+        special_image_mask = special_image_mask.unsqueeze(-1).expand_as(inputs_embeds).to(inputs_embeds.device)
+        if inputs_embeds[special_image_mask].numel() != image_features.numel():
+            raise ValueError(
+                f"Image features and image tokens do not match: tokens: {n_image_tokens}, features {image_features.shape[0]}"
+            )
+        return special_image_mask
+
+    @can_return_tuple
+    def forward(
+        self,
+        input_ids: Optional[torch.LongTensor] = None,
+        pixel_values: Optional[torch.FloatTensor] = None,
+        image_sizes: Optional[torch.LongTensor] = None,
+        attention_mask: Optional[torch.Tensor] = None,
+        position_ids: Optional[torch.LongTensor] = None,
+        past_key_values: Optional[Cache] = None,
+        inputs_embeds: Optional[torch.FloatTensor] = None,
+        vision_feature_layer: Optional[Union[int, list[int]]] = None,
+        vision_feature_select_strategy: Optional[str] = None,
+        use_cache: Optional[bool] = None,
+        output_attentions: Optional[bool] = None,
+        output_hidden_states: Optional[bool] = None,
+        return_dict: Optional[bool] = None,
+        **kwargs: Unpack[FlashAttentionKwargs],
+    ) -> Union[tuple, LlavaNextModelOutputWithPast]:
+        cache_position = kwargs.pop("cache_position", None)
+        output_attentions = output_attentions if output_attentions is not None else self.config.output_attentions
+        output_hidden_states = (
+            output_hidden_states if output_hidden_states is not None else self.config.output_hidden_states
+        )
+        return_dict = return_dict if return_dict is not None else self.config.return_dict
+        vision_feature_layer = (
+            vision_feature_layer if vision_feature_layer is not None else self.config.vision_feature_layer
+        )
+        vision_feature_select_strategy = (
+            vision_feature_select_strategy
+            if vision_feature_select_strategy is not None
+            else self.config.vision_feature_select_strategy
+        )
+
+        if (input_ids is None) ^ (inputs_embeds is not None):
+            raise ValueError("You must specify exactly one of input_ids or inputs_embeds")
+
+        if inputs_embeds is None:
+            inputs_embeds = self.get_input_embeddings()(input_ids)
+
+        # Extract deepstack + spatial features and prepare for layer-by-layer injection
+        deepstack_features = []
+        vision_mask = None
+        image_features = None
+        if pixel_values is not None and pixel_values.size(0) > 0:
+            image_features = self.get_image_features(
+                pixel_values,
+                image_sizes,
+                vision_feature_layer=vision_feature_layer,
+                vision_feature_select_strategy=vision_feature_select_strategy,
+            )
+
+            for idx, (llm_layer_idx, packed_features) in enumerate(image_features):
+                concat_features = torch.cat(packed_features, dim=0).to(
+                    inputs_embeds.device, inputs_embeds.dtype
+                )
+                if idx == 0:
+                    vision_mask = self.get_image_token_mask(
+                        input_ids, inputs_embeds=inputs_embeds, image_features=concat_features
+                    )
+                    inputs_embeds = inputs_embeds.masked_fill(vision_mask, 0.0)
+                deepstack_features.append((llm_layer_idx, concat_features))
+
+        # Custom forward pass with vision injection at specific LLM layers
+        hidden_states = inputs_embeds * self.language_model.embedding_multiplier
+
+        if use_cache and past_key_values is None:
+            past_key_values = DynamicCache(config=self.language_model.config)
+
+        if cache_position is None:
+            past_seen_tokens = past_key_values.get_seq_length() if past_key_values is not None else 0
+            cache_position = torch.arange(
+                past_seen_tokens, past_seen_tokens + inputs_embeds.shape[1], device=inputs_embeds.device
+            )
+        if position_ids is None:
+            position_ids = cache_position.unsqueeze(0)
+
+        # v4: input_embeds (positional), cache_position required
+        # v5: inputs_embeds (keyword), cache_position deprecated, past_key_values keyword-only
+        # v5 renamed input_embeds -> inputs_embeds; cache_position kept for BC
+        embeds_key = "inputs_embeds" if _V5 else "input_embeds"
+        mask_kwargs = dict(
+            config=self.language_model.config,
+            attention_mask=attention_mask,
+            cache_position=cache_position,
+            past_key_values=past_key_values,
+            position_ids=position_ids,
+        )
+        mask_kwargs[embeds_key] = inputs_embeds
+        causal_mask = create_causal_mask(**mask_kwargs)
+
+        position_embeddings = self.language_model.rotary_emb(hidden_states, position_ids)
+
+        all_hidden_states = () if output_hidden_states else None
+        all_self_attns = () if output_attentions else None
+
+        # v4: decoder_layer returns tuple (hidden_states, [attn_weights]), takes output_attentions/cache_position
+        # v5: decoder_layer returns bare tensor
+        layer_kwargs = dict(
+            attention_mask=causal_mask,
+            position_ids=position_ids,
+            past_key_values=past_key_values,
+            use_cache=use_cache,
+            position_embeddings=position_embeddings,
+        )
+        if not _V5:
+            layer_kwargs["output_attentions"] = output_attentions
+            layer_kwargs["cache_position"] = cache_position
+
+        # Layer-by-layer forward with vision injection
+        for layer_idx, decoder_layer in enumerate(self.language_model.layers):
+            # Inject vision features at this layer if configured
+            for target_layer, features_for_layer in deepstack_features:
+                if layer_idx == target_layer:
+                    hidden_states = hidden_states.masked_scatter(
+                        vision_mask,
+                        (hidden_states[vision_mask] + features_for_layer.flatten()).view(-1)
+                    )
+
+            if output_hidden_states:
+                all_hidden_states += (hidden_states,)
+
+            layer_result = decoder_layer(hidden_states, **layer_kwargs, **kwargs)
+            if _V5:
+                hidden_states = layer_result
+            else:
+                hidden_states = layer_result[0]
+                if output_attentions:
+                    all_self_attns += (layer_result[1],)
+
+        hidden_states = self.language_model.norm(hidden_states)
+
+        if output_hidden_states:
+            all_hidden_states += (hidden_states,)
+
+
+        return LlavaNextModelOutputWithPast(
+            last_hidden_state=hidden_states,
+            past_key_values=past_key_values if use_cache else None,
+            hidden_states=all_hidden_states,
+            attentions=all_self_attns,
+            image_hidden_states=image_features if pixel_values is not None else None,
+        )

preprocessor_config.json

@@ -0,0 +1,143 @@
+{
+  "crop_size": {
+    "height": 384,
+    "width": 384
+  },
+  "do_center_crop": true,
+  "do_convert_rgb": true,
+  "do_normalize": true,
+  "do_pad": true,
+  "do_rescale": true,
+  "do_resize": true,
+  "image_grid_pinpoints": [
+    [
+      384,
+      384
+    ],
+    [
+      384,
+      768
+    ],
+    [
+      384,
+      1152
+    ],
+    [
+      384,
+      1536
+    ],
+    [
+      384,
+      1920
+    ],
+    [
+      384,
+      2304
+    ],
+    [
+      384,
+      2688
+    ],
+    [
+      384,
+      3072
+    ],
+    [
+      384,
+      3456
+    ],
+    [
+      384,
+      3840
+    ],
+    [
+      768,
+      384
+    ],
+    [
+      768,
+      768
+    ],
+    [
+      768,
+      1152
+    ],
+    [
+      768,
+      1536
+    ],
+    [
+      768,
+      1920
+    ],
+    [
+      1152,
+      384
+    ],
+    [
+      1152,
+      768
+    ],
+    [
+      1152,
+      1152
+    ],
+    [
+      1536,
+      384
+    ],
+    [
+      1536,
+      768
+    ],
+    [
+      1920,
+      384
+    ],
+    [
+      1920,
+      768
+    ],
+    [
+      2304,
+      384
+    ],
+    [
+      2688,
+      384
+    ],
+    [
+      3072,
+      384
+    ],
+    [
+      3456,
+      384
+    ],
+    [
+      3840,
+      384
+    ]
+  ],
+  "image_mean": [
+    0.5,
+    0.5,
+    0.5
+  ],
+  "image_processor_type": "LlavaNextImageProcessor",
+  "image_std": [
+    0.5,
+    0.5,
+    0.5
+  ],
+  "processor_class": "Granite4VisionProcessor",
+  "auto_map": {
+    "AutoProcessor": "processing.Granite4VisionProcessor"
+  },
+  "resample": 3,
+  "rescale_factor": 0.00392156862745098,
+  "size": {
+    "height": 384,
+    "width": 384
+  }
+}

processing.py

@@ -0,0 +1,54 @@
+from fractions import Fraction
+
+from transformers import LlavaNextProcessor
+from transformers.image_processing_utils import select_best_resolution
+
+
+
+class Granite4VisionProcessor(LlavaNextProcessor):
+    model_type = "granite4_vision"
+
+    def __init__(
+        self,
+        image_processor=None,
+        tokenizer=None,
+        patch_size=None,
+        vision_feature_select_strategy=None,
+        chat_template=None,
+        image_token="<image>",  # set the default and let users change if they have peculiar special tokens in rare cases
+        num_additional_image_tokens=0,
+        downsample_rate=None,
+        **kwargs,
+    ):
+        super().__init__(image_processor=image_processor, 
+                         tokenizer=tokenizer,
+                         patch_size=patch_size,
+                         vision_feature_select_strategy=vision_feature_select_strategy,
+                         chat_template=chat_template,
+                         image_token=image_token,
+                         num_additional_image_tokens=num_additional_image_tokens,
+                         )
+        self.downsample_rate = downsample_rate
+
+    def _get_number_of_features(self, orig_height: int, orig_width: int, height: int, width: int) -> int:
+        image_grid_pinpoints = self.image_processor.image_grid_pinpoints
+
+        height_best_resolution, width_best_resolution = select_best_resolution(
+            [orig_height, orig_width], image_grid_pinpoints
+        )
+        scale_height, scale_width = height_best_resolution // height, width_best_resolution // width
+
+        patches_height = height // self.patch_size
+        patches_width = width // self.patch_size
+        if self.downsample_rate is not None:
+            ds_rate = Fraction(self.downsample_rate)
+            patches_height = int(patches_height * ds_rate)
+            patches_width = int(patches_width * ds_rate)
+            
+        unpadded_features, newline_features = self._get_unpadded_features(
+            orig_height, orig_width, patches_height, patches_width, scale_height, scale_width
+        )
+        # The base patch covers the entire image (+1 for the CLS)
+        base_features = patches_height * patches_width + self.num_additional_image_tokens
+        num_image_tokens = unpadded_features + newline_features + base_features
+        return num_image_tokens

special_tokens_map.json

@@ -0,0 +1,33 @@
+{
+  "additional_special_tokens": [
+    "<image>"
+  ],
+  "bos_token": {
+    "content": "<|end_of_text|>",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  },
+  "eos_token": {
+    "content": "<|end_of_text|>",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  },
+  "pad_token": {
+    "content": "<|pad|>",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  },
+  "unk_token": {
+    "content": "<|unk|>",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  }
+}

tokenizer_config.json

@@ -0,0 +1,796 @@
+{
+  "add_bos_token": false,
+  "add_prefix_space": false,
+  "added_tokens_decoder": {
+    "100256": {
+      "content": "<|pad|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "100257": {
+      "content": "<|end_of_text|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "100258": {
+      "content": "<|fim_prefix|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "100259": {
+      "content": "<|fim_middle|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "100260": {
+      "content": "<|fim_suffix|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "100261": {
+      "content": "<|fim_pad|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "100262": {
+      "content": "<|filename|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "100263": {
+      "content": "<|reponame|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "100264": {
+      "content": "<|start_of_role|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "100265": {
+      "content": "<|end_of_role|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "100266": {
+      "content": "<|unused_1|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "100267": {
+      "content": "<|start_of_plugin|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "100268": {
+      "content": "<|end_of_plugin|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "100269": {
+      "content": "<|unk|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "100270": {
+      "content": "<tool_call>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "100271": {
+      "content": "</tool_call>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "100272": {
+      "content": "<tool_response>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "100273": {
+      "content": "</tool_response>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "100274": {
+      "content": "<think>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "100275": {
+      "content": "</think>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "100276": {
+      "content": "<think_on>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "100277": {
+      "content": "<think_off>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "100278": {
+      "content": "<schema>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "100279": {
+      "content": "</schema>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "100280": {
+      "content": "<tools>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "100281": {
+      "content": "</tools>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "100282": {
+      "content": "<documents>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "100283": {
+      "content": "</documents>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "100284": {
+      "content": "<|unused_15|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "100285": {
+      "content": "<|unused_16|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "100286": {
+      "content": "<|unused_17|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "100287": {
+      "content": "<|unused_18|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "100288": {
+      "content": "<|unused_19|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "100289": {
+      "content": "<|unused_20|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "100290": {
+      "content": "<|unused_21|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "100291": {
+      "content": "<|unused_22|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "100292": {
+      "content": "<|unused_23|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "100293": {
+      "content": "<|unused_24|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "100294": {
+      "content": "<|unused_25|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "100295": {
+      "content": "<|unused_26|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "100296": {
+      "content": "<|unused_27|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "100297": {
+      "content": "<|unused_28|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "100298": {
+      "content": "<|unused_29|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "100299": {
+      "content": "<|unused_30|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "100300": {
+      "content": "<|unused_31|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "100301": {
+      "content": "<|unused_32|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "100302": {
+      "content": "<|unused_33|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "100303": {
+      "content": "<|unused_34|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "100304": {
+      "content": "<|unused_35|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "100305": {
+      "content": "<|unused_36|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "100306": {
+      "content": "<|unused_37|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "100307": {
+      "content": "<|unused_38|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "100308": {
+      "content": "<|unused_39|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "100309": {
+      "content": "<|unused_40|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "100310": {
+      "content": "<|unused_41|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "100311": {
+      "content": "<|unused_42|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "100312": {
+      "content": "<|unused_43|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "100313": {
+      "content": "<|unused_44|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "100314": {
+      "content": "<|unused_45|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "100315": {
+      "content": "<|unused_46|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "100316": {
+      "content": "<|unused_47|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "100317": {
+      "content": "<|unused_48|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "100318": {
+      "content": "<|unused_49|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "100319": {
+      "content": "<|unused_50|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "100320": {
+      "content": "<|unused_51|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "100321": {
+      "content": "<|unused_52|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "100322": {
+      "content": "<|unused_53|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "100323": {
+      "content": "<|unused_54|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "100324": {
+      "content": "<|unused_55|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "100325": {
+      "content": "<|unused_56|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "100326": {
+      "content": "<|unused_57|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "100327": {
+      "content": "<|unused_58|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "100328": {
+      "content": "<|unused_59|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "100329": {
+      "content": "<|unused_60|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "100330": {
+      "content": "<|unused_61|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "100331": {
+      "content": "<|unused_62|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "100332": {
+      "content": "<|unused_63|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "100333": {
+      "content": "<|unused_64|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "100334": {
+      "content": "<|unused_65|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "100335": {
+      "content": "<|unused_66|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "100336": {
+      "content": "<|unused_67|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "100337": {
+      "content": "<|unused_68|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "100338": {
+      "content": "<|unused_69|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "100339": {
+      "content": "<|unused_70|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "100340": {
+      "content": "<|unused_71|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "100341": {
+      "content": "<|unused_72|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "100342": {
+      "content": "<|unused_73|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "100343": {
+      "content": "<|unused_74|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "100344": {
+      "content": "<|unused_75|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "100345": {
+      "content": "<|unused_76|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "100346": {
+      "content": "<|unused_77|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "100347": {
+      "content": "<|unused_78|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "100348": {
+      "content": "<|unused_79|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "100349": {
+      "content": "<|unused_80|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "100350": {
+      "content": "<|unused_81|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "100351": {
+      "content": "<|unused_82|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "100352": {
+      "content": "<image>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    }
+  },
+  "additional_special_tokens": [
+    "<image>"
+  ],
+  "bos_token": "<|end_of_text|>",
+  "clean_up_tokenization_spaces": false,
+  "eos_token": "<|end_of_text|>",
+  "errors": "replace",
+  "extra_special_tokens": {},
+  "model_max_length": 1000000000000000019884624838656,
+  "pad_token": "<|pad|>",
+  "padding_side": "left",
+  "processor_class": "Granite4VisionProcessor",
+  "tokenizer_class": "GPT2Tokenizer",
+  "unk_token": "<|unk|>"
+}