Create ReadMe_Surgical_Vocab_Reduction_for_QWEN2B_DotOCRv1.5_V1.0untested_py.md

ReadMe_Surgical_Vocab_Reduction_for_QWEN2B_DotOCRv1.5_V1.0untested_py.md

@@ -0,0 +1,98 @@
+# Surgical Vocabulary & Parameter Reduction for Qwen2 / dots.ocr-1.5 (V1.0 Untested)
+
+This repository contains details and scripts for **Surgical_Vocab_Reduction_for_QWEN2B_DotOCRv1.5_V1.0untested.py**, a pipeline designed to prune the large multilingual vocabulary of the **dots.ocr-1.5** model (and other Qwen2-based Vision-Language Models). 
+
+By surgically removing non-target Unicode blocks (such as Chinese, Cyrillic, Arabic, Hebrew, Japanese, and Korean) while keeping English, French, Spanish, special layout coordinate symbols, and byte-fallback sequences, this script significantly reduces both the model's disk/VRAM footprint and its output logit calculation latency during local edge inference (such as on Intel Core Ultra NPUs or mobile devices).
+
+This implementation is inspired by and built upon the methodology outlined in [Method for Dynamically Reducing Logit Computation in LLMs](https://huggingface.co/MartialTerran/Method_for_Dynamically_Reducing_Logit_Computation_in_LLMs).
+
+---
+
+## 1. The Core Problem
+
+Modern, state-of-the-art multimodal models like **dots.ocr-1.5** are designed for global multilingual coverage. To achieve this, they utilize very large vocabularies (Qwen2 series, for example, features a vocabulary of **151,936 tokens**). 
+
+While this makes the model highly capable globally, it introduces significant overhead for deployment scenarios limited to Western European scripts (e.g., English, French, Spanish):
+
+1. **Parameter Inflation:** The input embedding layer (`model.embed_tokens.weight`) and the final Language Modeling head (`lm_head.weight`) scale linearly with vocabulary size. At $151,936$ tokens, these two layers alone consume hundreds of millions of parameters, adding several gigabytes of unnecessary storage and VRAM.
+2. **Computational Overhead (Logit Latency):** During autoregressive token generation, the final layer performs a matrix multiplication of shape `[batch_size * seq_len, hidden_dim] @ [hidden_dim, vocab_size]` to compute logits. Performing this massive projection on every decoded token introduces latency, particularly on resource-constrained CPUs and edge NPUs.
+
+---
+
+## 2. The Surgical Reduction Method
+
+The script executes a targeted, multi-stage compression process that shrinks the model's text processing layers without degrading its fundamental visual-spatial, OCR, or layout parsing intelligence:
+
+```
+[ Original VLM ] ──> [ 1. Scan Tokenizer & Filter Vocab ] ──> [ 2. Filter BPE Merges ] 
+                           │ (ASCII, Latin-1, & Special Tags Only)       │
+                           ▼                                             ▼
+[ Pruned Weights ] <── [ 4. Prune Embedding/Head Tensors ] <── [ 3. Re-index IDs ]
+```
+
+### Stage A: Tokenizer Parsing & Robust Filtering
+The script reads `tokenizer.json` and evaluates each token string against strict, multi-tier criteria to determine which to keep:
+* **Preserve Special & Coordinate Tokens:** Bounding box markers, segment boundaries, and visual layout tags (e.g., `<|image_1|>`, `<|box_start|>`, etc.) are detected and kept to protect the spatial parsing mechanism.
+* **Preserve Byte-Fallback Mappings:** To prevent the tokenizer from failing when encountering out-of-vocabulary characters, any token representing a single byte (Latin-1 range `0x00` - `0xFF` or length-1 strings) is retained.
+* **Targeted Script Deletion:** Multi-character tokens containing Chinese characters, Japanese Kana, Hangul, Cyrillic, Arabic, Devanagari, or Thai are discarded. 
+* **Targeted Language Retention:** Standard ASCII punctuation, numbers, and the accented character sets of English, French, and Spanish (using Latin-1 and Latin Extended-A Unicode ranges) are preserved.
+
+### Stage B: Re-Indexing & Merge List Sanitization
+* Discarded tokens are stripped, and the remaining tokens are sequentially re-indexed from $0$ to $N_{\text{new}} - 1$.
+* The Byte Pair Encoding (BPE) `"merges"` rules are filtered. If a merge pair contains a sub-token or output token that was deleted, the merge rule is safely omitted. This prevents tokenizer errors or lookup warnings.
+
+### Stage C: Surgical Tensor Pruning
+* The script scans the model directory for weights files (both PyTorch `.bin` and Safetensors `.safetensors` formats).
+* It locates the input embedding weights (`model.embed_tokens.weight` or `transformer.wte.weight`) and the output projection head weights (`lm_head.weight`).
+* It extracts only the slices of these matrices corresponding to the indices of the retained vocabulary, updating their shapes dynamically.
+
+### Stage D: Configuration Re-alignment
+* Updates `config.json` with the new `"vocab_size"`.
+* Remaps `bos_token_id`, `eos_token_id`, and `pad_token_id` to their new values inside `generation_config.json`.
+
+---
+
+## 3. Anticipated Benefits
+
+* **VRAM and Storage Compression:** Shifting from $151,936$ tokens down to an estimated $30,000 \text{--} 40,000$ tokens significantly shrinks the parameters of the text layers. For a hidden size of $D = 3584$, this reduction strips away over **800 million parameters**, translating to immediate savings of up to **1.6 GB of VRAM/storage** in FP16 (or comparable savings under quantized configurations).
+* **Improved Generation Latency:** Computing output logits against a vocabulary that is roughly $75\%$ smaller reduces the arithmetic intensity of the final `lm_head` projection layer by up to $75\%$, improving token-generation speed on resource-constrained CPUs and edge NPUs.
+* **Codebase & Edge Independence:** By outputting standard, structurally valid Hugging Face files, the resulting compressed model can be loaded natively by downstream optimization toolkits (such as OpenVINO or ONNX Runtime) without requiring custom code layers.
+
+---
+
+## 4. How to Use
+
+### Setup & Dependencies
+Ensure your python environment has the required libraries installed:
+```bash
+pip install torch safetensors transformers
+```
+
+### Directory Structure
+Place your original downloaded weights for `dots.ocr-1.5` in a source directory, and designate a destination directory for the pruned output. Example:
+```
+./weights/
+├── dots_ocr_1_5/                  <-- Original weights directory
+│   ├── config.json
+│   ├── tokenizer.json
+│   ├── model-00001-of-00002.safetensors
+│   └── ...
+└── dots_ocr_1_5_pruned/           <-- Target destination for pruned output
+```
+
+### Execution
+Run the script from your terminal:
+```bash
+python Surgical_Vocab_Reduction_for_QWEN2B_DotOCRv1.5_V1.0untested.py
+```
+Upon successful execution, the script will output the pruned weights, tokenizer structures, and updated configurations directly into your target directory. You can then load, quantize (e.g., using OpenVINO for Intel NPUs), or test the compressed model.
+
+---
+
+## 5. Disclaimer & Untested Notice
+
+Please note that this script is marked as **V1.0 Untested**. 
+
+While designed with strong safeguards—such as preserving all special layout coordinates, maintaining the byte-fallback system, and pruning the BPE merge sequences—**empirical validation is required**. 
+
+Users are encouraged to run evaluations and verification benchmarks (such as measuring layout extraction accuracy, tokenization rate, and logit output accuracy) on their pruned models to confirm that model performance meets expectations for their targeted language profiles.