--- license: apache-2.0 language: - en - ko library_name: onnxruntime pipeline_tag: visual-question-answering tags: - onnx - onnxruntime - document-question-answering - receipt - key-information-extraction - lora - routing --- # TinyReceiptVQA-StructuredQA-22M-BPE1536-DirectNoValue-LoRA-Router-e100 TinyReceiptVQA is a compact receipt-domain visual question-answering model. It takes a full receipt image and a question, then generates a structured rationale and final `` value. A learned router selects one of eight family slots; each slot has a post-encoder memory adapter and a post-decoder adapter for phone, address, store, item-row, item-math, item-lookup, math, or other questions. ## Files ```text encoder_model.onnx # CNN + multimodal encoder + router + memory adapter decoder_model.onnx # autoregressive decoder + selected decoder adapter encoder_model_int8.onnx # static W8A8 U8S8 QDQ encoder decoder_model_int8.onnx # static W8A8 U8S8 QDQ decoder manifest.json # graph contract, hashes, and runtime validation config.json vocab.json byte_bpe.py # dependency-free tokenizer runtime inference.py # ONNX Runtime greedy decoding examples/ receipt_en.jpg # verified English synthetic receipt receipt_ko.jpg # verified Korean synthetic receipt family_examples.json # 16 explicit/AUTO checks and expected answers test_all_families.py # emits decoded text, token IDs, and BPE tokens eval/ heldout_summary.json # heldout evaluation summary int8_w8a8_summary.json onnx_parity_summary.json comparison.json ``` The encoder is run once. The host runtime then calls the decoder with the growing output prefix until EOS; this avoids recomputing the receipt image and question encoder for every generated token. ## Usage ```bash pip install -r requirements.txt python inference.py \ --model-dir . \ --image examples/receipt_en.jpg \ --question "What is the street number in the store address?" # Expected JSON fields: "answer": "837", "well_formed": true # Use the Korean example with the static W8A8 U8S8 QDQ ONNX files. python inference.py \ --model-dir . \ --precision int8 \ --image examples/receipt_ko.jpg \ --question "영수증의 가게 위치에서 도로명 뒤 숫자는 무엇입니까?" # Expected JSON fields: "answer": "875", "well_formed": true ``` `--family auto` is the default and uses the learned router. An explicit family such as `phone` or `address` can be supplied for controlled debugging. The input image contract is float32 `[batch, 1, 320, 672]`: convert to grayscale, resize to 672x320 (width x height), scale to `[0,1]`, then normalize with `(x - 0.5) / 0.5`. ## Runtime interface The opset is 18. Batch, question length, and decoder-prefix length are dynamic; image height and width are fixed. - Encoder inputs: `image`, `question_ids`, and `family_ids` (`-1` means AUTO). - Encoder outputs: cached `memory`, padding mask, router logits, selected family. - Decoder inputs: generated prefix, cached memory/mask, selected family. - Decoder output: token logits. Use the included `inference.py` entry point to handle preprocessing, AUTO-routing, and greedy generation. The JSON result sets `well_formed` to `true` only when generation contains a complete `...` block. If that block is missing, `answer` is an empty string and `well_formed` is `false`; inspect `generated` for diagnostics. ## Byte-Fallback BPE and vocabulary This release uses only the packaged **Byte-Fallback BPE1536** tokenizer contract. `vocab.json` declares `type=byte_fallback_bpe`, version 1, NFC normalization, and exactly 1,536 tokens. Character-vocabulary artifacts are rejected instead of falling back silently. Its SHA-256 tokenizer fingerprint is `5801826ac9092bdc984210af805d2ff20f0398b80adaf5963ef13b04771a8584`. | Token IDs | Count | Purpose | | --- | ---: | --- | | 0–3 | 4 | ``, ``, ``, `` | | 4–11 | 8 | Atomic ``, ``, ``, and `` open/close tags | | 12–21 | 10 | Atomic digits `0` through `9` | | 22–277 | 256 | Every UTF-8 byte as `<0x00>` through `<0xFF>` | | 278–1018 | 741 | Unicode characters learned from training questions and targets | | 1019–1535 | 517 | Learned BPE merge outputs | The structured tags and digits never participate in merges. Whitespace is a hard merge boundary; in this vocabulary an ASCII space is `<0x20>` (ID 54). If a Unicode character is absent from the learned alphabet, its UTF-8 bytes are emitted instead, so valid Unicode input can round-trip without using ``: `decode(encode(text)) == NFC(text)`. Byte fallback guarantees representability, not that the vision model can recognize an unseen character correctly. Only training records were used to learn the alphabet and merges. The optional `tokenizers` library was used while building the vocabulary, but packaged loading, encoding, and decoding use the dependency-free `byte_bpe.py` reference runtime. Browser implementations must reproduce its NFC, whitespace-boundary, atomic-token, merge-order, and byte-fallback behavior exactly. The target schema is `direct_answer_no_value_v1`. For direct reads such as a phone number, store name, or item name, `` would merely duplicate the answer and is omitted. It remains for address extraction, lookup, and arithmetic when it carries distinct evidence. Training used BF16 autocast on the GPU. Saved source tensors and the FP32 ONNX graphs are FP32; this is expected and does not mean training was performed in FP32. The additional deployment graphs are static W8A8 INT8. ## Verified bilingual examples These receipts come from the synthetic validation split, not heldout data or INT8 calibration data. They were selected as readable demonstrations and are not a family-level benchmark. | English | Korean | | --- | --- | | ![English synthetic receipt](examples/receipt_en.jpg) | ![Korean synthetic receipt](examples/receipt_ko.jpg) | | Listed family / case | English question → answer | Korean question → answer | | --- | --- | --- | | `phone` | What is the store phone number? → `6533831` | 가게 전화번호는 무엇입니까? → `9113125` | | `address` | What is the street number in the store address? → `837` | 영수증의 가게 위치에서 도로명 뒤 숫자는 무엇입니까? → `875` | | `store` | What is the store name? → `Best price Mall` | 가게 이름은 무엇입니까? → `착한 도매슈퍼마켓` | | `item_row` | What is the name of item 1? → `Packed Yogurt` | 1번째 품목명은 무엇입니까? → `과일류` | | `item_math` | What is the sum of item totals? → `2200` | 품목 총합을 모두 더하면 얼마입니까? → `3600` | | `item_lookup` | What is the quantity of Packed Yogurt? → `3` | 과일류 개수는 몇 개입니까? → `4` | | `math` | Calculate item 1 unit price multiplied by quantity. → `1800` | 1번째 품목의 가격 곱하기 개수는 얼마입니까? → `400` | | extra `phone` position | What is the second number of the store's phone number? → `5` | 가게 전화번호의 뒤에서 3번째 숫자는 무엇입니까? → `1` | All 16 questions passed in every tested combination: | Routing | FP32 | Static W8A8 INT8 | Listed-family agreement | | --- | ---: | ---: | ---: | | Explicit family override | 16/16 | 16/16 | 16/16 | | Learned AUTO router | 16/16 | 16/16 | 14/16 | AUTO routes the two listed `math` questions to trained `item_math`; their final answers and decoded outputs remain exact. The first six listed families have training records. `math` and `other` have zero family-specific training records, so the explicit `math` rows are execution smoke tests, not evidence of general math-family accuracy. `other` is not included. ### Observed decoded output by family The following outputs were observed identically for FP32 and INT8, with both explicit and AUTO routing, on the English example. These are complete decoder outputs after removing ``/`` for display. | Family | Decoded output | | --- | --- | | `phone` | `phoneidentity6533831` | | `address` | `addrSeoul Hwasun Jodonan St. 837street_no837` | | `store` | `storeidentityBest price Mall` | | `item_row` | `itemitem_1_namePacked Yogurt` | | `item_math` | `items1800,400sum2200` | | `item_lookup` | `itemPacked Yogurtlookup_quantity3` | | `math` | `item600*3product1800` | The Korean outputs use the same schema: | Family | Decoded output | | --- | --- | | `phone` | `phoneidentity9113125` | | `address` | `addr의정부시 도봉구 엄치길 875street_no875` | | `store` | `storeidentity착한 도매슈퍼마켓` | | `item_row` | `itemitem_1_name과일류` | | `item_math` | `items400,3200sum3600` | | `item_lookup` | `item과일류lookup_quantity4` | | `math` | `item100*4product400` | ### Actual emitted BPE tokens These are the actual FP32 decoder tokens for the seven English family rows, not strings decoded and then re-encoded. The selected examples emitted the same token sequences in INT8. | Family | Emitted tokens, including generation boundaries | | --- | --- | | `phone` | ` · · phone · · · identity · · · 6 · 5 · 3 · 3 · 8 · 3 · 1 · · ` | | `address` | ` · · addr · · · Se · ou · l · <0x20> · H · w · as · un · <0x20> · J · od · on · an · <0x20> · St. · <0x20> · 8 · 3 · 7 · · · street_no · · · 8 · 3 · 7 · · ` | | `store` | ` · · store · · · identity · · · B · est · <0x20> · price · <0x20> · M · all · · ` | | `item_row` | ` · · item · · · item_ · 1 · _name · · · P · ack · ed · <0x20> · Y · o · g · u · r · t · · ` | | `item_math` | ` · · items · · · 1 · 8 · 0 · 0 · , · 4 · 0 · 0 · · · sum · · · 2 · 2 · 0 · 0 · · ` | | `item_lookup` | ` · · item · · · P · ack · ed · <0x20> · Y · o · g · u · r · t · · · lookup_quantity · · · 3 · · ` | | `math` | ` · · item · · · 6 · 0 · 0 · * · 3 · · · product · · · 1 · 8 · 0 · 0 · · ` | For the English phone-position question, the observed decoded output was: ```text phone6533831front_25 ``` Its actual token IDs and tokens were: ```text IDs: [1, 4, 1038, 5, 6, 18, 17, 15, 15, 20, 15, 13, 7, 8, 1063, 14, 9, 10, 17, 11, 2] Tokens: [, , phone, , , 6, 5, 3, 3, 8, 3, 1, , , front_, 2, , , 5, , ] ``` The Korean back-position question generated `phone9113125back_31`. Run the packaged test from the repository root. It reports decoded text, actual decoder IDs, token strings, selected families, and router logits: ```bash python examples/test_all_families.py \ --model-dir . \ --precision both \ --routing both # Learned router only (family_ids=-1). python examples/test_all_families.py --model-dir . --routing auto ``` The machine-readable questions and recorded answers are in [`examples/family_examples.json`](examples/family_examples.json). The INT8 variant uses static U8S8 W8A8 QDQ quantization for convolution, attention, FFN, adapter, router, and output-head `Conv`, `MatMul`, and `Gemm` operators. Weights are symmetric signed INT8 per output channel, activations are asymmetric UINT8 per tensor, and integer accumulation is INT32. Calibration uses only synthetic training records; heldout records are not calibration inputs. ONNX Runtime can fuse supported QDQ regions into integer kernels, while normalization, softmax, positional, and shape operations remain floating point. The two FP32 graphs total 84.3 MiB; the matching W8A8 graphs total 27.4 MiB, a 67.5% reduction. These are executable quantized ONNX graphs, not storage-only weights that are dequantized back to FP32 before inference. ## Validation All four model graphs pass ONNX Checker and CPU ONNX Runtime execution. Validation covered AUTO routing, all eight explicit adapters, dynamic sequence shapes, and greedy generation. The selected adapter ID matched exactly. On the 2,000-question AUTO phone/address heldout evaluation, the model achieved answer exact 97.05%, recomputed answer exact 97.00%, and target exact 45.30%. The source PyTorch reference and the final ONNX FP32 rerun agree exactly on all nine reported accuracy metrics across the overall, address, and phone subsets. ### Character error rate CER is the standard OCR edit-distance metric: `CER = (substitutions + deletions + insertions) / ground-truth characters`. For readability, the table reports character accuracy, `1 - CER`, alongside CER. These are NFC-normalized PyTorch FP32 AUTO results for the same 2,000 heldout examples. Lower CER and higher character accuracy are better. | Scope | Character accuracy | CER | | --- | ---: | ---: | | Overall target | 95.94% | 4.06% | | Overall `` | 78.26% | 21.74% | | Overall `` | 98.17% | 1.83% | | Address target | 93.07% | 6.93% | | Address `` | 72.84% | 27.16% | | Address `` | 97.83% | 2.17% | | Phone target | 99.82% | 0.18% | | Phone `` | 98.19% | 1.81% | | Phone `` | 99.28% | 0.72% | Full-target character metrics include repeated structured markup such as ``, ``, and ``. Those common tag characters are easy to generate and therefore make full-target character accuracy look better than the model's actual OCR/copy quality. Use `` CER to assess transcription of the visual evidence and `` exact/CER to assess final QA output. The final split ONNX files were also evaluated directly with ONNX Runtime 1.23.2 using CUDAExecutionProvider with CPU fallback enabled, with learned AUTO routing: | ONNX variant | Answer exact | Recomputed exact | Target exact | | --- | ---: | ---: | ---: | | FP32 | 97.05% | 97.00% | 45.30% | | W8A8 U8S8 QDQ | 96.80% | 96.90% | 45.05% | The two variants had 80.60% full prediction agreement, 99.25% answer agreement, and 100.00% selected-family agreement. There were 388 changed full predictions out of 2000; variant-specific accuracy is reported independently above. Accumulated model execution time for all records was 25.75 seconds for FP32 and 32.62 seconds for W8A8. W8A8 was 26.7% slower in this server-side provider configuration. This CUDA timing is not a proxy for browser WASM latency; validate performance in the target browser. ## Model design and architecture This configuration has **21,834,104 train-time parameters** (about 21.8M, rounded to 22M in the model name), and all of them were trainable. The token embedding and output projection share one weight matrix, so it is counted once. The LoRA branches account for 256,000 parameters during training; they are folded into dense FFN weights for ONNX inference and add no runtime branch. ```text 672x320 grayscale receipt -> stride-32 CNN -> 10x21 visual tokens -----------+ question -> Byte-Fallback BPE -> question embeddings -----------------------+--> concat question embeddings -> mean-pool router -> selected family -----------------+ | v 6-layer multimodal encoder -> selected memory adapter | cached memory | BOS / growing prefix -> 4-layer causal decoder + cross-attention -----------------+ -> selected decoder adapter -> tied vocabulary head -> next token ``` ### Vision and multimodal encoder The input is float32 grayscale `[B, 1, 320, 672]`. Five bias-free 3x3 convolutions downsample through `1 -> 48 -> 96 -> 192 -> 320 -> 320` channels. Conv blocks use GroupNorm and SiLU; four two-convolution residual blocks are interleaved with the downsampling stages. The total stride is 32, producing a `10x21` feature map that is flattened into 210 visual tokens of width 320. Visual tokens receive learned image positions and an image-type embedding. Question tokens use the shared `1536x320` token embedding, learned question positions, and a question-type embedding. The model concatenates `[visual tokens, question tokens]` and applies a 6-layer pre-LayerNorm Transformer encoder with 8 attention heads (head width 40), FFN width 1280, GELU, and dropout 0.1. The encoded memory shape is `[B, 210 + Q, 320]`. ### Router and family adapters The router sees question embeddings only, before image/question fusion. It mean-pools non-padding question tokens, then applies `Linear(320 -> 160)`, GELU, and `Linear(160 -> 8)`. AUTO routing takes the hard argmax; `family_ids=0..7` explicitly overrides it. The ordered slots are `phone`, `address`, `store`, `item_row`, `item_math`, `item_lookup`, `math`, and `other`. One selected residual bottleneck adapter is applied after the encoder and one after the decoder. Each is `320 -> 64 -> 320` with GELU, so this is eight routed family slots backed by 16 adapter MLPs, not a soft mixture of all adapters. Training used token cross-entropy plus `0.1 *` router-family cross-entropy. Family-specific training records cover the first six slots. `math` and `other` are untrained extension slots whose encoder and decoder adapter up-projections are exactly zero. Explicitly selecting either slot therefore applies an identity adapter; AUTO can instead route such prompts to a trained family. ### Decoder, LoRA, and deployment split The 4-layer pre-LayerNorm Transformer decoder uses causal self-attention and cross-attention to multimodal memory, with the same 320-wide, 8-head, 1280-wide-FFN design. A final LayerNorm and output bias produce token logits. The output projection is weight-tied to the input token embedding. Training added rank-8 LoRA (`alpha=16`, scale 2, dropout 0.05) to `linear1` and `linear2` in every encoder and decoder FFN: 20 Linear modules in total. This was LoRA-augmented **full-model training**, not a frozen-base PEFT run. Export folded all 20 deltas into their dense weights; the maximum pre/post-merge absolute difference was `5.245208740234375e-06`. The ONNX graphs contain no separate LoRA operators. The split encoder runs the CNN, multimodal encoder, router, and memory adapter once. It returns memory, its padding mask, router logits, and the selected family ID. The decoder receives those cached encoder outputs plus the growing prefix. There is **no decoder KV/past cache**: each decoding step recomputes the full prefix, while the image and question encoder are not recomputed. Generation uses greedy argmax from BOS until EOS or 192 tokens. The source checkpoint is epoch-100 `last.pt`. Heldout was excluded from training and intermediate checkpointing, then used post hoc to compare e79 `best.pt` with e100 `last.pt`; e100 `last.pt` was selected for higher heldout answer exact. ## Target serialization: `direct_answer_no_value_v1` This is the current training-target serialization identifier; it is not a model-architecture or tokenizer version. The complete serialized target is teacher-forced into the decoder and contributes to token cross-entropy loss: ```text ...[optional ...]...... ``` ``, ``, and `` are always retained. `` is omitted only when it would repeat the final answer: `identity`, `copy`, `count`, and `item_N_name`, `item_N_unit_price`, `item_N_quantity`, or `item_N_total`. It remains when it carries distinct evidence for extraction, lookup, or arithmetic. For example: ```text storeidentityABC마트 phone6533831front_25 ``` The separate `answer` property in an annotation is used for evaluation; it is not a second decoder-loss target. Using the `v1` identifier does not alter the learned output behavior or model weights, so retraining is unnecessary. ## Limitations ### Training and task scope This checkpoint learned a finite bilingual `structured_qa` task grammar for receipts. It is not an open-world document-VQA system, a general-purpose OCR engine, or an arbitrary table-query/calculation model. Training used synthetic receipts together with annotated real receipts, but this release does not ship the exact training-record count or per-family training distribution. | Trained family | Supervised coverage | Boundary not claimed | | --- | --- | --- | | `phone` | Full digits-only number; one digit from the front or back at positions **1–4**; prefix/suffix lengths **2–4**; digit count and digit sum | Position 5 or later, arbitrary substring extraction, and unseen wording are out of the generated task range | | `address` | Full address; final street/building number; first and second whitespace token; road name; whitespace-removed character positions **1–3** from either end | Geocoding and free-form address semantics are not trained; the component parser assumes a simple road name and a final digit group | | `store` | Full store name; first/last word; whitespace-removed character count; character positions **1–4** from either end | Other character positions, arbitrary spans, and merchant/entity normalization are not trained | | `item_row` | Item count; name, unit price, quantity, and row total for rows **1–4**; first character of those item names | Direct row-index questions for row 5 or later and fields outside this schema are not trained | | `item_lookup` | Name or sampled partial-name lookup of price, quantity, or total from the first **6** eligible non-address-like rows; absent-item negatives with the fixed answer `없음`; reverse lookup when a numeric value identifies one unique row | Ambiguous duplicate values, arbitrary fuzzy matching, and name-based lookup beyond row 6 are not covered | | `item_math` | Per-row `unit_price × quantity` for rows **1–4**; sums and min/max values or item names over valid total, unit-price, and quantity rows; sum and absolute difference of the first two row totals | General arithmetic, arbitrary expressions, and unconstrained table reasoning are not trained | The prompt generator selects from finite English and Korean template banks; question language may differ from the receipt text language. Byte fallback makes other Unicode text encodable, but it does not add multilingual OCR or QA capability. Close human paraphrases are therefore not guaranteed. For example, prefer an explicit learned form such as `전화번호 앞에서 1번째 숫자는?` or `What is the first digit of the store phone number?`. A shorter form such as `전화번호 첫번째 숫자`, wording with `index`, or a request for the fifth digit is outside the guaranteed template/range and may produce a plausible but wrong operation. Only `phone`, `address`, `store`, `item_row`, `item_math`, and `item_lookup` have family-specific training records. `math` and `other` are untrained extension slots with exact identity adapters; explicitly selecting either slot does not provide a separately trained math or catch-all capability. Direct extraction of dates, times, tax, payment method, printed subtotal/grand total, change, and other fields outside the table above has no task-specific accuracy claim. Images are converted to grayscale and resized to the fixed 672x320 input. Training augmentation covered modest contrast/brightness changes, blur, and small rotations, so tiny text, heavy blur, large rotation, unusual layouts, or very dense receipts may degrade recognition. CER in this card is informational for transcription within the evaluated tasks; it is not evidence of unrestricted full-document OCR quality. ### Output and evaluation scope - `well_formed: true` means only that a complete `...` block was generated. It does not verify that the selected family, ``, copied ``, or answer is semantically correct. - The 2,000-record heldout evaluation contains only supported phone/address prompts (1,032 address and 968 phone). It does not validate store/item families, every trained operation, or novel paraphrases. The bilingual family examples are selected smoke tests, not a family-level benchmark. - Heldout was excluded from training and intermediate checkpointing. After training it was used to compare e79 `best.pt` and e100 `last.pt` and to select e100 `last.pt`, so it is a development test rather than a final blind benchmark. - W8A8 output can differ from FP32 output because both weights and activations are quantized. Validate accuracy and latency in the target browser. - Integer-kernel fusion depends on the ONNX Runtime execution provider and target hardware.