ivere27/tiny-receipt-vqa-structured-qa-21m

TinyReceiptVQA-StructuredQA-22M-BPE1536-DirectNoValue-LoRA-Router-e100

TinyReceiptVQA is a compact receipt-domain visual question-answering model. It takes a full receipt image and a question, then generates a structured rationale and final <answer> value. A learned router selects one of eight family slots; each slot has a post-encoder memory adapter and a post-decoder adapter for phone, address, store, item-row, item-math, item-lookup, math, or other questions.

Files

encoder_model.onnx       # CNN + multimodal encoder + router + memory adapter
decoder_model.onnx       # autoregressive decoder + selected decoder adapter
encoder_model_int8.onnx  # static W8A8 U8S8 QDQ encoder
decoder_model_int8.onnx  # static W8A8 U8S8 QDQ decoder
manifest.json            # graph contract, hashes, and runtime validation
config.json
vocab.json
byte_bpe.py              # dependency-free tokenizer runtime
inference.py             # ONNX Runtime greedy decoding
examples/
  receipt_en.jpg         # verified English synthetic receipt
  receipt_ko.jpg         # verified Korean synthetic receipt
  family_examples.json   # 16 explicit/AUTO checks and expected answers
  test_all_families.py   # emits decoded text, token IDs, and BPE tokens
eval/
  heldout_summary.json   # heldout evaluation summary
  int8_w8a8_summary.json
  onnx_parity_summary.json
  comparison.json

The encoder is run once. The host runtime then calls the decoder with the growing output prefix until EOS; this avoids recomputing the receipt image and question encoder for every generated token.

Usage

pip install -r requirements.txt

python inference.py \
  --model-dir . \
  --image examples/receipt_en.jpg \
  --question "What is the street number in the store address?"
# Expected JSON fields: "answer": "837", "well_formed": true

# Use the Korean example with the static W8A8 U8S8 QDQ ONNX files.
python inference.py \
  --model-dir . \
  --precision int8 \
  --image examples/receipt_ko.jpg \
  --question "영수증의 가게 위치에서 도로명 뒤 숫자는 무엇입니까?"
# Expected JSON fields: "answer": "875", "well_formed": true

--family auto is the default and uses the learned router. An explicit family such as phone or address can be supplied for controlled debugging.

The input image contract is float32 [batch, 1, 320, 672]: convert to grayscale, resize to 672x320 (width x height), scale to [0,1], then normalize with (x - 0.5) / 0.5.

Runtime interface

The opset is 18. Batch, question length, and decoder-prefix length are dynamic; image height and width are fixed.

• Encoder inputs: image, question_ids, and family_ids (-1 means AUTO).
• Encoder outputs: cached memory, padding mask, router logits, selected family.
• Decoder inputs: generated prefix, cached memory/mask, selected family.
• Decoder output: token logits.

Use the included inference.py entry point to handle preprocessing, AUTO-routing, and greedy generation.

The JSON result sets well_formed to true only when generation contains a complete <answer>...</answer> block. If that block is missing, answer is an empty string and well_formed is false; inspect generated for diagnostics.

Byte-Fallback BPE and vocabulary

This release uses only the packaged Byte-Fallback BPE1536 tokenizer contract. vocab.json declares type=byte_fallback_bpe, version 1, NFC normalization, and exactly 1,536 tokens. Character-vocabulary artifacts are rejected instead of falling back silently. Its SHA-256 tokenizer fingerprint is 5801826ac9092bdc984210af805d2ff20f0398b80adaf5963ef13b04771a8584.

Token IDs	Count	Purpose
0–3	4	<pad>, <bos>, <eos>, <unk>
4–11	8	Atomic <field>, <value>, <op>, and <answer> open/close tags
12–21	10	Atomic digits 0 through 9
22–277	256	Every UTF-8 byte as <0x00> through <0xFF>
278–1018	741	Unicode characters learned from training questions and targets
1019–1535	517	Learned BPE merge outputs

The structured tags and digits never participate in merges. Whitespace is a hard merge boundary; in this vocabulary an ASCII space is <0x20> (ID 54). If a Unicode character is absent from the learned alphabet, its UTF-8 bytes are emitted instead, so valid Unicode input can round-trip without using <unk>: decode(encode(text)) == NFC(text). Byte fallback guarantees representability, not that the vision model can recognize an unseen character correctly.

Only training records were used to learn the alphabet and merges. The optional tokenizers library was used while building the vocabulary, but packaged loading, encoding, and decoding use the dependency-free byte_bpe.py reference runtime. Browser implementations must reproduce its NFC, whitespace-boundary, atomic-token, merge-order, and byte-fallback behavior exactly.

The target schema is direct_answer_no_value_v1. For direct reads such as a phone number, store name, or item name, <value> would merely duplicate the answer and is omitted. It remains for address extraction, lookup, and arithmetic when it carries distinct evidence.

Training used BF16 autocast on the GPU. Saved source tensors and the FP32 ONNX graphs are FP32; this is expected and does not mean training was performed in FP32. The additional deployment graphs are static W8A8 INT8.

Verified bilingual examples

These receipts come from the synthetic validation split, not heldout data or INT8 calibration data. They were selected as readable demonstrations and are not a family-level benchmark.

English	Korean

Listed family / case	English question → answer	Korean question → answer
phone	What is the store phone number? → 6533831	가게 전화번호는 무엇입니까? → 9113125
address	What is the street number in the store address? → 837	영수증의 가게 위치에서 도로명 뒤 숫자는 무엇입니까? → 875
store	What is the store name? → Best price Mall	가게 이름은 무엇입니까? → 착한 도매슈퍼마켓
item_row	What is the name of item 1? → Packed Yogurt	1번째 품목명은 무엇입니까? → 과일류
item_math	What is the sum of item totals? → 2200	품목 총합을 모두 더하면 얼마입니까? → 3600
item_lookup	What is the quantity of Packed Yogurt? → 3	과일류 개수는 몇 개입니까? → 4
math	Calculate item 1 unit price multiplied by quantity. → 1800	1번째 품목의 가격 곱하기 개수는 얼마입니까? → 400
extra phone position	What is the second number of the store's phone number? → 5	가게 전화번호의 뒤에서 3번째 숫자는 무엇입니까? → 1

All 16 questions passed in every tested combination:

Routing	FP32	Static W8A8 INT8	Listed-family agreement
Explicit family override	16/16	16/16	16/16
Learned AUTO router	16/16	16/16	14/16

AUTO routes the two listed math questions to trained item_math; their final answers and decoded outputs remain exact. The first six listed families have training records. math and other have zero family-specific training records, so the explicit math rows are execution smoke tests, not evidence of general math-family accuracy. other is not included.

Observed decoded output by family

The following outputs were observed identically for FP32 and INT8, with both explicit and AUTO routing, on the English example. These are complete decoder outputs after removing <bos>/<eos> for display.

Family	Decoded output
phone	<field>phone</field><op>identity</op><answer>6533831</answer>
address	<field>addr</field><value>Seoul Hwasun Jodonan St. 837</value><op>street_no</op><answer>837</answer>
store	<field>store</field><op>identity</op><answer>Best price Mall</answer>
item_row	<field>item</field><op>item_1_name</op><answer>Packed Yogurt</answer>
item_math	<field>items</field><value>1800,400</value><op>sum</op><answer>2200</answer>
item_lookup	<field>item</field><value>Packed Yogurt</value><op>lookup_quantity</op><answer>3</answer>
math	<field>item</field><value>600*3</value><op>product</op><answer>1800</answer>

The Korean outputs use the same schema:

Family	Decoded output
phone	<field>phone</field><op>identity</op><answer>9113125</answer>
address	<field>addr</field><value>의정부시 도봉구 엄치길 875</value><op>street_no</op><answer>875</answer>
store	<field>store</field><op>identity</op><answer>착한 도매슈퍼마켓</answer>
item_row	<field>item</field><op>item_1_name</op><answer>과일류</answer>
item_math	<field>items</field><value>400,3200</value><op>sum</op><answer>3600</answer>
item_lookup	<field>item</field><value>과일류</value><op>lookup_quantity</op><answer>4</answer>
math	<field>item</field><value>100*4</value><op>product</op><answer>400</answer>

Actual emitted BPE tokens

These are the actual FP32 decoder tokens for the seven English family rows, not strings decoded and then re-encoded. The selected examples emitted the same token sequences in INT8.

Family	Emitted tokens, including generation boundaries
phone	<bos> · <field> · phone · </field> · <op> · identity · </op> · <answer> · 6 · 5 · 3 · 3 · 8 · 3 · 1 · </answer> · <eos>
address	<bos> · <field> · addr · </field> · <value> · Se · ou · l · <0x20> · H · w · as · un · <0x20> · J · od · on · an · <0x20> · St. · <0x20> · 8 · 3 · 7 · </value> · <op> · street_no · </op> · <answer> · 8 · 3 · 7 · </answer> · <eos>
store	<bos> · <field> · store · </field> · <op> · identity · </op> · <answer> · B · est · <0x20> · price · <0x20> · M · all · </answer> · <eos>
item_row	<bos> · <field> · item · </field> · <op> · item_ · 1 · _name · </op> · <answer> · P · ack · ed · <0x20> · Y · o · g · u · r · t · </answer> · <eos>
item_math	<bos> · <field> · items · </field> · <value> · 1 · 8 · 0 · 0 · , · 4 · 0 · 0 · </value> · <op> · sum · </op> · <answer> · 2 · 2 · 0 · 0 · </answer> · <eos>
item_lookup	<bos> · <field> · item · </field> · <value> · P · ack · ed · <0x20> · Y · o · g · u · r · t · </value> · <op> · lookup_quantity · </op> · <answer> · 3 · </answer> · <eos>
math	<bos> · <field> · item · </field> · <value> · 6 · 0 · 0 · * · 3 · </value> · <op> · product · </op> · <answer> · 1 · 8 · 0 · 0 · </answer> · <eos>

For the English phone-position question, the observed decoded output was:

<field>phone</field><value>6533831</value><op>front_2</op><answer>5</answer>

Its actual token IDs and tokens were:

IDs:    [1, 4, 1038, 5, 6, 18, 17, 15, 15, 20, 15, 13, 7,
         8, 1063, 14, 9, 10, 17, 11, 2]
Tokens: [<bos>, <field>, phone, </field>, <value>, 6, 5, 3, 3, 8, 3, 1,
         </value>, <op>, front_, 2, </op>, <answer>, 5, </answer>, <eos>]

The Korean back-position question generated <field>phone</field><value>9113125</value><op>back_3</op><answer>1</answer>.

Run the packaged test from the repository root. It reports decoded text, actual decoder IDs, token strings, selected families, and router logits:

python examples/test_all_families.py \
  --model-dir . \
  --precision both \
  --routing both

# Learned router only (family_ids=-1).
python examples/test_all_families.py --model-dir . --routing auto

The machine-readable questions and recorded answers are in examples/family_examples.json.

The INT8 variant uses static U8S8 W8A8 QDQ quantization for convolution, attention, FFN, adapter, router, and output-head Conv, MatMul, and Gemm operators. Weights are symmetric signed INT8 per output channel, activations are asymmetric UINT8 per tensor, and integer accumulation is INT32. Calibration uses only synthetic training records; heldout records are not calibration inputs. ONNX Runtime can fuse supported QDQ regions into integer kernels, while normalization, softmax, positional, and shape operations remain floating point. The two FP32 graphs total 84.3 MiB; the matching W8A8 graphs total 27.4 MiB, a 67.5% reduction. These are executable quantized ONNX graphs, not storage-only weights that are dequantized back to FP32 before inference.

Validation

All four model graphs pass ONNX Checker and CPU ONNX Runtime execution. Validation covered AUTO routing, all eight explicit adapters, dynamic sequence shapes, and greedy generation. The selected adapter ID matched exactly.

On the 2,000-question AUTO phone/address heldout evaluation, the model achieved answer exact 97.05%, recomputed answer exact 97.00%, and target exact 45.30%.

The source PyTorch reference and the final ONNX FP32 rerun agree exactly on all nine reported accuracy metrics across the overall, address, and phone subsets.

Character error rate

CER is the standard OCR edit-distance metric:

CER = (substitutions + deletions + insertions) / ground-truth characters.

For readability, the table reports character accuracy, 1 - CER, alongside CER. These are NFC-normalized PyTorch FP32 AUTO results for the same 2,000 heldout examples. Lower CER and higher character accuracy are better.

Scope	Character accuracy	CER
Overall target	95.94%	4.06%
Overall <value>	78.26%	21.74%
Overall <answer>	98.17%	1.83%
Address target	93.07%	6.93%
Address <value>	72.84%	27.16%
Address <answer>	97.83%	2.17%
Phone target	99.82%	0.18%
Phone <value>	98.19%	1.81%
Phone <answer>	99.28%	0.72%

Full-target character metrics include repeated structured markup such as <field>, <op>, and <answer>. Those common tag characters are easy to generate and therefore make full-target character accuracy look better than the model's actual OCR/copy quality. Use <value> CER to assess transcription of the visual evidence and <answer> exact/CER to assess final QA output.

The final split ONNX files were also evaluated directly with ONNX Runtime 1.23.2 using CUDAExecutionProvider with CPU fallback enabled, with learned AUTO routing:

ONNX variant	Answer exact	Recomputed exact	Target exact
FP32	97.05%	97.00%	45.30%
W8A8 U8S8 QDQ	96.80%	96.90%	45.05%

The two variants had 80.60% full prediction agreement, 99.25% answer agreement, and 100.00% selected-family agreement. There were 388 changed full predictions out of 2000; variant-specific accuracy is reported independently above.

Accumulated model execution time for all records was 25.75 seconds for FP32 and 32.62 seconds for W8A8. W8A8 was 26.7% slower in this server-side provider configuration. This CUDA timing is not a proxy for browser WASM latency; validate performance in the target browser.

Model design and architecture

This configuration has 21,834,104 train-time parameters (about 21.8M, rounded to 22M in the model name), and all of them were trainable. The token embedding and output projection share one weight matrix, so it is counted once. The LoRA branches account for 256,000 parameters during training; they are folded into dense FFN weights for ONNX inference and add no runtime branch.

672x320 grayscale receipt -> stride-32 CNN -> 10x21 visual tokens -----------+
question -> Byte-Fallback BPE -> question embeddings -----------------------+--> concat
question embeddings -> mean-pool router -> selected family -----------------+      |
                                                                                   v
                              6-layer multimodal encoder -> selected memory adapter
                                                                                   |
                                                                               cached memory
                                                                                   |
BOS / growing prefix -> 4-layer causal decoder + cross-attention -----------------+
                      -> selected decoder adapter -> tied vocabulary head -> next token

Vision and multimodal encoder

The input is float32 grayscale [B, 1, 320, 672]. Five bias-free 3x3 convolutions downsample through 1 -> 48 -> 96 -> 192 -> 320 -> 320 channels. Conv blocks use GroupNorm and SiLU; four two-convolution residual blocks are interleaved with the downsampling stages. The total stride is 32, producing a 10x21 feature map that is flattened into 210 visual tokens of width 320.

Visual tokens receive learned image positions and an image-type embedding. Question tokens use the shared 1536x320 token embedding, learned question positions, and a question-type embedding. The model concatenates [visual tokens, question tokens] and applies a 6-layer pre-LayerNorm Transformer encoder with 8 attention heads (head width 40), FFN width 1280, GELU, and dropout 0.1. The encoded memory shape is [B, 210 + Q, 320].

Router and family adapters

The router sees question embeddings only, before image/question fusion. It mean-pools non-padding question tokens, then applies Linear(320 -> 160), GELU, and Linear(160 -> 8). AUTO routing takes the hard argmax; family_ids=0..7 explicitly overrides it. The ordered slots are phone, address, store, item_row, item_math, item_lookup, math, and other.

One selected residual bottleneck adapter is applied after the encoder and one after the decoder. Each is 320 -> 64 -> 320 with GELU, so this is eight routed family slots backed by 16 adapter MLPs, not a soft mixture of all adapters. Training used token cross-entropy plus 0.1 * router-family cross-entropy. Family-specific training records cover the first six slots. math and other are untrained extension slots whose encoder and decoder adapter up-projections are exactly zero. Explicitly selecting either slot therefore applies an identity adapter; AUTO can instead route such prompts to a trained family.

Decoder, LoRA, and deployment split

The 4-layer pre-LayerNorm Transformer decoder uses causal self-attention and cross-attention to multimodal memory, with the same 320-wide, 8-head, 1280-wide-FFN design. A final LayerNorm and output bias produce token logits. The output projection is weight-tied to the input token embedding.

Training added rank-8 LoRA (alpha=16, scale 2, dropout 0.05) to linear1 and linear2 in every encoder and decoder FFN: 20 Linear modules in total. This was LoRA-augmented full-model training, not a frozen-base PEFT run. Export folded all 20 deltas into their dense weights; the maximum pre/post-merge absolute difference was 5.245208740234375e-06. The ONNX graphs contain no separate LoRA operators.

The split encoder runs the CNN, multimodal encoder, router, and memory adapter once. It returns memory, its padding mask, router logits, and the selected family ID. The decoder receives those cached encoder outputs plus the growing prefix. There is no decoder KV/past cache: each decoding step recomputes the full prefix, while the image and question encoder are not recomputed. Generation uses greedy argmax from BOS until EOS or 192 tokens.

The source checkpoint is epoch-100 last.pt. Heldout was excluded from training and intermediate checkpointing, then used post hoc to compare e79 best.pt with e100 last.pt; e100 last.pt was selected for higher heldout answer exact.

Target serialization: direct_answer_no_value_v1

This is the current training-target serialization identifier; it is not a model-architecture or tokenizer version. The complete serialized target is teacher-forced into the decoder and contributes to token cross-entropy loss:

<field>...</field>[optional <value>...</value>]<op>...</op><answer>...</answer>

<field>, <op>, and <answer> are always retained. <value> is omitted only when it would repeat the final answer: identity, copy, count, and item_N_name, item_N_unit_price, item_N_quantity, or item_N_total. It remains when it carries distinct evidence for extraction, lookup, or arithmetic. For example:

<field>store</field><op>identity</op><answer>ABC마트</answer>
<field>phone</field><value>6533831</value><op>front_2</op><answer>5</answer>

The separate answer property in an annotation is used for evaluation; it is not a second decoder-loss target. Using the v1 identifier does not alter the learned output behavior or model weights, so retraining is unnecessary.

Limitations

Training and task scope

This checkpoint learned a finite bilingual structured_qa task grammar for receipts. It is not an open-world document-VQA system, a general-purpose OCR engine, or an arbitrary table-query/calculation model. Training used synthetic receipts together with annotated real receipts, but this release does not ship the exact training-record count or per-family training distribution.

Trained family	Supervised coverage	Boundary not claimed
phone	Full digits-only number; one digit from the front or back at positions 1–4; prefix/suffix lengths 2–4; digit count and digit sum	Position 5 or later, arbitrary substring extraction, and unseen wording are out of the generated task range
address	Full address; final street/building number; first and second whitespace token; road name; whitespace-removed character positions 1–3 from either end	Geocoding and free-form address semantics are not trained; the component parser assumes a simple road name and a final digit group
store	Full store name; first/last word; whitespace-removed character count; character positions 1–4 from either end	Other character positions, arbitrary spans, and merchant/entity normalization are not trained
item_row	Item count; name, unit price, quantity, and row total for rows 1–4; first character of those item names	Direct row-index questions for row 5 or later and fields outside this schema are not trained
item_lookup	Name or sampled partial-name lookup of price, quantity, or total from the first 6 eligible non-address-like rows; absent-item negatives with the fixed answer 없음; reverse lookup when a numeric value identifies one unique row	Ambiguous duplicate values, arbitrary fuzzy matching, and name-based lookup beyond row 6 are not covered
item_math	Per-row unit_price × quantity for rows 1–4; sums and min/max values or item names over valid total, unit-price, and quantity rows; sum and absolute difference of the first two row totals	General arithmetic, arbitrary expressions, and unconstrained table reasoning are not trained

The prompt generator selects from finite English and Korean template banks; question language may differ from the receipt text language. Byte fallback makes other Unicode text encodable, but it does not add multilingual OCR or QA capability. Close human paraphrases are therefore not guaranteed. For example, prefer an explicit learned form such as 전화번호 앞에서 1번째 숫자는? or What is the first digit of the store phone number?. A shorter form such as 전화번호 첫번째 숫자, wording with index, or a request for the fifth digit is outside the guaranteed template/range and may produce a plausible but wrong operation.

Only phone, address, store, item_row, item_math, and item_lookup have family-specific training records. math and other are untrained extension slots with exact identity adapters; explicitly selecting either slot does not provide a separately trained math or catch-all capability. Direct extraction of dates, times, tax, payment method, printed subtotal/grand total, change, and other fields outside the table above has no task-specific accuracy claim.

Images are converted to grayscale and resized to the fixed 672x320 input. Training augmentation covered modest contrast/brightness changes, blur, and small rotations, so tiny text, heavy blur, large rotation, unusual layouts, or very dense receipts may degrade recognition. CER in this card is informational for transcription within the evaluated tasks; it is not evidence of unrestricted full-document OCR quality.

Output and evaluation scope

• well_formed: true means only that a complete <answer>...</answer> block was generated. It does not verify that the selected family, <op>, copied <value>, or answer is semantically correct.
• The 2,000-record heldout evaluation contains only supported phone/address prompts (1,032 address and 968 phone). It does not validate store/item families, every trained operation, or novel paraphrases. The bilingual family examples are selected smoke tests, not a family-level benchmark.
• Heldout was excluded from training and intermediate checkpointing. After training it was used to compare e79 best.pt and e100 last.pt and to select e100 last.pt, so it is a development test rather than a final blind benchmark.
• W8A8 output can differ from FP32 output because both weights and activations are quantized. Validate accuracy and latency in the target browser.
• Integer-kernel fusion depends on the ONNX Runtime execution provider and target hardware.