trishpurkait/billstructai-qwen-lora

BillStructAI — Qwen2.5-1.5B-Instruct LoRA (Invoice OCR → JSON)

A QLoRA adapter for Qwen/Qwen2.5-1.5B-Instruct that converts noisy OCR text from invoices and receipts into structured JSON.

• Base model: Qwen2.5-1.5B-Instruct
• Method: QLoRA (4-bit NF4), rank 16, targeting all attention + MLP projection layers
• Task: structured invoice/receipt field extraction from raw OCR text
• Training data: rvl-cdip-invoice-extracted, filtered to high/medium-confidence examples
• Checkpoint: step 900 of 940 (selected by lowest validation loss, not the final step)

Full training/eval code, results, and a live demo: github.com/your-username/billstructai

Results

Evaluated on a 470-example held-out test set with an independent LLM judge (Qwen3-Next-80B), scoring 0–10 on four dimensions:

Metric	Base	Fine-tuned
Factual correctness	3.88	7.11
Schema following	3.34	8.66
Field completeness	3.71	6.91
Hallucination control	2.31	7.52
Overall	2.98	7.40

The base model frequently invents fields and values not present in the source OCR text (e.g. a fabricated email address, a fabricated timestamp); the fine-tuned model follows the target schema and defaults to null instead of guessing. See the repo's examples.md for annotated before/after cases.

Usage

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer, BitsAndBytesConfig
from peft import PeftModel

BASE_MODEL = "Qwen/Qwen2.5-1.5B-Instruct"
ADAPTER = "your-username/billstructai-qwen-lora"

bnb_config = BitsAndBytesConfig(
    load_in_4bit=True,
    bnb_4bit_quant_type="nf4",
    bnb_4bit_compute_dtype=torch.float16,
    bnb_4bit_use_double_quant=True,
)

tokenizer = AutoTokenizer.from_pretrained(BASE_MODEL)
base_model = AutoModelForCausalLM.from_pretrained(
    BASE_MODEL, quantization_config=bnb_config, device_map="auto"
)
model = PeftModel.from_pretrained(base_model, ADAPTER)
model.eval()

SYSTEM_PROMPT = """You are a precise invoice information extraction assistant.
Your task is to extract structured invoice data from noisy OCR text.
Return only valid JSON.
Do not explain.
Do not add markdown.
Use null for missing fields.
Do not hallucinate values that are not present in the OCR text."""

ocr_text = "PASTE OCR TEXT HERE"

messages = [
    {"role": "system", "content": SYSTEM_PROMPT},
    {"role": "user", "content": f"Extract invoice information from the OCR text below and return only valid JSON.\n\nOCR_TEXT:\n{ocr_text}"},
]
prompt = tokenizer.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
inputs = tokenizer(prompt, return_tensors="pt").to(model.device)

with torch.no_grad():
    output = model.generate(**inputs, max_new_tokens=768, do_sample=False, pad_token_id=tokenizer.pad_token_id or tokenizer.eos_token_id)

print(tokenizer.decode(output[0][inputs["input_ids"].shape[1]:], skip_special_tokens=True))

Output schema

{
  "vendor": {"name": null, "address": null, "tax_id": null, "phone": null, "email": null, "website": null},
  "customer": {"name": null, "address": null, "customer_id": null},
  "invoice_number": null,
  "invoice_date": null,
  "due_date": null,
  "purchase_order": null,
  "line_items": [{"description": null, "quantity": null, "unit": null, "unit_price": null, "amount": null, "tax_rate": null}],
  "subtotal": null,
  "tax_amount": null,
  "discount": null,
  "total_amount": null,
  "amount_paid": null,
  "balance_due": null,
  "currency": null,
  "payment_terms": null,
  "payment_method": null,
  "notes": null
}

Limitations

• Trained on English-language, largely US-format invoices/receipts sourced from scanned documents (RVL-CDIP); accuracy on other layouts, languages, or currencies is untested.
• Rule-based field accuracy (exact/fuzzy match against gold values) was measured on a 50-sample subset with the full-precision model; the LLM-judge scores above are the only metric verified on the full 470-sample, 4-bit-quantized configuration. See the linked repo for details.
• Not evaluated for adversarial or intentionally malformed OCR input.

Training details

LoRA rank	16
LoRA alpha	32
LoRA dropout	0.05
Target modules	q_proj, k_proj, v_proj, o_proj, gate_proj, up_proj, down_proj
Quantization	4-bit NF4, double quant
Epochs	2
Effective batch size	8 (1 × 8 grad. accumulation)
Learning rate	2e-4, cosine schedule
Selected checkpoint	step 900 / 940 (lowest eval_loss)