caiotheodoro/reconforge-recon-lora

ReconForge Recon — Financial Reconciliation Verdict Model

A LoRA fine-tune of Qwen3-1.7B for financial back-office reconciliation. It trades raw accuracy (0.805 vs DeepSeek v4-flash's 0.876) for severity-weighted recall (0.901 vs 0.872) and perfect recall on HIGH-severity exceptions — the error class that actually costs money — at zero API cost, offline, on Apple Silicon. Both the accuracy loss and the R_w gain are statistically significant (95% bootstrap CI over 10,000 resamples, paired on the same 800-task set — see Results). Trained in ~100 minutes on an M5.

Quick Start

# Block 1 — load the adapter
# Requires: mlx==0.32.0, mlx-lm==0.31.3, Python 3.11+ (pinned to this repo's
# training/eval environment — see reconforge/model/uv.lock)
from huggingface_hub import snapshot_download
from mlx_lm.lora import load

adapter_dir = snapshot_download("caiotheodoro/reconforge-recon-lora")
model, tokenizer = load(
    "mlx-community/Qwen3-1.7B-4bit",
    adapter_path=adapter_dir,
    tokenizer_config={"trust_remote_code": True},
)

# Block 2 — build the prompt
import json

SYSTEM_PROMPT = """You are ReconForge, a financial back-office reconciliation operations engine. \
You reconcile a single ledger entry against a single bank statement entry and return a structured verdict.

Your output MUST be exactly one JSON object with these keys:
- "verdict": "MATCH" | "EXCEPTION" | "ESCALATE"
- "exception_type": null or one of AMOUNT_MISMATCH, FX_CONVERSION_ERROR, \
BENEFICIARY_MISMATCH, COUNTERPARTY_MISMATCH, VALUE_DATE_MISMATCH, MISSING_MESSAGE, \
DUPLICATE, FIELD_CORRUPTION, PARTIAL_MATCH
- "severity": "LOW" | "MEDIUM" | "HIGH"
- "confidence": float in [0, 1]
- "reason": short reason, under 10 words
- "resolution": one of "auto-adjust", "escalate", "reject", "rebook", "flag-review"
"""

user_prompt = """Reconcile the following ledger entry against the bank statement.

LEDGER ENTRY:
{
  "amount": "10000.00", "ccy": "USD", "counterparty": "Acme Corp",
  "beneficiary": "Acme Corp", "value_date": "2024-01-15",
  "message_type": "MT300", "ref": "OUR-REF-000001"
}

BANK STATEMENT:
{
  "amount": "9999.50", "ccy": "USD", "counterparty": "Acme Corp",
  "beneficiary": "Acme Corp", "value_date": "2024-01-15",
  "message_type": "MT940", "ref": "CP-PAY-000001"
}

Return the verdict JSON object only."""

# Block 3 — generate
from mlx_lm.generate import generate

for attr in ("has_thinking", "enable_thinking"):
    if hasattr(tokenizer, attr):
        setattr(tokenizer, attr, False)

messages = [
    {"role": "system", "content": SYSTEM_PROMPT},
    {"role": "user", "content": user_prompt},
]
prompt = tokenizer.apply_chat_template(
    messages, add_generation_prompt=True, enable_thinking=False, return_dict=False
)
response = generate(
    model, tokenizer, prompt, max_tokens=128,   # single greedy sample --
                                                 # a minimal usage example,
                                                 # NOT the eval config
)
verdict = json.loads(response)
print(f"{verdict['verdict']} / {verdict['exception_type']} / {verdict['severity']}")

This Quick Start uses one greedy sample. It is a minimal usage example, not the eval configuration. The Results below were produced with self-consistency sampling (5 samples, temp=0.6, majority vote) — see the Run configuration table. A single greedy sample will not reproduce the reported numbers.

The model is trained in non-thinking mode (enable_thinking=False) and emits ~38 tokens per verdict on average. Leaving thinking enabled changes the output format and breaks the parse.

Results

Eval set: ReconEval v0.1.0, 800 held-out tasks, seed 777, exact-overlap contamination 0/800 (near-duplicate rate 3.6% at Jaccard ≥ 0.8 — see recon-eval).

All CIs are 95% bootstrap intervals, 10,000 resamples, seed 11, over the same 800-task set per model (reconforge/model/scripts/intervals.py).

Model	Params	Accuracy [95% CI]	R_w [95% CI]	HIGH Recall [95% CI]	Parse	Cost
ReconForge Recon	1.7B	0.805 [0.778, 0.833]	0.901 [0.876, 0.924]	1.000 [1.000, 1.000]	1.000	$0
DeepSeek v4-flash	—	0.876 [0.853, 0.899]	0.872 [0.843, 0.899]	0.995 [0.983, 1.000]	0.996	API (per-token; not independently verified here)
Base Qwen3-1.7B	1.7B	0.678 [0.645, 0.710]	0.600 [0.548, 0.650]	0.875 [0.825, 0.921]	0.999	$0

Champion vs. DeepSeek v4-flash, paired bootstrap (same 800 tasks, 10,000 resamples):

Comparison	Difference	95% CI	Significant (α=0.05)
Accuracy (Recon − DeepSeek)	−0.071	[−0.096, −0.046]	Yes — DeepSeek higher
R_w (Recon − DeepSeek)	+0.029	[0.008, 0.052]	Yes — ReconForge Recon higher

Both gaps are real, not noise: ReconForge Recon trades a statistically significant amount of raw accuracy for a statistically significant gain in severity-weighted recall.

Run configuration:

	ReconForge Recon	DeepSeek v4-flash	Base Qwen3-1.7B
Temperature	0.6	0.0	0.0
top_p	1.0	(provider default)	1.0
max tokens	256	1024	256
Samples per task	5 (self-consistency, majority vote)	1	1 (greedy)
Prompt version	v1	v1	v1
Thinking mode	off	n/a	off
Eval date	2026-08-08	2026-08-08	2026-08-08
Model revision	adapters/champion @ this repo	API snapshot 2026-08-08 (provider exposes no pinned revision id)	mlx-community/Qwen3-1.7B-4bit, no adapter

Full machine-readable run configs: docs/validation/runconfig.json in the source repo.

R_w is severity-weighted recall over the exception subset only (381 of the 800 tasks). It is not computed over the full eval set. Parse failures count as misses and remain in the denominator (reconforge_model/metrics.py), the conservative choice.

ECE for ReconForge Recon is 0.0875 (self-consistency ECE, from the champion eval run). DeepSeek and base Qwen do not use self-consistency sampling in this eval, so a comparable ECE was not computed for them here — reported as not available rather than estimated.

Per-Exception Recall (ReconForge Recon)

Exact verdict+type match recall, with 95% bootstrap CI, grouped by the expected exception type:

Exception Type	Recall [95% CI]	n	R_w weight
AMOUNT_MISMATCH	1.000 [1.000, 1.000]	73	1.0
FX_CONVERSION_ERROR	1.000 [1.000, 1.000]	32	1.0
BENEFICIARY_MISMATCH	1.000 [1.000, 1.000]	42	0.9
MISSING_MESSAGE	1.000 [1.000, 1.000]	45	0.6
COUNTERPARTY_MISMATCH	0.865 [0.744, 0.970]	37	0.9
VALUE_DATE_MISMATCH	0.692 [0.564, 0.814]	52	0.6
PARTIAL_MATCH	0.688 [0.517, 0.846]	32	0.5
FIELD_CORRUPTION	0.270 [0.133, 0.421]	37	0.2
DUPLICATE	0.000 [0.000, 0.000]	31	0.2

The weight column is included deliberately: R_w = 0.901 despite two recall figures below 0.3 (FIELD_CORRUPTION, DUPLICATE), because those two types carry the lowest weights — a reader who cannot see that will assume the headline is inflated. Note the CI widths at n = 31–52: COUNTERPARTY_MISMATCH's point estimate (0.865) carries a CI spanning [0.744, 0.970] — treat single decimal-place differences between runs at this sample size as noise, not signal.

Training

Parameter	Value
Base model	mlx-community/Qwen3-1.7B-4bit
Method	MLX-LoRA
LoRA rank / alpha / dropout	16 / 32 / 0.05
Target modules	All linear layers (16 layers)
Optimizer	AdamW
Learning rate	1e-5
Batch size	2
Max sequence length	2048
Grad checkpointing	Enabled
Seed	7
Iterations	740 (early stopped at plateau)
Train loss	2.4 → 0.088

Compute infrastructure: Apple M5, 16 GB unified memory. Wall time ~100 min (740 iterations). Peak memory 3.346 GB. No GPU cluster, no distributed training. macOS version at training time was not captured — not restated here as a placeholder.

Framework versions: mlx==0.32.0, mlx-lm==0.31.3, Python 3.11.15 (from reconforge/model/uv.lock).

CO2: Not measured. Order-of-magnitude: ~100 minutes of M5 CPU/GPU package power on a laptop-class chip — negligible relative to any cloud training run, but not formally estimated (co2_eq_emissions was not computed and is not asserted here).

Training data: 3,198 synthetic reconciliation pairs (train, seed 7) + 802 (val). Stratified split by (difficulty decile, exception type). Contamination guard: SHA-256 field-level pair signatures, zero exact overlap between train and eval — see recon-eval for the full audit including the near-duplicate rate this exact-hash check does not cover.

Ablation — DUPLICATE recall across two checkpoints

Run	Train pairs	Self-consistency	DUPLICATE recall	R_w	Accuracy
champion (740 iters)	3,198	x5	0.000 (0/31)	0.901	0.805
b2 (590 iters)	3,201	x3	0.032 (1/31)	0.723	0.769

Both runs used almost identical training-pair counts (3,198 vs 3,201) — so this pair of runs does not cleanly isolate a data-scale effect: b2 also differs in checkpoint (590 vs 740 iterations) and eval self-consistency (x3 vs x5), both of which move R_w and accuracy on their own. What the data does support: at comparable data volume, DUPLICATE recall stayed at or near zero across both checkpoints (0/31 and 1/31), while every other metric moved substantially between the two runs. That is consistent with — but does not prove — the discriminating signal for DUPLICATE (statement.reference == ledger.reference while all other fields match) being a single exact-equality predicate the model doesn't reliably attend to, rather than a data-volume problem. A clean data-scaling ablation (same checkpoint, different train-set sizes) has not been run; treat the "architecture/scale limit, not data limit" framing as a hypothesis this ablation is consistent with, not a proven conclusion. Either way, DUPLICATE detection is free and exact with a deterministic reference-equality pre-check — see Out-of-scope use.

Uses

Direct use. Classifying ledger↔statement reconciliation pairs into MATCH / EXCEPTION(type, severity) / ESCALATE with a structured JSON verdict.

Downstream use. As one scorer inside a larger reconciliation pipeline, behind a rule pre-check and in front of a human review queue.

Out-of-scope use — read this before deploying.

• Duplicate detection. Recall is 0.000 on the eval set. Use a deterministic reference-equality check. Routing duplicates to this model means missing (nearly) all of them.
• Uncertainty signalling. The model emits ESCALATE zero times in this eval. It has no demonstrated way to say "unsure." The system layer must compensate: every HIGH-severity verdict routes to human review unconditionally. This is a deployment requirement, not a caveat.
• Live financial data. Trained on synthetic pairs only. Performance on real ledgers is unmeasured.
• Autonomous judging. As an LLM judge against the oracle on a 100-task golden set, it reaches Cohen's kappa 0.74 (computed directly from docs/validation/golden-100.jsonl), below a 0.85 threshold for unsupervised use.
• Any non-reconciliation domain. Single-domain by construction; does not transfer to payment repair or settlement.

Limitations and Bias

1. Synthetic data only. Generated from a parameterized oracle, not sampled from production systems. Real reconciliation traffic has correlations, seasonality, and counterparty long tails this data does not model. Treat the absolute numbers as an upper bound.
2. DUPLICATE recall ~0. See Ablation above — not fixable by the data-volume change tested so far.
3. Zero escalations observed in this eval run — see Out-of-scope use.
4. Judge calibration gap. Kappa 0.74 vs oracle. ECE 0.0875 self-consistency — treat any single-sample deployment as uncalibrated relative to this number, which was measured under 5-sample self-consistency.
5. Single-domain. No transfer evidence to payment repair or settlement.
6. Generator bias. The eval set inherits every bias of the generator, including its exception-type mix and difficulty distribution. A model tuned to that generator will look better than it is on any other distribution. The near-duplicate audit (3.6% at Jaccard ≥ 0.8) bounds train/eval leakage; it says nothing about generalization beyond the generator's distribution.

Card metadata

• Authors: Caio Theodoro
• Contact: via the HF repo discussions tab
• Repository: https://github.com/caiotheodoro/reconforge

Synthetic data. All tasks are generated, not drawn from live financial systems. No real counterparties, account identifiers, or personal data are present. The methodology is the contribution; the data is an instrument for measuring it.

Not production-validated. Nothing here is financial advice or a validated control. Any deployment touching real money requires independent validation and a human-in-the-loop review path for high-severity cases.

Citation

@misc{theodoro2026reconforge,
  title  = {ReconForge: Severity-Weighted Evaluation for Financial Reconciliation Agents},
  author = {Caio Theodoro},
  year   = {2026},
  url    = {https://github.com/caiotheodoro/reconforge},
  note   = {LoRA adapter for Qwen3-1.7B, Apache-2.0}
}