license: apache-2.0
base_model: mlx-community/Qwen3-1.7B-4bit
base_model_relation: adapter
library_name: mlx
tags:
- fine-tune
- finance
- reconciliation
- mlx
- lora
- structured-outputs
- json-mode
- agent-evaluation
- severity-weighted
pipeline_tag: text-generation
metrics:
- accuracy
- recall
datasets:
- caiotheodoro/recon-eval
model-index:
- name: reconforge-recon-lora
results:
- task:
type: text-generation
name: Financial Reconciliation Verdict Classification
dataset:
name: ReconEval
type: caiotheodoro/recon-eval
revision: v0.1.0
metrics:
- name: Accuracy
type: accuracy
value: 0.805
- name: Severity-Weighted Recall
type: recall
value: 0.9007
- name: HIGH-Severity Recall
type: recall
value: 1
- name: Structured Output Parse Rate
type: accuracy
value: 1
- name: Expected Calibration Error
type: accuracy
value: 0.0875
ReconForge Recon β Financial Reconciliation Verdict Model
A LoRA fine-tune of Qwen3-1.7B for financial back-office reconciliation. It trades raw accuracy (0.805 vs DeepSeek v4-flash's 0.876) for severity-weighted recall (0.901 vs 0.872) and perfect recall on HIGH-severity exceptions β the error class that actually costs money β at zero API cost, offline, on Apple Silicon. Both the accuracy loss and the R_w gain are statistically significant (95% bootstrap CI over 10,000 resamples, paired on the same 800-task set β see Results). Trained in ~100 minutes on an M5.
Quick Start
# Block 1 β load the adapter
# Requires: mlx==0.32.0, mlx-lm==0.31.3, Python 3.11+ (pinned to this repo's
# training/eval environment β see reconforge/model/uv.lock)
from huggingface_hub import snapshot_download
from mlx_lm.lora import load
adapter_dir = snapshot_download("caiotheodoro/reconforge-recon-lora")
model, tokenizer = load(
"mlx-community/Qwen3-1.7B-4bit",
adapter_path=adapter_dir,
tokenizer_config={"trust_remote_code": True},
)
# Block 2 β build the prompt
import json
SYSTEM_PROMPT = """You are ReconForge, a financial back-office reconciliation operations engine. \
You reconcile a single ledger entry against a single bank statement entry and return a structured verdict.
Your output MUST be exactly one JSON object with these keys:
- "verdict": "MATCH" | "EXCEPTION" | "ESCALATE"
- "exception_type": null or one of AMOUNT_MISMATCH, FX_CONVERSION_ERROR, \
BENEFICIARY_MISMATCH, COUNTERPARTY_MISMATCH, VALUE_DATE_MISMATCH, MISSING_MESSAGE, \
DUPLICATE, FIELD_CORRUPTION, PARTIAL_MATCH
- "severity": "LOW" | "MEDIUM" | "HIGH"
- "confidence": float in [0, 1]
- "reason": short reason, under 10 words
- "resolution": one of "auto-adjust", "escalate", "reject", "rebook", "flag-review"
"""
user_prompt = """Reconcile the following ledger entry against the bank statement.
LEDGER ENTRY:
{
"amount": "10000.00", "ccy": "USD", "counterparty": "Acme Corp",
"beneficiary": "Acme Corp", "value_date": "2024-01-15",
"message_type": "MT300", "ref": "OUR-REF-000001"
}
BANK STATEMENT:
{
"amount": "9999.50", "ccy": "USD", "counterparty": "Acme Corp",
"beneficiary": "Acme Corp", "value_date": "2024-01-15",
"message_type": "MT940", "ref": "CP-PAY-000001"
}
Return the verdict JSON object only."""
# Block 3 β generate
from mlx_lm.generate import generate
for attr in ("has_thinking", "enable_thinking"):
if hasattr(tokenizer, attr):
setattr(tokenizer, attr, False)
messages = [
{"role": "system", "content": SYSTEM_PROMPT},
{"role": "user", "content": user_prompt},
]
prompt = tokenizer.apply_chat_template(
messages, add_generation_prompt=True, enable_thinking=False, return_dict=False
)
response = generate(
model, tokenizer, prompt, max_tokens=128, # single greedy sample --
# a minimal usage example,
# NOT the eval config
)
verdict = json.loads(response)
print(f"{verdict['verdict']} / {verdict['exception_type']} / {verdict['severity']}")
This Quick Start uses one greedy sample. It is a minimal usage example, not the eval configuration. The Results below were produced with self-consistency sampling (5 samples, temp=0.6, majority vote) β see the Run configuration table. A single greedy sample will not reproduce the reported numbers.
The model is trained in non-thinking mode (
enable_thinking=False) and emits ~38 tokens per verdict on average. Leaving thinking enabled changes the output format and breaks the parse.
Results
Eval set: ReconEval v0.1.0, 800 held-out tasks, seed 777, exact-overlap
contamination 0/800 (near-duplicate rate 3.6% at Jaccard β₯ 0.8 β see
recon-eval).
All CIs are 95% bootstrap intervals, 10,000 resamples, seed 11, over the same
800-task set per model (reconforge/model/scripts/intervals.py).
| Model | Params | Accuracy [95% CI] | R_w [95% CI] | HIGH Recall [95% CI] | Parse | Cost |
|---|---|---|---|---|---|---|
| ReconForge Recon | 1.7B | 0.805 [0.778, 0.833] | 0.901 [0.876, 0.924] | 1.000 [1.000, 1.000] | 1.000 | $0 |
| DeepSeek v4-flash | β | 0.876 [0.853, 0.899] | 0.872 [0.843, 0.899] | 0.995 [0.983, 1.000] | 0.996 | API (per-token; not independently verified here) |
| Base Qwen3-1.7B | 1.7B | 0.678 [0.645, 0.710] | 0.600 [0.548, 0.650] | 0.875 [0.825, 0.921] | 0.999 | $0 |
Champion vs. DeepSeek v4-flash, paired bootstrap (same 800 tasks, 10,000 resamples):
| Comparison | Difference | 95% CI | Significant (Ξ±=0.05) |
|---|---|---|---|
| Accuracy (Recon β DeepSeek) | β0.071 | [β0.096, β0.046] | Yes β DeepSeek higher |
| R_w (Recon β DeepSeek) | +0.029 | [0.008, 0.052] | Yes β ReconForge Recon higher |
Both gaps are real, not noise: ReconForge Recon trades a statistically significant amount of raw accuracy for a statistically significant gain in severity-weighted recall.
Run configuration:
| ReconForge Recon | DeepSeek v4-flash | Base Qwen3-1.7B | |
|---|---|---|---|
| Temperature | 0.6 | 0.0 | 0.0 |
| top_p | 1.0 | (provider default) | 1.0 |
| max tokens | 256 | 1024 | 256 |
| Samples per task | 5 (self-consistency, majority vote) | 1 | 1 (greedy) |
| Prompt version | v1 |
v1 |
v1 |
| Thinking mode | off | n/a | off |
| Eval date | 2026-08-08 | 2026-08-08 | 2026-08-08 |
| Model revision | adapters/champion @ this repo |
API snapshot 2026-08-08 (provider exposes no pinned revision id) | mlx-community/Qwen3-1.7B-4bit, no adapter |
Full machine-readable run configs: docs/validation/runconfig.json in the
source repo.
R_w is severity-weighted recall over the exception subset only (381 of the 800 tasks). It is not computed over the full eval set. Parse failures count as misses and remain in the denominator (
reconforge_model/metrics.py), the conservative choice.ECE for ReconForge Recon is 0.0875 (self-consistency ECE, from the champion eval run). DeepSeek and base Qwen do not use self-consistency sampling in this eval, so a comparable ECE was not computed for them here β reported as not available rather than estimated.
Per-Exception Recall (ReconForge Recon)
Exact verdict+type match recall, with 95% bootstrap CI, grouped by the expected exception type:
| Exception Type | Recall [95% CI] | n | R_w weight |
|---|---|---|---|
| AMOUNT_MISMATCH | 1.000 [1.000, 1.000] | 73 | 1.0 |
| FX_CONVERSION_ERROR | 1.000 [1.000, 1.000] | 32 | 1.0 |
| BENEFICIARY_MISMATCH | 1.000 [1.000, 1.000] | 42 | 0.9 |
| MISSING_MESSAGE | 1.000 [1.000, 1.000] | 45 | 0.6 |
| COUNTERPARTY_MISMATCH | 0.865 [0.744, 0.970] | 37 | 0.9 |
| VALUE_DATE_MISMATCH | 0.692 [0.564, 0.814] | 52 | 0.6 |
| PARTIAL_MATCH | 0.688 [0.517, 0.846] | 32 | 0.5 |
| FIELD_CORRUPTION | 0.270 [0.133, 0.421] | 37 | 0.2 |
| DUPLICATE | 0.000 [0.000, 0.000] | 31 | 0.2 |
The weight column is included deliberately: R_w = 0.901 despite two recall figures below 0.3 (FIELD_CORRUPTION, DUPLICATE), because those two types carry the lowest weights β a reader who cannot see that will assume the headline is inflated. Note the CI widths at n = 31β52: COUNTERPARTY_MISMATCH's point estimate (0.865) carries a CI spanning [0.744, 0.970] β treat single decimal-place differences between runs at this sample size as noise, not signal.
Training
| Parameter | Value |
|---|---|
| Base model | mlx-community/Qwen3-1.7B-4bit |
| Method | MLX-LoRA |
| LoRA rank / alpha / dropout | 16 / 32 / 0.05 |
| Target modules | All linear layers (16 layers) |
| Optimizer | AdamW |
| Learning rate | 1e-5 |
| Batch size | 2 |
| Max sequence length | 2048 |
| Grad checkpointing | Enabled |
| Seed | 7 |
| Iterations | 740 (early stopped at plateau) |
| Train loss | 2.4 β 0.088 |
Compute infrastructure: Apple M5, 16 GB unified memory. Wall time ~100 min (740 iterations). Peak memory 3.346 GB. No GPU cluster, no distributed training. macOS version at training time was not captured β not restated here as a placeholder.
Framework versions: mlx==0.32.0, mlx-lm==0.31.3, Python 3.11.15
(from reconforge/model/uv.lock).
CO2: Not measured. Order-of-magnitude: ~100 minutes of M5 CPU/GPU package
power on a laptop-class chip β negligible relative to any cloud training run,
but not formally estimated (co2_eq_emissions was not computed and is not
asserted here).
Training data: 3,198 synthetic reconciliation pairs (train, seed 7) + 802 (val). Stratified split by (difficulty decile, exception type). Contamination guard: SHA-256 field-level pair signatures, zero exact overlap between train and eval β see recon-eval for the full audit including the near-duplicate rate this exact-hash check does not cover.
Ablation β DUPLICATE recall across two checkpoints
| Run | Train pairs | Self-consistency | DUPLICATE recall | R_w | Accuracy |
|---|---|---|---|---|---|
| champion (740 iters) | 3,198 | x5 | 0.000 (0/31) | 0.901 | 0.805 |
| b2 (590 iters) | 3,201 | x3 | 0.032 (1/31) | 0.723 | 0.769 |
Both runs used almost identical training-pair counts (3,198 vs 3,201) β so
this pair of runs does not cleanly isolate a data-scale effect: b2 also
differs in checkpoint (590 vs 740 iterations) and eval self-consistency (x3
vs x5), both of which move R_w and accuracy on their own. What the data does
support: at comparable data volume, DUPLICATE recall stayed at or near zero
across both checkpoints (0/31 and 1/31), while every other metric moved
substantially between the two runs. That is consistent with β but does not
prove β the discriminating signal for DUPLICATE (statement.reference == ledger.reference while all other fields match) being a single
exact-equality predicate the model doesn't reliably attend to, rather than a
data-volume problem. A clean data-scaling ablation (same checkpoint,
different train-set sizes) has not been run; treat the "architecture/scale
limit, not data limit" framing as a hypothesis this ablation is consistent
with, not a proven conclusion. Either way, DUPLICATE detection is free and
exact with a deterministic reference-equality pre-check β see Out-of-scope
use.
Uses
Direct use. Classifying ledgerβstatement reconciliation pairs into MATCH / EXCEPTION(type, severity) / ESCALATE with a structured JSON verdict.
Downstream use. As one scorer inside a larger reconciliation pipeline, behind a rule pre-check and in front of a human review queue.
Out-of-scope use β read this before deploying.
- Duplicate detection. Recall is 0.000 on the eval set. Use a deterministic reference-equality check. Routing duplicates to this model means missing (nearly) all of them.
- Uncertainty signalling. The model emits ESCALATE zero times in this eval. It has no demonstrated way to say "unsure." The system layer must compensate: every HIGH-severity verdict routes to human review unconditionally. This is a deployment requirement, not a caveat.
- Live financial data. Trained on synthetic pairs only. Performance on real ledgers is unmeasured.
- Autonomous judging. As an LLM judge against the oracle on a 100-task
golden set, it reaches Cohen's kappa 0.74 (computed directly from
docs/validation/golden-100.jsonl), below a 0.85 threshold for unsupervised use. - Any non-reconciliation domain. Single-domain by construction; does not transfer to payment repair or settlement.
Limitations and Bias
- Synthetic data only. Generated from a parameterized oracle, not sampled from production systems. Real reconciliation traffic has correlations, seasonality, and counterparty long tails this data does not model. Treat the absolute numbers as an upper bound.
- DUPLICATE recall ~0. See Ablation above β not fixable by the data-volume change tested so far.
- Zero escalations observed in this eval run β see Out-of-scope use.
- Judge calibration gap. Kappa 0.74 vs oracle. ECE 0.0875 self-consistency β treat any single-sample deployment as uncalibrated relative to this number, which was measured under 5-sample self-consistency.
- Single-domain. No transfer evidence to payment repair or settlement.
- Generator bias. The eval set inherits every bias of the generator, including its exception-type mix and difficulty distribution. A model tuned to that generator will look better than it is on any other distribution. The near-duplicate audit (3.6% at Jaccard β₯ 0.8) bounds train/eval leakage; it says nothing about generalization beyond the generator's distribution.
Card metadata
- Authors: Caio Theodoro
- Contact: via the HF repo discussions tab
- Repository: https://github.com/caiotheodoro/reconforge
Synthetic data. All tasks are generated, not drawn from live financial systems. No real counterparties, account identifiers, or personal data are present. The methodology is the contribution; the data is an instrument for measuring it.
Not production-validated. Nothing here is financial advice or a validated control. Any deployment touching real money requires independent validation and a human-in-the-loop review path for high-severity cases.
Citation
@misc{theodoro2026reconforge,
title = {ReconForge: Severity-Weighted Evaluation for Financial Reconciliation Agents},
author = {Caio Theodoro},
year = {2026},
url = {https://github.com/caiotheodoro/reconforge},
note = {LoRA adapter for Qwen3-1.7B, Apache-2.0}
}