cakradana-app/cakradana-v1-docie-deepseek-r1-distill-llama-8b-adapter

🧾 Cakradana v1 — Document-IE Adapter

A LoRA adapter that turns Indonesian campaign-finance filing text into structured donation records

Built with Llama

---

📋 Table of Contents

• 🎯 What this is
• 📊 Evaluation
• 🚀 Usage
• 🗂️ Output schema
• 🔧 Training
• ⚠️ Limitations
• 📜 Licence

---

🎯 What this is

Indonesian candidates and parties file campaign-finance reports — LADK, LPSDK, LPPDK — with the KPU (Komisi Pemilihan Umum). Turning those filings into auditable data means reading a donor table and emitting one record per donation, without inventing any.

This is a LoRA adapter for unsloth/DeepSeek-R1-Distill-Llama-8B (83.9 M trainable parameters) that does that. It is inert on its own — it needs the base model.

It pairs with cakradana-v1-ocr-deepseek-3b-adapter, which produces the text this model reads. Both were built for Cakradana, an AI system for transparency in Indonesian election financing, which placed 3rd in the Open Sector at the TrackAML 2.0 hackathon run by PPATK, Indonesia's Financial Transaction Reports and Analysis Center.

---

📊 Evaluation

Compare only at equal token budget. The base model spends its output budget reasoning: at 1,024 tokens, 44 of 100 baseline outputs never closed </think> and so never reached any JSON. Scoring those as failures credits this adapter for R1-Distill's verbosity rather than for extraction.

The middle column re-runs 30 of the identical chunks at 3,072 tokens (the eval seeds its selection, so those rows are a strict subset). All three columns below are those same 30 chunks.

metric	base @1024	base @3072	tuned @1024
parse rate	0.333	0.733	1.000
fabrication rate	0.444	0.643	0.000
span validity	0.867	0.935	0.918
F1 (strict)	0.050	0.198	0.418
recall	0.025	0.136	0.322
cut off mid-reasoning	13	6	0

Fabrication is the result

89% of real filing chunks carry no donation at all — covers, decision letters, expenditure tables. A model that invents records on those is worse than useless no matter how it scores elsewhere.

The base model fabricates on roughly two thirds of the empty chunks it manages to answer, and giving it three times the budget does not help (0.444 → 0.643; more budget means more outputs parse, which reveals more fabrication). The tuned model fabricates on none — 0 of 60 hard-null chunks in the full run, while still emitting 218 records on the chunks that do carry them. It learned to discriminate, not to refuse.

Span validity is not an improvement

At equal budget the base model scores 0.935 against this adapter's 0.918. The apparent 0.792 → 0.931 gain in the naive 1,024-token comparison is a budget artefact, and it is not quoted here as a win.

Secondary: the full 100-chunk run

Both columns at 1,024 tokens, so the parse-rate and recall gaps are inflated by the budget effect above. Reported for completeness, not as the headline.

metric	base	tuned
parse rate	0.370	0.990
fabrication rate	0.636	0.000
F1 (strict)	0.030	0.578
F1 (amount-only)	0.039	0.745

The matcher is not doing the work: folding titles, legal forms and punctuation moves strict F1 by 0.007. The gap to amount-only matching is real signal — where this model errs, it more often has the donation right and the donor name wrong than the reverse.

Reproduce either run:

python scripts/eval_extraction.py                  # tuned
python scripts/eval_extraction.py --adapter ''     # baseline

The published figures were measured against the private pre-redaction corpus. The script reads the public release — a redacted subset, 6,174 pairs against 8,016 — so it selects different rows and will not reproduce them to the digit. It reproduces the comparison.

---

🚀 Usage

The prompt is not a suggestion. This adapter was trained under vocabulary.build_system_prompt() / build_user_prompt(), shipped in this repo. Substituting your own instructions serves the model something it never saw during training.

pip install "transformers==4.57.3" peft unsloth
pip uninstall -y torchao      # peft calls is_torchao_available(), which raises
                              # rather than returning False on some builds

python scripts/extract.py filing.txt

from scripts.extract import load, extract, check_spans

model, tokenizer = load()                      # base + this adapter
records, raw = extract(model, tokenizer, document_text)
bad = check_spans(records, document_text)      # every *_span must be quotable

add_generation_prompt=True ends the prompt at <｜Assistant｜><think>\n, and the model continues </think>\n[...]. Training used exactly that state, so the standard chat-template call is the correct one — no prompt surgery, and no reasoning trace to pay for. The <think> block is deliberately empty: this task teaches an output format, and the span rule is what guards correctness.

---

🗂️ Output schema

A JSON array of objects, each carrying only the fields the document supports:

[{"sender": "Hj. Nessy Ariyani", "sender_type": "individual",
  "amount": 750000, "date": "2015-10-15",
  "sender_span": "Hj. Nessy Ariyani", "amount_span": "750.000",
  "date_span": "15 OKTOBER 2015"}]

Every field carries a *_span, and a span is the raw quoted substring — not an offset. The value is normalised (750000); the span is what is printed on the page ("750.000"). This is the contract that makes an extraction checkable rather than merely plausible: check_spans() in scripts/extract.py verifies each one against the source, and a span that is not in the document is a fabricated quote however right the value looks.

An empty array is a valid and common answer. It is the correct answer for most filing pages.

---

🔧 Training

base	unsloth/DeepSeek-R1-Distill-Llama-8B-unsloth-bnb-4bit, 4-bit
method	LoRA, r=32, alpha=32, dropout 0, bias none
trainable	83,886,080 parameters, 448 tensors
scope	all 7 projections × all 32 layers
schedule	2 epochs, 338 steps, batch 2 × grad-accum 4
optimiser	adamw_8bit, lr 1e-4 linear, warmup 0.03, weight decay 0.01
precision	bf16, max_seq_length 4096
seed	20260819
hardware	1× A100-SXM4-40GB
runtime	1,114.6 s (18 m 35 s)
final train loss	0.0679

Data. 8,016 pairs → 7,984 after deduplicating chunk placements → 343 record-bearing and 6,179 hard-null in train. Hard nulls are downsampled to 3:1 against record-bearing chunks (the corpus ships at 17:1), giving 1,372 rows, then 1,349 after dropping 23 over the sequence limit.

That 3:1 ratio is what produced the fabrication result. It was not swept; a different ratio would likely trade fabrication against recall.

Loss falls on the answer only, via train_on_responses_only with the <｜User｜> / <｜Assistant｜> markers. Note those are U+FF5C FULLWIDTH VERTICAL LINE, not ASCII | — an ASCII pipe matches nothing and silently supervises zero tokens. scripts/train_extraction.py decodes the supervised span and aborts unless </think> appears in it, rather than training for hours on nothing.

The published dataset is a redacted subset of what this run saw. Training used the private pre-redaction corpus; the public release (cakradana-kpu-filings-14k-pages) withholds pages carrying personal identifiers, leaving 6,174 pairs against the 8,016 used here. It is the corresponding citable dataset.

---

⚠️ Limitations

• Recall collapses on record-dense chunks: 0.608 (1–5 records), 0.652 (6–20), 0.241 (21+). The cause is in the recipe above, not the model. The max_seq_length=4096 filter dropped 13 record-bearing training rows whose median was 60 records (max 172), leaving a training set that tops out at 45. Of 40 record-bearing eval outputs only 1 was truncated by the output cap; 23 emitted fewer records than expected and then closed the array — 142 records never emitted. Raising --max-seq to 8192 is the obvious v2, and the F1 above is its baseline.
• Raise max_new_tokens for dense pages. One eval output was truncated mid-record at 1,024. scripts/extract.py defaults to 2,048.
• eval_clean covers 4 publishing hosts, 166 documents. Nothing here supports a claim about an unseen publisher.
• Both evaluation references are teacher-generated, so absolute scores describe agreement with a teacher, not ground truth.
• 19.1% of record placements in the training split are duplicated across overlapping chunks (touching 40.2% of record-bearing chunks). Oversampling, not leakage: train and eval share zero documents, zero hosts and zero identical records.
• Indonesian campaign-finance documents specifically. Not a general information-extraction model.
• Output is a reading, not a finding. Nothing this model emits is evidence of wrongdoing by anyone named in a document it read.

---

📜 Licence

Built with Llama. Governed by the Llama 3.1 Community License, inherited through DeepSeek-R1-Distill-Llama-8B from Llama-3.1-8B-Base. That covers the weights; it does not release you from the training corpus's terms, which prohibit attempting to re-identify individuals and use for harassment or profiling. See LICENSE.

Made with ❤️ by the Cakradana Team · cakradana.faizath.com