Restructure extraction into tidy tables, scoped fields, and an audit trail

- Split output into two tidy (one-fact-per-row) tables instead of one
wide sparse CSV: extracted fields and notification hits, plus a third
run-metadata CSV documenting the exact settings (categories, keywords,
custom fields, match filter, chat model/transcript) that produced the
run - a lineage record for auditing why data was or wasn't pulled.
- Standard field extraction (PO/SAP/invoice/cert codes) is now scoped to
the selected categories via CATEGORY_FIELD_MAP instead of always
running every pattern on every document.
- Added a generic label->value custom field extractor (extract_custom_fields)
so users can name arbitrary fields to pull as data (e.g. "reference
number", "gross weight") separate from notification keywords, which
only raise a flag. The chat assistant now configures both.

README.md

@@ -23,25 +23,59 @@ and key fields out of paper-heavy workflows:
 - **Business finance** — past-due invoices, payment rejections, credit
   holds, invoice numbers, dollar amounts, due dates.
 
+## Notifications vs. fields
+
+The tool treats two kinds of requests differently, rather than running
+one blanket extraction pass over every document:
+
+- **Notifications** — keyword/phrase hits that flag an event (a boolean
+  "this happened on this page"), e.g. "past due", "non-compliant",
+  "change order". Each category (construction/compliance/finance/general)
+  has its own built-in phrase list; you can add your own custom
+  notification keywords too.
+- **Fields** — labeled values extracted as data, not just flagged.
+  Standard fields are *scoped to the categories you select* (see
+  `CATEGORY_FIELD_MAP` in `extractors.py`) so a compliance document
+  doesn't generate empty invoice-number columns: construction gets PO/SAP
+  numbers, compliance gets certification codes, finance gets invoice
+  numbers/percentages, and dates/amounts apply where relevant. On top of
+  that, you can name your own **custom fields** — any label that appears
+  in the document followed by a value (e.g. "reference number", "gross
+  weight", "delivery date") — and the app finds the label and captures
+  what follows it.
+
 ## How it works
 
 1. **Tailor extraction (chat tab)** — describe in plain language what you
-   want flagged and pulled (e.g. *"only flag overdue invoices over $10k
-   and non-compliance notices, and pull SAP PO numbers"*). A Hugging
-   Face-hosted chat model turns that into extraction settings
-   (categories, custom keywords, export filter), which populate the
-   Upload & Run tab automatically. You can also skip the chat and set
-   these by hand.
+   want flagged vs. pulled as data (e.g. *"flag overdue invoices over
+   $10k and non-compliance notices, and pull the reference number and
+   gross weight"*). A Hugging Face-hosted chat model turns that into
+   extraction settings (categories, notification keywords, custom fields,
+   export filter), which populate the Upload & Run tab automatically. You
+   can also skip the chat and set everything by hand there.
 2. Upload one or more PDFs or images.
 3. Each page's text is pulled directly from the PDF when it has a text
    layer; scanned/image-only pages are rendered and run through
    Tesseract OCR.
-4. The text is scanned against keyword sets for the categories you pick
-   (or your own custom keywords), and structured fields (PO/SAP/invoice
-   numbers, dates, dollar amounts, percentages, certification codes) are
-   pulled out with regex.
-5. Results are shown in a table and exported as a CSV, one row per page
-   that has a hit (or every page, if you turn that filter off).
+4. The text is scanned for notification keywords and both standard and
+   custom fields, per the scoping above.
+5. Results are shown as two tidy (one-fact-per-row) tables — extracted
+   fields and notification hits — and exported as three CSVs:
+   - `ocr_fields_*.csv` — file, page, field_type (standard/custom),
+     field_name, field_value, OCR source/confidence.
+   - `ocr_notifications_*.csv` — file, page, category, keyword, context,
+     OCR source/confidence.
+   - `ocr_run_metadata_*.csv` — an audit record of exactly what produced
+     the run: categories, notification keywords, custom fields, the
+     match filter, the chat model used, and the full chat transcript.
+     This is the lineage trail for anyone auditing why a given row was
+     (or wasn't) extracted.
+
+   Long/tidy tables were chosen over one wide fixed-schema table so that
+   irrelevant fields never show up as blank columns, and so the output
+   loads cleanly into a database, pivot table, or BI tool without
+   reshaping — consistent with standard data-quality practice (e.g. DAMA
+   DMBOK's completeness, validity, and lineage dimensions).
 
 ## Chat assistant setup
 
@@ -74,7 +108,8 @@ python app.py
 
 - `app.py` — Gradio UI and orchestration.
 - `ocr_engine.py` — PDF/image loading and OCR (PyMuPDF + Tesseract).
-- `extractors.py` — keyword notification scanning and structured field
-  extraction (regex-based, easy to extend per domain).
+- `extractors.py` — keyword notification scanning, category-scoped
+  standard field extraction, and generic label->value custom field
+  extraction.
 - `chat_config.py` — chat assistant that turns natural-language
   instructions into extraction settings via a Hugging Face chat model.

app.py

@@ -2,11 +2,15 @@
 
 Upload scanned PDFs or images (construction reports, compliance
 certificates, invoices, etc.), scan the extracted text for domain
-"notification" keywords, pull out structured fields (PO/SAP/invoice
-numbers, dates, amounts, cert codes), and export everything to CSV.
+"notification" keywords, pull out labeled data fields (standard IDs/dates/
+amounts plus any custom labels the user names), and export everything to
+CSV as tidy (one-fact-per-row) tables, alongside an audit record of the
+settings that produced the run.
 """
+import json
 import os
 import tempfile
+import uuid
 from datetime import datetime
 
 import gradio as gr
@@ -21,14 +25,21 @@ CATEGORY_CHOICES = chat_config.CATEGORY_CHOICES
 DEFAULT_SETTINGS = {
     "categories": CATEGORY_CHOICES,
     "custom_keywords": [],
+    "custom_fields": [],
     "only_matches": True,
 }
 
-COLUMNS = [
-    "file_name", "page", "text_source", "ocr_confidence",
-    "notification_flags", "notification_context",
-    "po_numbers", "sap_doc_numbers", "invoice_numbers", "cert_codes",
-    "dates", "amounts", "percentages",
+FIELDS_COLUMNS = [
+    "run_id", "file_name", "page", "field_type", "field_name", "field_value",
+    "text_source", "ocr_confidence",
+]
+NOTIFICATIONS_COLUMNS = [
+    "run_id", "file_name", "page", "category", "keyword", "context",
+    "text_source", "ocr_confidence",
+]
+METADATA_COLUMNS = [
+    "run_id", "timestamp", "files", "categories", "custom_keywords",
+    "custom_fields", "only_matches_filter", "chat_model", "chat_transcript",
 ]
 
 
@@ -36,22 +47,41 @@ def _file_path(f):
     return f if isinstance(f, str) else f.name
 
 
-def process_files(files, categories, custom_keywords_str, only_matches):
+def _parse_comma_list(s):
+    return [k.strip() for k in s.split(",") if k.strip()] if s and s.strip() else []
+
+
+def _format_transcript(history):
+    if not history:
+        return ""
+    lines = []
+    for turn in history:
+        role = turn.get("role", "?")
+        content = turn.get("content", "")
+        lines.append(f"{role}: {content}")
+    return "\n".join(lines)
+
+
+def process_files(files, categories, custom_keywords_str, custom_fields_str, only_matches, chat_history):
+    empty = (pd.DataFrame(columns=FIELDS_COLUMNS), pd.DataFrame(columns=NOTIFICATIONS_COLUMNS))
     if not files:
-        return "Upload at least one PDF or image file.", pd.DataFrame(columns=COLUMNS), None, ""
+        return "Upload at least one PDF or image file.", empty[0], empty[1], None, "", {}
 
-    custom_keywords = None
-    if custom_keywords_str and custom_keywords_str.strip():
-        custom_keywords = [k.strip() for k in custom_keywords_str.split(",") if k.strip()]
+    custom_keywords = _parse_comma_list(custom_keywords_str)
+    custom_field_labels = _parse_comma_list(custom_fields_str)
 
-    rows = []
+    run_id = uuid.uuid4().hex[:8]
+    field_rows = []
+    notification_rows = []
     full_text_log = []
     warnings = []
     total_pages = 0
+    file_names = []
 
     for f in files:
         path = _file_path(f)
         file_name = os.path.basename(path)
+        file_names.append(file_name)
         try:
             pages = ocr_engine.extract_pages(path)
         except Exception as exc:
@@ -61,57 +91,78 @@ def process_files(files, categories, custom_keywords_str, only_matches):
         for page_data in pages:
             total_pages += 1
             text = page_data["text"]
+            source = page_data["source"]
+            confidence = page_data["ocr_confidence"]
+
             notifications = extractors.find_notifications(text, categories, custom_keywords)
-            fields = extractors.extract_fields(text)
-            has_content = bool(notifications) or any(fields.values())
+            std_fields = extractors.extract_fields(text, categories)
+            custom_field_values = extractors.extract_custom_fields(text, custom_field_labels)
 
-            full_text_log.append(
-                f"--- {file_name} | page {page_data['page']} | {page_data['source']} ---\n{text}\n"
-            )
+            full_text_log.append(f"--- {file_name} | page {page_data['page']} | {source} ---\n{text}\n")
 
+            has_content = bool(notifications) or any(std_fields.values()) or any(custom_field_values.values())
             if only_matches and not has_content:
                 continue
 
-            contexts = []
+            for field_name, values in std_fields.items():
+                for value in values:
+                    field_rows.append({
+                        "run_id": run_id, "file_name": file_name, "page": page_data["page"],
+                        "field_type": "standard", "field_name": field_name, "field_value": value,
+                        "text_source": source, "ocr_confidence": confidence,
+                    })
+            for field_name, values in custom_field_values.items():
+                for value in values:
+                    field_rows.append({
+                        "run_id": run_id, "file_name": file_name, "page": page_data["page"],
+                        "field_type": "custom", "field_name": field_name, "field_value": value,
+                        "text_source": source, "ocr_confidence": confidence,
+                    })
             for n in notifications:
-                if n["context"] not in contexts:
-                    contexts.append(n["context"])
-
-            rows.append({
-                "file_name": file_name,
-                "page": page_data["page"],
-                "text_source": page_data["source"],
-                "ocr_confidence": page_data["ocr_confidence"],
-                "notification_flags": "; ".join(
-                    sorted({f"[{n['category']}] {n['keyword']}" for n in notifications})
-                ),
-                "notification_context": " | ".join(contexts[:5]),
-                "po_numbers": "; ".join(fields["po_numbers"]),
-                "sap_doc_numbers": "; ".join(fields["sap_doc_numbers"]),
-                "invoice_numbers": "; ".join(fields["invoice_numbers"]),
-                "cert_codes": "; ".join(fields["cert_codes"]),
-                "dates": "; ".join(fields["dates"]),
-                "amounts": "; ".join(fields["amounts"]),
-                "percentages": "; ".join(fields["percentages"]),
-            })
-
-    df = pd.DataFrame(rows, columns=COLUMNS)
-
-    csv_path = None
-    if not df.empty:
+                notification_rows.append({
+                    "run_id": run_id, "file_name": file_name, "page": page_data["page"],
+                    "category": n["category"], "keyword": n["keyword"], "context": n["context"],
+                    "text_source": source, "ocr_confidence": confidence,
+                })
+
+    fields_df = pd.DataFrame(field_rows, columns=FIELDS_COLUMNS)
+    notifications_df = pd.DataFrame(notification_rows, columns=NOTIFICATIONS_COLUMNS)
+
+    metadata_row = {
+        "run_id": run_id,
+        "timestamp": datetime.now().isoformat(timespec="seconds"),
+        "files": "; ".join(file_names),
+        "categories": "; ".join(categories),
+        "custom_keywords": "; ".join(custom_keywords),
+        "custom_fields": "; ".join(custom_field_labels),
+        "only_matches_filter": only_matches,
+        "chat_model": chat_config.DEFAULT_MODEL if chat_history else "",
+        "chat_transcript": _format_transcript(chat_history),
+    }
+    metadata_df = pd.DataFrame([metadata_row], columns=METADATA_COLUMNS)
+
+    csv_paths = []
+    if not fields_df.empty or not notifications_df.empty:
         timestamp = datetime.now().strftime("%Y%m%d_%H%M%S")
-        csv_path = os.path.join(tempfile.gettempdir(), f"ocr_extract_{timestamp}.csv")
-        df.to_csv(csv_path, index=False)
+        tmp_dir = tempfile.gettempdir()
+        fields_path = os.path.join(tmp_dir, f"ocr_fields_{timestamp}_{run_id}.csv")
+        notifications_path = os.path.join(tmp_dir, f"ocr_notifications_{timestamp}_{run_id}.csv")
+        metadata_path = os.path.join(tmp_dir, f"ocr_run_metadata_{timestamp}_{run_id}.csv")
+        fields_df.to_csv(fields_path, index=False)
+        notifications_df.to_csv(notifications_path, index=False)
+        metadata_df.to_csv(metadata_path, index=False)
+        csv_paths = [fields_path, notifications_path, metadata_path]
 
     status_lines = [
         f"Processed {len(files)} file(s), {total_pages} page(s) total.",
-        f"{len(df)} row(s) exported" + (" (matches only)." if only_matches else "."),
+        f"{len(fields_df)} field value(s), {len(notifications_df)} notification hit(s)"
+        + (" (matches-only pages)." if only_matches else " (all pages)."),
     ]
     if warnings:
         status_lines.append("Warnings: " + "; ".join(warnings))
     status = "\n".join(status_lines)
 
-    return status, df, csv_path, "\n".join(full_text_log)
+    return status, fields_df, notifications_df, csv_paths, "\n".join(full_text_log), metadata_row
 
 
 def handle_chat(history, message, settings, hf_token):
@@ -119,7 +170,8 @@ def handle_chat(history, message, settings, hf_token):
     if not message or not message.strip():
         return (
             history, "", settings, settings,
-            settings["categories"], ", ".join(settings["custom_keywords"]), settings["only_matches"],
+            settings["categories"], ", ".join(settings["custom_keywords"]),
+            ", ".join(settings["custom_fields"]), settings["only_matches"],
         )
 
     reply, new_settings = chat_config.chat_update(history, message.strip(), settings, hf_token)
@@ -127,10 +179,10 @@ def handle_chat(history, message, settings, hf_token):
         {"role": "user", "content": message.strip()},
         {"role": "assistant", "content": reply},
     ]
-    custom_kw_str = ", ".join(new_settings["custom_keywords"])
     return (
         history, "", new_settings, new_settings,
-        new_settings["categories"], custom_kw_str, new_settings["only_matches"],
+        new_settings["categories"], ", ".join(new_settings["custom_keywords"]),
+        ", ".join(new_settings["custom_fields"]), new_settings["only_matches"],
     )
 
 
@@ -139,19 +191,23 @@ with gr.Blocks(title="OCR Notification & Data Extractor") as demo:
         "# OCR Notification & Data Extractor\n"
         "Upload scanned PDFs or images (construction reports, compliance "
         "certificates, invoices, SAP printouts, etc.). The app OCRs each "
-        "page, flags notification-worthy keywords per domain, extracts "
-        "structured fields (PO/SAP/invoice numbers, dates, amounts, cert "
-        "codes), and exports everything to CSV."
+        "page, flags notification-worthy keywords per domain, and pulls out "
+        "labeled data fields - standard ones scoped to the categories you "
+        "pick, plus any custom labels you name (e.g. \"gross weight\", "
+        "\"delivery date\"). Every run exports three tidy CSVs: extracted "
+        "fields, notification hits, and a metadata/audit record of exactly "
+        "what settings produced them."
     )
 
     settings_state = gr.State(dict(DEFAULT_SETTINGS))
 
     with gr.Tab("1. Tailor extraction (chat)"):
         gr.Markdown(
-            "Describe what you want flagged and pulled, in plain language, "
-            "e.g. *\"Only flag overdue invoices over $10k and non-compliance "
-            "notices, and pull SAP PO numbers\"*. This updates the settings "
-            "used in the **Upload & Run** tab — you can still edit them by "
+            "Describe what you want **flagged** (notifications) vs. **pulled "
+            "as data** (fields), in plain language, e.g. *\"flag overdue "
+            "invoices and non-compliance notices, and pull the reference "
+            "number, gross weight, and delivery date\"*. This updates the "
+            "settings used in **Upload & Run** - you can still edit them by "
             "hand there too."
         )
         hf_token_input = gr.Textbox(
@@ -163,7 +219,7 @@ with gr.Blocks(title="OCR Notification & Data Extractor") as demo:
         with gr.Row():
             chat_input = gr.Textbox(
                 label="Message", scale=4,
-                placeholder="e.g. Only flag overdue invoices and pull PO numbers, ignore construction stuff",
+                placeholder="e.g. Flag overdue invoices; pull reference number, gross weight, delivery date",
             )
             chat_send = gr.Button("Send", scale=1, variant="primary")
         settings_display = gr.JSON(label="Active extraction settings", value=DEFAULT_SETTINGS)
@@ -179,10 +235,17 @@ with gr.Blocks(title="OCR Notification & Data Extractor") as demo:
                     choices=CATEGORY_CHOICES,
                     value=CATEGORY_CHOICES,
                     label="Notification categories",
+                    info="Also scopes which standard fields (PO/SAP/invoice/cert) get extracted - see CATEGORY_FIELD_MAP.",
                 )
                 custom_keywords_input = gr.Textbox(
-                    label="Custom keywords (comma-separated, optional)",
+                    label="Custom notification keywords (comma-separated)",
                     placeholder="e.g. re-inspection, warranty claim, hold payment",
+                    info="Phrases that just raise a flag - not extracted as values.",
+                )
+                custom_fields_input = gr.Textbox(
+                    label="Custom fields to extract as data (comma-separated labels)",
+                    placeholder="e.g. reference number, gross weight, delivery date",
+                    info="Labels looked up in the document; the value after each label is captured.",
                 )
                 only_matches_input = gr.Checkbox(
                     value=True,
@@ -192,28 +255,26 @@ with gr.Blocks(title="OCR Notification & Data Extractor") as demo:
 
             with gr.Column(scale=2):
                 status_output = gr.Textbox(label="Status", lines=3, interactive=False)
-                table_output = gr.Dataframe(label="Extracted records", headers=COLUMNS, wrap=True)
-                csv_output = gr.File(label="Download CSV")
+                fields_table_output = gr.Dataframe(label="Extracted fields (tidy)", headers=FIELDS_COLUMNS, wrap=True)
+                notifications_table_output = gr.Dataframe(
+                    label="Notification hits (tidy)", headers=NOTIFICATIONS_COLUMNS, wrap=True
+                )
+                csv_output = gr.Files(label="Download CSVs (fields, notifications, run metadata)")
+                run_metadata_output = gr.JSON(label="This run's settings (audit record)")
                 with gr.Accordion("Full OCR text (debug)", open=False):
                     text_output = gr.Textbox(label="Raw extracted text", lines=20)
 
-    chat_send.click(
-        fn=handle_chat,
-        inputs=[chatbot, chat_input, settings_state, hf_token_input],
-        outputs=[chatbot, chat_input, settings_state, settings_display,
-                 categories_input, custom_keywords_input, only_matches_input],
-    )
-    chat_input.submit(
-        fn=handle_chat,
-        inputs=[chatbot, chat_input, settings_state, hf_token_input],
-        outputs=[chatbot, chat_input, settings_state, settings_display,
-                 categories_input, custom_keywords_input, only_matches_input],
-    )
+    chat_outputs = [
+        chatbot, chat_input, settings_state, settings_display,
+        categories_input, custom_keywords_input, custom_fields_input, only_matches_input,
+    ]
+    chat_send.click(fn=handle_chat, inputs=[chatbot, chat_input, settings_state, hf_token_input], outputs=chat_outputs)
+    chat_input.submit(fn=handle_chat, inputs=[chatbot, chat_input, settings_state, hf_token_input], outputs=chat_outputs)
 
     run_button.click(
         fn=process_files,
-        inputs=[files_input, categories_input, custom_keywords_input, only_matches_input],
-        outputs=[status_output, table_output, csv_output, text_output],
+        inputs=[files_input, categories_input, custom_keywords_input, custom_fields_input, only_matches_input, chatbot],
+        outputs=[status_output, fields_table_output, notifications_table_output, csv_output, text_output, run_metadata_output],
     )
 
 if __name__ == "__main__":

chat_config.py

@@ -1,6 +1,6 @@
 """Conversational assistant that turns free-form instructions into
-extraction settings (categories, custom keywords, export filter) using a
-Hugging Face-hosted chat model.
+extraction settings (categories, notification keywords, value fields,
+export filter) using a Hugging Face-hosted chat model.
 """
 import json
 import os
@@ -16,38 +16,55 @@ DEFAULT_MODEL = os.environ.get("HF_CHAT_MODEL", "HuggingFaceH4/zephyr-7b-beta")
 
 SYSTEM_PROMPT = f"""You are a configuration assistant for a document OCR/data-extraction tool.
 
-The tool can flag document pages using built-in keyword categories, each with a
-preset list of "notification" phrases for that domain:
-- construction: change order, delay notice, punch list, non-conformance, RFI,
-  submittal rejected, back charge, stop work, safety violation, schedule slip, etc.
-- compliance: non-compliant, violation, deficiency, corrective action, audit
-  finding, expired, recall, citation, penalty, failed inspection, etc.
-- finance: past due, overdue, payment rejected, invoice disputed, credit hold,
-  chargeback, insufficient funds, write-off, collections, late fee, etc.
-- general: urgent, cancelled, rejected, approved, pending approval, on hold.
-
-It always extracts these structured fields wherever present, regardless of
-category: PO numbers, SAP document numbers, invoice numbers, certification
-codes (ISO/OSHA/ASTM/ANSI), dates, dollar amounts, percentages.
-
-The user will describe in plain language what they want flagged or pulled.
-Given their message and the current settings, respond with:
+The tool distinguishes two different kinds of things a user can ask for:
+
+1. NOTIFICATIONS - keyword/phrase hits that flag an event worth a reviewer's
+   attention (a boolean "this happened somewhere on the page"). Built-in
+   category keyword sets:
+   - construction: change order, delay notice, punch list, non-conformance,
+     RFI, submittal rejected, back charge, stop work, safety violation,
+     schedule slip, etc.
+   - compliance: non-compliant, violation, deficiency, corrective action,
+     audit finding, expired, recall, citation, penalty, failed inspection.
+   - finance: past due, overdue, payment rejected, invoice disputed, credit
+     hold, chargeback, insufficient funds, write-off, collections, late fee.
+   - general: urgent, cancelled, rejected, approved, pending approval, on hold.
+   Users can also name their own free-text notification phrases.
+
+2. FIELDS - labeled values to pull out of the document as data, not just
+   flag. Two kinds:
+   - Standard fields, auto-scoped to the categories selected above: PO
+     numbers + SAP document numbers (construction), certification codes
+     ISO/OSHA/ASTM/ANSI (compliance), invoice numbers + percentages
+     (finance), and dates/dollar amounts wherever relevant.
+   - Custom fields: any label the user names that appears in the document
+     followed by a value, e.g. "reference number", "gross weight",
+     "delivery date", "customer PO". These are looked up generically by
+     label text, so use the user's own wording for the label.
+
+The user will describe in plain language what they want flagged vs. pulled
+as data. Given their message and the current settings, respond with:
 1. A short (1-3 sentence) conversational reply confirming what you changed,
    or asking a clarifying question if the request is ambiguous.
 2. On its own line, a fenced json block with the FULL updated settings:
 ```json
-{{"categories": ["finance"], "custom_keywords": ["warranty claim"], "only_matches": true}}
+{{"categories": ["finance"], "custom_keywords": ["warranty claim"], "custom_fields": ["reference number", "gross weight"], "only_matches": true}}
 ```
 
 Rules:
 - "categories" must only contain values from {CATEGORY_CHOICES}.
-- "custom_keywords" is a list of extra free-text phrases/terms to flag that
-  aren't already covered by a category (empty list if none requested).
-- "only_matches" is true if only pages with a hit should be exported, false
-  to export every page regardless of match.
-- Always include all three keys, carrying over prior values the user didn't
-  ask to change.
-- Never invent a category, field, or capability that isn't listed above.
+- "custom_keywords" is a list of extra free-text NOTIFICATION phrases to
+  flag (empty list if none requested).
+- "custom_fields" is a list of label names to extract as VALUES, in the
+  user's own wording (empty list if none requested).
+- "only_matches" is true if only pages with a hit/field should be
+  exported, false to export every page regardless.
+- Always include all four keys, carrying over prior values the user
+  didn't ask to change.
+- If the user names something ambiguous, prefer treating it as a
+  custom_field when they clearly want a value (a number, date, code) back,
+  and as a custom_keyword when they clearly want an alert/flag instead.
+- Never invent a category or capability that isn't listed above.
 """
 
 
@@ -66,6 +83,7 @@ def _extract_json_block(text):
 def _sanitize_settings(parsed, current_settings):
     categories = current_settings.get("categories", [])
     custom_keywords = current_settings.get("custom_keywords", [])
+    custom_fields = current_settings.get("custom_fields", [])
     only_matches = current_settings.get("only_matches", True)
 
     if isinstance(parsed, dict):
@@ -73,12 +91,15 @@ def _sanitize_settings(parsed, current_settings):
             categories = [c for c in parsed["categories"] if c in CATEGORY_CHOICES]
         if isinstance(parsed.get("custom_keywords"), list):
             custom_keywords = [str(k).strip() for k in parsed["custom_keywords"] if str(k).strip()]
+        if isinstance(parsed.get("custom_fields"), list):
+            custom_fields = [str(k).strip() for k in parsed["custom_fields"] if str(k).strip()]
         if isinstance(parsed.get("only_matches"), bool):
             only_matches = parsed["only_matches"]
 
     return {
         "categories": categories,
         "custom_keywords": custom_keywords,
+        "custom_fields": custom_fields,
         "only_matches": only_matches,
     }
 

extractors.py

@@ -61,7 +61,19 @@ FIELD_PATTERNS = {
     ),
 }
 
+# Which standard field patterns are relevant to each domain. Extraction is
+# scoped to the categories a user actually selects rather than running every
+# pattern against every document ("blanket" extraction) - a compliance
+# certificate has no business generating empty invoice_number columns.
+CATEGORY_FIELD_MAP = {
+    "construction": ["po_numbers", "sap_doc_numbers", "dates", "amounts"],
+    "compliance": ["cert_codes", "dates"],
+    "finance": ["invoice_numbers", "amounts", "dates", "percentages"],
+    "general": ["dates"],
+}
+
 CONTEXT_WINDOW = 60
+MAX_CUSTOM_VALUE_LEN = 80
 
 
 def _dedupe(values):
@@ -73,10 +85,25 @@ def _dedupe(values):
     return seen
 
 
-def extract_fields(text):
-    """Pull structured fields out of a page/block of text."""
+def fields_for_categories(categories):
+    """Union of standard field names relevant to the given categories."""
+    names = []
+    for cat in categories:
+        for name in CATEGORY_FIELD_MAP.get(cat, []):
+            if name not in names:
+                names.append(name)
+    return names
+
+
+def extract_fields(text, categories):
+    """Pull the standard fields relevant to `categories` out of a page of text.
+
+    Returns {field_name: [values]} scoped to only the patterns those
+    categories map to - see CATEGORY_FIELD_MAP.
+    """
     fields = {}
-    for name, pattern in FIELD_PATTERNS.items():
+    for name in fields_for_categories(categories):
+        pattern = FIELD_PATTERNS[name]
         matches = pattern.findall(text)
         flat = []
         for m in matches:
@@ -88,6 +115,54 @@ def extract_fields(text):
     return fields
 
 
+def _postprocess_custom_value(label, raw_value):
+    raw_value = raw_value.strip()[:MAX_CUSTOM_VALUE_LEN]
+    if "date" in label:
+        m = FIELD_PATTERNS["dates"].search(raw_value)
+        if m:
+            return m.group(0)
+    if any(k in label for k in ("weight", "amount", "qty", "quantity", "total", "count", "price")):
+        m = re.match(r"[\d,]+\.?\d*\s?[a-zA-Z%]*", raw_value)
+        if m and m.group(0).strip():
+            return m.group(0).strip()
+    # Generic fallback: a handful of tokens, cut before an obvious next column
+    # (two-or-more spaces, which OCR/tables often use as a column separator).
+    trimmed = re.split(r"\s{2,}", raw_value)[0]
+    m = re.match(r"\S+(?:\s\S+){0,4}", trimmed)
+    return m.group(0).strip() if m else trimmed.strip()
+
+
+def extract_custom_fields(text, field_labels):
+    """Generic label->value extraction for user-defined fields.
+
+    For labels like "reference number" or "gross weight", finds the label in
+    the text and captures whatever follows it on the same line (or the next
+    line, if the label sits alone on its own line - common in OCR'd forms).
+    Returns {label: [values]}.
+    """
+    results = {label: [] for label in field_labels if label.strip()}
+    if not results:
+        return results
+
+    lines = text.splitlines()
+    for i, line in enumerate(lines):
+        lower_line = line.lower()
+        for label in results:
+            label_clean = label.strip().lower()
+            idx = lower_line.find(label_clean)
+            if idx == -1:
+                continue
+            after = line[idx + len(label_clean):]
+            after = re.sub(r"^[\s:.\-–—]+", "", after).strip()
+            if not after and i + 1 < len(lines):
+                after = lines[i + 1].strip()
+            if after:
+                value = _postprocess_custom_value(label_clean, after)
+                if value and value not in results[label]:
+                    results[label].append(value)
+    return results
+
+
 def find_notifications(text, categories, custom_keywords=None):
     """Scan text for keyword hits across the given categories.