ds-EkaCare commited on
Commit
cc6acca
Β·
verified Β·
1 Parent(s): ca2525b

Lean README: three-task framing (class + medical + handwritten), remove quality head docs

Browse files
Files changed (1) hide show
  1. README.md +36 -78
README.md CHANGED
@@ -9,45 +9,29 @@ tags:
9
  library_name: transformers
10
  ---
11
 
12
- # Hierarchical SigLIP2 Document Classifier (~97M)
13
 
14
- A **frozen SigLIP2 base** vision tower (`google/siglip2-base-patch16-naflex`, single
15
- high-res forward at `max_num_patches=1024`) plus lightweight heads that classify a
16
- document image across several tasks from one shared pooled embedding:
17
 
18
- | Task | Output |
19
- |------|--------|
20
- | `l2` (flat, **default**) | leaf class β€” flat 27-way head; **L1 inferred** from it |
21
- | `l1` | document group β€” one of 5 (Document, Miscellaneous, Diagnostic imaging, Card / credential, Body images) |
22
- | `l2` (leaf) | leaf class via a per-L1 head, when the group is fixed/predicted |
23
- | `medical` | sigmoid β€” `medical` / `non_medical` (+ `p_positive`) |
24
- | `handwritten` | sigmoid β€” `handwritten` / `printed` (+ `p_positive`); tuned on documents |
25
- | `quality` | sigmoid β€” a single **`score` in 1–100** (P(good) mapped onto 1–100; higher = better), no label |
26
 
27
- ## L2 has two paths, selected by `scope`
28
 
29
- There is **both** a flat 27-way L2 head (the backbone's own end-to-end classifier,
30
- run on the raw pooled feature) **and** a per-L1 leaf head per group (run on the
31
- standardized embedding). You choose how L2 / L1 are produced:
32
 
33
- | `scope` | what runs | L1 | L2 |
34
- |---------|-----------|----|----|
35
- | `"flat"` *(default)* | flat 27-way head | **inferred** from the L2 prediction | flat |
36
- | `"l1"` | L1 group head only | predicted | β€” (not produced) |
37
- | `"hierarchical"` | L1 head β†’ that group's leaf head | predicted | leaf |
38
 
39
- **Fix the group yourself** by passing `l1="Document"` (any group name): L2 is then
40
- read from that group's leaf head, overriding the scope's routing.
 
 
 
41
 
42
- ```
43
- image β†’ frozen SigLIP2 tower β†’ pooled embedding ──► flat L2 head ─► L2 (+ inferred L1) [scope="flat"]
44
- β”‚
45
- β”œβ”€ standardize (mu/sd) β†’ L1 head ─► L1 [scope="l1"]
46
- β”‚ └─► leaf L2 head[L1] ─► L2 [scope="hierarchical" / fixed l1]
47
- └─ medical Β· handwritten Β· quality (sigmoid heads, independent)
48
- ```
49
 
50
- ### Taxonomy (5 L1 groups β†’ 27 L2 leaves)
51
 
52
  ```
53
  Document (10 leaves)
@@ -61,27 +45,27 @@ Document (10 leaves)
61
  β”œβ”€ Form (key: form)
62
  β”œβ”€ Invoice / bill (key: invoice_bill)
63
  β”œβ”€ Certificate (key: certificate)
64
- Miscellaneous (2 leaves)
65
- β”œβ”€ Other (Miscellaneous) (key: other_miscellaneous)
66
- β”œβ”€ Medication image (key: medication_image)
67
- Diagnostic imaging (5 leaves)
68
- β”œβ”€ Ultrasound (key: ultrasound)
69
- β”œβ”€ X-ray (key: x_ray)
70
- β”œβ”€ MRI (key: mri)
71
- β”œβ”€ CT (key: ct)
72
- β”œβ”€ Other (Diagnostic imaging) (key: other_diagnostic_imaging)
73
  Card / credential (7 leaves)
74
  β”œβ”€ ABHA card (key: abha_card)
75
- β”œβ”€ Voter ID (key: voter_id)
76
  β”œβ”€ Aadhaar card (key: aadhaar_card)
77
- β”œβ”€ Insurance e-card (key: insurance_e_card)
78
  β”œβ”€ PAN card (key: pan_card)
 
79
  β”œβ”€ PMJAY / Ayushman (key: pmjay_ayushman)
 
80
  β”œβ”€ Other card (key: other_card)
 
 
 
 
 
 
81
  Body images (3 leaves)
82
- β”œβ”€ Headshot (key: headshot)
83
  β”œοΏ½οΏ½ Skin / wound (key: skin_wound)
 
84
  β”œβ”€ Other (Body images) (key: other_body_images)
 
 
 
85
  ```
86
 
87
  ## Usage
@@ -93,46 +77,34 @@ from PIL import Image
93
  model = AutoModel.from_pretrained("ekacare/med-doc-classifier", trust_remote_code=True).eval()
94
  img = Image.open("doc.jpg")
95
 
96
- model.classify(img) # default: flat L2 + inferred L1 + binaries
97
  ```
98
 
99
  ```jsonc
100
- // example return value (scope="flat")
101
  {
102
  "l2": { "source": "flat", "key": "lab_diagnostic_report",
103
  "value": "Lab / diagnostic report", "confidence": 0.97 },
104
  "l1": { "value": "Document", "source": "inferred_from_flat_l2" },
105
  "medical": { "value": "medical", "confidence": 0.99, "p_positive": 0.99 },
106
- "handwritten": { "value": "printed", "confidence": 0.98, "p_positive": 0.02 },
107
- "quality": { "score": 94, "p_good": 0.94 }
108
  }
109
  ```
110
 
111
- ### Choosing the L2 scope
112
 
113
  ```python
114
- model.classify(img) # scope="flat" (default): flat L2, L1 inferred
115
  model.classify(img, scope="l1") # L1 group only (no L2)
116
- model.classify(img, scope="hierarchical") # L1 head β†’ leaf L2
117
- model.classify(img, l1="Document") # FIX L1 = Document β†’ that group's leaf L2
118
- model.classify(img, medical=False, handwritten=False, quality=False) # L2/L1 only
119
- ```
120
-
121
- `medical`, `handwritten`, `quality` are independent sigmoid heads β€” toggle them
122
- freely. `quality` reports a single **`score` in 1–100** (the sigmoid P(good) mapped
123
- onto the range; higher = better quality) plus the raw `p_good` β€” there is no
124
- poor/good label. `handwritten` is trained on document images.
125
-
126
- ### Top-k
127
-
128
- ```python
129
  model.classify(img, top_k=3) # top-3 candidates for the multi-class heads
130
  ```
131
 
 
 
132
  ## Quantized loading (optimum-quanto β€” CPU & GPU)
133
 
134
- int8 **and int4** run on CPU as well as GPU. Only the vision tower is quantized;
135
- `patch_embedding` and the heads stay full precision.
136
 
137
  ```python
138
  # pip install optimum-quanto
@@ -141,17 +113,3 @@ from modeling_siglip2_hier import load_classifier # downloaded by trust_remote
141
  model = load_classifier("ekacare/med-doc-classifier", quantization="int4") # None | "int8" | "int4"
142
  model.classify(img)
143
  ```
144
-
145
- Flat-L2 accuracy by precision (vision tower quantized; full internal test = 16,204
146
- images, external OOD = 953; see `benchmark_results.json`):
147
-
148
- | precision | internal acc / mF1 (cluster) | external acc / mF1 (cluster) |
149
- |-----------|------------------------------|------------------------------|
150
- | bf16 | 96.45 / 0.965 (97.36) | 80.90 / 0.702 (90.95) |
151
- | int8 | 96.43 / 0.965 (97.36) | 80.90 / 0.653 (91.15) |
152
- | int4 | 95.22 / 0.955 (94.23) | 77.23 / 0.630 (85.60) |
153
-
154
- **int8 is effectively lossless on accuracy** (internal βˆ’0.02, external acc identical;
155
- only the external macro-F1 dips as rare classes lose a little balance). **int4** trades
156
- ~1.2pt internal / ~3.7pt external for the smallest footprint. Base weights ship in bf16
157
- (~193 MB); int8/int4 quantize the vision tower only (heads + `patch_embedding` stay fp).
 
9
  library_name: transformers
10
  ---
11
 
12
+ # Eka Doc Classifier (~96M)
13
 
14
+ A compact vision model for classifying health-app uploads β€” prescriptions shot at an angle, X-rays, Aadhaar cards, discharge summaries, the occasional accidental selfie. A frozen **SigLIP2-base** tower (`google/siglip2-base-patch16-naflex`) with lightweight heads answers three questions about an image in a single pass:
 
 
15
 
16
+ - **What it is** β€” a two-level label: 5 groups (L1) β†’ 27 leaf classes (L2)
17
+ - **Is it medical?** β€” binary head
18
+ - **Is it handwritten?** β€” binary head
 
 
 
 
 
19
 
20
+ The model was built by training many specialised models, ensembling them, and distilling the ensemble into this one compact student.
21
 
22
+ ## Results
 
 
23
 
24
+ Full internal test = 16,204 images (see `benchmark_results.json`).
 
 
 
 
25
 
26
+ | Precision | Size | Accuracy (27-class) | Macro-F1 |
27
+ |-----------|------|---------------------|----------|
28
+ | bf16 | 193 MB | 96.45 | 0.965 |
29
+ | int8 | 103 MB | 96.43 | 0.965 |
30
+ | int4 | 60 MB | 95.22 | 0.955 |
31
 
32
+ int8 is effectively lossless; int4 trades about a point of accuracy for a model over 3Γ— smaller that runs comfortably on a plain CPU. It matches or beats models several times its size, including a 428M SigLIP2-SO400M variant (96.24%) and a 256M generative VLM baseline (90.42%).
 
 
 
 
 
 
33
 
34
+ ## Taxonomy (5 L1 groups β†’ 27 L2 leaves)
35
 
36
  ```
37
  Document (10 leaves)
 
45
  β”œβ”€ Form (key: form)
46
  β”œβ”€ Invoice / bill (key: invoice_bill)
47
  β”œβ”€ Certificate (key: certificate)
 
 
 
 
 
 
 
 
 
48
  Card / credential (7 leaves)
49
  β”œβ”€ ABHA card (key: abha_card)
 
50
  β”œβ”€ Aadhaar card (key: aadhaar_card)
 
51
  β”œβ”€ PAN card (key: pan_card)
52
+ β”œβ”€ Voter ID (key: voter_id)
53
  β”œβ”€ PMJAY / Ayushman (key: pmjay_ayushman)
54
+ β”œβ”€ Insurance e-card (key: insurance_e_card)
55
  β”œβ”€ Other card (key: other_card)
56
+ Diagnostic imaging (5 leaves)
57
+ β”œβ”€ X-ray (key: x_ray)
58
+ β”œβ”€ CT (key: ct)
59
+ β”œβ”€ MRI (key: mri)
60
+ β”œβ”€ Ultrasound (key: ultrasound)
61
+ β”œβ”€ Other (Diagnostic imaging) (key: other_diagnostic_imaging)
62
  Body images (3 leaves)
 
63
  β”œοΏ½οΏ½ Skin / wound (key: skin_wound)
64
+ β”œβ”€ Headshot (key: headshot)
65
  β”œβ”€ Other (Body images) (key: other_body_images)
66
+ Miscellaneous (2 leaves)
67
+ β”œβ”€ Medication image (key: medication_image)
68
+ β”œβ”€ Other (Miscellaneous) (key: other_miscellaneous)
69
  ```
70
 
71
  ## Usage
 
77
  model = AutoModel.from_pretrained("ekacare/med-doc-classifier", trust_remote_code=True).eval()
78
  img = Image.open("doc.jpg")
79
 
80
+ model.classify(img)
81
  ```
82
 
83
  ```jsonc
84
+ // example return value
85
  {
86
  "l2": { "source": "flat", "key": "lab_diagnostic_report",
87
  "value": "Lab / diagnostic report", "confidence": 0.97 },
88
  "l1": { "value": "Document", "source": "inferred_from_flat_l2" },
89
  "medical": { "value": "medical", "confidence": 0.99, "p_positive": 0.99 },
90
+ "handwritten": { "value": "printed", "confidence": 0.98, "p_positive": 0.02 }
 
91
  }
92
  ```
93
 
94
+ By default L2 comes from a flat 27-way head and L1 is inferred from it. Other options:
95
 
96
  ```python
 
97
  model.classify(img, scope="l1") # L1 group only (no L2)
98
+ model.classify(img, scope="hierarchical") # L1 head β†’ that group's leaf L2 head
99
+ model.classify(img, l1="Document") # fix L1 β†’ that group's leaf L2 head
 
 
 
 
 
 
 
 
 
 
 
100
  model.classify(img, top_k=3) # top-3 candidates for the multi-class heads
101
  ```
102
 
103
+ `medical` and `handwritten` are independent binary heads β€” disable them with `medical=False` / `handwritten=False`.
104
+
105
  ## Quantized loading (optimum-quanto β€” CPU & GPU)
106
 
107
+ Only the vision tower is quantized; the heads stay full precision.
 
108
 
109
  ```python
110
  # pip install optimum-quanto
 
113
  model = load_classifier("ekacare/med-doc-classifier", quantization="int4") # None | "int8" | "int4"
114
  model.classify(img)
115
  ```