Image Classification
Transformers
Safetensors
siglip2_hier_doc
feature-extraction
siglip2
document-classification
hierarchical
multi-task
custom_code
Instructions to use ekacare/med-doc-classifier with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- Transformers
How to use ekacare/med-doc-classifier with Transformers:
# Use a pipeline as a high-level helper from transformers import pipeline pipe = pipeline("image-classification", model="ekacare/med-doc-classifier", trust_remote_code=True) pipe("https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/hub/parrots.png")# Load model directly from transformers import AutoModel model = AutoModel.from_pretrained("ekacare/med-doc-classifier", trust_remote_code=True, device_map="auto") - Notebooks
- Google Colab
- Kaggle
Lean README: three-task framing (class + medical + handwritten), remove quality head docs
Browse files
README.md
CHANGED
|
@@ -9,45 +9,29 @@ tags:
|
|
| 9 |
library_name: transformers
|
| 10 |
---
|
| 11 |
|
| 12 |
-
#
|
| 13 |
|
| 14 |
-
A
|
| 15 |
-
high-res forward at `max_num_patches=1024`) plus lightweight heads that classify a
|
| 16 |
-
document image across several tasks from one shared pooled embedding:
|
| 17 |
|
| 18 |
-
|
| 19 |
-
|
| 20 |
-
|
| 21 |
-
| `l1` | document group β one of 5 (Document, Miscellaneous, Diagnostic imaging, Card / credential, Body images) |
|
| 22 |
-
| `l2` (leaf) | leaf class via a per-L1 head, when the group is fixed/predicted |
|
| 23 |
-
| `medical` | sigmoid β `medical` / `non_medical` (+ `p_positive`) |
|
| 24 |
-
| `handwritten` | sigmoid β `handwritten` / `printed` (+ `p_positive`); tuned on documents |
|
| 25 |
-
| `quality` | sigmoid β a single **`score` in 1β100** (P(good) mapped onto 1β100; higher = better), no label |
|
| 26 |
|
| 27 |
-
|
| 28 |
|
| 29 |
-
|
| 30 |
-
run on the raw pooled feature) **and** a per-L1 leaf head per group (run on the
|
| 31 |
-
standardized embedding). You choose how L2 / L1 are produced:
|
| 32 |
|
| 33 |
-
|
| 34 |
-
|---------|-----------|----|----|
|
| 35 |
-
| `"flat"` *(default)* | flat 27-way head | **inferred** from the L2 prediction | flat |
|
| 36 |
-
| `"l1"` | L1 group head only | predicted | β (not produced) |
|
| 37 |
-
| `"hierarchical"` | L1 head β that group's leaf head | predicted | leaf |
|
| 38 |
|
| 39 |
-
|
| 40 |
-
|
|
|
|
|
|
|
|
|
|
| 41 |
|
| 42 |
-
|
| 43 |
-
image β frozen SigLIP2 tower β pooled embedding βββΊ flat L2 head ββΊ L2 (+ inferred L1) [scope="flat"]
|
| 44 |
-
β
|
| 45 |
-
ββ standardize (mu/sd) β L1 head ββΊ L1 [scope="l1"]
|
| 46 |
-
β βββΊ leaf L2 head[L1] ββΊ L2 [scope="hierarchical" / fixed l1]
|
| 47 |
-
ββ medical Β· handwritten Β· quality (sigmoid heads, independent)
|
| 48 |
-
```
|
| 49 |
|
| 50 |
-
##
|
| 51 |
|
| 52 |
```
|
| 53 |
Document (10 leaves)
|
|
@@ -61,27 +45,27 @@ Document (10 leaves)
|
|
| 61 |
ββ Form (key: form)
|
| 62 |
ββ Invoice / bill (key: invoice_bill)
|
| 63 |
ββ Certificate (key: certificate)
|
| 64 |
-
Miscellaneous (2 leaves)
|
| 65 |
-
ββ Other (Miscellaneous) (key: other_miscellaneous)
|
| 66 |
-
ββ Medication image (key: medication_image)
|
| 67 |
-
Diagnostic imaging (5 leaves)
|
| 68 |
-
ββ Ultrasound (key: ultrasound)
|
| 69 |
-
ββ X-ray (key: x_ray)
|
| 70 |
-
ββ MRI (key: mri)
|
| 71 |
-
ββ CT (key: ct)
|
| 72 |
-
ββ Other (Diagnostic imaging) (key: other_diagnostic_imaging)
|
| 73 |
Card / credential (7 leaves)
|
| 74 |
ββ ABHA card (key: abha_card)
|
| 75 |
-
ββ Voter ID (key: voter_id)
|
| 76 |
ββ Aadhaar card (key: aadhaar_card)
|
| 77 |
-
ββ Insurance e-card (key: insurance_e_card)
|
| 78 |
ββ PAN card (key: pan_card)
|
|
|
|
| 79 |
ββ PMJAY / Ayushman (key: pmjay_ayushman)
|
|
|
|
| 80 |
ββ Other card (key: other_card)
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 81 |
Body images (3 leaves)
|
| 82 |
-
ββ Headshot (key: headshot)
|
| 83 |
βοΏ½οΏ½ Skin / wound (key: skin_wound)
|
|
|
|
| 84 |
ββ Other (Body images) (key: other_body_images)
|
|
|
|
|
|
|
|
|
|
| 85 |
```
|
| 86 |
|
| 87 |
## Usage
|
|
@@ -93,46 +77,34 @@ from PIL import Image
|
|
| 93 |
model = AutoModel.from_pretrained("ekacare/med-doc-classifier", trust_remote_code=True).eval()
|
| 94 |
img = Image.open("doc.jpg")
|
| 95 |
|
| 96 |
-
model.classify(img)
|
| 97 |
```
|
| 98 |
|
| 99 |
```jsonc
|
| 100 |
-
// example return value
|
| 101 |
{
|
| 102 |
"l2": { "source": "flat", "key": "lab_diagnostic_report",
|
| 103 |
"value": "Lab / diagnostic report", "confidence": 0.97 },
|
| 104 |
"l1": { "value": "Document", "source": "inferred_from_flat_l2" },
|
| 105 |
"medical": { "value": "medical", "confidence": 0.99, "p_positive": 0.99 },
|
| 106 |
-
"handwritten": { "value": "printed", "confidence": 0.98, "p_positive": 0.02 }
|
| 107 |
-
"quality": { "score": 94, "p_good": 0.94 }
|
| 108 |
}
|
| 109 |
```
|
| 110 |
|
| 111 |
-
|
| 112 |
|
| 113 |
```python
|
| 114 |
-
model.classify(img) # scope="flat" (default): flat L2, L1 inferred
|
| 115 |
model.classify(img, scope="l1") # L1 group only (no L2)
|
| 116 |
-
model.classify(img, scope="hierarchical") # L1 head β leaf L2
|
| 117 |
-
model.classify(img, l1="Document")
|
| 118 |
-
model.classify(img, medical=False, handwritten=False, quality=False) # L2/L1 only
|
| 119 |
-
```
|
| 120 |
-
|
| 121 |
-
`medical`, `handwritten`, `quality` are independent sigmoid heads β toggle them
|
| 122 |
-
freely. `quality` reports a single **`score` in 1β100** (the sigmoid P(good) mapped
|
| 123 |
-
onto the range; higher = better quality) plus the raw `p_good` β there is no
|
| 124 |
-
poor/good label. `handwritten` is trained on document images.
|
| 125 |
-
|
| 126 |
-
### Top-k
|
| 127 |
-
|
| 128 |
-
```python
|
| 129 |
model.classify(img, top_k=3) # top-3 candidates for the multi-class heads
|
| 130 |
```
|
| 131 |
|
|
|
|
|
|
|
| 132 |
## Quantized loading (optimum-quanto β CPU & GPU)
|
| 133 |
|
| 134 |
-
|
| 135 |
-
`patch_embedding` and the heads stay full precision.
|
| 136 |
|
| 137 |
```python
|
| 138 |
# pip install optimum-quanto
|
|
@@ -141,17 +113,3 @@ from modeling_siglip2_hier import load_classifier # downloaded by trust_remote
|
|
| 141 |
model = load_classifier("ekacare/med-doc-classifier", quantization="int4") # None | "int8" | "int4"
|
| 142 |
model.classify(img)
|
| 143 |
```
|
| 144 |
-
|
| 145 |
-
Flat-L2 accuracy by precision (vision tower quantized; full internal test = 16,204
|
| 146 |
-
images, external OOD = 953; see `benchmark_results.json`):
|
| 147 |
-
|
| 148 |
-
| precision | internal acc / mF1 (cluster) | external acc / mF1 (cluster) |
|
| 149 |
-
|-----------|------------------------------|------------------------------|
|
| 150 |
-
| bf16 | 96.45 / 0.965 (97.36) | 80.90 / 0.702 (90.95) |
|
| 151 |
-
| int8 | 96.43 / 0.965 (97.36) | 80.90 / 0.653 (91.15) |
|
| 152 |
-
| int4 | 95.22 / 0.955 (94.23) | 77.23 / 0.630 (85.60) |
|
| 153 |
-
|
| 154 |
-
**int8 is effectively lossless on accuracy** (internal β0.02, external acc identical;
|
| 155 |
-
only the external macro-F1 dips as rare classes lose a little balance). **int4** trades
|
| 156 |
-
~1.2pt internal / ~3.7pt external for the smallest footprint. Base weights ship in bf16
|
| 157 |
-
(~193 MB); int8/int4 quantize the vision tower only (heads + `patch_embedding` stay fp).
|
|
|
|
| 9 |
library_name: transformers
|
| 10 |
---
|
| 11 |
|
| 12 |
+
# Eka Doc Classifier (~96M)
|
| 13 |
|
| 14 |
+
A compact vision model for classifying health-app uploads β prescriptions shot at an angle, X-rays, Aadhaar cards, discharge summaries, the occasional accidental selfie. A frozen **SigLIP2-base** tower (`google/siglip2-base-patch16-naflex`) with lightweight heads answers three questions about an image in a single pass:
|
|
|
|
|
|
|
| 15 |
|
| 16 |
+
- **What it is** β a two-level label: 5 groups (L1) β 27 leaf classes (L2)
|
| 17 |
+
- **Is it medical?** β binary head
|
| 18 |
+
- **Is it handwritten?** β binary head
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 19 |
|
| 20 |
+
The model was built by training many specialised models, ensembling them, and distilling the ensemble into this one compact student.
|
| 21 |
|
| 22 |
+
## Results
|
|
|
|
|
|
|
| 23 |
|
| 24 |
+
Full internal test = 16,204 images (see `benchmark_results.json`).
|
|
|
|
|
|
|
|
|
|
|
|
|
| 25 |
|
| 26 |
+
| Precision | Size | Accuracy (27-class) | Macro-F1 |
|
| 27 |
+
|-----------|------|---------------------|----------|
|
| 28 |
+
| bf16 | 193 MB | 96.45 | 0.965 |
|
| 29 |
+
| int8 | 103 MB | 96.43 | 0.965 |
|
| 30 |
+
| int4 | 60 MB | 95.22 | 0.955 |
|
| 31 |
|
| 32 |
+
int8 is effectively lossless; int4 trades about a point of accuracy for a model over 3Γ smaller that runs comfortably on a plain CPU. It matches or beats models several times its size, including a 428M SigLIP2-SO400M variant (96.24%) and a 256M generative VLM baseline (90.42%).
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 33 |
|
| 34 |
+
## Taxonomy (5 L1 groups β 27 L2 leaves)
|
| 35 |
|
| 36 |
```
|
| 37 |
Document (10 leaves)
|
|
|
|
| 45 |
ββ Form (key: form)
|
| 46 |
ββ Invoice / bill (key: invoice_bill)
|
| 47 |
ββ Certificate (key: certificate)
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 48 |
Card / credential (7 leaves)
|
| 49 |
ββ ABHA card (key: abha_card)
|
|
|
|
| 50 |
ββ Aadhaar card (key: aadhaar_card)
|
|
|
|
| 51 |
ββ PAN card (key: pan_card)
|
| 52 |
+
ββ Voter ID (key: voter_id)
|
| 53 |
ββ PMJAY / Ayushman (key: pmjay_ayushman)
|
| 54 |
+
ββ Insurance e-card (key: insurance_e_card)
|
| 55 |
ββ Other card (key: other_card)
|
| 56 |
+
Diagnostic imaging (5 leaves)
|
| 57 |
+
ββ X-ray (key: x_ray)
|
| 58 |
+
ββ CT (key: ct)
|
| 59 |
+
ββ MRI (key: mri)
|
| 60 |
+
ββ Ultrasound (key: ultrasound)
|
| 61 |
+
ββ Other (Diagnostic imaging) (key: other_diagnostic_imaging)
|
| 62 |
Body images (3 leaves)
|
|
|
|
| 63 |
βοΏ½οΏ½ Skin / wound (key: skin_wound)
|
| 64 |
+
ββ Headshot (key: headshot)
|
| 65 |
ββ Other (Body images) (key: other_body_images)
|
| 66 |
+
Miscellaneous (2 leaves)
|
| 67 |
+
ββ Medication image (key: medication_image)
|
| 68 |
+
ββ Other (Miscellaneous) (key: other_miscellaneous)
|
| 69 |
```
|
| 70 |
|
| 71 |
## Usage
|
|
|
|
| 77 |
model = AutoModel.from_pretrained("ekacare/med-doc-classifier", trust_remote_code=True).eval()
|
| 78 |
img = Image.open("doc.jpg")
|
| 79 |
|
| 80 |
+
model.classify(img)
|
| 81 |
```
|
| 82 |
|
| 83 |
```jsonc
|
| 84 |
+
// example return value
|
| 85 |
{
|
| 86 |
"l2": { "source": "flat", "key": "lab_diagnostic_report",
|
| 87 |
"value": "Lab / diagnostic report", "confidence": 0.97 },
|
| 88 |
"l1": { "value": "Document", "source": "inferred_from_flat_l2" },
|
| 89 |
"medical": { "value": "medical", "confidence": 0.99, "p_positive": 0.99 },
|
| 90 |
+
"handwritten": { "value": "printed", "confidence": 0.98, "p_positive": 0.02 }
|
|
|
|
| 91 |
}
|
| 92 |
```
|
| 93 |
|
| 94 |
+
By default L2 comes from a flat 27-way head and L1 is inferred from it. Other options:
|
| 95 |
|
| 96 |
```python
|
|
|
|
| 97 |
model.classify(img, scope="l1") # L1 group only (no L2)
|
| 98 |
+
model.classify(img, scope="hierarchical") # L1 head β that group's leaf L2 head
|
| 99 |
+
model.classify(img, l1="Document") # fix L1 β that group's leaf L2 head
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 100 |
model.classify(img, top_k=3) # top-3 candidates for the multi-class heads
|
| 101 |
```
|
| 102 |
|
| 103 |
+
`medical` and `handwritten` are independent binary heads β disable them with `medical=False` / `handwritten=False`.
|
| 104 |
+
|
| 105 |
## Quantized loading (optimum-quanto β CPU & GPU)
|
| 106 |
|
| 107 |
+
Only the vision tower is quantized; the heads stay full precision.
|
|
|
|
| 108 |
|
| 109 |
```python
|
| 110 |
# pip install optimum-quanto
|
|
|
|
| 113 |
model = load_classifier("ekacare/med-doc-classifier", quantization="int4") # None | "int8" | "int4"
|
| 114 |
model.classify(img)
|
| 115 |
```
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|