Publish KoharuLayout RF-DETR Seg 2XL 1152 (SafeTensors)

README.md

@@ -0,0 +1,182 @@
+---
+license: other
+library_name: rfdetr
+pipeline_tag: image-segmentation
+datasets:
+  - mayocream/manga109-segmentation
+language:
+  - ja
+tags:
+  - manga
+  - comics
+  - rf-detr
+  - instance-segmentation
+  - object-detection
+  - layout-analysis
+  - text-detection
+---
+
+# KoharuLayout-RFDETR-Seg-2XL-1152
+
+KoharuLayout is a high-resolution RF-DETR Seg 2XL model for manga page layout
+analysis. It predicts bounding boxes and instance masks for four classes:
+
+| ID | Class | Meaning |
+|---:|---|---|
+| 0 | `text` | Dialogue, captions, titles, credits, and other non-COO text |
+| 1 | `onomatopoeia` | Comic onomatopoeia and sound effects (COO) |
+| 2 | `bubble` | Speech and text balloons |
+| 3 | `panel` | Manga panels and frames |
+
+The model does **not** perform OCR or reading-order prediction.
+
+## Files
+
+- `model.safetensors` — RF-DETR inference weights (SafeTensors only)
+- `load_model.py` — strict RF-DETR Seg 2XL loader
+- `inference_config.json` — class mapping and recommended inference settings
+- `validation_metrics.json` — final held-out validation results
+
+The deployment weights contain only RF-DETR tensors. The auxiliary dense
+typography branch used during fine-tuning has been removed because it is not
+part of RF-DETR inference.
+
+## Usage
+
+```bash
+pip install "rfdetr==1.7.0" "safetensors>=0.5" huggingface_hub pillow
+```
+
+```python
+from huggingface_hub import hf_hub_download
+from PIL import Image
+import importlib.util
+
+weights = hf_hub_download(
+    repo_id="mayocream/koharu-layout-rfdetr-seg-2xl-1152",
+    filename="model.safetensors",
+)
+loader_path = hf_hub_download(
+    repo_id="mayocream/koharu-layout-rfdetr-seg-2xl-1152",
+    filename="load_model.py",
+)
+spec = importlib.util.spec_from_file_location("koharu_layout_loader", loader_path)
+loader = importlib.util.module_from_spec(spec)
+spec.loader.exec_module(loader)
+model = loader.load_model(weights)
+
+image = Image.open("page.jpg").convert("RGB")
+detections = model.predict(
+    image,
+    threshold=0.25,
+    shape=(1152, 1152),
+    include_source_image=False,
+)
+
+print(detections.xyxy)       # bounding boxes
+print(detections.mask)       # instance masks
+print(detections.class_id)   # 0=text, 1=COO, 2=bubble, 3=panel
+print(detections.confidence)
+```
+
+CUDA is strongly recommended. The model was trained and evaluated at 1152 px.
+
+## Recommended thresholds
+
+A single `0.25` threshold maximizes recall, but class-specific filtering is
+usually better:
+
+| Class | Suggested threshold |
+|---|---:|
+| Text | 0.25–0.30 |
+| COO | 0.40–0.50 |
+| Bubble | 0.50 |
+| Panel | 0.50 |
+
+The lower text threshold helps retain titles, credits, and back matter. Highly
+decorative pages may require tiling or a separate fallback detector.
+
+## Validation results
+
+The final checkpoint was evaluated on the held-out 1,070-page Manga109 test
+split at 1152 px.
+
+| Metric | Score |
+|---|---:|
+| Box mAP50–95 | 0.7929 |
+| Box mAP50 | 0.8941 |
+| Box mAP75 | 0.8373 |
+| Mask mAP50–95 | 0.5187 |
+| Mask mAP50 | 0.7206 |
+| Detection precision | 0.8061 |
+| Detection recall | 0.7452 |
+| Detection F1 | 0.7672 |
+
+Per-class box AP:
+
+| Class | AP |
+|---|---:|
+| Text | 0.8762 |
+| COO | 0.4285 |
+| Bubble | 0.9094 |
+| Panel | 0.9575 |
+
+## Out-of-domain review
+
+The model was also run qualitatively on 212 full-color Blue Archive comic pages
+and 79 monochrome Marriage Toxin pages. These folders have no ground-truth
+annotations, so the observations below are visual rather than accuracy claims.
+
+- Dialogue text, bubbles, and conventional panel layouts generalized well.
+- Marriage Toxin dialogue, bubbles, and panels were particularly consistent.
+- Large isolated COO was often detected, but smaller effects were fragmented or
+  confused with ordinary text.
+- Covers, logos, credits, and dense collage layouts remain difficult.
+- A highly decorative collage page produced 140 low-confidence predictions,
+  approaching the configured 160-candidate limit.
+
+## Training
+
+- Architecture: RF-DETR Seg 2XL (`rfdetr==1.7.0`)
+- Resolution: 1152 × 1152
+- Main supervision: Manga109 Segmentation v1.1.0
+- Classes: text, onomatopoeia, bubble, panel
+- Final continuation stage: 6 epochs
+- Final-stage global batch: 32 (8 GPUs × 2 images × 2 accumulation steps)
+- Optimizer learning rate: `3e-5`; encoder learning rate: `1e-5`
+- EMA enabled; seed 42
+- Additional normal-text supervision was distilled from PP-DocLayoutV3
+
+The auxiliary typography head used for training improved mask recall but did
+not materially improve box AP. It is intentionally excluded from this
+deployment checkpoint.
+
+## Limitations
+
+- COO is substantially weaker than text, bubble, and panel detection.
+- Large display titles and credits may be missed or assigned low confidence.
+- Character nameplates and small icons can be confused with bubbles.
+- Dense collages can create many duplicate or low-confidence instances.
+- The model has no OCR, semantic reading order, or text-to-bubble relationship
+  output.
+- The training distribution is predominantly Japanese manga. Results on other
+  comic styles and languages may differ.
+
+## License and training-data terms
+
+The RF-DETR software is distributed under Apache-2.0. The model was fine-tuned
+using Manga109 images, which are distributed separately under Manga109's
+academic-use terms. This repository does not redistribute Manga109 images.
+
+The model repository therefore uses a custom/`other` license designation.
+Users are responsible for obtaining Manga109 and complying with its terms and
+all applicable rights. This model card does not grant rights to Manga109 source
+images or to third-party comic content.
+
+## Weight integrity
+
+SHA-256:
+
+```text
+0d730ea52064051f3a1669bd146a2625d511b78072f1c176e200f70cbdb9042c
+```

inference_config.json

@@ -0,0 +1,20 @@
+{
+  "architecture": "RFDETRSeg2XLarge",
+  "rfdetr_version": "1.7.0",
+  "resolution": 1152,
+  "num_select": 160,
+  "classes": {
+    "0": "text",
+    "1": "onomatopoeia",
+    "2": "bubble",
+    "3": "panel"
+  },
+  "recommended_thresholds": {
+    "text": 0.25,
+    "onomatopoeia": 0.4,
+    "bubble": 0.5,
+    "panel": 0.5
+  },
+  "checkpoint": "model.safetensors",
+  "checkpoint_sha256": "0d730ea52064051f3a1669bd146a2625d511b78072f1c176e200f70cbdb9042c"
+}

load_model.py

@@ -0,0 +1,29 @@
+"""SafeTensors-only loader for KoharuLayout RF-DETR Seg 2XL."""
+
+from __future__ import annotations
+
+from pathlib import Path
+import warnings
+
+from rfdetr import RFDETRSeg2XLarge
+from rfdetr.config import PretrainWeightsCompatibilityWarning
+from safetensors.torch import load_file
+
+
+CLASS_NAMES = ["text", "onomatopoeia", "bubble", "panel"]
+
+
+def load_model(weights: str | Path) -> RFDETRSeg2XLarge:
+    with warnings.catch_warnings():
+        warnings.simplefilter("ignore", PretrainWeightsCompatibilityWarning)
+        model = RFDETRSeg2XLarge(
+            pretrain_weights=None,
+            resolution=1152,
+            num_select=160,
+            num_classes=len(CLASS_NAMES),
+        )
+    incompatible = model.model.model.load_state_dict(load_file(str(weights), device="cpu"), strict=True)
+    if incompatible.missing_keys or incompatible.unexpected_keys:
+        raise RuntimeError(f"Incompatible weights: {incompatible}")
+    model.model.class_names = CLASS_NAMES.copy()
+    return model

model.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:0d730ea52064051f3a1669bd146a2625d511b78072f1c176e200f70cbdb9042c
+size 161292684

requirements.txt

@@ -0,0 +1,4 @@
+rfdetr==1.7.0
+safetensors>=0.5
+huggingface_hub>=1.0
+pillow

validation_metrics.json

@@ -0,0 +1,24 @@
+{
+  "dataset": "Manga109 Segmentation v1.1.0 held-out test split",
+  "pages": 1070,
+  "resolution": 1152,
+  "checkpoint_epoch": 5,
+  "metrics": {
+    "box_map_50_95": 0.79288250207901,
+    "box_map_50": 0.8940883278846741,
+    "box_map_75": 0.8372876644134521,
+    "box_mar": 0.850690484046936,
+    "precision": 0.806089460849762,
+    "recall": 0.745160698890686,
+    "f1": 0.767235279083252,
+    "mask_map_50_95": 0.5187143087387085,
+    "mask_map_50": 0.7205575704574585,
+    "box_ap_by_class": {
+      "text": 0.8761724829673767,
+      "onomatopoeia": 0.4284520149230957,
+      "bubble": 0.9094059467315674,
+      "panel": 0.9574995636940002
+    }
+  }
+}
+