Add model card

README.md

@@ -0,0 +1,114 @@
+---
+license: other
+license_name: mixed-upstream-see-card
+library_name: mlx
+pipeline_tag: text-to-image
+tags:
+  - instantid
+  - sdxl
+  - mlx
+  - apple-silicon
+  - face-id
+  - controlnet
+  - ip-adapter
+  - identity-preservation
+---
+
+# SceneWorks/instantid-mlx
+
+Converted weights for running **InstantID** identity-preserving SDXL **natively on Apple Silicon
+with MLX** — zero Python at inference time. These are the three artifacts the
+[`mlx-gen-instantid`](https://github.com/michaeltrefry/mlx-gen) provider loads to compose InstantID
+out of the SDXL backbone + the native MLX face stack (`mlx-gen-face`).
+
+This repo holds **only the InstantID-specific glue weights**. The SDXL base (e.g.
+`SG161222/RealVisXL_V5.0` or `stabilityai/stable-diffusion-xl-base-1.0`), the IdentityNet
+ControlNet (`InstantX/InstantID` → `ControlNetModel/`), and the OpenPose ControlNet for pose mode
+(`xinsir/controlnet-openpose-sdxl-1.0`) are loaded directly from their own diffusers repos — no
+conversion needed for those.
+
+## Files
+
+| File | Size | What it is | Source | Converter |
+|---|---|---|---|---|
+| `ip-adapter.safetensors` | 1.57 GB | The InstantID face **IP-Adapter**: the image-projection **Resampler** (`image_proj.*`, ArcFace 512-d → 16×2048 face tokens) + the 70 decoupled cross-attention **K/V pairs** (`ip_adapter.*`). Re-serialized from the upstream torch **pickle** `ip-adapter.bin` into safetensors (MLX's loader reads safetensors, not pickle). | [`InstantX/InstantID`](https://huggingface.co/InstantX/InstantID) → `ip-adapter.bin` | `tools/convert_instantid.py` |
+| `scrfd_10g.safetensors` | 16 MB | **SCRFD** 5-point face detector (bbox + landmarks) — the detection half of the native face stack. Ported from the insightface `antelopev2` `scrfd_10g_bnkps` ONNX graph. | insightface `antelopev2` (`scrfd_10g_bnkps.onnx`) | `tools/convert_scrfd.py` |
+| `arcface_iresnet100.safetensors` | 248 MB | **ArcFace** `iresnet100` 512-d recognition embedder — the identity-fidelity half. Ported from the insightface `antelopev2` `glintr100` ONNX graph. | insightface `antelopev2` (`glintr100.onnx`) | `tools/convert_glintr100.py` |
+
+### Checksums (sha256)
+
+```
+fa5608b6121ffaa40228e76ac96e10f56e39b3aba2f6c4905ff7ef9046391c29  ip-adapter.safetensors
+7b40147a85771139e70a8d9fe6be27ffcf32f4c911770ef24b5b05c29f534eda  scrfd_10g.safetensors
+9deff2fef8fe1b3e357a99c01f28cc478dd8acbeab0d3749d252f6d69990ee39  arcface_iresnet100.safetensors
+```
+
+## Usage
+
+### In `mlx-gen-instantid` (Rust / MLX)
+
+```rust
+use mlx_gen::weights::Weights;
+use mlx_gen::WeightsSource;
+use mlx_gen_instantid::{InstantId, InstantIdPaths, InstantIdRequest};
+
+let model = InstantId::load(&InstantIdPaths {
+    sdxl_base:   "/path/to/RealVisXL_V5.0".into(),          // diffusers SDXL snapshot
+    identitynet: WeightsSource::Dir("/path/to/InstantX--InstantID/ControlNetModel".into()),
+    ip_adapter:  "ip-adapter.safetensors".into(),           // <- from this repo
+})?
+.with_face(
+    &Weights::from_file("scrfd_10g.safetensors")?,          // <- from this repo
+    &Weights::from_file("arcface_iresnet100.safetensors")?, // <- from this repo
+)?;
+
+let out = model.generate(&InstantIdRequest { /* prompt, w/h, steps, guidance, scales, seed */ ..Default::default() }, &reference_image)?;
+```
+
+For pose mode add `.with_openpose(&WeightsSource::Dir("/path/to/xinsir--controlnet-openpose-sdxl-1.0".into()))?`
+and call `generate_pose(req, &reference, &keypoints)`; for the ADetailer-style face-restore pass
+call `restore_face(req, &base, &reference_embedding)`.
+
+### In SceneWorks (download-on-first-use)
+
+The SceneWorks Rust GPU worker fetches these three files from this repo on first use into its app
+cache (mirroring the `SceneWorks/yolo11m-person-detect-mlx` and `SceneWorks/sam2-mlx` pattern). You
+can pre-stage them with the env override `SCENEWORKS_INSTANTID_WEIGHTS=/dir/with/the/three/files`.
+
+### Validation (real-weight, MLX, RealVisXL_V5.0 @ 1024²/30, fp16)
+
+| Mode | Metric | Result |
+|---|---|---|
+| Single identity (`generate`) | ArcFace-cosine(ref, generated) | **0.8731** (torch baseline ≈ 0.876) |
+| Angle set (`generate_angle`, three-quarter right) | ArcFace-cosine | **0.8343** |
+| Pose mode (`generate_pose`, full-body) | ArcFace-cosine | **0.7129** (small full-body face) |
+| Face-restore (`restore_face`) | ArcFace-cosine | base 0.7370 → **0.8338** |
+
+## Reproducing the conversion
+
+All three converters live in [`mlx-gen/tools/`](https://github.com/michaeltrefry/mlx-gen/tree/main/tools)
+and run in a torch venv (torch + safetensors; insightface for SCRFD/ArcFace ONNX import):
+
+```bash
+python tools/convert_instantid.py    # InstantX/InstantID ip-adapter.bin  -> ip-adapter.safetensors
+python tools/convert_scrfd.py        # antelopev2 scrfd_10g_bnkps.onnx     -> scrfd_10g.safetensors
+python tools/convert_glintr100.py    # antelopev2 glintr100.onnx           -> arcface_iresnet100.safetensors
+```
+
+## Provenance & licensing
+
+These are **format conversions** of third-party weights; the upstream licenses govern use. Verify
+you comply with each before using them:
+
+- **`ip-adapter.safetensors`** — derived from [`InstantX/InstantID`](https://huggingface.co/InstantX/InstantID)
+  (Apache-2.0). InstantID research: *"InstantID: Zero-shot Identity-Preserving Generation in Seconds"*
+  (Wang et al., 2024).
+- **`scrfd_10g.safetensors`** and **`arcface_iresnet100.safetensors`** — derived from the
+  [InsightFace](https://github.com/deepinsight/insightface) `antelopev2` model pack (`scrfd_10g_bnkps`
+  + `glintr100`). **The InsightFace pretrained models are released for non-commercial research
+  purposes only** — see the InsightFace repository for their terms. Do not use these two files in a
+  commercial setting without securing appropriate rights from the upstream authors.
+
+`license: other` reflects this mix; this card is the authoritative license statement. No additional
+license is granted by the conversion. Conversions produced by the
+[`mlx-gen`](https://github.com/michaeltrefry/mlx-gen) tooling (Apache-2.0 code).