maxffarrell/embeddinggemma-2-CoreAI

EmbeddingGemma 2 — Apple Core AI

Apple Core AI .aimodel conversion of google/embeddinggemma-2, revision 914f7f89142e33e77833254d9c9b90c3cef7303b, Apache-2.0. This is the new 740M multimodal model, distinct from EmbeddingGemma 300M.

Supports text/code, images, video frames, audio, and interleaved combinations within the limits below. Returns normalized 768-dimensional embeddings. All neural inference, projection, masked mean pooling and initial L2 normalization run in Core AI. The host uses Transformers for tokenization/media preprocessing and NumPy for feature placement and optional truncated-vector normalization. No source PyTorch checkpoint is needed for inference.

Limits and platform

• Batch size one; 1,024 combined text and media tokens, including special tokens. The source supports 8,192; this conversion is a fixed smaller export. Oversized inputs raise an error instead of being truncated.
• Vision: 2,520 patches per image/frame, at most 280 soft tokens. Preserves the upstream aspect-ratio-aware image processor and padding. Each image/frame is processed sequentially; total context capacity still applies.
• Audio: one mono 16 kHz clip per encode call, maximum 11.2 seconds (1,120 mel frames / 280 soft tokens). No resampling in the adapter. Longer clips must be handled separately by the caller; this does not implement the source's minute-scale audio context.
• Video accepts an already sampled NumPy [frames,height,width,channels] array. Validation used two frames. Decode and sample video files before calling the adapter; no file decoding, temporal sampling or audio-track extraction is implemented by this adapter. The upstream per-frame image processing is preserved.
• Text task prefixes are preserved via prompt_name, e.g. SearchQuery, Document, CodeRetrieval. For mixed content, manually supply placeholder text (<|image|>, <|audio|>, <|video|>) in the desired order and omit prompt_name.
• Optional Matryoshka truncation re-normalizes the vector; dimensions 128, 256 and 512 were exercised.
• FP32 inference with the source weights promoted from BF16; assets total approximately 3 GB. Tested on macOS 27.0.1 / Apple Silicon with default Core AI specialization. No independent iOS, ANE or GPU placement claim.
• Experimental community conversion. Numerical fixtures are conversion tests, not a retrieval-quality benchmark.

Download and run

hf download maxffarrell/embeddinggemma-2-CoreAI --local-dir embeddinggemma-2-CoreAI
cd embeddinggemma-2-CoreAI
python3 -m venv .venv
source .venv/bin/activate
pip install -r requirements.txt

import asyncio
import numpy as np
from PIL import Image
from coreai_embeddinggemma2 import EmbeddingGemma2CoreAI

async def main():
    model = await EmbeddingGemma2CoreAI.load(".")
    query = await model.encode("What causes the northern lights?", prompt_name="SearchQuery")
    document = await model.encode("Charged particles from the sun cause auroras.", prompt_name="Document")
    print(float(query @ document))
    image = await model.encode(images=Image.open("photo.jpg"))
    audio = await model.encode(audio=np.zeros(16000, dtype=np.float32), sampling_rate=16000)
    small = await model.encode("An example sentence", truncate_dim=256)

asyncio.run(main())

Validation

validation.json: 21 cases against the full pinned PyTorch FP32 model with native-length source inputs, testing task prefixes, code, Spanish, Japanese, short/long/near-capacity text, images, mixed text/images, two images, short/long/limit audio, text/audio, video, text/video and all four modalities together. Worst elementwise error: 9.76026058e-07; minimum cosine similarity: 0.999999881. Predeclared final limits: cosine > 0.99999 and maximum elementwise error < 0.00005, plus finite unit vectors.

smoke.json: separate source-checkpoint-free native runtime smoke in five text/media cases with PyTorch neural model loading explicitly disabled, plus invalid-input rejection. Task-independent retrieval accuracy has not been benchmarked.

Reproduce

From a parent directory containing this bundle under bundles/embeddinggemma-2-CoreAI, download the pinned source into upstream/embeddinggemma-2, install requirements.txt, and run convert_gemma2.py, validate_gemma2.py, then smoke_gemma2.py. The validation/smoke scripts default to that directory layout. The converter uses the exact official Transformers commit in requirements because stable 5.17 predates support. coreai-torch 0.4.3 warns it was validated through PyTorch 2.13; this export was tested on 2.14.

Graphs: embedding.aimodel (text backbone + projection + pooling + normalization), vision.aimodel (vision encoder + projection + valid-token mask), audio.aimodel (audio encoder + projection + valid-token mask). Integer inputs are int32. manifest.json declares signatures, static shapes and output names. An equivalent slice/stack decomposition replaces unsupported aten.unfold in audio; the source weights are unchanged. The text mask preserves the source's inclusive bidirectional sliding window.

SOURCE.json records the verified upstream weight hash; SHA256SUMS covers every release file except itself. Original model documentation is in UPSTREAM_README.md. For Clef Flash, an existing Core AI conversion was found and verified to use current source weight hashes; see RELATED_MODEL_RESEARCH.md.