ettin-reranker-400m-v1 — LiteRT

cross-encoder/ettin-reranker-400m-v1 converted to LiteRT (.tflite) for on-device inference. A true cross-encoder reranker on the ettin-400m ModernBERT backbone: the query and the passage go through the model together and the graph returns one relevance score — the highest-accuracy reranking shape, at one invoke per candidate. Fully offline, on CPU.

The full scoring head (CLS pooling → Dense+GELU → LayerNorm → Dense) is inside the graph.

File	Recipe	Signatures	Size	Peak RSS*
ettin-reranker-400m-v1_wi8fc.tflite	int8 dynamic-range	128, 256, 512	413 MB	1469 MiB	recommended
ettin-reranker-400m-v1_fp16.tflite	fp16 weights, float compute	128, 256, 512	797 MB	6379 MiB	desktop only

* load + invoke all three signatures, XNNPACK, 8 threads, M4 Max Mac. XNNPACK expands fp16 weights to fp32 per signature subgraph, so the fp16 file peaks at ~8× its file size — treat it as a desktop artifact.

The score is a raw logit — do not sigmoid

The reference stack pins activation_fn to Identity for this model (config_sentence_transformers.json), so CrossEncoder.predict returns raw logits and the base card's own example scores ([3.6875, 11.6875, 4.75, 9.375]) are raw logits. This graph returns the same raw logit. Applying a sigmoid does not change the ranking (it is monotonic), but it does change the scale — if you want numbers comparable to the base card, use the score as-is.

Host contract

• Input = one (query, passage) pair per invoke, encoded exactly as the tokenizer's pair template: [CLS] query [SEP] passage [SEP] — i.e. tokenizer(query, passage, truncation="longest_first", max_length=S). No token_type_ids.
• Pad with token id 50283 ([PAD]) to the signature length; attention mask 1 on real tokens, 0 on pads.
• Output = float32 [1, 1], the raw relevance logit. Higher is more relevant.
• To rerank N candidates: N invokes, sort by score.

Signatures

Batch-1, right-padded static shapes: input_ids int32 [1, S], attention_mask int32 [1, S], for S in 128 / 256 / 512. Route each pair to the smallest signature that fits; pairs longer than 512 are truncated (longest_first trims the longer of query/passage first). Padding is masked inside the graph, so the same pair returns the same score through every signature (measured below).

Usage (Python)

import numpy as np
from ai_edge_litert.interpreter import Interpreter
from transformers import AutoTokenizer

PAD_ID = 50283
tok = AutoTokenizer.from_pretrained("cross-encoder/ettin-reranker-400m-v1")
it = Interpreter(model_path="ettin-reranker-400m-v1_wi8fc.tflite", num_threads=8)

LENS = sorted(int(n.split("_")[1]) for n in it.get_signature_list())
runners = {s: it.get_signature_runner(f"score_{s}") for s in LENS}

def score(query, passage):
    e = tok(query, passage, truncation="longest_first",
            max_length=LENS[-1])["input_ids"]
    S = next(s for s in LENS if len(e) <= s)
    x = np.full((1, S), PAD_ID, np.int32)
    m = np.zeros((1, S), np.int32)
    x[0, :len(e)] = e
    m[0, :len(e)] = 1
    return float(list(runners[S](input_ids=x, attention_mask=m).values())[0][0, 0])

query = "Which planet is known as the Red Planet?"
passages = [
    "Venus is often called Earth's twin because of its similar size and proximity.",
    "Mars, known for its reddish appearance, is often referred to as the Red Planet.",
    "Jupiter, the largest planet in our solar system, has a prominent red spot.",
    "Saturn, famous for its rings, is sometimes mistaken for the Red Planet.",
]
ranked = sorted(passages, key=lambda p: -score(query, p))

Quality

Checks against the PyTorch fp32 reference.

1. The base card's own usage example: the converted fp32 reference scores the four passages [3.6339, 11.7127, 4.7563, 9.3698] — the base card publishes [3.6875, 11.6875, 4.75, 9.375] from a bfloat16 run, and the fp32 values match to bf16 tolerance (max |Δ| 0.054) with the ranking identical. Every variant preserves the full ranking.

2. NanoSciFact reranking (50 claims × 20 candidates each — gold abstracts + random negatives; the deployment shape, one invoke per candidate): PyTorch nDCG@10 0.9637 / MRR@10 0.950 / hit@1 0.920; fp32 identical to PyTorch with zero score difference; fp16 identical in every metric with zero pairwise rank inversions; int8 0.9563 / 0.940 / 0.900 with 2.8% of within-list candidate pairs inverted. If the last point of ranking accuracy matters and you have the RAM, use fp16; on-device, int8's cost is measured above.

3. Mechanics: the same pair scored through score_128/256/512 returns exactly the same score (spread 0.0, every variant); pad-content invariance exactly 0; the sliding-window edge case (short pair in a long signature) is guarded inside the graph and verified finite in fp32.

Corpora are subsampled, so absolute numbers are not comparable to published benchmarks.

Speed

CPU/XNNPACK, median of 10 runs, 75%-full signatures:

Variant	Machine	score_256	score_512
wi8fc	M4 Max Mac, 16 threads	89.5 ms	144.1 ms
fp16	M4 Max Mac, 16 threads	110.7 ms	188.7 ms

A cross-encoder pays per candidate: reranking 20 candidates at score_256 costs 20 invokes. Route each pair to the smallest signature that fits.

Conversion

Encoder lane — a direct multi-signature litert_torch trace of the HF ModernBERT backbone (not an LLM export), with the full-attention and sliding-window (±64) masks built by hand inside the traced wrapper, and the Sentence Transformers v5 head modules (CLS pooling, Dense 1024→1024 + GELU, LayerNorm, Dense 1024→1) implemented in-graph from the repo's module weights. Gated on: bitwise agreement with the vendor's own mask path, pad-content invariance, finiteness of the score for short pairs in long signatures (the sliding-window edge case), cross-signature agreement, and the base card's published example.

Script and full notes: hf-to-litertlm.

License

Apache 2.0, inherited from the base model (Sentence Transformers / JHU CLSP ettin encoder).

Modification notice: these files are converted, not original. The weights were exported to LiteRT and quantized (int8 dynamic-range / fp16); the scoring head was folded into the graph. No fine-tuning or weight modification beyond quantization was performed.