toxic: on-device multilingual hate-speech and abuse triage

A multilingual toxicity classifier for on-device hate speech detection and content moderation triage: five content heads (HATEFUL, HARASSMENT, THREAT, SEXUAL, SELF_HARM) plus protected-group target heads, covering 23 languages. Inference is 100% on-device: no text leaves the device to be scored.

Headline: 0.785 macro-F1 averaged over seven real-Multilingual-HateCheck languages (German, French, Dutch, Portuguese, Spanish, Italian, Polish), measured on the torch reference. The shipped, quantized artifacts hold 0.777 (iOS Core ML int4, Android LiteRT int4) to 0.769 (web ONNX int4e) of that same 7-language average, re-measured on the exact bytes that ship. On the seven non-English languages, toxic's shipped artifacts beat every runnable competitor build (full-precision and int4 alike) by 12 to 29 points of macro-F1, at a fraction of the disk. Full tables are in How it compares.

Provenance is load-bearing in this card and never blurred. Eight of the 23 languages (English plus the 7-EU set) are scored on real Multilingual HateCheck, the published, peer-reviewed benchmark. The other 15 are scored on translate-and-audit synthetic evaluation sets built for this project; those numbers are development-benchmark estimates, not MHC-comparable, and are never averaged into the 7-EU headline. Three of the 15 (Lithuanian, Latvian, Estonian) carry an extra caveat: even the translation audit could not fully verify them. Every table below labels which tier a number belongs to.

Looking for English only? See toxic-en, the English specialist. Same idea as Whisper's .en models: this multilingual model covers English too (0.771 shipped, LiteRT int4), but a model trained only on English does the job better for English-first apps (toxic-en ships 0.808 to 0.817). Use toxic when you need coverage across languages in one model; use toxic-en when your traffic is English and you want the extra points.

Quality numbers throughout are macro-F1 on HateCheck and its multilingual extension for the real-MHC languages, and on translate-and-audit synthetic sets for the other 15: hand-written or translated test cases built to trap classifiers with hard hate (slurs, spelling evasion, implied hate) and deliberately confusing non-hate (quotes, counter-speech, negation). Macro-F1 averages the score for catching hate with the score for letting clean text through, so a model cannot look good by simply flagging everything (or nothing); 1.000 is perfect. HateCheck is eval-only: never trained on.

Triage, not verdict. Outputs are escalation signals for human review or a heavier local tier, not autonomous removal decisions. The failure modes below are published on purpose: read them before wiring the model into any enforcement path.

Only HATEFUL is benchmarked. Every number in this card is the HATEFUL head. HARASSMENT, THREAT and SEXUAL have never been measured against any external baseline or any held-out human-adjudicated set, so this card makes no quality claim about them and you should not infer one. An earlier version of this card described THREAT as "a high-precision head intended for resource routing with a human in the loop". No measurement supported that, and it has been removed rather than restated.

Do not use SELF_HARM in this release, and treat SEXUAL as unvalidated. The v0.2.0 training mix contains zero positive SELF_HARM examples and only 884 positive SEXUAL examples out of 563,197 rows. Measured on 3,728 English HateCheck rows, SELF_HARM never exceeds 0.008 and SEXUAL fires once, so SELF_HARM has effectively zero recall and cannot route anything. If you need self-harm detection, do not rely on this head. Both are being removed from the flagged rule or retrained before the next release.

HARASSMENT over-fires on English. It was trained against a generic toxicity mapping, so it fires on undirected profanity ("I'm so fucking excited"), on counter-speech that quotes hate in order to condemn it, and on reclaimed in-group usage. Gating on flagged gives a false-positive rate of 0.43 on English HateCheck against 0.11 for hateful alone. Gate on hateful until this is fixed; raising the HARASSMENT threshold does not fix it, because the head ranks undirected profanity above directed abuse.

Files

File	Format	Size	Contents
toxic.tflite	LiteRT / TFLite (int4 blockwise-32)	~90.5 MB	Android and Linux, native LiteRT kernels, 7-EU 0.777
toxic.onnx	ONNX (int4e)	~100.1 MB	Browser artifact for ONNX Runtime Web, 7-EU 0.769
toxic.mlmodelc	Core ML package (4-bit palettized)	~79.9 MB	iOS / macOS, ANE-resident, 7-EU 0.777
model.int8.onnx	ONNX (int8)	~159.5 MB	On request; higher fidelity (7-EU 0.783), larger
model.int8.tflite	LiteRT (int8)	~162.0 MB	On request; not yet HateCheck-measured
model.pt	PyTorch state dict (fp32)	~609 MB	Torch reference weights (159M params)
config.json	JSON	tiny	Encoder + head config (base, labels, max_len, vocab size)
tokenizer.json, tokenizer_config.json	JSON	~	Trimmed (95,552-piece) SentencePiece-Unigram tokenizer (XLM-R lineage)
labels.json	JSON	tiny	id2label / label2id for both heads
toxic_meta.json	JSON	tiny	Version, languages, labels, default threshold and the per-language threshold menu

Taxonomy

Two multi-label heads over one shared encoder. Text can fire several labels at once; thresholds apply per label.

• Content heads (5): HATEFUL (scoped to the EU notion of illegal hate speech: public incitement to violence or hatred against a protected group, Framework Decision 2008/913/JHA), HARASSMENT (directed abuse: insult or bullying aimed at a person or a group; a HATEFUL message is normally also HARASSMENT), THREAT (threat of violence), SEXUAL (sexual harassment / unsolicited sexual aggression), SELF_HARM (self-harm / suicide content, for resource routing).

  HARASSMENT changed meaning in taxonomy v3. It previously read "targeted insult / bullying of an individual (not group-protected)", which described it as a sibling of HATEFUL. It is now a superset: both human-annotated training corpora treat an attack on a protected group as also being abuse directed at someone. Integrations that presented the two heads as mutually exclusive need updating.

  SELF_HARM is untrained. Do not use it. No registered training corpus annotates self-harm, so the head has no positive examples and no supervised negatives either: it is untrained, not merely rare. Its output is whatever the shared trunk happens to produce. Measured on 3,728 English HateCheck rows it never exceeds 0.008, so in practice it does not fire, but that is an observation about this checkpoint and not a guarantee.

  Be precise about how it is switched off, because it depends which file you read. The SDKs bundle a sidecar meta that pins this head's threshold to 1.0, which a sigmoid cannot reach. The toxic_meta.json published in this repo carries no per-head threshold block at all, so a consumer that reads the published meta directly and applies the usual 0.5 default gets an untrained head wired into a flag-if-any-head-fires contract. If you built your own pipeline on this file rather than on one of our SDKs, pin SELF_HARM yourself. The next artifact carries the threshold block in the published meta so both paths agree; this one does not.

  Do not route self-harm resources off this output under any configuration.

• Target heads (10): the 2008/913/JHA protected grounds RACE, COLOUR, RELIGION, DESCENT, NATIONAL_ETHNIC_ORIGIN, plus the extended grounds SEXUAL_ORIENTATION, GENDER, DISABILITY, AGE, OTHER. Only meaningful when HATEFUL fires; used for per-ground fairness reporting.

Architecture

• Encoder: XLM-R-base lineage (12 layers, hidden 768), vocabulary trimmed from 250k to 95,552 pieces. 159M total params (73M embedding + 86M body), fine-tuned with focal BCE on the two heads.
• Static shapes: fixed [1, 128] input, no dynamic control flow; ANE-native reformulation for the Core ML export, int4 for the LiteRT and ONNX exports.
• Runtime: tokenize off-graph (SentencePiece-Unigram, tokenizer.json), run the encoder, apply sigmoid per label, threshold per the menu below. Default threshold 0.40.

Phone-class latency (iPhone 13/15, mid-range Android) is not yet measured for this release.

Measured quality (the shipped artifacts, not the checkpoint)

Every number below is HateCheck (or, for the 15 synthetic languages, the translate-and-audit synthetic set) re-run on the exact artifact in this repo, threshold 0.40.

7-EU headline (real Multilingual HateCheck: de, fr, nl, pt, es, it, pl)

platform	ship artifact	size	7-EU macro-F1	en macro-F1	all-23 (secondary)
iOS / macOS (Core ML)	toxic.mlmodelc	79.9 MB	0.777	0.719	(8-lang only)
Android / Linux (LiteRT)	toxic.tflite	90.5 MB	0.777	0.771	0.743
Web (ONNX)	toxic.onnx	100.1 MB	0.769	0.758	0.739
On request (ONNX int8)	model.int8.onnx	159.5 MB	0.783	0.759	0.748
Reference	model.pt (torch)	~609 MB (fp32)	0.785	0.779	0.755

all-23 blends the real-MHC 8 languages with the 15 synthetic-eval languages and is a secondary coverage figure, not comparable to the 7-EU headline; treat it as a rough sense of "how does this do everywhere," not a quality claim.

Core ML palettization (4-bit) costs the most on English: 0.719 vs the 0.779 torch reference, a 6-point drop, while still holding the 7-EU headline at 0.777, tied with the Android build. English is where int4 quantization hurts most across every platform we ship. Per-language source tables: reports/v2ship_coreml_int4_real8.md, reports/v2ship_tflite_int4b32.md, reports/v2ship_onnx_int4e.md, reports/v2ship_onnx_int8.md, and the torch reference in reports/E24_trimmed_multilingual.md.

Per-language, real-MHC (threshold 0.40)

lang	torch	LiteRT int4b32	Core ML int4	ONNX int4e
en	0.779	0.771	0.719	0.758
de	0.814	0.809	0.800	0.805
fr	0.819	0.809	0.792	0.797
nl	0.780	0.774	0.780	0.765
pt	0.768	0.760	0.783	0.756
es	0.781	0.771	0.778	0.764
it	0.772	0.763	0.772	0.754
pl	0.759	0.750	0.735	0.744
7-EU mean	0.785	0.777	0.777	0.769

The 15 synthetic-eval languages (translate-and-audit, NOT MHC-comparable)

These are not scored on real HateCheck. They are scored on a translate-and-audit synthetic evaluation set built for this project (data/eval/mhc_v2/), used as a development benchmark. Numbers below are useful for relative comparison across this model's own artifacts, not for comparison against any real-MHC number or any competitor. Lithuanian, Latvian and Estonian carry an extra caveat: the translation audit could not fully verify them, so treat those three as provisional.

lang	torch (synthetic)	LiteRT int4b32	ONNX int4e
da	0.818	0.806	0.809
sv	0.808	0.808	0.801
ro	0.790	0.778	0.786
bg	0.782	0.765	0.757
hr	0.759	0.741	0.749
el	0.743	0.741	0.727
sk	0.742	0.720	0.725
cs	0.737	0.718	0.718
fi	0.718	0.697	0.685
et (caveated)	0.718	0.702	0.708
lv (caveated)	0.716	0.699	0.700
sl	0.715	0.690	0.695
hu	0.708	0.698	0.690
lt (caveated)	0.676	0.664	0.659
ga	0.661	0.655	0.655

Source: reports/v2ship_tflite_int4b32.md, reports/v2ship_onnx_int4e.md, reports/E24_trimmed_multilingual.md (torch).

How it compares (measured, same scorer, same suite)

Every runnable competitor was re-run through the identical HateCheck harness, in both its full-precision (bf16) build and its shipping int4 quant, so quantized is compared against quantized and full against full. Generative guards are scored on their parsed safety verdict (Unsafe = hate); their on-disk sizes are published Q4_K_M GGUF bytes. 7-EU = mean over de, fr, nl, pt, es, it, pl.

system	build	on-disk size	7-EU macro-F1	en macro-F1
toxic	LiteRT int4b32 / Core ML int4 (shipped)	90.5 / 79.9 MB	0.777	0.771 / 0.719
toxic	torch reference	~609 MB (fp32)	0.785	0.779
Llama-Guard-3-1B	bf16 (server)	n/a	0.649	0.814
Llama-Guard-3-1B	int4	955.4 MB	0.633	0.794
Qwen3Guard-Gen-0.6B	bf16 (server)	n/a	0.652	0.770
Qwen3Guard-Gen-0.6B	int4	484.2 MB	0.622	0.750
Detoxify multilingual	fp32	~280 MB	0.487	0.642

What the table says:

• On the seven non-English languages, toxic's shipped artifacts beat every competitor build outright, at roughly 5 to 12x less disk (Core ML int4 79.9 MB / LiteRT int4b32 90.5 MB vs Qwen3Guard int4 484.2 MB and Llama-Guard int4 955.4 MB). The margin is never small: shipped toxic (7-EU 0.777) beats Llama-Guard bf16 (0.649) by 12.8 points and Llama-Guard int4 (0.633, 955 MB) by 14.4 points; it beats Qwen3Guard bf16 (0.652) by 12.5 points and Detoxify fp32 (0.487) by 29 points.
• On English, the competitors' server bf16 builds lead. Llama-Guard-3-1B bf16 scores 0.814 on English vs toxic's 0.779 torch reference and 0.719-0.771 shipped. This is exactly the gap toxic-en is built to close: see the English specialist card for numbers that match or beat Llama-Guard's own shipping int4 build.
• Detoxify multilingual, the incumbent same-architecture-class classifier, trails toxic's shipped artifacts on every one of the 7-EU languages, and the gap is uneven: 12-17 points in French, Portuguese, Spanish and Italian, but it effectively collapses in German, Dutch and Polish (48-52 points behind).

Source: reports/leaderboard.md, reports/V2_RELEASE_DATASHEET.md.

Per-language, head to head (measured)

The aggregate above hides where the difference actually comes from, so here is every shipped language against the two generative guards, one row per language.

How these numbers were produced

• Our real decision rule, not a favourable one. These rows score the shipped artifact exactly as the SDK does: flagged fires when ANY content head clears its own threshold from toxic_meta.json (HATEFUL 0.40, the other four heads 0.50). The tables higher up this card report the HATEFUL head alone, which is the HateCheck literature convention and the right number to compare against published HateCheck results. Here we use the product rule because a generative guard answers one question, "is this unsafe", and is credited when it flags for any reason. Scoring only our HATEFUL head against that would be asymmetric. The two conventions differ by about 2 points on 7-EU (0.777 vs 0.801) and under 1 point on the synthetic tier.
• That rule costs us on false positives, and we report it. Flagging on any head raises recall and also raises the false-positive rate: on 7-EU from 0.25 to 0.29, and on English from 0.108 to 0.427, because heads like HARASSMENT fire on reclaimed slurs and profanity-without-hate that HateCheck labels non-hateful. If you gate purely on flagged, that is the rate you get; gate on hateful for the narrower signal.
• One harness, no per-language tuning. Every system is scored by toxic_training.hatecheck.scores on the same sentences, metric is macro-F1 over the hateful / non-hateful decision, so a model cannot win by flagging everything. Neither we nor the competitors get per-language threshold tuning.
• Competitors are shown at their best. Llama-Guard-3-1B and Qwen3Guard-Gen-0.6B are their full-precision bf16 server builds, not the quantized weights you would ship on a phone. Qwen also emits a third verdict, "Controversial"; we score it whichever way is more favourable to Qwen per language, so these margins are the conservative ones.
• Generative guards are read, not thresholded. Both are prompted with their own chat template and decoded greedily; the parsed safety verdict is the prediction. Unparsable generations would count against the guard, though in practice there were none.
• Two provenance tiers, never mixed. The first table is published Multilingual HateCheck (Röttger et al., WOAH 2022). The second is our own translate-and-audit set for the fifteen languages MHC does not cover: every system is scored on identical sentences, so the comparison between systems is sound, but the absolute values are not comparable to real MHC and are never averaged into a real-MHC figure.

Known limits of the synthetic tier

A seven-way independent audit of those fifteen sets (roughly 1,000 rows read by native-language reviewers) found no gold-label errors, no encoding corruption and no untranslated rows, but three limits worth stating plainly:

• Part of the training data for these same fifteen languages was generated by the same model that built this eval set. We verified zero exact and zero near-duplicate overlap between them, and the untrimmed reference model scores higher than the shipped one, so vocabulary selection bought nothing. A shared-generator style effect still cannot be ruled out, so treat this tier as directional and the real-MHC tier as authoritative.
• About 21% of rows are deliberate character-obfuscation tests (leetspeak, inserted spaces, character swaps). Our lead is much wider there, because the SDK normalizes text before tokenizing and the guards receive it raw. On clean, unobfuscated rows the margin is roughly +20 points rather than +57.
• Irish is missing most of its slur_h rows (16 against 144 for every other language) and all of slur_homonym_nh. The Irish headline holds, the Irish slur sub-slice is not a fair comparison point.

Real Multilingual HateCheck (authoritative) published Röttger et al. sets

lang	toxic (shipped rule)	Llama-Guard-3-1B	Qwen3Guard-Gen-0.6B	toxic vs best
en (English)	0.767	0.814	0.770	-4.7
de (German)	0.827	0.686	0.674	+14.0
fr (French)	0.825	0.654	0.691	+13.4
nl (Dutch)	0.807	0.668	0.647	+13.9
pt (Portuguese)	0.798	0.684	0.671	+11.4
es (Spanish)	0.797	0.685	0.692	+10.5
it (Italian)	0.772	0.597	0.642	+13.0
pl (Polish)	0.784	0.566	0.547	+21.7
mean (8 languages)	0.797	0.669	0.667	+12.7

Translate-and-audit sets (synthetic-eval provenance) our own sets, NOT comparable to real MHC

lang	toxic (shipped rule)	Llama-Guard-3-1B	Qwen3Guard-Gen-0.6B	toxic vs best
da (Danish)	0.811	0.505	0.529	+28.3
sv (Swedish)	0.825	0.581	0.550	+24.4
fi (Finnish)	0.704	0.375	0.485	+22.0
hu (Hungarian)	0.703	0.485	0.525	+17.8
cs (Czech)	0.716	0.594	0.556	+12.2
sk (Slovak)	0.720	0.471	0.524	+19.5
sl (Slovenian)	0.690	0.399	0.492	+19.8
hr (Croatian)	0.745	0.443	0.485	+26.0
bg (Bulgarian)	0.773	0.503	0.550	+22.3
el (Greek)	0.745	0.576	0.538	+16.9
ro (Romanian)	0.786	0.576	0.572	+21.1
lt (Lithuanian)	0.676	0.316	0.511	+16.5
lv (Latvian)	0.706	0.333	0.484	+22.2
et (Estonian)	0.708	0.303	0.501	+20.7
ga (Irish)	0.654	0.234	0.281	+37.4
mean (15 languages)	0.731	0.446	0.505	+22.5

False positives in context

Two different numbers describe "how often it flags clean text," and conflating them misreads the model.

On adversarial trap cases: roughly 20 to 27%, real-MHC 7-EU languages, at threshold 0.40. HateCheck's non-hate cases are not ordinary text; they are traps built from the exact patterns that break hate classifiers: counter_quote / counter_ref quote or denounce hate to condemn it, slur_reclaimed is in-group reclaimed slur use, negate_neg is negated hate ("no one should ever hurt them"). On the flagship LiteRT artifact the overall non-hate false-positive rate ranges from 19.6% (Dutch) to 27.4% (Italian) across the 7-EU languages at threshold 0.40 (reports/v2ship_tflite_int4b32.md).

On natural text: single digits. On the held-out test split of real corpora (Civil Comments, Measuring Hate Speech, GAHD; never seen in training), the HATEFUL head fires on 1.7% of clean rows overall, and any of the five heads fires on 7.7% (the extra fires are mostly HARASSMENT at 6.7%). The one natural-corpus outlier is German GAHD at 26.6%, and that is by design: GAHD is an adversarially constructed dataset of contrast pairs (quotes, counter-speech, borderline rephrasings), so it measures the trap rate, not the organic rate. Source: reports/E24_extended.md.

Beware the low-FPR trap when comparing. Llama-Guard-3-1B posts a low average non-hate false-positive rate on the seven non-English languages in bf16 (0.187, averaged from the per-language rows in reports/leaderboard.md), noticeably lower than toxic's own 7-EU FPR range (0.196-0.274 on the shipped LiteRT artifact, above). But Llama-Guard's 7-EU macro-F1 is only 0.649, 12.8 points below toxic's 0.777 shipped, which means the low FPR is bought with weaker hate-class performance, not superior precision at matched recall. A low false-positive rate bought by under-firing is not the same as good triage, which is why the headline metric is macro-F1 (balanced by construction across both classes) and why a low FPR number should always be read next to the paired macro-F1. If your application needs a lower FPR from toxic specifically, buy it explicitly with the threshold menu below and keep the recall cost on the table.

Threshold menu (per-language operating points, torch reference)

macro-F1 is the 2-class macro; FPR is the false-positive rate on non-hate cases, the number to watch if you fear over-blocking. Full sweep (0.30 to 0.70, all 23 languages) is in reports/E24_sweep.md.

lang	tuned threshold	macro-F1 @ tuned	FPR @ tuned	macro-F1 @ 0.40	FPR @ 0.40
en	0.30	0.811	0.200	0.779	0.135
de	0.40	0.814	0.260	0.814	0.260
fr	0.50	0.819	0.199	0.819	0.225
nl	0.35	0.782	0.220	0.780	0.204
pt	0.30	0.772	0.286	0.768	0.260
es	0.50	0.784	0.230	0.781	0.265
it	0.30	0.774	0.278	0.772	0.248
pl	0.30	0.764	0.281	0.759	0.258

At per-language tuned thresholds, the 7-EU average (de, fr, nl, pt, es, it, pl) rises to 0.787 from 0.785 at the flat 0.40 default; French, German and Spanish clear 0.78-0.82 at their tuned point. Every language's improvement from tuning is small (a few tenths of a point) because 0.40 is already close to optimal almost everywhere; the exception is English, where dropping to 0.30 buys +3.2 points of macro-F1 at the cost of a higher FPR (0.200 vs 0.135). The shipped default is a flat 0.40 for every language (no threshold is fit to the benchmark); the full per-language sweep for all 23 languages (0.30 to 0.70) is in reports/E24_sweep.md as tuning guidance, so the operating point stays yours to set per application.

Failure modes (read before deploying)

This section is the practical read: what actually goes wrong, what it looks like in an app, and what to do about it. Publishing these is part of the product.

1. It sometimes flags people quoting or condemning hate. News reports, moderation discussions, and counter-speech repeat the hateful words they are arguing against, and the model reacts to the words. On the real-MHC languages, counter_quote_nh and counter_ref_nh false-positive rates on the flagship LiteRT artifact range from roughly 7% (Dutch) to 53% (German, counter_ref_nh). Mitigation: route flags to human review or a heavier model; never auto-remove on this signal alone.

2. Positive or neutral mentions of identity groups can trip it, especially outside English. Sentences like "Proud to be a trans woman and having a great day" are sometimes flagged simply because an identity term appears. ident_pos_nh false-positive rates on the flagship LiteRT artifact range from 23% (Spanish) to 37% (Polish) across the 7-EU languages, and target_group_nh false positives run from 28% (German) to 46% (Italian). Mitigation: use the higher per-language thresholds from the menu; treat short identity-statement texts as low-confidence.

3. Reclaimed slurs and lookalike words, most measurable in English. The real Multilingual HateCheck translation only carries the reclaimed-slur and slur-homonym trap cells (slur_reclaimed_nh, slur_homonym_nh) for English, not for the other 7 real-MHC languages, so this failure mode is only directly measured there: on the flagship LiteRT artifact English scores 7% FPR on reclaimed slurs and 33% FPR on slur homonyms. Mitigation: human review for user-report flows in communities where reclaimed language is common.

4. Bare-slur hate is a harder hate pattern than direct insults, especially in English. slur_h accuracy (an error here is a miss) on the flagship LiteRT artifact ranges from 0.667 (English, the worst) to 0.907 (Italian, the best); Polish (0.705) and German (0.758) also trail. Mitigation: if slur coverage is critical, pair the model with a per-language slur wordlist; the two failure profiles are complementary.

5. Hate phrased as a question or an implication is harder than direct insults. phrase_question_h accuracy on the flagship LiteRT artifact is 0.471-0.700 across the 7-EU languages, and derog_impl_h (implied derogation) is 0.586-0.890, well below direct threat detection (threat_dir_h, threat_norm_h), which is 0.89-1.00 in every language measured. Mitigation: none needed for triage use; do not promise users that subtle hate is always caught.

6. Quantization costs are largest on English, across every platform. Core ML 4-bit palettization costs English 6.0 points (0.719 vs 0.779 torch) while holding the 7-EU average flat (0.777 vs 0.785 torch, a smaller 0.8-point loss). LiteRT int4b32 costs English 0.8 points (0.771 vs 0.779). ONNX int4e costs English 2.1 points (0.758 vs 0.779); the on-request ONNX int8 build does not meaningfully recover this (0.759, effectively the same as int4e). If English dominates your traffic, toxic-en is the better choice: it is not subject to this multilingual-model quantization pattern and scores higher on English at every comparable size.

7. Not yet measured: latency, on any platform, for this v0.2.0 release. No phone-class or desktop-class timing has been re-run against the v2 artifacts; do not assume the prior generation's numbers carry over, since the vocabulary, quantization scheme and shipped file sizes all changed.

The pattern behind 1-3: counter-speech, negated hate, reclaimed slurs and positive or neutral identity mentions are the dominant false-positive sources in every language; bare slurs are among the harder misses. If your application is FPR-sensitive, use the higher per-language thresholds above and route positives to review instead of auto-actioning.

Per-language deep-dive (real-MHC, flagship LiteRT int4b32 artifact)

HateCheck groups its test cases into named "functionalities" (cells), each isolating one linguistic pattern. Cells ending _h are hateful (an error is a miss); cells ending _nh are non-hateful (an error is a false positive).

lang	strongest cells (acc)	weakest cells (acc)
en	profanity_nh 0.980, threat_norm_h 0.971, target_obj_nh 0.969	phrase_question_h 0.521, derog_impl_h 0.536, spell_char_del_h 0.579
de	ref_subs_clause_h 0.979, spell_char_swap_h 0.971, threat_dir_h 0.971	counter_ref_nh 0.471, ident_pos_nh 0.657, phrase_question_h 0.700
fr	threat_dir_h 1.000, spell_space_add_h 0.994, spell_char_swap_h 0.993	ref_subs_sent_h 0.496, derog_impl_h 0.629, target_indiv_nh 0.631
nl	threat_dir_h 1.000, counter_quote_nh 0.976, counter_ref_nh 0.929	phrase_question_h 0.471, ref_subs_sent_h 0.571, derog_impl_h 0.629
pt	threat_dir_h 0.993, threat_norm_h 0.936, profanity_h 0.921	target_group_nh 0.554, phrase_question_h 0.586, target_obj_nh 0.600
es	threat_dir_h 1.000, spell_char_swap_h 0.957, threat_norm_h 0.952	target_obj_nh 0.585, target_indiv_nh 0.600, target_group_nh 0.615
it	threat_dir_h 1.000, spell_space_add_h 0.981, threat_norm_h 0.971	phrase_question_h 0.486, target_group_nh 0.538, ref_subs_sent_h 0.564
pl	threat_dir_h 1.000, counter_quote_nh 0.977, threat_norm_h 0.943	negate_neg_nh 0.579, target_indiv_nh 0.585, derog_impl_h 0.586

Read: explicit threats are near-perfect in every language; errors concentrate in exactly the trap families described in the failure modes above. Source: reports/v2ship_tflite_int4b32.md (per-functionality breakdown for all 23 languages).

Fairness across protected groups

Measured on the held-out natural (non-synthetic) test split, threshold 0.40.

Does it recognize hate equally, whoever the target is? On the held-out test rows with human-annotated target groups (Measuring Hate Speech corpus), HATEFUL recall per protected ground:

ground	n	HATEFUL recall
AGE	21	1.000
DISABILITY	27	0.889
SEXUAL_ORIENTATION	116	0.853
GENDER	227	0.806
RELIGION	84	0.809
DESCENT	116	0.750
RACE	186	0.753

Spread is 25 points (0.750 to 1.000), wider than we would like: RACE and DESCENT recall trail GENDER, RELIGION, SEXUAL_ORIENTATION and DISABILITY by 5-15 points. AGE's 1.000 is on the smallest group (n=21) and should be read with that in mind. This is a monitored target for the next data iteration.

Source: reports/E24_extended.md. Note: the mention-trap style ranking analysis (BPSN/BNSP AUCs, published for the prior 8-language release) has not yet been re-run against this v2 model; we are not carrying over the prior generation's BPSN/BNSP numbers here because they do not measure this model. That analysis is planned for the next iteration rather than reported now.

How these numbers were made

The evaluation discipline is the product as much as the weights are:

• Every quality number is re-measured on the exact shipped bytes, not extrapolated from the training checkpoint. Quantized artifacts are re-run through the full scorer per platform (Core ML, LiteRT, ONNX, torch).
• Provenance is tracked per language and never blurred. Real Multilingual HateCheck (8 languages) is the only source for the headline; the 15 translate-and-audit synthetic languages are reported separately, with Lithuanian, Latvian and Estonian further caveated as provisional.
• 444 rows in the synthetic sets are unverified and still counted. The translate-and-audit pass flags rows it could not confirm, most often an English slur that survived translation verbatim, so the row may be testing the wrong construct in that language. They range from 10 rows (ga) to 60 (da), 0.81% of the synthetic total. They are counted rather than dropped, because "needs a native check" means uncertain and not wrong, and quietly removing uncertain rows biases a score toward whatever is left. The 8 real-MHC languages, including every headline number, contain none of them.
• Benchmarks are eval-only, with one honest qualification. No HateCheck row appears in training, in any language: that is enforced mechanically by a hash-intersection check over every generated file, and the natural-corpus test split is carved by content hash before training and never touched. But HateCheck is a development set for this corpus at the construct level, and pretending otherwise would be misleading. The template generator mirrors HateCheck's functional cell names, the clean-side generator targets its non-hate cells by name, and the per-language register hints were written by reading its slur cells. So "no verbatim overlap" is what we verify, and "held out" is not what we can claim. Numbers on real in-the-wild corpora (textdetox, offenseval2020) are the ones to weigh if you want a figure untouched by that dependency.
• Competitors run through the identical harness, same scorer, same threshold discipline, with their quantized and full-precision builds both measured rather than assumed.

License

Desert Ant Labs Source-Available License. Free for most apps; a commercial license is required at scale. Full terms at the link. Licensing: licensing@desertant.com.

Built exclusively from commercially clean components (CC0 / CC-BY / MIT / Apache-2.0 training data, MIT base encoder). Attributions, dataset citations and generator credits are in THIRD_PARTY_NOTICES.md.