Upload folder using huggingface_hub

README.md

@@ -0,0 +1,95 @@
+---
+license: apache-2.0
+library_name: transformers
+tags: [text-classification, llm-routing, llm-d, semantic-router, triage]
+base_model: sentence-transformers/all-MiniLM-L6-v2
+---
+
+# llm-d-sc-complexity-v3
+
+`triage` classifier for [llm-d semantic
+classification](https://github.com/llm-d-incubation/llm-d-semantic-classifier).
+Labels: `TRIVIAL`, `WORK`.
+
+Architecture: **sequence-classification head (requires a runtime that reads logits)**,
+base `sentence-transformers/all-MiniLM-L6-v2`.
+
+## Accuracy
+
+Read the real-traffic row first.
+
+| eval set | n | accuracy | 95% CI | macro F1 |
+|---|---:|---:|---|---:|
+| **real traffic, refined gold** (high-effort re-adjudication) | 376 | 0.9601 | 0.935 – 0.976 | 0.9391 |
+| real traffic (WildChat, unanimous 3-model jury) | 418 | 0.9593 | 0.936 – 0.974 | 0.9375 |
+
+> **Run-to-run variance.** RECOMMENDED COMPLEXITY MODEL for llm-d-sc routing. Supersedes llm-d-sc-complexity-v2 (4-tier, 0.8963) and llm-d-sc-route-gate (2-tier SIMPLE+MEDIUM split, 0.9269) for backend selection.
+
+Published seed is the FIRST one run (11), not the best. Both seeds score 0.9601 on refined gold -- not a duplicate: different parameter sums, logits differing by up to 1.48, 10 of 376 predictions disagree and happen to cancel.
+
+  accuracy 96.01%   majority baseline 80.32%   lift +15.69   p50 4.37 ms
+  TRIVIAL recall 94.59%, precision 86.42%  (TRIVIAL is 19.7% of the eval)
+
+WHAT IT DECIDES. TRIVIAL (the 4-tier ladder's SIMPLE) versus WORK (MEDIUM, COMPLEX, REASONING): can a small model or a cache serve this, or does it need the main model?
+
+WHY THIS FOLD RATHER THAN THE ONE PRAXIS CURRENTLY USES. Enumerating EVERY contiguous fold of the 4-tier ladder and ranking by three-juror agreement:
+    SIMPLE | MEDIUM+COMPLEX+REASONING   86.9% agreement   <- this model
+    SIMPLE+MEDIUM | COMPLEX+REASONING   82.0% agreement   <- the deployed split
+Their majority baselines are 80.32% and 80.59%, so the comparison is like-for-like: +3.32 points for choosing the split by measurement instead of by which one the router happens to implement. Tier-exact accuracy on the full 4-tier taxonomy is 0.8963 and is capped near 0.926 by inter-juror agreement; this decision is not.
+
+TWO LIMITATIONS THAT MATTER MORE THAN THE HEADLINE.
+
+  1. CONTESTED ROWS. On the 176 rows where the three-model jury SPLIT, this model scores 80.11% against an 82.95% majority baseline -- BELOW chance -- with TRIVIAL recall at 43.33%. Contested rows are about 32% of real traffic.
+
+  2. ABSTENTION DOES NOT HELP IT. For the other gates in this family, model confidence tracks jury disagreement at 1.7-1.9x enrichment, so routing the least-confident slice to the large model fixes most of the contested-row problem. This model sits at 1.0x: its least-confident 5% contains contested rows at exactly the base rate. It is confidently WRONG on hard rows rather than uncertain about them, so no confidence threshold rescues it -- and at 99% recall it false-fires on 74% of traffic.
+
+PRACTICAL GUIDANCE. Deploy at full coverage and accept 80.11% on the split rows, or send TRIVIAL predictions to the cheap path only when the cheap path degrades gracefully -- TRIVIAL precision is 86.42%, so about 5% of WORK prompts land there. A three-way TRIVIAL/STANDARD/HARD variant is being tested to see whether a middle tier fixes the blind-confidence problem, as it did for the egress gate; if it does, this card will point at it.
+
+### The eval has a measured ceiling
+
+Gold labels were audited by blind paired adjudication in two strata — the rows
+this model got wrong, and a sample of the rows it got right — with the judge
+shown two candidate labels in random order and no indication of provenance.
+**Roughly 4.9% of the gold labels are themselves wrong**, so a PERFECT
+classifier scored against this eval would reach about **0.95, not 1.0**.
+
+Read the real-traffic accuracy against that ceiling, not against 100%. Auditing
+only a model's mistakes would move the number up artificially; sampling the
+correct rows too is what makes the estimate honest, and it revealed that on
+~3.3% of "correct" rows the model agreed with a bad label — meaning measured
+accuracy is very slightly overstated.
+
+### How the eval was built
+
+Real-traffic rows come from [WildChat-1M](https://huggingface.co/datasets/allenai/WildChat-1M)
+(ungated real assistant traffic). Each prompt was labelled independently by three
+models (`claude-opus-5`, `claude-sonnet-5`, `claude-fable-5-1`) from the task
+rubric alone -- **no labeller ever saw a proposed label**, so agreement is
+evidence rather than assent. Only unanimous rows are scored.
+
+Those three agree unanimously on roughly 70-74% of real prompts. The remaining
+prompts are published as a `contested` split rather than discarded: they measure
+how much real traffic this taxonomy does not resolve, which no single accuracy
+figure can express.
+
+### Training data
+
+418 rows from `triage-v2+triage-real+triage-active+triage-distill+triage-real-contested`, mixing jury-labelled real traffic
+(register and class prior) with rubric-grounded synthetic data (coverage of tiers
+that are rare in real traffic). Training prior: `None`.
+Held-out eval prompts are excluded by content hash.
+
+## Latency
+
+CPU single-request: **p50 4.37 ms, p99 9.78 ms**
+(Apple M-series, single thread). llm-d-sc serves the classifier on CPU, so model
+size trades directly against per-replica throughput.
+
+## Limitations
+
+- WildChat is consumer traffic. For `sensitivity` it is ~93% `PUBLIC` and cannot
+  measure the tiers that gate egress; the enterprise row above covers those.
+- Labels come from LLM jurors, not human annotators. The rubric was validated by
+  reproducing the project's hand-authored gold labels
+  (complexity 0.9875, cost 1.000, sensitivity 1.000) before use.
+- Not independently reproduced.

config.json

@@ -0,0 +1,38 @@
+{
+  "add_cross_attention": false,
+  "architectures": [
+    "BertForSequenceClassification"
+  ],
+  "attention_probs_dropout_prob": 0.1,
+  "bos_token_id": null,
+  "classifier_dropout": null,
+  "dtype": "float32",
+  "eos_token_id": null,
+  "gradient_checkpointing": false,
+  "hidden_act": "gelu",
+  "hidden_dropout_prob": 0.1,
+  "hidden_size": 384,
+  "id2label": {
+    "0": "TRIVIAL",
+    "1": "WORK"
+  },
+  "initializer_range": 0.02,
+  "intermediate_size": 1536,
+  "is_decoder": false,
+  "label2id": {
+    "TRIVIAL": 0,
+    "WORK": 1
+  },
+  "layer_norm_eps": 1e-12,
+  "max_position_embeddings": 512,
+  "model_type": "bert",
+  "num_attention_heads": 12,
+  "num_hidden_layers": 6,
+  "pad_token_id": 0,
+  "position_embedding_type": "absolute",
+  "tie_word_embeddings": true,
+  "transformers_version": "5.16.1",
+  "type_vocab_size": 2,
+  "use_cache": false,
+  "vocab_size": 30522
+}

model.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:c5001488fdaf9d5b724f4e24ed39f51686de830ddd5d401074f4cc506bccb7ee
+size 90867952

tokenizer_config.json

@@ -0,0 +1,24 @@
+{
+  "backend": "tokenizers",
+  "cls_token": "[CLS]",
+  "do_basic_tokenize": true,
+  "do_lower_case": true,
+  "is_local": false,
+  "local_files_only": false,
+  "mask_token": "[MASK]",
+  "max_length": 128,
+  "model_max_length": 512,
+  "never_split": null,
+  "pad_to_multiple_of": null,
+  "pad_token": "[PAD]",
+  "pad_token_type_id": 0,
+  "padding_side": "right",
+  "sep_token": "[SEP]",
+  "stride": 0,
+  "strip_accents": null,
+  "tokenize_chinese_chars": true,
+  "tokenizer_class": "BertTokenizer",
+  "truncation_side": "right",
+  "truncation_strategy": "longest_first",
+  "unk_token": "[UNK]"
+}