Evaluation: Vela 2.0 9B
Model card Β· Usage Β· Training
The tables retain the full released results, including comparisons and task breakdowns. Test texts are held out of training; calibration and release-selection protocols are described separately below.
Family results
| Model | Safety (AUC, 14 sets) | PII, 8K (F1; Vela 2.0 shipped calibration) | Hallucination (F1) | RAGTruth (F1) | fast-decisions | Jev Index (Vela 2.0: noul_calibration=True) |
|---|---|---|---|---|---|---|
| Vela 2.0 9B | 0.921 | 0.940 | 0.885 | 0.774 | 62.5 | 41.63 |
| Vela 2.0 0.3B (encoder) | 0.871 | 0.929 | 0.848 | 0.706 | 49.4 | β |
| Vela 2.0 0.8B | 0.875 | 0.603 | 0.828 | 0.652 | 56.8 | 16.01 |
| Vela 2.0 4B | 0.921 | 0.906 | 0.874 | 0.770 | 61.5 | 31.91 |
| Vela 1.0 specialists (one model per task) | β | 0.908 | 0.875 | 0.747 | β | β |
| LettuceDetect v2 qwen-2b (generative, hallucination only) | β | β | 0.921 | 0.817 | β | β |
| LettuceDetect v2 mmBERT-base (hallucination only) | β | β | 0.869 | 0.743 | β | β |
| GLiNER2.5-Decide | 0.704 | β | β | β | 62.9 | β |
| Decision-2.0-Lux-9B (our base, Decision 2.0) | β | β | β | β | β | 46.23 |
The decoder checkpoints, temperatures and thresholds were selected on dev data only. The 0.3B encoder follows a different release-selection protocol: final release selection also considered test results. Its 0.929 shipped 8K PII result uses a threshold floor lowered after the test effect was seen; the test-blind 0.001 floor gives 0.896, and the research scorer gives 0.894. The 0.3B fast-decisions result is not an independent zero-shot evaluation for that checkpoint. See the 0.3B evaluation disclosures.
| Comparison detail | Protocol |
|---|---|
| LettuceDetect v2, RAGTruth | Re-scored by us on the same 2,700 rows, using its own evaluation input; reproduces its model cards |
| LettuceDetect v2, 10,698 examples | Values from its model cards on the same test set |
| Jev Decision Index | Edition 0.2.1, 38 general decision benchmarks; optional Noul calibration enabled |
| This decoder, default Jev mode | 41.09 with optional Noul calibration off |
| Decision 2.0 bases | Same harness, reproducing released cards within 0.1: Eos 20.14, Nox 42.55, Lux 46.23 |
The routing recipe trades part of the base model's general-decision score for the router tasks. The family table keeps the base and specialist references visible so that this tradeoff can be assessed.
Router tasks against per-task references
PII rows in this table use the research scorer's per-length dev thresholds. All other metrics retain the released task definitions. The delta column is paired against the named task reference.
| Task | Reference model | Reference score | Vela 2.0 9B (research PII thresholds) | Ξ points [95% CI] | Vela 2.0 0.8B | Vela 2.0 4B | Vela 2.0 0.3B |
|---|---|---|---|---|---|---|---|
| Request harm, RTP-LX (AUC) | Safety | 0.761 | 0.801 | +4.0 [+3.4, +4.5] | 0.710 | 0.774 | 0.728 |
| Hate speech, Multilingual HateCheck (AUC) | Safety | 0.646 | 0.855 | +20.9 [+20.3, +21.5] | 0.653 | 0.769 | 0.662 |
| Request harm, CultureGuard (AUC) | Safety | 0.863 | 0.902 | +4.0 [+3.6, +4.2] | 0.850 | 0.898 | 0.900 |
| Prompt attacks, unseen families (AUC) | Guard | 0.792 | 0.989 | +19.8 [+15.4, +23.9] | 0.940 | 0.985 | 0.882 |
| Domain (macro-F1) | Domain | 0.831 | 0.844 | +1.3 [β0.3, +2.8] | 0.759 | 0.820 | 0.825 |
| Modality (macro-F1) | Modality | 0.889 | 1.000 | +11.1 [+9.4, +12.7] | 1.000 | 1.000 | 0.961 |
| Fact-check needed (macro-F1) | FactCheck | 0.501 | 1.000 | +49.9 [+46.1, +53.5] | 0.999 | 1.000 | 0.997 |
| Feedback (macro-F1) | Feedback | 0.425 | 0.939 | +51.3 [+48.8, +54.1] | 0.904 | 0.920 | 0.989 |
| PII, short texts (F1) | PII | 0.976 | 0.985 | +0.9 [+0.1, +1.9] | 0.984 | 0.987 | 0.995 |
| PII, 4K-token documents (F1) | PII | 0.928 | 0.965 | +3.6 [β0.3, +6.8] | 0.624 | 0.898 | 0.987 |
| PII, 8K-token documents (F1) | PII | 0.908 | 0.940 | +3.2 [β1.1, +7.0] | 0.603 | 0.905 | 0.894 |
| Hallucination, 10,698 examples (F1) | Halu | 0.875 | 0.885 | +1.0 [+0.4, +1.7] | 0.828 | 0.874 | 0.848 |
| RAGTruth (F1) | LettuceDetect v2 mmBERT | 0.743 | 0.774 | +3.0 [+0.5, +5.4] | 0.652 | 0.770 | 0.706 |
| Toxic spans (char-F1) | mmBERT span head | 0.451 | 0.446 | β0.5 [β1.2, +0.2] | 0.366 | 0.379 | 0.423 |
PII calibration modes
| Mode for this checkpoint | Short / 4K / 8K F1 |
|---|---|
| Research scorer, per-length dev thresholds | 0.985 / 0.965 / 0.940 |
| Shipped calibration, end to end | 0.987 / 0.959 / 0.940 |
The family summary uses shipped 8K calibration; the router comparison uses research-scoring thresholds. Both decoder modes are dev-selected. The 0.3B encoder's research and shipped results, and its test-informed release selection, are disclosed next to the family table above.
Safety, all 14 sets (AUC)
| set | Vela 2.0 9B | Vela 2.0 0.8B | Vela 2.0 4B | Vela 2.0 0.3B | Vela 1.0 | GLiNER2.5-Decide |
|---|---|---|---|---|---|---|
| RTP-LX, request harm | 0.801 | 0.710 | 0.774 | 0.728 | 0.761 | 0.642 |
| Multilingual HateCheck | 0.855 | 0.653 | 0.769 | 0.662 | 0.646 | 0.602 |
| XSTest, request harm | 0.990 | 0.854 | 0.967 | 0.890 | 0.782 | 0.807 |
| CultureGuard, request harm | 0.902 | 0.850 | 0.898 | 0.900 | 0.863 | 0.674 |
| Prompt attacks, unseen families | 0.989 | 0.940 | 0.985 | 0.882 | 0.792 | 0.893 |
| Prompt attacks, known families | 0.962 | 0.948 | 0.956 | 0.950 | 0.820 | 0.569 |
| deepset prompt injections | 0.990 | 0.970 | 0.989 | 0.957 | 0.779 | 0.761 |
| CultureGuard, response harm | 0.941 | 0.860 | 0.937 | 0.916 | 0.828 | 0.624 |
| AEGIS 2.0, response harm | 0.931 | 0.883 | 0.934 | 0.917 | 0.872 | 0.660 |
| PolyGuard, response harm | 0.957 | 0.901 | 0.925 | 0.936 | 0.884 | 0.759 |
| Do-Not-Answer, response harm | 0.777 | 0.883 | 0.931 | 0.656 | 0.418 | 0.416 |
| Do-Not-Answer, refusal | 0.913 | 0.918 | 0.893 | 0.896 | β | 0.886 |
| XSTest, refusal | 0.989 | 0.973 | 0.992 | 0.955 | β | 0.887 |
| PolyGuard, refusal | 0.894 | 0.907 | 0.945 | 0.947 | β | 0.683 |
| mean | 0.921 | 0.875 | 0.921 | 0.871 | β | 0.704 |
These comparisons cover trained safety families. They are not zero-shot comparisons against GLiNER2.5-Decide, which was not trained on these families. Vela 1.0 uses Safety for harm and Guard for attacks; it has no refusal output.
General and held-out decisions
| benchmark | Vela 2.0 9B | Vela 2.0 0.8B | Vela 2.0 4B | Vela 2.0 0.3B | reference |
|---|---|---|---|---|---|
fastino/fast-decisions (17 domains) |
62.5 | 56.8 | 61.5 | 49.4 | GLiNER2.5-Decide 62.9 |
| MASSIVE, unseen scenarios: intent / scenario accuracy | 0.763 / 0.871 | 0.544 / 0.632 | 0.727 / 0.814 | 0.510 / 0.575 | β |
| QASC, 8-way science QA (held out) | 0.644 | 0.434 | 0.640 | 0.251 | chance 0.125 |
| Super-NI task 1568 (held out) | 0.574 | 0.340 | 0.494 | 0.370 | β |
Differences are paired bootstrap 95% intervals with 2,000 resamples. This decoder's checkpoint, temperatures and thresholds were chosen on dev data only. The 0.3B fast-decisions figure follows the encoder's separately disclosed release-selection protocol and is not an independent zero-shot result for that checkpoint.
Open extraction benchmarks
The broad span head is evaluated on open-label NER, relations, extractive QA and evidence. SQuAD v2 dev is not zero-shot for this model: its training split is in the broad-head data. NER and ACL-Verbatim are evaluated with the held-out protocols below.
| Model | NER, 7 sets (F1) | Relations, CoNLL04 (F1) | SQuAD v2 dev (F1) | ACL-Verbatim evidence (word F1) | Latency, 3K-token context (ms) |
|---|---|---|---|---|---|
| Vela 2.0 9B, broad span head | 43.5 | 10.1 | 71.5 | 24.5 | 2288 |
| GLiNER2.5-base | 55.9 | 8.5 | 45.5 | 2.8 | 360 |
| GLiNER2.5-Decide | 51.5 | 1.5 | 46.0 | 2.3 | 415 |
| GLiFormer-base | 60.4 | 8.6 | 53.1 | 2.7 | 386 |
| GLiFormer-large | 63.6 | 6.2 | 47.0 | 7.0 | 930 |
| GLiNER large-v2.5 | 61.3 | 7.5 | 54.9 | 4.6 | 496 |
| ACL-Verbatim ModernBERT (trained in-domain, evidence only) | β | β | β | 53.4 | β |
NER per set (CrossNER and MIT, strict micro-F1, label names only):
| Model | AI | Literature | Music | Politics | Science | Movie | Restaurant | mean |
|---|---|---|---|---|---|---|---|---|
| Vela 2.0 9B, broad span head | 39.6 | 34.7 | 40.2 | 50.9 | 56.7 | 45.7 | 36.7 | 43.5 |
| GLiNER2.5-base | 52.5 | 52.2 | 57.4 | 63.2 | 59.3 | 60.6 | 46.4 | 55.9 |
| GLiNER2.5-Decide | 48.8 | 48.0 | 56.2 | 63.5 | 47.0 | 55.5 | 41.6 | 51.5 |
| GLiFormer-base | 56.4 | 62.6 | 67.6 | 71.4 | 67.5 | 56.8 | 40.5 | 60.4 |
| GLiFormer-large | 54.2 | 60.3 | 66.7 | 71.8 | 70.0 | 60.4 | 61.9 | 63.6 |
| GLiNER large-v2.5 | 56.4 | 65.6 | 71.4 | 72.4 | 65.0 | 53.7 | 44.9 | 61.3 |
| Evaluation | Protocol |
|---|---|
| Shared harness | Every model is scored by us on the same rows; reproduces the GLiFormer-base card on all seven NER sets and the gliner_large-v2.5 card average within 0.2 |
| NER | CrossNER and MIT; dataset's full label list per sentence, label names only; strict entity-level micro-F1 following GLiNER evaluation |
| Thresholds | GLiNER-family baselines: 0.5 defaults; this model: shipped broad-head threshold 0.5 |
| Relations | Per (entity, relation type), ask βWhich spans are the relation of entity?β over the sentence; tail-span micro-F1 on CoNLL04, SciERC, NYT11 and FewRel (one split) |
| Relation baselines | GLiNER-family models receive the same query as an entity label |
| SQuAD v2 | Train split is in the broad-head data; dev F1 is not zero-shot |
| ACL-Verbatim | Held out; paragraph-scale evidence in scientific papers |
| ACL-Verbatim ModernBERT reference | Trained on ACL-Verbatim itself, evidence only |
| Latency | One question per call, approximately 3K-token multi-passage context, mean over 100 questions after five warm-up calls, one A40 GPU for every model |
Router versus broad head on zero-shot dev
These ablations are on held-out dev sets, distinct from the benchmark test results above. Values use the original 0β1 F1 scale.
| Task | Router span head | Broad span head |
|---|---|---|
| NER | 0.219 | 0.427 |
| Relations | 0.230 | 0.510 |
| Verbatim evidence | 0.424 | 0.487 |
The broad head was added with the backbone, candidate head and router span head frozen. Router outputs were checked to be bitwise identical on 700 rows; see training stages.