Hugging Face's logo

jatshi
/
StreamSense-Serve-v4-Router

Feature Extraction
sentence-transformers
multimodal
routing
uncertainty
evidence-grounding
chinese
Model card Files
xet
 Community
jatshi commited on
Commit
4ff5147
·
verified ·
1 Parent(s): 70e1ff6

Publish verified StreamSense-Serve 4.0 router

Browse files
Files changed (7) hide show
  1. MODEL_CARD.md +97 -0
  2. README.md +97 -0
  3. SHA256SUMS +6 -0
  4. e2e_diagnostics.json +115 -0
  5. e2e_summary.json +51 -0
  6. router_artifact.json +1125 -0
  7. router_test_metrics.json +343 -0
MODEL_CARD.md ADDED
@@ -0,0 +1,97 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ ---
2
+ tags:
3
+ - multimodal
4
+ - routing
5
+ - uncertainty
6
+ - evidence-grounding
7
+ - sentence-transformers
8
+ - chinese
9
+ base_model: BAAI/bge-small-zh-v1.5
10
+ pipeline_tag: feature-extraction
11
+ ---
12
+
13
+ # StreamSense-Serve 4.0 Router
14
+
15
+ This repository contains the trained, auditable routing artifact used by
16
+ [StreamSense-Serve 4.0](https://github.com/Jatshi/StreamSense-Serve). It is not a copy of the
17
+ Qwen or BGE base-model weights. The artifact stores four lightweight decision branches and their
18
+ jointly calibrated thresholds:
19
+
20
+ 1. safety risk;
21
+ 2. lightweight-evidence uncertainty;
22
+ 3. cross-modal conflict;
23
+ 4. visual dependency.
24
+
25
+ The runtime escalates to visual enhancement when **any** branch crosses its threshold. Risk and
26
+ visual dependency use frozen `BAAI/bge-small-zh-v1.5` embeddings followed by logistic heads;
27
+ uncertainty and conflict use physical signal and cross-modal features. The BGE revision is pinned
28
+ inside `router_artifact.json` and must be downloaded under its upstream terms.
29
+
30
+ ## Files
31
+
32
+ - `router_artifact.json`: coefficients, normalizers, frozen encoder revision, four thresholds,
33
+ training-data digest, and grouped-OOF metrics;
34
+ - `router_test_metrics.json`: results on 576 physical media cases from eight unseen semantic
35
+ groups, including baselines, condition metrics, robustness slices, and group bootstrap;
36
+ - `e2e_summary.json`: formal 32-video × three-system result, including answer, citation, visual-call,
37
+ and total-backend-call rates;
38
+ - `e2e_diagnostics.json`: group-bootstrap confidence intervals, paired system differences, and
39
+ failure-mode counts;
40
+ - `SHA256SUMS`: integrity hashes for the published artifacts;
41
+ - `MODEL_CARD.md`: this card in a standalone filename for local archives.
42
+
43
+ ## Measured result
44
+
45
+ | Metric | Value |
46
+ |---|---:|
47
+ | Development semantic groups | 24 |
48
+ | Grouped OOF folds | 4 |
49
+ | OOF escalation recall | 95.88% |
50
+ | Held-out semantic groups | 8 |
51
+ | Held-out physical cases | 576 |
52
+ | Held-out escalation recall | 94.81% |
53
+ | Held-out risk recall | 98.61% |
54
+ | Visual-enhancement rate | 94.97% |
55
+
56
+ The held-out split is deliberately difficult: 520/576 cases are oracle-positive, so even a
57
+ perfect router could avoid visual enhancement on at most 9.72% of this split. The measured 5.03%
58
+ reduction is not a production-traffic cost estimate.
59
+
60
+ The separate formal end-to-end run used 32 real MP4 cases sampled evenly from eight untouched
61
+ semantic groups and executed learned, always-visual, and never-visual systems once each (96 system
62
+ records). Quality pass rates were 65.63%, 68.75%, and 78.13%, respectively. The preregistered H5
63
+ hypothesis therefore failed: in this test, OCR already resolved the two visual groups and the
64
+ free-text VLM evidence introduced state inconsistencies instead of net quality. This negative
65
+ result is retained because it defines the next work item: non-OCR visual tasks and structured VLM
66
+ evidence. Latency values in this run are not comparable across systems because ASR/OCR outputs
67
+ were memoized in execution order.
68
+
69
+ ## Reproduce
70
+
71
+ ```bash
72
+ python -m pip install -e ".[semantic-router]"
73
+ python scripts/evaluate_v4_router.py \
74
+ --features /path/to/features.jsonl \
75
+ --router router_artifact.json \
76
+ --output reproduced_metrics.json \
77
+ --bootstrap-iterations 1000
78
+ ```
79
+
80
+ The complete media builder, Faster-Whisper/RapidOCR extraction, group-isolated OOF training, and
81
+ end-to-end Qwen-VL/EvidenceAgent evaluation are in the GitHub repository. See the
82
+ [implementation status](https://github.com/Jatshi/StreamSense-Serve/blob/main/docs/V4_IMPLEMENTATION_STATUS_ZH.md),
83
+ [release notes and failure diary](https://github.com/Jatshi/StreamSense-Serve/blob/main/docs/V4_RELEASE_NOTES_ZH.md),
84
+ and [learning guide](https://github.com/Jatshi/StreamSense-Serve/blob/main/docs/V4_LEARNING_AND_INTERVIEW_ZH.md).
85
+
86
+ ## Intended use and limitations
87
+
88
+ This artifact is an engineering benchmark router for evidence-grounded audio/video assistants.
89
+ It was trained on 32 generated Chinese semantic scenarios expanded into 2,304 physical corruption
90
+ conditions. It is not validated for medical decisions, surveillance, identity recognition, or
91
+ autonomous safety actions. The corpus uses one synthetic voice family and controlled visual
92
+ templates; accents, spontaneous speech, natural meetings, and deployment priors require separate
93
+ evaluation.
94
+
95
+ The endpoint experiment uses one text EvidenceAgent call per case and adds one image-bearing VLM
96
+ call only when the router escalates. Therefore `visual-enhancement rate` does not mean that all
97
+ non-escalated cases avoid every large-model call.
README.md ADDED
@@ -0,0 +1,97 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ ---
2
+ tags:
3
+ - multimodal
4
+ - routing
5
+ - uncertainty
6
+ - evidence-grounding
7
+ - sentence-transformers
8
+ - chinese
9
+ base_model: BAAI/bge-small-zh-v1.5
10
+ pipeline_tag: feature-extraction
11
+ ---
12
+
13
+ # StreamSense-Serve 4.0 Router
14
+
15
+ This repository contains the trained, auditable routing artifact used by
16
+ [StreamSense-Serve 4.0](https://github.com/Jatshi/StreamSense-Serve). It is not a copy of the
17
+ Qwen or BGE base-model weights. The artifact stores four lightweight decision branches and their
18
+ jointly calibrated thresholds:
19
+
20
+ 1. safety risk;
21
+ 2. lightweight-evidence uncertainty;
22
+ 3. cross-modal conflict;
23
+ 4. visual dependency.
24
+
25
+ The runtime escalates to visual enhancement when **any** branch crosses its threshold. Risk and
26
+ visual dependency use frozen `BAAI/bge-small-zh-v1.5` embeddings followed by logistic heads;
27
+ uncertainty and conflict use physical signal and cross-modal features. The BGE revision is pinned
28
+ inside `router_artifact.json` and must be downloaded under its upstream terms.
29
+
30
+ ## Files
31
+
32
+ - `router_artifact.json`: coefficients, normalizers, frozen encoder revision, four thresholds,
33
+ training-data digest, and grouped-OOF metrics;
34
+ - `router_test_metrics.json`: results on 576 physical media cases from eight unseen semantic
35
+ groups, including baselines, condition metrics, robustness slices, and group bootstrap;
36
+ - `e2e_summary.json`: formal 32-video × three-system result, including answer, citation, visual-call,
37
+ and total-backend-call rates;
38
+ - `e2e_diagnostics.json`: group-bootstrap confidence intervals, paired system differences, and
39
+ failure-mode counts;
40
+ - `SHA256SUMS`: integrity hashes for the published artifacts;
41
+ - `MODEL_CARD.md`: this card in a standalone filename for local archives.
42
+
43
+ ## Measured result
44
+
45
+ | Metric | Value |
46
+ |---|---:|
47
+ | Development semantic groups | 24 |
48
+ | Grouped OOF folds | 4 |
49
+ | OOF escalation recall | 95.88% |
50
+ | Held-out semantic groups | 8 |
51
+ | Held-out physical cases | 576 |
52
+ | Held-out escalation recall | 94.81% |
53
+ | Held-out risk recall | 98.61% |
54
+ | Visual-enhancement rate | 94.97% |
55
+
56
+ The held-out split is deliberately difficult: 520/576 cases are oracle-positive, so even a
57
+ perfect router could avoid visual enhancement on at most 9.72% of this split. The measured 5.03%
58
+ reduction is not a production-traffic cost estimate.
59
+
60
+ The separate formal end-to-end run used 32 real MP4 cases sampled evenly from eight untouched
61
+ semantic groups and executed learned, always-visual, and never-visual systems once each (96 system
62
+ records). Quality pass rates were 65.63%, 68.75%, and 78.13%, respectively. The preregistered H5
63
+ hypothesis therefore failed: in this test, OCR already resolved the two visual groups and the
64
+ free-text VLM evidence introduced state inconsistencies instead of net quality. This negative
65
+ result is retained because it defines the next work item: non-OCR visual tasks and structured VLM
66
+ evidence. Latency values in this run are not comparable across systems because ASR/OCR outputs
67
+ were memoized in execution order.
68
+
69
+ ## Reproduce
70
+
71
+ ```bash
72
+ python -m pip install -e ".[semantic-router]"
73
+ python scripts/evaluate_v4_router.py \
74
+ --features /path/to/features.jsonl \
75
+ --router router_artifact.json \
76
+ --output reproduced_metrics.json \
77
+ --bootstrap-iterations 1000
78
+ ```
79
+
80
+ The complete media builder, Faster-Whisper/RapidOCR extraction, group-isolated OOF training, and
81
+ end-to-end Qwen-VL/EvidenceAgent evaluation are in the GitHub repository. See the
82
+ [implementation status](https://github.com/Jatshi/StreamSense-Serve/blob/main/docs/V4_IMPLEMENTATION_STATUS_ZH.md),
83
+ [release notes and failure diary](https://github.com/Jatshi/StreamSense-Serve/blob/main/docs/V4_RELEASE_NOTES_ZH.md),
84
+ and [learning guide](https://github.com/Jatshi/StreamSense-Serve/blob/main/docs/V4_LEARNING_AND_INTERVIEW_ZH.md).
85
+
86
+ ## Intended use and limitations
87
+
88
+ This artifact is an engineering benchmark router for evidence-grounded audio/video assistants.
89
+ It was trained on 32 generated Chinese semantic scenarios expanded into 2,304 physical corruption
90
+ conditions. It is not validated for medical decisions, surveillance, identity recognition, or
91
+ autonomous safety actions. The corpus uses one synthetic voice family and controlled visual
92
+ templates; accents, spontaneous speech, natural meetings, and deployment priors require separate
93
+ evaluation.
94
+
95
+ The endpoint experiment uses one text EvidenceAgent call per case and adds one image-bearing VLM
96
+ call only when the router escalates. Therefore `visual-enhancement rate` does not mean that all
97
+ non-escalated cases avoid every large-model call.
SHA256SUMS ADDED
@@ -0,0 +1,6 @@
 
 
 
 
 
 
 
1
+ c3db7638c60276ef0451a0175a547390411ee0201080d9069e16f241ffb0be32 README.md
2
+ c3db7638c60276ef0451a0175a547390411ee0201080d9069e16f241ffb0be32 MODEL_CARD.md
3
+ 21ef08a058dc6299f330cdf2a00cc41c7b1431ea6ce53139ddf33725ee2917f8 router_artifact.json
4
+ 8ed77b38b4f6273ddd4bbf6e7c7e3f4354c4eb622ac526cfa10a6f1b73d39dc3 router_test_metrics.json
5
+ 39dfeb6b4d4c9da26862990354479a51e382878c39c46b29628c1b41b7450ec4 e2e_summary.json
6
+ 967ec34a209d27703cfbb932f50e946025c7ee8ff8658493d66e0db8a533f822 e2e_diagnostics.json
e2e_diagnostics.json ADDED
@@ -0,0 +1,115 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "schema_version": 1,
3
+ "bootstrap_unit": "group_id",
4
+ "bootstrap_iterations": 10000,
5
+ "independent_groups": 8,
6
+ "systems": {
7
+ "always_vlm": {
8
+ "cases": 32,
9
+ "quality_pass_rate": 0.6875,
10
+ "group_bootstrap_95_ci": [
11
+ 0.46875,
12
+ 0.875
13
+ ],
14
+ "failure_counts": {
15
+ "state": 9,
16
+ "answer_fact": 1,
17
+ "citation": 4
18
+ },
19
+ "per_group_quality_pass_rate": {
20
+ "meeting-conflict-003": 0.75,
21
+ "meeting-conflict-005": 0.25,
22
+ "meeting-normal-001": 1.0,
23
+ "meeting-visual-001": 1.0,
24
+ "meeting-visual-008": 1.0,
25
+ "safety-risk-003": 0.25,
26
+ "safety-risk-004": 0.5,
27
+ "support-normal-001": 0.75
28
+ }
29
+ },
30
+ "learned": {
31
+ "cases": 32,
32
+ "quality_pass_rate": 0.65625,
33
+ "group_bootstrap_95_ci": [
34
+ 0.40625,
35
+ 0.875
36
+ ],
37
+ "failure_counts": {
38
+ "state": 10,
39
+ "answer_fact": 1,
40
+ "citation": 5
41
+ },
42
+ "per_group_quality_pass_rate": {
43
+ "meeting-conflict-003": 0.75,
44
+ "meeting-conflict-005": 0.0,
45
+ "meeting-normal-001": 1.0,
46
+ "meeting-visual-001": 1.0,
47
+ "meeting-visual-008": 1.0,
48
+ "safety-risk-003": 0.25,
49
+ "safety-risk-004": 0.5,
50
+ "support-normal-001": 0.75
51
+ }
52
+ },
53
+ "never_vlm": {
54
+ "cases": 32,
55
+ "quality_pass_rate": 0.78125,
56
+ "group_bootstrap_95_ci": [
57
+ 0.53125,
58
+ 0.96875
59
+ ],
60
+ "failure_counts": {
61
+ "state": 6,
62
+ "answer_fact": 1,
63
+ "citation": 5
64
+ },
65
+ "per_group_quality_pass_rate": {
66
+ "meeting-conflict-003": 0.75,
67
+ "meeting-conflict-005": 0.0,
68
+ "meeting-normal-001": 1.0,
69
+ "meeting-visual-001": 1.0,
70
+ "meeting-visual-008": 1.0,
71
+ "safety-risk-003": 0.75,
72
+ "safety-risk-004": 1.0,
73
+ "support-normal-001": 0.75
74
+ }
75
+ }
76
+ },
77
+ "paired_quality": {
78
+ "always_vlm_minus_learned": {
79
+ "paired_cases": 32,
80
+ "quality_rate_delta": 0.03125,
81
+ "group_bootstrap_95_ci": [
82
+ 0.0,
83
+ 0.09375
84
+ ],
85
+ "both_pass": 21,
86
+ "left_only_pass": 1,
87
+ "right_only_pass": 0,
88
+ "both_fail": 10
89
+ },
90
+ "always_vlm_minus_never_vlm": {
91
+ "paired_cases": 32,
92
+ "quality_rate_delta": -0.09375,
93
+ "group_bootstrap_95_ci": [
94
+ -0.25,
95
+ 0.0625
96
+ ],
97
+ "both_pass": 21,
98
+ "left_only_pass": 1,
99
+ "right_only_pass": 4,
100
+ "both_fail": 6
101
+ },
102
+ "learned_minus_never_vlm": {
103
+ "paired_cases": 32,
104
+ "quality_rate_delta": -0.125,
105
+ "group_bootstrap_95_ci": [
106
+ -0.3125,
107
+ 0.0
108
+ ],
109
+ "both_pass": 21,
110
+ "left_only_pass": 0,
111
+ "right_only_pass": 4,
112
+ "both_fail": 7
113
+ }
114
+ }
115
+ }
e2e_summary.json ADDED
@@ -0,0 +1,51 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "schema_version": 1,
3
+ "physical_cases_per_system": 32,
4
+ "independent_test_groups": 8,
5
+ "latency_comparable_across_systems": false,
6
+ "latency_note": "Observed wall latency is retained for execution audit only. ASR/OCR observations are memoized across systems in requested order, so later systems exclude analyzer work and their latency must not be compared with the first system.",
7
+ "systems": {
8
+ "always_vlm": {
9
+ "cases": 32,
10
+ "quality_pass_rate": 0.6875,
11
+ "state_accuracy": 0.71875,
12
+ "exact_state_accuracy": 0.6875,
13
+ "answer_fact_coverage": 0.96875,
14
+ "citation_validity": 0.875,
15
+ "mean_answer_char_f1": 0.6598538614163614,
16
+ "visual_backend_call_rate": 1.0,
17
+ "text_answer_backend_call_rate": 1.0,
18
+ "backend_calls_per_case": 2.0,
19
+ "mean_observed_wall_latency_ms": 2576.251625374425,
20
+ "backend_call_savings_vs_always": 0.0
21
+ },
22
+ "learned": {
23
+ "cases": 32,
24
+ "quality_pass_rate": 0.65625,
25
+ "state_accuracy": 0.6875,
26
+ "exact_state_accuracy": 0.6875,
27
+ "answer_fact_coverage": 0.96875,
28
+ "citation_validity": 0.84375,
29
+ "mean_answer_char_f1": 0.6769566939286767,
30
+ "visual_backend_call_rate": 0.90625,
31
+ "text_answer_backend_call_rate": 1.0,
32
+ "backend_calls_per_case": 1.90625,
33
+ "mean_observed_wall_latency_ms": 12376.805822888855,
34
+ "backend_call_savings_vs_always": 0.046875
35
+ },
36
+ "never_vlm": {
37
+ "cases": 32,
38
+ "quality_pass_rate": 0.78125,
39
+ "state_accuracy": 0.8125,
40
+ "exact_state_accuracy": 0.8125,
41
+ "answer_fact_coverage": 0.96875,
42
+ "citation_validity": 0.84375,
43
+ "mean_answer_char_f1": 0.6423656732759995,
44
+ "visual_backend_call_rate": 0.0,
45
+ "text_answer_backend_call_rate": 1.0,
46
+ "backend_calls_per_case": 1.0,
47
+ "mean_observed_wall_latency_ms": 1510.641771601513,
48
+ "backend_call_savings_vs_always": 0.5
49
+ }
50
+ }
51
+ }
router_artifact.json ADDED
@@ -0,0 +1,1125 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "schema_version": 1,
3
+ "model_type": "four_gate_grouped_oof_logistic_with_frozen_bge",
4
+ "feature_names": [
5
+ "asr_confidence",
6
+ "ocr_confidence",
7
+ "snr_normalized",
8
+ "clipping_ratio",
9
+ "frame_change_score",
10
+ "text_overlap"
11
+ ],
12
+ "means": [
13
+ 0.5984041843291966,
14
+ 0.9886692975920228,
15
+ 0.38740253817260956,
16
+ 0.0006088501837213004,
17
+ 0.0590113267180487,
18
+ 0.11119001997114039
19
+ ],
20
+ "scales": [
21
+ 0.19605187939554394,
22
+ 0.009141897375754,
23
+ 0.15228899124350764,
24
+ 0.0022157638735014664,
25
+ 0.046181809711483425,
26
+ 0.0952325744684164
27
+ ],
28
+ "coefficients": [
29
+ -0.8040053602731648,
30
+ 0.20367841517418075,
31
+ 0.14121080495192506,
32
+ -0.37204370318636215,
33
+ -0.06489212943272263,
34
+ 0.07756360217093984
35
+ ],
36
+ "intercept": 0.014577317251189882,
37
+ "escalation_threshold": 0.3912382767204465,
38
+ "hard_risk_threshold": 0.630421353807803,
39
+ "hard_conflict_threshold": 0.34661490612654,
40
+ "visual_threshold": 0.419183177731241,
41
+ "training_data_sha256": "ccf888d64f5689627c97c392071e2a88617810d96c8c1aad06eb1728f79b8c66",
42
+ "seed": 20260824,
43
+ "metrics": {
44
+ "grouped_oof_recall": 0.9588014981273408,
45
+ "grouped_oof_escalation_rate": 0.9618055555555556,
46
+ "grouped_oof_uncertainty_brier": 0.26623208283603056,
47
+ "development_groups": 24.0,
48
+ "grouped_folds": 4.0
49
+ },
50
+ "risk_text_model": null,
51
+ "visual_text_model": null,
52
+ "risk_embedding_model": {
53
+ "model_id": "BAAI/bge-small-zh-v1.5",
54
+ "revision": "7999e1d3359715c523056ef9478215996d62a620",
55
+ "embedding_dimension": 512,
56
+ "coefficients": [
57
+ 0.007662025744155098,
58
+ -0.7141694968998114,
59
+ 0.07623148950627853,
60
+ -0.041841657154868026,
61
+ 0.09477789395133969,
62
+ 0.0516977000967392,
63
+ 0.13228885298290316,
64
+ -0.0898557050811419,
65
+ 0.004945581270215789,
66
+ 0.12420252906302309,
67
+ -0.07711460211292227,
68
+ 0.1589011530514023,
69
+ 0.47722869026988224,
70
+ -0.24324179713049604,
71
+ 0.09984767046879676,
72
+ -0.576556302311756,
73
+ -0.39808355157122555,
74
+ -0.22880586013740964,
75
+ 0.1824920081298546,
76
+ -0.10680153842187337,
77
+ -0.26594435687970436,
78
+ -0.2177030437621377,
79
+ -0.05877818433920755,
80
+ -0.5037888997689209,
81
+ 0.05934769577700187,
82
+ -0.12740502986179844,
83
+ 0.010317929860146034,
84
+ 0.0787552486318577,
85
+ 0.0763396131057123,
86
+ 0.6046805182295106,
87
+ 0.06260612827553966,
88
+ -0.22831529187887997,
89
+ -0.1344487580676067,
90
+ 0.10175192351946613,
91
+ -0.2775976090199446,
92
+ 0.5363078276970322,
93
+ -0.3756882270839027,
94
+ 0.4557577141916217,
95
+ -0.0913873660383468,
96
+ -0.10035824635763645,
97
+ -0.3080232730266825,
98
+ 0.15319022121238976,
99
+ -0.37295079871355985,
100
+ -0.18829016673097923,
101
+ -0.1786138428091699,
102
+ 0.2149902372504248,
103
+ 0.1364458015035106,
104
+ -0.07904528876275846,
105
+ 0.5662700780211531,
106
+ -0.31992363843949034,
107
+ -0.06569531793964628,
108
+ 0.17997482829009637,
109
+ -0.40389918177155715,
110
+ 0.6051421738675398,
111
+ -0.21483677777742588,
112
+ 0.23247725060761276,
113
+ -0.46087889624802275,
114
+ -0.17925984780038764,
115
+ 0.3082819631034047,
116
+ -0.05103668987874846,
117
+ -0.10724141815031,
118
+ 0.23559425313822369,
119
+ -0.18688021633749247,
120
+ -0.07688499614630412,
121
+ -0.6274533029478508,
122
+ -0.10280996122948247,
123
+ -0.30231546892097694,
124
+ -0.32765236965873173,
125
+ 0.07588269173223448,
126
+ 0.204684872605058,
127
+ 0.16298018240531603,
128
+ 0.1347243199024299,
129
+ 0.43973894311410766,
130
+ 0.12491655797592838,
131
+ -0.25750410456921363,
132
+ -0.142609108738015,
133
+ 0.5013876913902239,
134
+ 0.4196998196247118,
135
+ 0.13167721590598336,
136
+ 0.24007764661610534,
137
+ 0.009109725303352252,
138
+ 0.05316415242528987,
139
+ 0.0508388532390468,
140
+ -0.12227210283937803,
141
+ 0.3852370778921867,
142
+ -0.002705710978800487,
143
+ 0.1173270485689071,
144
+ 0.15073449522247181,
145
+ -0.4609885225964628,
146
+ 0.06164885456132461,
147
+ 0.33490539053572965,
148
+ 0.06642302387350119,
149
+ -0.6189965101757012,
150
+ -0.16809586656958883,
151
+ 0.11744579177722593,
152
+ 0.16743671563626697,
153
+ -0.2794510352754829,
154
+ -0.6088942372207904,
155
+ 0.15300233706185598,
156
+ -0.5711098172358764,
157
+ 0.12716377626925446,
158
+ 0.1791143080639329,
159
+ 0.47013678350403104,
160
+ 0.45536366907710274,
161
+ 0.02634822099529112,
162
+ -0.21278061907011864,
163
+ 0.3813689671240984,
164
+ -0.024166492385778393,
165
+ -0.6132727376949348,
166
+ -0.4460075031340679,
167
+ -0.09710738135730512,
168
+ 0.4289227764669695,
169
+ -0.16224599148511284,
170
+ -0.09599947610112894,
171
+ -0.022486719683222106,
172
+ -0.06710291272281972,
173
+ 0.06591180123186682,
174
+ 0.11116125061537693,
175
+ -0.38096338848472083,
176
+ 0.14460825284200077,
177
+ -0.41421773537491874,
178
+ -0.8345309294316035,
179
+ 0.019669209625756034,
180
+ -0.20265457557580413,
181
+ 0.22654246126790104,
182
+ 0.011504554253017754,
183
+ 0.22270216039416077,
184
+ 0.2886650598975271,
185
+ 0.05740812457448501,
186
+ -0.046920468138516254,
187
+ 0.05057312213109944,
188
+ 0.14887697125993055,
189
+ -0.157954179963204,
190
+ -0.20096302443367867,
191
+ -0.18089475278092576,
192
+ -0.03515098127667268,
193
+ 0.002669510093847192,
194
+ 0.18711643779391163,
195
+ -0.059304837262174635,
196
+ 0.029525313316748184,
197
+ 0.20774847386584636,
198
+ -0.07431333649395627,
199
+ -0.07796377136448762,
200
+ 0.11360167619743992,
201
+ 0.1319633202424443,
202
+ 0.6064827462323653,
203
+ -0.11289456653491989,
204
+ 0.19856084051583817,
205
+ 0.25053203065063173,
206
+ -0.25218619522484026,
207
+ 0.33396379352420624,
208
+ -0.09569953417193534,
209
+ -0.15569151138707493,
210
+ 0.6734913155690451,
211
+ 0.22724073945903875,
212
+ -0.14139785932782648,
213
+ -0.3446471336041315,
214
+ 0.4396182190387566,
215
+ 0.13945704538668185,
216
+ 0.2638607325334584,
217
+ -0.22329885382355166,
218
+ 0.32088719357019513,
219
+ 0.16566444713204237,
220
+ 0.301500597329245,
221
+ 0.06057584585985235,
222
+ -0.21560193283624218,
223
+ -0.005093293718479465,
224
+ 0.19676476533798468,
225
+ 0.06606663365488884,
226
+ 0.15915383208657435,
227
+ -0.2626697549762528,
228
+ 0.05434197818113439,
229
+ -0.30801211128260086,
230
+ 0.023810692680091526,
231
+ 0.27204294092965603,
232
+ -0.14262845545411584,
233
+ -0.7173418996087871,
234
+ 0.3962809516352104,
235
+ 0.0466797956256157,
236
+ 0.251996339455604,
237
+ -0.3272187376030538,
238
+ 0.14268725324119075,
239
+ -0.1423478557456533,
240
+ -0.3090472198480806,
241
+ -0.35590959686083845,
242
+ 0.6106828994519948,
243
+ -0.18224801032637036,
244
+ 0.03839163505228328,
245
+ -0.41122096445941825,
246
+ 0.015037811794301638,
247
+ -0.0500993921990379,
248
+ 0.09473134818011593,
249
+ 0.39863232252991065,
250
+ -0.11339872799793693,
251
+ -0.17778451424844047,
252
+ -0.2748610682500473,
253
+ -0.4474903564215646,
254
+ 0.11057627953427282,
255
+ -0.42724375038617673,
256
+ 0.5607463697068104,
257
+ 0.11895700533050364,
258
+ -0.039853077322335444,
259
+ 0.2735825998548635,
260
+ -0.2690108922669548,
261
+ -0.1333596764913424,
262
+ 0.3429742742291783,
263
+ 0.0998652756194102,
264
+ 0.48621493647353226,
265
+ 0.16476097869979145,
266
+ -0.2530559857769199,
267
+ -0.2940961129691533,
268
+ 0.2239828091775879,
269
+ -0.16767411516829053,
270
+ 0.2121914956972058,
271
+ -0.2579265356205543,
272
+ -0.00023089170073640648,
273
+ 0.13905005611097954,
274
+ -0.19020641452595552,
275
+ 0.3374513870316398,
276
+ -0.2474260956196787,
277
+ 0.7940328657858415,
278
+ -0.2606026086011829,
279
+ -0.05224597834531189,
280
+ 0.4153862091010088,
281
+ 0.15079248776375304,
282
+ -0.187260503629211,
283
+ 0.27448596049195595,
284
+ -0.08517899323824756,
285
+ -0.12456489109225952,
286
+ 0.018256527649130754,
287
+ 0.4217123456300277,
288
+ 0.47575761154585255,
289
+ 0.13560553593752658,
290
+ -0.6036253422193626,
291
+ 0.14537745885806927,
292
+ 0.2409195990005231,
293
+ 0.010460335319622044,
294
+ -0.06481033391553385,
295
+ -0.07321235928695526,
296
+ -0.10836941585894107,
297
+ 0.24656901528401962,
298
+ 0.20006373911566971,
299
+ -0.0038623983893992363,
300
+ -0.05654023188138607,
301
+ 0.020413776701463385,
302
+ -0.14728908975237576,
303
+ -0.28580825979917707,
304
+ -0.6894495538415679,
305
+ -0.08277941058080167,
306
+ 0.43052501106108454,
307
+ 0.059018829293383285,
308
+ -0.13913525449034,
309
+ 0.00014284107944296116,
310
+ -0.0010710981919843424,
311
+ -0.3171700754914986,
312
+ -0.03384766486740618,
313
+ 0.34507734780638194,
314
+ -0.0005615219774957316,
315
+ 0.45120417346005437,
316
+ -0.0700585767668361,
317
+ 0.2778774569889433,
318
+ -0.47608606045289076,
319
+ 0.2514703552604392,
320
+ 0.39081613344310684,
321
+ -0.18179950256284255,
322
+ -0.5848282562229613,
323
+ -0.14700744585330733,
324
+ -0.004133645882029186,
325
+ 0.06965661189551023,
326
+ -0.4464319836574728,
327
+ 0.30519064108341687,
328
+ 0.1369466371678625,
329
+ 0.31192016728112176,
330
+ -0.5152417314511427,
331
+ 0.541849879648225,
332
+ 0.22435207780048455,
333
+ 0.7546146196743289,
334
+ 0.21725036920421326,
335
+ 0.21699397092575284,
336
+ 0.3563760005715661,
337
+ -0.04893356436286871,
338
+ 0.6597826242456876,
339
+ 0.04937790876192553,
340
+ -0.32266087755679396,
341
+ 0.35290658383895906,
342
+ -0.16730018503909627,
343
+ 0.17174128014012466,
344
+ -0.3244840780507594,
345
+ 0.5783053075761347,
346
+ -0.37025823778783273,
347
+ 0.3295934434057696,
348
+ -0.28867122548500856,
349
+ 0.5329074921440071,
350
+ 0.04491110098727547,
351
+ -0.2797877209320986,
352
+ -0.6310142621529669,
353
+ 0.2908405721913978,
354
+ 0.01707878800468819,
355
+ 0.1569749155739621,
356
+ -0.09246659447773187,
357
+ -0.028737371059247955,
358
+ 0.003524293875593761,
359
+ -0.17492663918733417,
360
+ 0.12574672158534203,
361
+ -0.5269074765324523,
362
+ 0.014632993380400925,
363
+ 0.14922693924769487,
364
+ 0.265929592860134,
365
+ -0.22829319548629606,
366
+ 0.4499386093350679,
367
+ -0.09152308408707262,
368
+ 0.16209452655010176,
369
+ 0.05074943631920171,
370
+ -0.1622995483596696,
371
+ -0.1503149925592783,
372
+ -0.05657272361499154,
373
+ 0.08814221548533721,
374
+ 0.052130341637250555,
375
+ 0.07968035055395906,
376
+ 0.45422588786464557,
377
+ -0.0898391590687328,
378
+ 0.697605215240023,
379
+ -0.3556641765778416,
380
+ -0.26551031584075035,
381
+ -0.11110639407008298,
382
+ 0.010552882402446914,
383
+ -0.05167816009306504,
384
+ -0.11538293171445987,
385
+ -0.12332180225315376,
386
+ 0.6860751942245118,
387
+ 0.0403584380651553,
388
+ -0.13239785738929102,
389
+ 0.4371214096523252,
390
+ -0.544798272552446,
391
+ 0.1759630364332709,
392
+ 0.06737216697015039,
393
+ -0.058048032531371196,
394
+ 0.007184005357349283,
395
+ 0.23999215339352328,
396
+ -0.4877465509189127,
397
+ -0.3500250761240196,
398
+ 0.438450799516787,
399
+ 0.06959471946881816,
400
+ -0.23128599419559218,
401
+ 0.1706456041278082,
402
+ -0.3340440063217172,
403
+ 0.0913176789427986,
404
+ -0.9020671545144728,
405
+ 0.023287607993034122,
406
+ -0.28302081893779557,
407
+ -0.39052460382185966,
408
+ -0.07596959352571803,
409
+ 0.026179328007219102,
410
+ -0.09464087913718891,
411
+ 0.19530935498958543,
412
+ 0.043468386438953024,
413
+ 0.17654290495479316,
414
+ -0.16853137477561286,
415
+ -0.24678329147535336,
416
+ 0.12396175712815506,
417
+ -0.04271116635833899,
418
+ -0.05884194903076503,
419
+ 0.2071054753412583,
420
+ -0.13742437525774262,
421
+ 0.2669439383287722,
422
+ 0.3090872845375883,
423
+ -0.02255834153486707,
424
+ -0.17369984123162266,
425
+ -0.25818602609016855,
426
+ 0.07829264421375824,
427
+ 0.36051752890357863,
428
+ -0.04895145986977088,
429
+ 0.7209612922412801,
430
+ -0.409194793377698,
431
+ 0.43815463533568855,
432
+ -0.06028240136540023,
433
+ 0.1534623296864694,
434
+ 0.4760155792012816,
435
+ 0.03865756479307296,
436
+ -0.038876433912575725,
437
+ 0.03675308792162826,
438
+ -0.25541982767678917,
439
+ 0.35801306748878253,
440
+ -0.3387843224248126,
441
+ -0.0708957362147179,
442
+ -0.18441784179511034,
443
+ 0.14591623812683838,
444
+ 0.4830739261003857,
445
+ -0.06126600513334302,
446
+ -0.20404930073615776,
447
+ 0.17360804214980632,
448
+ -0.024391581505170342,
449
+ -0.18069334797484124,
450
+ -0.346187623061213,
451
+ -0.4534139959270948,
452
+ -0.6249294174266691,
453
+ -0.33505661269339243,
454
+ -0.3224539022936928,
455
+ 0.04705610384883693,
456
+ 0.07121132202190623,
457
+ -0.2424150283008643,
458
+ -0.0671267261473386,
459
+ 0.16636758399088944,
460
+ 0.11637217802032948,
461
+ 0.29438134010474504,
462
+ 0.06954288263089385,
463
+ 0.30003460501325996,
464
+ -0.3045853502820464,
465
+ 0.2059802555934281,
466
+ 0.29059244262719164,
467
+ 0.07051033996327596,
468
+ -0.07613257785150264,
469
+ -0.20247373290622275,
470
+ 0.02805448047363747,
471
+ 0.5796336679783479,
472
+ -0.5755152636522294,
473
+ 0.267283641985174,
474
+ 0.0017175777114517429,
475
+ -0.5294743248884337,
476
+ -0.4099790942088795,
477
+ 0.07046508092323862,
478
+ -0.03637066337973398,
479
+ 0.3308120176753581,
480
+ 0.38462121864682286,
481
+ 0.21388138204639395,
482
+ -0.39494856528831274,
483
+ -0.06541264525419045,
484
+ 0.20057994109041313,
485
+ 0.6786386645703181,
486
+ 0.5747399484133838,
487
+ -0.13188186841672064,
488
+ 0.15514505048907443,
489
+ -0.15759701792680156,
490
+ 0.057494236741077544,
491
+ 0.11576582263150735,
492
+ 0.4344526124179084,
493
+ 0.020189204093114466,
494
+ 0.1929567458643577,
495
+ -0.23979032485363616,
496
+ -0.11740417378081978,
497
+ 0.04976472356120376,
498
+ 0.4629660900997577,
499
+ -0.27822338532868296,
500
+ 0.12871061824657742,
501
+ 0.11409004863233921,
502
+ 0.04859311190132807,
503
+ -0.9276086728569232,
504
+ -0.19161971405340944,
505
+ -0.36004468917391474,
506
+ 0.2564053622978546,
507
+ -0.47645411251204584,
508
+ 0.05528792988525499,
509
+ 0.06821092161237784,
510
+ -0.22793119710894788,
511
+ 0.20570076780422478,
512
+ -0.3515596400366589,
513
+ 0.023625988525648574,
514
+ -0.22735020283141574,
515
+ -0.1802117963904701,
516
+ 0.16648851591983477,
517
+ -0.2656857754831091,
518
+ -0.48910341920354594,
519
+ 0.12022017824398784,
520
+ -0.510763109794434,
521
+ 0.25794055656417597,
522
+ 0.07239611059395042,
523
+ -0.012492712964815897,
524
+ 0.5067809656800218,
525
+ -0.03622064171427367,
526
+ -0.2077428256868999,
527
+ 0.17523851593890225,
528
+ -0.036537226101597806,
529
+ -0.12795348049854122,
530
+ -0.3399095681291818,
531
+ -0.4709974401302609,
532
+ -0.43235290208369237,
533
+ -0.1647664430002508,
534
+ -0.09317024837957168,
535
+ 0.597587508850784,
536
+ -0.026571431198142502,
537
+ -0.09948352171809821,
538
+ -0.15835642136496936,
539
+ 0.2848669224222134,
540
+ -0.24869111932224522,
541
+ 0.15882237744139172,
542
+ 0.23491645433570943,
543
+ 0.6093286166913296,
544
+ 0.0007851043258982056,
545
+ -0.08254284259663536,
546
+ -0.06448074671615701,
547
+ -0.05562934150695031,
548
+ -0.27340376054513293,
549
+ -0.08668594422455984,
550
+ 0.16164114010717998,
551
+ -0.21616107594355138,
552
+ 0.1493444737814225,
553
+ 0.1100617270258033,
554
+ -0.3784245367811329,
555
+ -0.6233360706735349,
556
+ 0.49192062023624367,
557
+ -0.1392229350922768,
558
+ -0.46833916605070447,
559
+ 0.22617792341639947,
560
+ 0.10374892512219298,
561
+ -0.02541066910869443,
562
+ 0.04342958313663511,
563
+ -0.5219714444081168,
564
+ -0.07132046055949809,
565
+ 0.4165036455990968,
566
+ -0.18831057121889316,
567
+ -0.45722289625299556,
568
+ 0.8215948261029691
569
+ ],
570
+ "intercept": -0.5483732154783423,
571
+ "max_length": 256
572
+ },
573
+ "visual_embedding_model": {
574
+ "model_id": "BAAI/bge-small-zh-v1.5",
575
+ "revision": "7999e1d3359715c523056ef9478215996d62a620",
576
+ "embedding_dimension": 512,
577
+ "coefficients": [
578
+ 0.15288017948681573,
579
+ 0.28034001661956126,
580
+ 0.35256594958874093,
581
+ 0.10774142275803182,
582
+ 0.04896336025547128,
583
+ 0.3133324496319549,
584
+ -0.08375958097711962,
585
+ 0.025428989865681068,
586
+ 0.2697144543253356,
587
+ 0.0018151025507232323,
588
+ 0.1302811151004063,
589
+ -0.051197380186417216,
590
+ -0.2754912321349183,
591
+ -0.24963773761993324,
592
+ -0.24625828419387305,
593
+ -0.259441494145526,
594
+ 0.40048789410631386,
595
+ 0.026503038111362554,
596
+ -0.16116411580644657,
597
+ 0.5485996540784291,
598
+ -0.2594022888154592,
599
+ -0.6591321982071502,
600
+ 0.01902073165146004,
601
+ 0.05040649192954869,
602
+ -0.09438900150382808,
603
+ 0.12755102386433734,
604
+ -0.05104046095517025,
605
+ -0.04647113006135928,
606
+ 0.23995473289028577,
607
+ -0.08406062162561131,
608
+ -0.3010841368223521,
609
+ -0.19111346243983124,
610
+ -0.29251352860535623,
611
+ 0.15425020751095367,
612
+ -0.22547809051236464,
613
+ 0.015051714095651715,
614
+ -0.043606081675281745,
615
+ -0.13268611939320807,
616
+ 0.15400596381149576,
617
+ 0.2693766038106768,
618
+ 0.3307065033950017,
619
+ -0.12577243408758987,
620
+ -0.26705349250390403,
621
+ -0.6277011718403798,
622
+ -0.08181967237657925,
623
+ -0.07367935034355308,
624
+ -0.35209218327725816,
625
+ -0.043870210763814615,
626
+ -0.05999503672891342,
627
+ -0.184658385003852,
628
+ 0.022181560757546744,
629
+ -0.19183693518618075,
630
+ -0.013107803913601475,
631
+ 0.20256432079888925,
632
+ 0.3347624673700205,
633
+ 0.6793641532967644,
634
+ -0.011957192189081927,
635
+ -0.3112350998125864,
636
+ 0.11553873649084787,
637
+ 0.2480383517104258,
638
+ 0.33127214889037865,
639
+ 0.31610808978229393,
640
+ 0.565561128491159,
641
+ -0.12608476352367187,
642
+ -0.07618272294713427,
643
+ 0.006259098402294123,
644
+ 0.12977069251468307,
645
+ -0.26076218796065537,
646
+ -0.21280027983137367,
647
+ 0.04229665151345746,
648
+ 0.13138472327907724,
649
+ 0.15944197716146943,
650
+ -0.22875833923826805,
651
+ -0.0997006050161596,
652
+ -0.1603833645217211,
653
+ 0.43734472243047473,
654
+ 0.1793372628265469,
655
+ 0.06477955383026043,
656
+ 0.5709087270287277,
657
+ -0.5530601828923516,
658
+ -0.07383826528321398,
659
+ 0.019365969216293775,
660
+ -0.09410865097118686,
661
+ 0.055075411240995825,
662
+ -0.23563564016826272,
663
+ -0.3454289429961646,
664
+ -0.223201359020439,
665
+ 0.46385937250940595,
666
+ -0.17965611828946146,
667
+ -0.10696524252791621,
668
+ -0.3957249924056434,
669
+ -0.08847000367683246,
670
+ -0.4027214014667627,
671
+ 0.1867292838952492,
672
+ -0.171558789564516,
673
+ 0.10764719406589017,
674
+ 0.12361033195566219,
675
+ -0.5011957565205546,
676
+ -0.2826463445470648,
677
+ -0.008786908097207741,
678
+ 0.16759027173683913,
679
+ 0.10846984324418614,
680
+ 0.45246354996665616,
681
+ 0.017742702264512786,
682
+ 0.6577997643121655,
683
+ -0.3766605336159885,
684
+ -0.27681776568958694,
685
+ -0.027053773210424535,
686
+ -0.24742035331211248,
687
+ 0.08430531176350635,
688
+ 0.22604262097074204,
689
+ -0.6286753445355091,
690
+ 0.6673689202103827,
691
+ -0.1353186710740125,
692
+ 0.2671016693711405,
693
+ -0.40636530193326376,
694
+ -0.04250760280591084,
695
+ 0.20207416609175255,
696
+ 0.07655563676108867,
697
+ 0.5232631791033037,
698
+ -0.1558715782028673,
699
+ 0.2884800344368237,
700
+ -0.2786132071549931,
701
+ 0.5483306435646782,
702
+ 0.0432838093339513,
703
+ -0.1551819301246636,
704
+ -0.017638318904504607,
705
+ 0.00209213275336865,
706
+ -0.12421207724583051,
707
+ 0.06930540100150906,
708
+ 0.5383979747714221,
709
+ -0.2776937608965831,
710
+ 0.11281609927063467,
711
+ -0.12029005106458164,
712
+ 0.03627680495429228,
713
+ 0.21372803826399156,
714
+ -0.48377103556093726,
715
+ 0.4347822294445627,
716
+ 0.29839512160167936,
717
+ -0.23036315308019292,
718
+ 0.5386950248278709,
719
+ -0.1691345910175734,
720
+ 0.37601366463265296,
721
+ 0.0773869487551093,
722
+ -0.3321221501588597,
723
+ -0.1879477598569936,
724
+ -0.3041643887669381,
725
+ 0.12869116604991251,
726
+ 0.20393255348651945,
727
+ -0.07397802505225169,
728
+ 0.2811222249133581,
729
+ 0.432764484322847,
730
+ 0.26027505703964476,
731
+ 0.017521840693003887,
732
+ 0.11340726974814608,
733
+ -0.18565602984242233,
734
+ -0.03355455523441567,
735
+ 0.3940070426182175,
736
+ 0.40323882641475284,
737
+ 0.05537831683249908,
738
+ -0.0478096328354154,
739
+ -0.31962818983758257,
740
+ -0.010868168999457561,
741
+ -0.07960958714068812,
742
+ -0.21292288989860633,
743
+ 0.06539084864907137,
744
+ 0.3477943168567016,
745
+ -0.2966754151650263,
746
+ -0.3817202650485718,
747
+ -0.11005824426348727,
748
+ -0.788217441459477,
749
+ -0.2790149117664826,
750
+ 0.17059766738704032,
751
+ -0.384080241039986,
752
+ 0.2522123193921934,
753
+ 0.292694380694104,
754
+ 0.2926495386429877,
755
+ 0.05648116133427689,
756
+ 0.19165469343631414,
757
+ -0.07588795525359147,
758
+ -0.15409710238116409,
759
+ -0.170896258124394,
760
+ -0.006286948936159283,
761
+ 0.4362882511826906,
762
+ 0.30681756920079,
763
+ -0.12903860954650634,
764
+ -0.6130169739111501,
765
+ -0.189217432381222,
766
+ -0.08692855205367189,
767
+ -0.02813193350319749,
768
+ -0.3100558228219196,
769
+ 0.36187001536305996,
770
+ 0.16765962082297525,
771
+ -0.3500802090956713,
772
+ 0.21943000153653894,
773
+ 0.06305769323559719,
774
+ 0.19893091533660956,
775
+ 0.2962307947844857,
776
+ -0.5218711602301793,
777
+ 0.138010598336156,
778
+ 0.6511335215422157,
779
+ 0.2925307390099801,
780
+ 0.480552120719635,
781
+ -0.017393925868054657,
782
+ -0.11486885929960945,
783
+ -0.03108455232403387,
784
+ -0.17447977776759233,
785
+ -0.2550372111536348,
786
+ 0.19761196580190202,
787
+ -0.03533224047202053,
788
+ -0.21788616941927585,
789
+ -0.506894501691044,
790
+ 0.3386977339852481,
791
+ 0.3062162341897937,
792
+ -0.50778709647438,
793
+ 0.2702886646760909,
794
+ 0.25803107459245284,
795
+ 0.27058409563124314,
796
+ 0.035883960545613706,
797
+ 0.2675683018700185,
798
+ -0.11137463212247642,
799
+ -0.3807733850535155,
800
+ 0.2259539370040709,
801
+ 0.06174887113281623,
802
+ 0.02981023852941477,
803
+ 0.27460837546709116,
804
+ 0.13660942352943453,
805
+ -0.5624728759359434,
806
+ 0.14628618087848402,
807
+ 0.5229297227013479,
808
+ -0.23272287257416308,
809
+ 0.11490094237123163,
810
+ 0.09474533107500503,
811
+ 0.23299837034048607,
812
+ 0.3408311146836348,
813
+ 0.3456170329805354,
814
+ 0.24524439266019118,
815
+ -0.17279840267936264,
816
+ -0.3439506542151117,
817
+ 0.19753631964714544,
818
+ 0.21112034006540115,
819
+ -0.24054262353771394,
820
+ -0.24543172293112542,
821
+ -0.7022749329439336,
822
+ -0.24792250253335696,
823
+ 0.13655373247991237,
824
+ -0.15664777168235158,
825
+ -0.3348708553696203,
826
+ -0.15571540636938921,
827
+ -0.024511485819360832,
828
+ 0.2024301230206176,
829
+ 0.09699593226823643,
830
+ 0.14575121324049561,
831
+ 0.15258113094707,
832
+ -0.3028212556383117,
833
+ -0.42688376467663713,
834
+ 0.18395328296515193,
835
+ -0.6426756358836201,
836
+ 0.38772072018841663,
837
+ -0.011986579383063528,
838
+ 0.025161149145428045,
839
+ -0.15381125335033688,
840
+ 0.0678554760863422,
841
+ 0.3905030465018981,
842
+ 0.0300642561708253,
843
+ 0.07642049592718832,
844
+ 0.19409744425944692,
845
+ -0.40312634110059675,
846
+ 0.14558158784541472,
847
+ -0.23420846966524686,
848
+ 0.23241536289786477,
849
+ -0.1495777637750904,
850
+ 0.23077064904709602,
851
+ 0.090140562056183,
852
+ -0.2596734720136315,
853
+ -0.02965371139432268,
854
+ 0.225105606336047,
855
+ 0.42617469542610914,
856
+ 0.2673232454024721,
857
+ -0.0852230325184744,
858
+ 0.05685713865334481,
859
+ 0.20446664021811212,
860
+ -0.7353212951681461,
861
+ -0.2231748075653183,
862
+ 0.36790322724751384,
863
+ 0.2559592924678584,
864
+ 0.012175953638741802,
865
+ -0.4491313458468742,
866
+ 0.24107629430816976,
867
+ -0.11300882871786004,
868
+ -0.031573116674652756,
869
+ 0.11168642338913444,
870
+ 0.2260499658827945,
871
+ -0.3060389522711188,
872
+ -0.3429918620117447,
873
+ -0.10827835194856027,
874
+ 0.25888838381186363,
875
+ 0.1463269160146203,
876
+ -0.4301249988598537,
877
+ -0.07134965166193265,
878
+ 0.11186821844343349,
879
+ -0.19678755563621525,
880
+ 0.5168784688312629,
881
+ -0.27281272445420585,
882
+ 0.26935279942667384,
883
+ -0.48878484897918195,
884
+ 0.28227352986979476,
885
+ -0.09263925261053138,
886
+ -0.19457108172054113,
887
+ -0.28499760491421455,
888
+ -0.03654506262251398,
889
+ 0.08262340655846385,
890
+ -0.32826629303097277,
891
+ -0.4855071110573556,
892
+ 0.33728918903882926,
893
+ -0.07839335851147236,
894
+ 0.46618844523374176,
895
+ -0.09414373801911907,
896
+ 0.31413890180363624,
897
+ 0.12158895855446844,
898
+ -0.30878205998746927,
899
+ 0.15683064735221283,
900
+ 0.4401738194741865,
901
+ 0.11305190614841115,
902
+ -0.34730529767321966,
903
+ 0.05081549578482955,
904
+ -0.6356485310819298,
905
+ 0.25999675219963103,
906
+ -0.43202773399964994,
907
+ -0.010256280874764037,
908
+ 0.19932270620813222,
909
+ -0.736190513532644,
910
+ 0.07331398481948644,
911
+ 0.054880241522645486,
912
+ -0.07460784362592962,
913
+ -0.28857519403079784,
914
+ 0.28018097420803983,
915
+ 0.157313392883975,
916
+ 0.27838573813102196,
917
+ 0.9869710676913036,
918
+ -0.11515281345567109,
919
+ -0.11608698855854264,
920
+ 0.19456836241840758,
921
+ -0.025743113431268685,
922
+ -0.3929685104245213,
923
+ -0.5078152553432957,
924
+ 0.20174605633694556,
925
+ -0.3075244604486868,
926
+ -0.016447246574448945,
927
+ -0.44220197983915366,
928
+ 0.45716968971082744,
929
+ 0.0071083201661456255,
930
+ -0.08561944157400685,
931
+ 0.25913479667766753,
932
+ 1.0021675598476822,
933
+ -0.15368538556203115,
934
+ 0.13108829576863248,
935
+ 0.4639860678148545,
936
+ 0.11352989672654673,
937
+ 0.34532443928657286,
938
+ -0.33014114475838985,
939
+ 0.004437541380570821,
940
+ -0.3256240471349774,
941
+ 0.4106288618849622,
942
+ 0.05484160631033312,
943
+ 0.0946800281889622,
944
+ -0.2959466984088272,
945
+ 0.2443893200854875,
946
+ -0.2025670012135975,
947
+ 0.02710647429913176,
948
+ -0.06544140946191422,
949
+ -0.44812620385082674,
950
+ 0.02644650747990818,
951
+ -0.004154462504482658,
952
+ -0.157580763710019,
953
+ -0.16720826237518124,
954
+ 0.19157153962566112,
955
+ 0.31838462775320386,
956
+ 0.18500415196158476,
957
+ -0.04122942056167146,
958
+ -0.2134590679554055,
959
+ 0.02675183400960697,
960
+ -0.1523039998271947,
961
+ -0.07086830002318338,
962
+ -0.17700199137084052,
963
+ -0.15090910410300828,
964
+ 0.045595852771433854,
965
+ -0.3931571122788859,
966
+ -0.288604061578777,
967
+ -0.3112747647507781,
968
+ -0.18656894109215916,
969
+ 0.3761834475995305,
970
+ 0.248845575784508,
971
+ 0.08648650414588042,
972
+ 0.2089184102164945,
973
+ 0.22525147529458683,
974
+ 0.1976961033017284,
975
+ 0.36835272046133577,
976
+ -0.061107670488005944,
977
+ 0.548755197673712,
978
+ -0.10758113322795078,
979
+ 0.001850454792062723,
980
+ 0.11330108531202239,
981
+ -0.1548870227734925,
982
+ 0.14468463599842804,
983
+ 0.22997174601596476,
984
+ -0.35384529526018205,
985
+ -0.22447089923644037,
986
+ 0.07457897578649998,
987
+ 0.09466797658053117,
988
+ 0.07277198026198381,
989
+ -0.27165952045453406,
990
+ -0.35186881653854624,
991
+ 0.24879217244787732,
992
+ -0.0995495504136883,
993
+ -0.38563973289667935,
994
+ 0.48863739831725445,
995
+ 0.1954380998853885,
996
+ 0.00005662420147029327,
997
+ -0.016913494518015142,
998
+ -0.3150108480241563,
999
+ 0.3018603777352657,
1000
+ -0.06694396994858627,
1001
+ 0.35736150028813435,
1002
+ 0.09240476839000383,
1003
+ -0.10627125884264449,
1004
+ 0.1541485555202587,
1005
+ -0.03397623908488935,
1006
+ 0.22561822326122932,
1007
+ 0.16610138199051105,
1008
+ -0.23120047390952297,
1009
+ 0.060500326657201224,
1010
+ -0.2258312426568248,
1011
+ 0.545985039112117,
1012
+ -0.2414634434480881,
1013
+ 0.33564943714347284,
1014
+ -0.05770508461708763,
1015
+ 0.0472299503154265,
1016
+ -0.3152405939681583,
1017
+ -0.34988909686117986,
1018
+ -0.33774379351093514,
1019
+ 0.3709969323987613,
1020
+ 0.28187110553100836,
1021
+ -0.2369852369313458,
1022
+ -0.5660561388432268,
1023
+ -0.19878322612364208,
1024
+ -0.5063774145742979,
1025
+ -0.6536567863670162,
1026
+ 0.22339927058171383,
1027
+ 0.059128493229686684,
1028
+ -0.7844711693765513,
1029
+ 0.28782070995760184,
1030
+ 0.31678461603275987,
1031
+ 0.20952085755231115,
1032
+ -0.1934076054459525,
1033
+ -0.5331731343700408,
1034
+ -0.033271020108989864,
1035
+ 0.0880994782707277,
1036
+ 0.24583069257611975,
1037
+ 0.38835536844270646,
1038
+ -0.45285814964677884,
1039
+ -0.334739455509092,
1040
+ -0.35770477835571607,
1041
+ -0.26570114692502583,
1042
+ -0.2791689665064966,
1043
+ -0.5394928587399432,
1044
+ 0.5446459026173321,
1045
+ 0.185168961584898,
1046
+ -0.30384005796392083,
1047
+ 0.17172809609637468,
1048
+ 0.18363197821938293,
1049
+ -0.2580222204742076,
1050
+ 0.2745952076518502,
1051
+ -0.2418218677347761,
1052
+ 0.03165181781167774,
1053
+ 0.2098790710761288,
1054
+ 0.13763465767062774,
1055
+ 0.2308443407362417,
1056
+ 0.1934305565199791,
1057
+ -0.33452862380568993,
1058
+ -0.14101528904142338,
1059
+ 0.42380524014299703,
1060
+ 0.26096120564881614,
1061
+ -0.33666514101533246,
1062
+ 0.2070146430110362,
1063
+ -0.46922658534025485,
1064
+ 0.06659364827062465,
1065
+ 0.08557426339902977,
1066
+ -0.10337293781304253,
1067
+ -1.0761481101932995,
1068
+ -0.6132166881621847,
1069
+ -0.14206431999481442,
1070
+ 0.5248349011579194,
1071
+ 0.2078209036151541,
1072
+ 0.24452859517418465,
1073
+ -0.3089836012259404,
1074
+ 0.270079874912177,
1075
+ 0.11720034929113866,
1076
+ -0.28126414940137917,
1077
+ -0.07628341330882753,
1078
+ -0.24795241097158033,
1079
+ 0.02460305502070274,
1080
+ 0.38589171458168153,
1081
+ 0.6832371545518914,
1082
+ 0.37082341977542266,
1083
+ 0.24719368720288967,
1084
+ -0.37963434829257714,
1085
+ -0.1079354792164504,
1086
+ 0.12460049434422721,
1087
+ -0.24704586218067576,
1088
+ -0.3672044262746687,
1089
+ -0.08731767807191713
1090
+ ],
1091
+ "intercept": -0.41452725177336386,
1092
+ "max_length": 256
1093
+ },
1094
+ "conflict_model": {
1095
+ "feature_names": [
1096
+ "text_overlap",
1097
+ "numeric_mismatch",
1098
+ "polarity_mismatch",
1099
+ "asr_confidence",
1100
+ "ocr_confidence"
1101
+ ],
1102
+ "means": [
1103
+ 0.11119001997114039,
1104
+ 0.2048611111111111,
1105
+ 0.3003472222222222,
1106
+ 0.5984041843291966,
1107
+ 0.9886692975920228
1108
+ ],
1109
+ "scales": [
1110
+ 0.0952325744684164,
1111
+ 0.40360009448144424,
1112
+ 0.45840895314730934,
1113
+ 0.19605187939554394,
1114
+ 0.009141897375754
1115
+ ],
1116
+ "coefficients": [
1117
+ -0.7937534320725415,
1118
+ 0.826873047713812,
1119
+ 0.6796950658930959,
1120
+ 0.23732890514997987,
1121
+ 0.3594398815082791
1122
+ ],
1123
+ "intercept": -0.3723765750283073
1124
+ }
1125
+ }
router_test_metrics.json ADDED
@@ -0,0 +1,343 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "schema_version": 1,
3
+ "evaluation_split": "test",
4
+ "independent_groups": 8,
5
+ "examples": 576,
6
+ "bootstrap_unit": "group_id",
7
+ "bootstrap_iterations": 1000,
8
+ "systems": {
9
+ "learned_hybrid": {
10
+ "examples": 576,
11
+ "positives": 520,
12
+ "true_positives": 493,
13
+ "false_positives": 54,
14
+ "false_negatives": 27,
15
+ "recall": 0.948076923076923,
16
+ "precision": 0.9012797074954296,
17
+ "specificity": 0.03571428571428571,
18
+ "escalation_rate": 0.9496527777777778,
19
+ "brier": 0.25443746488730223,
20
+ "ece": 0.4136310890206015,
21
+ "auroc": 0.8687843406593406,
22
+ "recall_95ci": [
23
+ 0.8519230769230769,
24
+ 1.0
25
+ ],
26
+ "escalation_rate_95ci": [
27
+ 0.8628472222222222,
28
+ 1.0
29
+ ],
30
+ "estimated_vlm_gpu_seconds": 191.45,
31
+ "vlm_call_savings_vs_always_vlm": 0.05034722222222221
32
+ },
33
+ "v3_rule_router": {
34
+ "examples": 576,
35
+ "positives": 520,
36
+ "true_positives": 155,
37
+ "false_positives": 3,
38
+ "false_negatives": 365,
39
+ "recall": 0.2980769230769231,
40
+ "precision": 0.9810126582278481,
41
+ "specificity": 0.9464285714285714,
42
+ "escalation_rate": 0.2743055555555556,
43
+ "brier": 0.6388888888888888,
44
+ "ece": 0.6388888888888888,
45
+ "auroc": 0.6222527472527473,
46
+ "recall_95ci": [
47
+ 0.03125,
48
+ 0.6319444444444444
49
+ ],
50
+ "escalation_rate_95ci": [
51
+ 0.03125,
52
+ 0.5954861111111112
53
+ ],
54
+ "estimated_vlm_gpu_seconds": 55.3,
55
+ "vlm_call_savings_vs_always_vlm": 0.7256944444444444
56
+ },
57
+ "always_vlm": {
58
+ "examples": 576,
59
+ "positives": 520,
60
+ "true_positives": 520,
61
+ "false_positives": 56,
62
+ "false_negatives": 0,
63
+ "recall": 1.0,
64
+ "precision": 0.9027777777777778,
65
+ "specificity": 0.0,
66
+ "escalation_rate": 1.0,
67
+ "brier": 0.09722222222222222,
68
+ "ece": 0.09722222222222221,
69
+ "auroc": 0.5,
70
+ "recall_95ci": [
71
+ 1.0,
72
+ 1.0
73
+ ],
74
+ "escalation_rate_95ci": [
75
+ 1.0,
76
+ 1.0
77
+ ],
78
+ "estimated_vlm_gpu_seconds": 201.6,
79
+ "vlm_call_savings_vs_always_vlm": 0.0
80
+ },
81
+ "never_vlm": {
82
+ "examples": 576,
83
+ "positives": 520,
84
+ "true_positives": 0,
85
+ "false_positives": 0,
86
+ "false_negatives": 520,
87
+ "recall": 0.0,
88
+ "precision": 1.0,
89
+ "specificity": 1.0,
90
+ "escalation_rate": 0.0,
91
+ "brier": 0.9027777777777778,
92
+ "ece": 0.9027777777777778,
93
+ "auroc": 0.5,
94
+ "recall_95ci": [
95
+ 0.0,
96
+ 0.0
97
+ ],
98
+ "escalation_rate_95ci": [
99
+ 0.0,
100
+ 0.0
101
+ ],
102
+ "estimated_vlm_gpu_seconds": 0.0,
103
+ "vlm_call_savings_vs_always_vlm": 1.0
104
+ }
105
+ },
106
+ "v4_condition_metrics": {
107
+ "risk": {
108
+ "examples": 576,
109
+ "positives": 144,
110
+ "true_positives": 142,
111
+ "false_positives": 0,
112
+ "false_negatives": 2,
113
+ "recall": 0.9861111111111112,
114
+ "precision": 1.0,
115
+ "specificity": 1.0,
116
+ "escalation_rate": 0.2465277777777778,
117
+ "brier": 0.06036160690906683,
118
+ "ece": 0.22424719773002424,
119
+ "auroc": 1.0
120
+ },
121
+ "uncertainty": {
122
+ "examples": 576,
123
+ "positives": 240,
124
+ "true_positives": 175,
125
+ "false_positives": 179,
126
+ "false_negatives": 65,
127
+ "recall": 0.7291666666666666,
128
+ "precision": 0.4943502824858757,
129
+ "specificity": 0.46726190476190477,
130
+ "escalation_rate": 0.6145833333333334,
131
+ "brier": 0.2293827050503838,
132
+ "ece": 0.07416748310606931,
133
+ "auroc": 0.6601066468253968
134
+ },
135
+ "conflict": {
136
+ "examples": 576,
137
+ "positives": 144,
138
+ "true_positives": 131,
139
+ "false_positives": 173,
140
+ "false_negatives": 13,
141
+ "recall": 0.9097222222222222,
142
+ "precision": 0.4309210526315789,
143
+ "specificity": 0.5995370370370371,
144
+ "escalation_rate": 0.5277777777777778,
145
+ "brier": 0.12098456100215688,
146
+ "ece": 0.21078960513805695,
147
+ "auroc": 0.9134677211934157
148
+ },
149
+ "visual": {
150
+ "examples": 576,
151
+ "positives": 216,
152
+ "true_positives": 169,
153
+ "false_positives": 132,
154
+ "false_negatives": 47,
155
+ "recall": 0.7824074074074074,
156
+ "precision": 0.5614617940199336,
157
+ "specificity": 0.6333333333333333,
158
+ "escalation_rate": 0.5225694444444444,
159
+ "brier": 0.16616077941724472,
160
+ "ece": 0.1971511931456856,
161
+ "auroc": 0.8709362139917696
162
+ }
163
+ },
164
+ "v4_robustness_slices": {
165
+ "background_id": {
166
+ "pink": {
167
+ "examples": 192,
168
+ "positives": 169,
169
+ "true_positives": 157,
170
+ "false_positives": 21,
171
+ "false_negatives": 12,
172
+ "recall": 0.9289940828402367,
173
+ "precision": 0.8820224719101124,
174
+ "specificity": 0.08695652173913043,
175
+ "escalation_rate": 0.9270833333333334,
176
+ "brier": 0.31135914332280196,
177
+ "ece": 0.46788093565946537,
178
+ "auroc": 0.8662207357859532
179
+ },
180
+ "reverb": {
181
+ "examples": 192,
182
+ "positives": 181,
183
+ "true_positives": 177,
184
+ "false_positives": 11,
185
+ "false_negatives": 4,
186
+ "recall": 0.9779005524861878,
187
+ "precision": 0.9414893617021277,
188
+ "specificity": 0.0,
189
+ "escalation_rate": 0.9791666666666666,
190
+ "brier": 0.16562298844441178,
191
+ "ece": 0.3286566249272864,
192
+ "auroc": 0.8679055750878956
193
+ },
194
+ "white": {
195
+ "examples": 192,
196
+ "positives": 170,
197
+ "true_positives": 159,
198
+ "false_positives": 22,
199
+ "false_negatives": 11,
200
+ "recall": 0.9352941176470588,
201
+ "precision": 0.8784530386740331,
202
+ "specificity": 0.0,
203
+ "escalation_rate": 0.9427083333333334,
204
+ "brier": 0.2863302628946927,
205
+ "ece": 0.44435570647505257,
206
+ "auroc": 0.8737967914438503
207
+ }
208
+ },
209
+ "snr_db": {
210
+ "0.0": {
211
+ "examples": 144,
212
+ "positives": 133,
213
+ "true_positives": 133,
214
+ "false_positives": 9,
215
+ "false_negatives": 0,
216
+ "recall": 1.0,
217
+ "precision": 0.9366197183098591,
218
+ "specificity": 0.18181818181818182,
219
+ "escalation_rate": 0.9861111111111112,
220
+ "brier": 0.15135230694589374,
221
+ "ece": 0.33521018341449565,
222
+ "auroc": 0.9958988380041012
223
+ },
224
+ "15.0": {
225
+ "examples": 144,
226
+ "positives": 130,
227
+ "true_positives": 119,
228
+ "false_positives": 14,
229
+ "false_negatives": 11,
230
+ "recall": 0.9153846153846154,
231
+ "precision": 0.8947368421052632,
232
+ "specificity": 0.0,
233
+ "escalation_rate": 0.9236111111111112,
234
+ "brier": 0.2976779995627198,
235
+ "ece": 0.45929345723247444,
236
+ "auroc": 0.7461538461538462
237
+ },
238
+ "30.0": {
239
+ "examples": 144,
240
+ "positives": 126,
241
+ "true_positives": 110,
242
+ "false_positives": 18,
243
+ "false_negatives": 16,
244
+ "recall": 0.873015873015873,
245
+ "precision": 0.859375,
246
+ "specificity": 0.0,
247
+ "escalation_rate": 0.8888888888888888,
248
+ "brier": 0.36197668627847096,
249
+ "ece": 0.5134423018117535,
250
+ "auroc": 0.8276014109347443
251
+ },
252
+ "5.0": {
253
+ "examples": 144,
254
+ "positives": 131,
255
+ "true_positives": 131,
256
+ "false_positives": 13,
257
+ "false_negatives": 0,
258
+ "recall": 1.0,
259
+ "precision": 0.9097222222222222,
260
+ "specificity": 0.0,
261
+ "escalation_rate": 1.0,
262
+ "brier": 0.20674286676212414,
263
+ "ece": 0.3714515093453985,
264
+ "auroc": 0.9301233118027011
265
+ }
266
+ },
267
+ "blur_radius": {
268
+ "0.0": {
269
+ "examples": 192,
270
+ "positives": 174,
271
+ "true_positives": 165,
272
+ "false_positives": 17,
273
+ "false_negatives": 9,
274
+ "recall": 0.9482758620689655,
275
+ "precision": 0.9065934065934066,
276
+ "specificity": 0.05555555555555555,
277
+ "escalation_rate": 0.9479166666666666,
278
+ "brier": 0.26204597095700016,
279
+ "ece": 0.42548523065516897,
280
+ "auroc": 0.8805874840357599
281
+ },
282
+ "1.5": {
283
+ "examples": 192,
284
+ "positives": 174,
285
+ "true_positives": 167,
286
+ "false_positives": 18,
287
+ "false_negatives": 7,
288
+ "recall": 0.9597701149425287,
289
+ "precision": 0.9027027027027027,
290
+ "specificity": 0.0,
291
+ "escalation_rate": 0.9635416666666666,
292
+ "brier": 0.23997474327744386,
293
+ "ece": 0.41915464769013605,
294
+ "auroc": 0.9613665389527458
295
+ },
296
+ "3.0": {
297
+ "examples": 192,
298
+ "positives": 172,
299
+ "true_positives": 161,
300
+ "false_positives": 19,
301
+ "false_negatives": 11,
302
+ "recall": 0.936046511627907,
303
+ "precision": 0.8944444444444445,
304
+ "specificity": 0.05,
305
+ "escalation_rate": 0.9375,
306
+ "brier": 0.26129168042746237,
307
+ "ece": 0.40869253006803635,
308
+ "auroc": 0.7604651162790698
309
+ }
310
+ },
311
+ "occlusion_ratio": {
312
+ "0.0": {
313
+ "examples": 288,
314
+ "positives": 261,
315
+ "true_positives": 248,
316
+ "false_positives": 26,
317
+ "false_negatives": 13,
318
+ "recall": 0.9501915708812261,
319
+ "precision": 0.9051094890510949,
320
+ "specificity": 0.037037037037037035,
321
+ "escalation_rate": 0.9513888888888888,
322
+ "brier": 0.24373840488323129,
323
+ "ece": 0.41284212751138916,
324
+ "auroc": 0.8809422449269193
325
+ },
326
+ "0.25": {
327
+ "examples": 288,
328
+ "positives": 259,
329
+ "true_positives": 245,
330
+ "false_positives": 28,
331
+ "false_negatives": 14,
332
+ "recall": 0.9459459459459459,
333
+ "precision": 0.8974358974358975,
334
+ "specificity": 0.034482758620689655,
335
+ "escalation_rate": 0.9479166666666666,
336
+ "brier": 0.265136524891373,
337
+ "ece": 0.4240392733918322,
338
+ "auroc": 0.8691252829183864
339
+ }
340
+ }
341
+ },
342
+ "limitations": "Routing quality only. Final answer quality must be measured by the separate end-to-end VLM/EvidenceAgent evaluation."
343
+ }