Publish causal YellowCab benchmark and model card

EVALUATION.md

@@ -0,0 +1,94 @@
+# YellowCab evaluation
+
+## Question
+
+Can three chronological road images combined with past-and-current telemetry
+forecast a GPS-derived taxi maneuver approximately 20 seconds ahead better than
+tested single-modality and rule-based baselines?
+
+## Frozen evaluation contract
+
+- Published model version: `v0.1-causal`
+- Published Safetensors SHA-256:
+  `af1a98cd96bb68d642f7a0adeedfacac37b230917afc15f3c87230a99a15b8a2`
+- Training manifest SHA-256:
+  `8b92ad563892776cc25148f39fec6bf207e1e442c618b58fb07af7e2b899364d`
+- Split SHA-256:
+  `bfc5e960c8af32d0ea75a5e293bb95b3125111ace2cf8452eb42feb4bc2a1a5e`
+- Seed: `20260726`
+- Primary metric: macro F1
+
+The split is forward in time by complete route. The earliest 26 routes are used
+for training, the next five for validation, and the newest five for testing.
+Frames never cross route boundaries between splits.
+
+## Causal-input audit
+
+The original dataset builder could use a future-bearing fallback when the
+past-to-current displacement was too small to establish a reliable heading. The
+causal manifest removes that fallback. When displacement is below four metres,
+heading is represented as unknown using `(sin=0, cos=1)`.
+
+The audit changed 5,974 of 74,680 rows:
+
+- 4,552 training rows
+- 819 validation rows
+- 603 test rows
+
+The released `v0.1-causal` checkpoint was trained on the corrected manifest.
+Labels still use future trajectory by definition; model inputs do not.
+
+## Metrics
+
+Macro F1 is primary because the test set is imbalanced. Accuracy, balanced
+accuracy, weighted F1, log loss, Brier score, and 15-bin expected calibration
+error are secondary.
+
+The macro-F1 confidence interval is computed with 500 bootstrap resamples of
+complete test routes. This preserves within-route dependence instead of treating
+8,594 sequential examples as independent observations.
+
+## Reproduction from published predictions
+
+The repository publishes labels, predictions, probabilities, and generic route
+groups for every test example in `eval_predictions.csv`. It excludes images,
+capture identifiers, timestamps, grid cells, and geographic data.
+
+```bash
+python -m pip install numpy pandas scikit-learn
+python recompute_metrics.py
+```
+
+The script checks that probability rows sum to one and reproduces the core
+point estimates and route-clustered macro-F1 interval.
+
+## Selective prediction
+
+`selective_metrics.csv` evaluates the confidence-based abstention rule. At the
+default 0.45 threshold, the model retains 5,971 of 8,594 examples (69.5%
+coverage). Accuracy among retained examples is 0.6583 and macro F1 is 0.4327.
+
+Selective metrics describe only the accepted subset. They do not mean the
+abstained examples disappear operationally; a real system must route them to a
+fallback or human review process.
+
+## Interpretation
+
+YellowCab has the highest macro F1 among the tested models. It does not win every
+metric: telemetry gradient boosting has higher balanced accuracy, and a majority
+classifier has higher raw accuracy because most examples are `continue`.
+
+This supports use as a ranking, indexing, and research model. It does not support
+claims of safe actuation, universal geographic generalization, or global
+state-of-the-art status.
+
+## Remaining evaluation work
+
+A stronger future release should add:
+
+- an independently human-labeled gold test set;
+- a separate calibration split;
+- evaluation across additional cities, camera mounts, weather, and daylight;
+- selective-risk curves for the abstention threshold;
+- an operational video-retrieval study at fixed event recall; and
+- independent replication on a public, task-compatible benchmark.

README.md

@@ -4,18 +4,42 @@ license: apache-2.0
 tags:
   - driving
   - multimodal
+  - computer-vision
   - time-series
+  - maneuver-forecasting
   - custom-handler
   - research
+metrics:
+  - f1
+  - accuracy
+model-index:
+  - name: YellowCab v0.1-causal
+    results:
+      - task:
+          type: image-classification
+          name: Multimodal maneuver forecasting
+        dataset:
+          name: YellowCab private forward-route holdout
+          type: private
+          split: test
+        metrics:
+          - type: f1
+            name: Macro F1
+            value: 0.3915542658
+          - type: accuracy
+            name: Balanced accuracy
+            value: 0.3915633404
+          - type: accuracy
+            name: Accuracy
+            value: 0.5755178031
 ---
 
 # YellowCab
 
-YellowCab is a multimodal PyTorch model that predicts one of five near-future
-taxi maneuver labels from three chronologically ordered camera frames and
-twelve telemetry features.
-
-The output labels are:
+YellowCab is a compact multimodal model for forecasting a taxi's observed
+maneuver approximately 20 seconds ahead. It combines three chronological road
+images with twelve past-and-current telemetry signals and returns calibrated
+probabilities for:
 
 - `continue`
 - `slow`
@@ -23,31 +47,153 @@ The output labels are:
 - `turn_left`
 - `turn_right`
 
-## Model
+The model has **5.29 million parameters**. It is intended for fleet-video
+indexing, maneuver analytics, multimodal research, and prototyping—not vehicle
+control.
 
-YellowCab combines:
+## What is technically useful about it?
 
-- an EfficientNet-B0 visual encoder;
-- a one-layer unidirectional GRU over three image embeddings;
-- a telemetry MLP over twelve normalized features; and
-- a five-class fusion head with temperature scaling and optional abstention.
+YellowCab tests a practical hypothesis: recent visual context and vehicle motion
+history are more useful together than either signal alone.
 
-The checkpoint and its exact loading configuration are included in this
-repository. The checkpoint SHA-256 is recorded in `config.json`.
+On the route-held-out silver benchmark, the causal temporal-fusion checkpoint
+reaches **0.3916 macro F1**, compared with **0.3414** for the strongest tested
+past-only telemetry baseline and **0.3050** for the tested current-frame
+vision baseline. This supports uses such as:
 
-## Data and privacy
+- assigning likely maneuver tags to fleet footage;
+- retrieving candidate stop and turn events for human review;
+- prioritizing uncertain sequences for data collection or annotation;
+- studying disagreement between visual and motion signals; and
+- prototyping calibrated, abstaining multimodal classifiers.
 
-The model was trained on privacy-redacted imagery collected from the operator's
-taxi fleet under contributor agreements authorizing model training. No source
-imagery, route manifests, or personal identifiers are distributed here.
+These results do not establish a global rank. YellowCab uses a private,
+task-specific benchmark and has not been evaluated under a directly comparable
+public autonomous-driving leaderboard protocol.
+
+## Architecture
+
+- Frozen ImageNet-pretrained EfficientNet-B0 visual encoder
+- Three frames at nominal offsets `t-20 s`, `t-10 s`, and `t`
+- Layer normalization followed by a one-layer unidirectional GRU
+- Telemetry MLP over twelve normalized causal features
+- Fusion classifier over five maneuver classes
+- Temperature scaling and optional confidence-based abstention
+
+The repository contains the exact architecture, configuration, integrity-checked
+Safetensors checkpoint, and custom Hugging Face endpoint handler.
+
+## Evaluation
+
+### Protocol
+
+- **74,680** examples across **36** routes
+- **55,711** training examples from 26 routes
+- **10,375** validation examples from 5 routes
+- **8,594** test examples from the newest 5 completely held-out routes
+- Complete forward-in-time route separation
+- No frame from a validation or test route enters training
+- Model inputs use only information available at or before the newest frame
+- Labels are GPS-derived maneuver proxies approximately 20 seconds ahead
+- Primary metric: macro F1
+- 95% confidence interval: 500 route-clustered bootstrap samples
+
+### Headline results
+
+| Metric | YellowCab |
+| --- | ---: |
+| Macro F1 | **0.3916** |
+| Route-bootstrap 95% CI | **0.3476–0.4281** |
+| Balanced accuracy | **0.3916** |
+| Accuracy | **0.5755** |
+| Weighted F1 | **0.5807** |
+| Log loss | **1.0841** |
+| Brier score | **0.5563** |
+| ECE, 15 bins | **0.0118** |
+
+Raw accuracy is not the primary metric because `continue` is the majority
+class. A majority classifier reaches 0.6232 accuracy while scoring only 0.1536
+macro F1.
+
+### Baselines
+
+| Model | Macro F1 | Balanced accuracy | Accuracy |
+| --- | ---: | ---: | ---: |
+| **YellowCab: temporal vision + telemetry** | **0.3916** | 0.3916 | 0.5755 |
+| Telemetry gradient boosting | 0.3414 | **0.4217** | 0.4147 |
+| Current-frame vision linear model | 0.3050 | 0.2968 | 0.5752 |
+| Past-motion rule | 0.2235 | 0.2238 | 0.4550 |
+| Telemetry logistic regression | 0.1887 | 0.2872 | 0.1963 |
+| Majority class | 0.1536 | 0.2000 | **0.6232** |
+
+YellowCab improves macro F1 over every tested baseline, but the gradient-boosted
+telemetry baseline has higher balanced accuracy. We report both rather than
+claiming uniform dominance.
+
+### Per-class results
+
+| Class | Precision | Recall | F1 | Support |
+| --- | ---: | ---: | ---: | ---: |
+| Continue | 0.7595 | 0.7394 | **0.7493** | 5,356 |
+| Slow | 0.4031 | 0.3544 | **0.3772** | 951 |
+| Stop | 0.3754 | 0.3423 | **0.3581** | 634 |
+| Turn left | 0.2447 | 0.3157 | **0.2757** | 833 |
+| Turn right | 0.1897 | 0.2061 | **0.1975** | 820 |
+
+Turn performance is the clearest weakness. YellowCab should be used to produce
+ranked candidates or analytics, not treated as a reliable maneuver oracle.
+
+### Confidence-based abstention
+
+The default confidence threshold is 0.45. On this held-out set it retains
+**69.5%** of examples, with **0.6583 selective accuracy** and **0.4327 selective
+macro F1** among retained examples. Raising the threshold trades coverage for
+higher accuracy:
+
+| Threshold | Coverage | Selective accuracy | Selective macro F1 |
+| --- | ---: | ---: | ---: |
+| 0.00 | 100.0% | 0.5755 | 0.3916 |
+| 0.35 | 91.0% | 0.6007 | 0.4061 |
+| **0.45** | **69.5%** | **0.6583** | **0.4327** |
+| 0.55 | 47.2% | 0.7324 | 0.4702 |
+| 0.65 | 30.2% | 0.8092 | 0.5275 |
+
+This makes the model more useful for candidate retrieval and review queues, but
+these thresholds must be recalibrated after domain shift. Full results are in
+[`selective_metrics.csv`](selective_metrics.csv).
+
+![Benchmark comparison](benchmark.png)
+
+![Held-out confusion matrix](confusion_matrix.png)
+
+The complete checkpoint-specific report is in
+[`evaluation.json`](evaluation.json). Recomputable, privacy-sanitized held-out
+probabilities are in [`eval_predictions.csv`](eval_predictions.csv), and the
+metric script is in [`recompute_metrics.py`](recompute_metrics.py). Additional
+methodology and limitations are documented in
+[`EVALUATION.md`](EVALUATION.md).
+
+## Checkpoint identity
+
+- Version: `v0.1-causal`
+- Published checkpoint SHA-256:
+  `af1a98cd96bb68d642f7a0adeedfacac37b230917afc15f3c87230a99a15b8a2`
+- Training manifest SHA-256:
+  `8b92ad563892776cc25148f39fec6bf207e1e442c618b58fb07af7e2b899364d`
+- Split SHA-256:
+  `bfc5e960c8af32d0ea75a5e293bb95b3125111ace2cf8452eb42feb4bc2a1a5e`
+- Training seed: `20260726`
+
+The causal-manifest audit replaced unavailable current headings with the neutral
+encoding `(sin=0, cos=1)` rather than deriving them from future motion. The
+published checkpoint was trained and evaluated on that corrected manifest.
 
 ## Input
 
 Requests require exactly three base64-encoded JPEG or PNG frames ordered oldest
-to newest. Their nominal offsets are approximately `t-20 s`, `t-10 s`, and `t`.
-Optional frame timestamps must be strictly increasing.
+to newest. Optional frame timestamps must be strictly increasing.
 
-All telemetry fields documented in `config.json` are required:
+All telemetry fields are required:
 
 `current_speed_mps`, `past_speed_mps`, `acceleration_mps2`,
 `past_turn_degrees`, `past_yaw_rate_deg_s`, `heading_sin`, `heading_cos`,
@@ -77,29 +223,10 @@ All telemetry fields documented in `config.json` are required:
 }
 ```
 
-The custom `EndpointHandler` in `handler.py` validates inputs and returns the
-predicted label, ordered class probabilities, confidence, and abstention state.
-This custom multimodal architecture is not compatible with the standard
-image-classification `pipeline()` or widget.
-
-## Evaluation
-
-The release process included leakage-free held-out evaluation. Numeric results
-are omitted until the corresponding report is published for this exact
-checkpoint digest. Users should evaluate performance and calibration on their
-own target domain.
-
-## Intended use and limitations
-
-YellowCab is intended for research, fleet analytics, and prototyping. Its labels
-are GPS-derived maneuver proxies rather than human safety annotations.
-Performance may vary with geography, weather, camera placement, traffic
-conditions, and telemetry quality.
-
-Telemetry supplied at inference time must be computed only from observations
-available at or before the newest frame. YellowCab is not a safety-certified
-system and should not be used as the sole input to steering, braking, throttle,
-or other safety-critical decisions.
+The custom `EndpointHandler` validates the request and returns the predicted
+label, ordered class probabilities, confidence, and abstention state. This
+custom multimodal architecture is not compatible with the standard
+image-classification `pipeline()` or Hub widget.
 
 ## Local use
 
@@ -116,9 +243,64 @@ predict = EndpointHandler(".")
 result = predict(request_body)
 ```
 
+## Training data and privacy
+
+The model was trained on privacy-redacted imagery collected from the operator's
+taxi fleet under contributor agreements authorizing model training. No source
+imagery, exact GPS records, route manifests, or personal identifiers are
+distributed in this repository.
+
+The published evaluation predictions use sequential example identifiers and
+generic held-out route groups. Capture identifiers, timestamps, grid cells, and
+locations have been removed.
+
+## Intended uses
+
+- Offline fleet-video indexing and search
+- Maneuver analytics and aggregate research
+- Candidate-event retrieval followed by human review
+- Multimodal classification and calibration research
+- Prototyping and collection-planning experiments
+
+## Out-of-scope uses
+
+Do not use YellowCab:
+
+- as the sole input to steering, braking, throttle, or dispatch decisions;
+- as a safety-certified driver-assistance component;
+- to identify drivers, passengers, pedestrians, or locations;
+- to infer intent, fault, liability, or legal compliance; or
+- outside a target domain without measuring performance and calibration again.
+
+## Limitations
+
+- Ground truth is GPS-derived silver labeling, not independent human annotation.
+- The evaluation covers five held-out routes from one operating domain.
+- Geography, weather, camera placement, traffic, and telemetry quality can shift
+  performance.
+- Rare classes are substantially weaker than `continue`.
+- Temperature scaling used the validation split rather than a separate
+  calibration split.
+- No membership-inference, model-inversion, or differential-privacy assessment
+  has been completed.
+- Benchmark results are author-reported and have not been independently
+  replicated on a public dataset.
+
 ## License
 
 Unless otherwise noted, the model weights and original code, configuration, and
-documentation in this repository are licensed under the Apache License 2.0.
-Third-party components remain subject to their respective terms; see
-`THIRD_PARTY_NOTICES.md`.
+documentation are licensed under the Apache License 2.0. Third-party components
+remain subject to their respective terms; see
+[`THIRD_PARTY_NOTICES.md`](THIRD_PARTY_NOTICES.md).
+
+## Citation
+
+```bibtex
+@software{yellowcab_v0_1_causal_2026,
+  author = {{The General Data Corporation}},
+  title = {YellowCab: Multimodal Taxi Maneuver Forecasting},
+  year = {2026},
+  version = {v0.1-causal},
+  url = {https://huggingface.co/generaldata/YellowCab}
+}
+```

benchmark.csv

@@ -0,0 +1,7 @@
+model,samples,accuracy,balanced_accuracy,macro_f1,weighted_f1,log_loss,brier_score,ece_15_bin,p95_latency_ms,continue_recall,continue_f1,slow_recall,slow_f1,stop_recall,stop_f1,turn_left_recall,turn_left_f1,turn_right_recall,turn_right_f1,macro_f1_route_ci_low,macro_f1_route_ci_high
+Temporal efficientnet_b0 + telemetry,8594,0.5755178031184547,0.39156334041323,0.3915542657655546,0.5807008272180106,1.0841176199931875,0.5563128377237149,0.011817411647322817,95.37229499110252,0.7393577296489918,0.7492904446546831,0.35436382754994744,0.37716843872411865,0.3422712933753943,0.3580858085808581,0.3157262905162065,0.27568134171907754,0.20609756097560974,0.19754529514903565,0.34762239048615595,0.42805096076291543
+Telemetry gradient boosting,8594,0.4147079357691413,0.42166330996553986,0.341392924380058,0.45303568638063824,1.2490630281069546,0.6566714449096165,0.12167008852186818,,0.4051530993278566,0.5513911828230212,0.6466876971608833,0.3416666666666667,0.4889589905362776,0.29865125240847784,0.2797118847539016,0.2583148558758315,0.28780487804878047,0.25694066412629285,0.3149153283503967,0.34939970154465316
+Vision linear (efficientnet_b0),8594,0.5751687223644403,0.2967510353079744,0.30503460507355407,0.5344562546214928,10.24602122531317,0.8349054717132016,0.40913361870747145,,0.8269230769230769,0.7406354515050168,0.15562565720294427,0.21480406386066764,0.25236593059936907,0.25559105431309903,0.15006002400960383,0.18102824040550325,0.09878048780487805,0.13311421528348397,0.26780560056303193,0.32731434371479995
+Past-motion rule,8594,0.4549685827321387,0.22377382659168304,0.22352977859537337,0.45325423166470347,2.058490921490782,0.966304980218757,0.44503141726786155,,0.6562733383121733,0.6534064504136072,0.10199789695057834,0.09627791563275434,0.0,0.0,0.17767106842737096,0.17928528164748636,0.18292682926829268,0.18867924528301888,0.18734638205859894,0.23451957590710315
+Telemetry logistic,8594,0.19629974400744707,0.2871892331606019,0.1886698746182765,0.18614071585068767,1.5832758165931655,0.8215841116420504,0.1432688177654283,,0.109596713965646,0.18162128712871287,0.601472134595163,0.25060240963855424,0.37066246056782337,0.17609591607343575,0.19567827130852342,0.17212249208025343,0.15853658536585366,0.16290726817042606,0.1839562547318565,0.20443593471582872
+Majority class,8594,0.6232255061670933,0.2,0.15357706093189966,0.4785657076746885,1.1837257781136636,0.5768401032344809,0.03274965759140802,,1.0,0.7678853046594982,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.14609150326797388,0.1554760284937855

bundle_manifest.json

@@ -1,53 +1,93 @@
 {
-  "schema": "yellowcab_huggingface_bundle_v1",
-  "model_id": "generaldata/YellowCab",
-  "model_version": "v0",
-  "license": "Apache-2.0",
-  "note": "bundle_manifest.json intentionally does not hash itself",
+  "schema": "gdc_yellowcab_public_bundle_v2",
   "files": {
     ".gitattributes": {
       "bytes": 69,
       "sha256": "14281383ff036613975d5f219e22a1102041eb14de55c3f34db8223bb0784145"
     },
-    "LICENSE": {
-      "bytes": 11357,
-      "sha256": "c71d239df91726fc519c6eb72d318ec65820627232b2f796219e87dcf35d0ab4"
+    "benchmark.csv": {
+      "bytes": 2524,
+      "sha256": "177d024f407c9e064b63eff4b0e594deb0ae6b861ce342c63e9bfc2458ff83e0"
     },
-    "NOTICE": {
-      "bytes": 622,
-      "sha256": "1983f5396c5d59083231fdf3516feacc267b13bbe8090b31804260fde5c39c72"
+    "benchmark.png": {
+      "bytes": 98848,
+      "sha256": "31e491792a60685e15a373cb1919bb732aaa675ca0bce5b5f38270c32d6c6a9a"
     },
-    "README.md": {
-      "bytes": 3787,
-      "sha256": "ca01c46bbd8f7b87a5a5353d3c6ccf2ac7b8032bb339033346e64d396713ef73"
+    "class_metrics.csv": {
+      "bytes": 2679,
+      "sha256": "bf35d02ca99e765be4926d95d6f197488569bf902c944eca97d2f954a5c87003"
     },
-    "THIRD_PARTY_NOTICES.md": {
-      "bytes": 1454,
-      "sha256": "38f7bb94a0824965ac7cff7cc2f4378a0a44cbe08aad02b97468d86243b2da1d"
+    "classification_report.json": {
+      "bytes": 1128,
+      "sha256": "3c10ff529643649eb2b72f79244f6370c2ff9fe0c34e27b0600fe951047a7969"
     },
     "config.json": {
-      "bytes": 9339,
-      "sha256": "1d268c27e8f4ebf76cb52cacd72e7a6f27a44e7076b857da15ef5253139b227f"
+      "bytes": 10944,
+      "sha256": "2a221f468f350360ed704f6552400312f67057fdc5610e795162a711a5636b9b"
+    },
+    "confusion_matrix.png": {
+      "bytes": 96286,
+      "sha256": "31fa8526560d76bc9600a40a6e48b2d6ef929934c54d8371e855de93431a5482"
+    },
+    "eval_predictions.csv": {
+      "bytes": 1156310,
+      "sha256": "d76846904ed54093d53a77610f040e2e9a6476cd2c06bd13edbf121ca9875dff"
+    },
+    "evaluation.json": {
+      "bytes": 12235,
+      "sha256": "e8890addffda4ec495a219f7439bb4d9ece5934f536e35d7f340830b475cbe21"
+    },
+    "EVALUATION.md": {
+      "bytes": 3606,
+      "sha256": "31d29ab99ac4761662a1bd4646e4c317deedb60c79577b844472485085f6a710"
     },
     "handler.py": {
       "bytes": 17681,
       "sha256": "a8a3c804e71e4864ac26cae394d9f1e32af86d38ae939226066e1c9767ac2eaa"
     },
+    "LICENSE": {
+      "bytes": 11357,
+      "sha256": "c71d239df91726fc519c6eb72d318ec65820627232b2f796219e87dcf35d0ab4"
+    },
     "model.py": {
       "bytes": 8419,
       "sha256": "48cf1aafb242a497be2ec9ad7abd6211608cf04e98c9fae20d5878ba8a09bc52"
     },
     "model.safetensors": {
-      "bytes": 21384660,
-      "sha256": "241f8335bdbe45e9f88e45640b35e3426e2c1e8fd58b9e0bec774a417988d57c"
+      "bytes": 21384748,
+      "sha256": "af1a98cd96bb68d642f7a0adeedfacac37b230917afc15f3c87230a99a15b8a2"
+    },
+    "NOTICE": {
+      "bytes": 622,
+      "sha256": "1983f5396c5d59083231fdf3516feacc267b13bbe8090b31804260fde5c39c72"
+    },
+    "README.md": {
+      "bytes": 10588,
+      "sha256": "6012ebee2b3d34d3117aaac5e2c1143e07f583b9c61fe6bc8cf1483c820cd8a4"
+    },
+    "recompute_metrics.py": {
+      "bytes": 3963,
+      "sha256": "dd2f8aeda429cfecb5204e6b2ed563a3d269500b0112ab6e7558a576f50dac5f"
     },
     "requirements.txt": {
       "bytes": 80,
       "sha256": "92f03418ab85f5cb1239971fb4515cf1d2acc9aed7013d0aca74b17daa3f97ef"
     },
+    "selective_metrics.csv": {
+      "bytes": 614,
+      "sha256": "e39afd4baf6104eced16f0658f19b3abb7f5278f29fe7cc641a544910eb28606"
+    },
+    "subset_metrics.csv": {
+      "bytes": 2430,
+      "sha256": "7de2bb82af604e2d5acc2276c345b7d01467b596f2196511394d2f3c60c569e4"
+    },
     "THIRD_PARTY_LICENSES/torchvision-BSD-3-Clause.txt": {
       "bytes": 1516,
       "sha256": "2b63e4f6569e0e58260881a1f7ea8b1d9802859ab3cb4b32a45cba246245c8da"
+    },
+    "THIRD_PARTY_NOTICES.md": {
+      "bytes": 1454,
+      "sha256": "38f7bb94a0824965ac7cff7cc2f4378a0a44cbe08aad02b97468d86243b2da1d"
     }
   }
 }

class_metrics.csv

@@ -0,0 +1,31 @@
+model,class,precision,recall,f1,support
+Majority class,continue,0.6232255061670933,1.0,0.7678853046594982,5356
+Majority class,slow,0.0,0.0,0.0,951
+Majority class,stop,0.0,0.0,0.0,634
+Majority class,turn_left,0.0,0.0,0.0,833
+Majority class,turn_right,0.0,0.0,0.0,820
+Past-motion rule,continue,0.6505645012030353,0.6562733383121733,0.6534064504136072,5356
+Past-motion rule,slow,0.09116541353383459,0.10199789695057834,0.09627791563275434,951
+Past-motion rule,stop,0.0,0.0,0.0,634
+Past-motion rule,turn_left,0.18092909535452323,0.17767106842737096,0.17928528164748636,833
+Past-motion rule,turn_right,0.19480519480519481,0.18292682926829268,0.18867924528301888,820
+Telemetry logistic,continue,0.529783393501805,0.109596713965646,0.18162128712871287,5356
+Telemetry logistic,slow,0.15827338129496402,0.601472134595163,0.25060240963855424,951
+Telemetry logistic,stop,0.11547911547911548,0.37066246056782337,0.17609591607343575,634
+Telemetry logistic,turn_left,0.1536286522148916,0.19567827130852342,0.17212249208025343,833
+Telemetry logistic,turn_right,0.16752577319587628,0.15853658536585366,0.16290726817042606,820
+Telemetry gradient boosting,continue,0.8628230616302187,0.4051530993278566,0.5513911828230212,5356
+Telemetry gradient boosting,slow,0.232163080407701,0.6466876971608833,0.3416666666666667,951
+Telemetry gradient boosting,stop,0.21497919556171982,0.4889589905362776,0.29865125240847784,634
+Telemetry gradient boosting,turn_left,0.2399588053553038,0.2797118847539016,0.2583148558758315,833
+Telemetry gradient boosting,turn_right,0.232055063913471,0.28780487804878047,0.25694066412629285,820
+Vision linear (efficientnet_b0),continue,0.6706541490006057,0.8269230769230769,0.7406354515050168,5356
+Vision linear (efficientnet_b0),slow,0.34660421545667447,0.15562565720294427,0.21480406386066764,951
+Vision linear (efficientnet_b0),stop,0.2588996763754045,0.25236593059936907,0.25559105431309903,634
+Vision linear (efficientnet_b0),turn_left,0.2281021897810219,0.15006002400960383,0.18102824040550325,833
+Vision linear (efficientnet_b0),turn_right,0.2040302267002519,0.09878048780487805,0.13311421528348397,820
+Temporal efficientnet_b0 + telemetry,continue,0.759493670886076,0.7393577296489918,0.7492904446546831,5356
+Temporal efficientnet_b0 + telemetry,slow,0.40311004784689,0.35436382754994744,0.37716843872411865,951
+Temporal efficientnet_b0 + telemetry,stop,0.3754325259515571,0.3422712933753943,0.3580858085808581,634
+Temporal efficientnet_b0 + telemetry,turn_left,0.24465116279069768,0.3157262905162065,0.27568134171907754,833
+Temporal efficientnet_b0 + telemetry,turn_right,0.18967452300785634,0.20609756097560974,0.19754529514903565,820

classification_report.json

@@ -0,0 +1,45 @@
+{
+  "continue": {
+    "precision": 0.759493670886076,
+    "recall": 0.7393577296489918,
+    "f1-score": 0.7492904446546831,
+    "support": 5356.0
+  },
+  "slow": {
+    "precision": 0.40311004784689,
+    "recall": 0.35436382754994744,
+    "f1-score": 0.37716843872411865,
+    "support": 951.0
+  },
+  "stop": {
+    "precision": 0.3754325259515571,
+    "recall": 0.3422712933753943,
+    "f1-score": 0.3580858085808581,
+    "support": 634.0
+  },
+  "turn_left": {
+    "precision": 0.24465116279069768,
+    "recall": 0.3157262905162065,
+    "f1-score": 0.27568134171907754,
+    "support": 833.0
+  },
+  "turn_right": {
+    "precision": 0.18967452300785634,
+    "recall": 0.20609756097560974,
+    "f1-score": 0.19754529514903565,
+    "support": 820.0
+  },
+  "accuracy": 0.5755178031184547,
+  "macro avg": {
+    "precision": 0.3944723860966154,
+    "recall": 0.39156334041323,
+    "f1-score": 0.3915542657655546,
+    "support": 8594.0
+  },
+  "weighted avg": {
+    "precision": 0.5874514202574582,
+    "recall": 0.5755178031184547,
+    "f1-score": 0.5807008272180106,
+    "support": 8594.0
+  }
+}

config.json

@@ -2,7 +2,7 @@
   "schema": "gdc_taxi_maneuver_release_config_v1",
   "model_id": "generaldata/YellowCab",
   "model_name": "YellowCab",
-  "model_version": "v0",
+  "model_version": "v0.1-causal",
   "task": "predict a GPS-derived maneuver proxy approximately 20 seconds ahead",
   "release_status": "public_research_release",
   "actuation_allowed": false,
@@ -212,34 +212,34 @@
       "source_example_count": 55711,
       "standard_deviation": "population_ddof_0",
       "missing_value_imputation_used": false,
-      "arithmetic": "parse as float64, subtract float64 mean, divide by float64 std, then cast to float32",
+      "arithmetic": "parse as float64, subtract float32 training mean, divide by float32 training standard deviation, then cast to float32",
       "mean": [
-        5.157084550141056,
-        5.055907504105811,
-        0.005347144408599654,
-        -0.7335137853872429,
-        -0.03728550645057143,
-        -0.016535674384678782,
-        0.000406427351177191,
-        4.231490900988432,
-        -0.24272832272310887,
-        0.6515114276493843,
-        0.08530019265689594,
-        -0.07634009204840192
+        5.1570844650268555,
+        5.055907726287842,
+        0.005347144324332476,
+        -0.7335137724876404,
+        -0.03728550672531128,
+        -0.01271896529942751,
+        0.08169970661401749,
+        4.2314910888671875,
+        -0.2427283227443695,
+        0.6515114307403564,
+        0.08530019223690033,
+        -0.07634009420871735
       ],
       "std": [
-        5.041778031679111,
-        5.164075560213916,
-        0.1699205233493807,
-        34.07308988288982,
-        1.7367109919849695,
-        0.7173100379931885,
-        0.6965577619147336,
-        1.9923153400250748,
-        0.6107164796141349,
-        0.3790002671740555,
-        0.6974503447543176,
-        0.7074313281731461
+        5.041778087615967,
+        5.1640753746032715,
+        0.16992051899433136,
+        34.073089599609375,
+        1.736711025238037,
+        0.6867451071739197,
+        0.7221804261207581,
+        1.9923152923583984,
+        0.6107164621353149,
+        0.3790002763271332,
+        0.6974503397941589,
+        0.7074313163757324
       ]
     }
   },
@@ -261,8 +261,8 @@
   },
   "calibration": {
     "method": "temperature_scaling",
-    "temperature": 0.9275532364845276,
-    "fit_split": "validation_after_restoring_best_epoch_3",
+    "temperature": 1.106177568435669,
+    "fit_split": "validation_after_restoring_best_epoch_4",
     "independent_calibration_split": false
   },
   "abstention": {
@@ -272,18 +272,55 @@
   },
   "checkpoint": {
     "file": "model.safetensors",
-    "sha256": "241f8335bdbe45e9f88e45640b35e3426e2c1e8fd58b9e0bec774a417988d57c",
-    "bytes": 21384660,
+    "sha256": "af1a98cd96bb68d642f7a0adeedfacac37b230917afc15f3c87230a99a15b8a2",
+    "bytes": 21384748,
     "parameter_count": 5294233,
     "state_tensor_count": 376,
     "state_numel": 5336298,
     "state_dtypes": {
       "torch.float32": 327,
       "torch.int64": 49
-    }
+    },
+    "source_checkpoint_sha256": "0cb9c71d5be8f7671029e459c24179863ae2abff3ccfec1b5837c169d7d6c703"
   },
   "warnings": [
     "research model; not a safety-certified vehicle-control system",
     "validate performance and calibration for the intended domain before use"
-  ]
+  ],
+  "training": {
+    "manifest_sha256": "8b92ad563892776cc25148f39fec6bf207e1e442c618b58fb07af7e2b899364d",
+    "split_sha256": "bfc5e960c8af32d0ea75a5e293bb95b3125111ace2cf8452eb42feb4bc2a1a5e",
+    "samples": 74680,
+    "routes": 36,
+    "split_samples": {
+      "train": 55711,
+      "validation": 10375,
+      "test": 8594
+    },
+    "split_routes": {
+      "train": 26,
+      "validation": 5,
+      "test": 5
+    },
+    "seed": 20260726,
+    "epochs_completed": 8,
+    "best_validation_macro_f1": 0.37644976936491265,
+    "causal_input_policy": "When past-to-current displacement is below 4 m, encode unknown current heading as sin=0 and cos=1. No future bearing is used.",
+    "future_derived_model_inputs": 0
+  },
+  "evaluation": {
+    "artifact": "evaluation.json",
+    "checkpoint_specific": true,
+    "primary_metric": "macro_f1",
+    "test_samples": 8594,
+    "test_routes": 5,
+    "macro_f1": 0.3915542657655546,
+    "macro_f1_route_ci_95": [
+      0.34762239048615595,
+      0.42805096076291543
+    ],
+    "balanced_accuracy": 0.39156334041323,
+    "accuracy": 0.5755178031184547,
+    "ece_15_bin": 0.011817411647322817
+  }
 }

evaluation.json

@@ -0,0 +1,344 @@
+{
+  "schema": "gdc_yellowcab_public_evaluation_v1",
+  "checkpoint": {
+    "source_file_sha256": "0cb9c71d5be8f7671029e459c24179863ae2abff3ccfec1b5837c169d7d6c703",
+    "published_file": "model.safetensors",
+    "published_sha256": "af1a98cd96bb68d642f7a0adeedfacac37b230917afc15f3c87230a99a15b8a2",
+    "training_manifest_sha256": "8b92ad563892776cc25148f39fec6bf207e1e442c618b58fb07af7e2b899364d",
+    "split_sha256": "bfc5e960c8af32d0ea75a5e293bb95b3125111ace2cf8452eb42feb4bc2a1a5e"
+  },
+  "causal_input_audit": {
+    "policy": "When past-to-current displacement is below 4 m, encode unknown current heading as sin=0 and cos=1. No future bearing is used.",
+    "corrected_rows": 5974,
+    "total_rows": 74680,
+    "future_derived_model_inputs": 0
+  },
+  "protocol": {
+    "label_type": "GPS-derived silver maneuver proxy",
+    "forecast_horizon": "approximately 20 seconds",
+    "split_policy": "Complete forward-in-time route holdout; no frame from a validation or test route enters training.",
+    "dataset_samples": 74680,
+    "dataset_routes": 36,
+    "test_samples": 8594,
+    "test_routes": 5,
+    "training_seed": 20260726,
+    "route_bootstrap_iterations": 500
+  },
+  "primary_result": {
+    "model": "Temporal efficientnet_b0 + telemetry",
+    "samples": 8594,
+    "accuracy": 0.5755178031184547,
+    "balanced_accuracy": 0.39156334041323,
+    "macro_f1": 0.3915542657655546,
+    "weighted_f1": 0.5807008272180106,
+    "log_loss": 1.0841176199931875,
+    "brier_score": 0.5563128377237149,
+    "ece_15_bin": 0.011817411647322817,
+    "p95_latency_ms": 95.37229499110252,
+    "continue_recall": 0.7393577296489918,
+    "continue_f1": 0.7492904446546831,
+    "slow_recall": 0.35436382754994744,
+    "slow_f1": 0.37716843872411865,
+    "stop_recall": 0.3422712933753943,
+    "stop_f1": 0.3580858085808581,
+    "turn_left_recall": 0.3157262905162065,
+    "turn_left_f1": 0.27568134171907754,
+    "turn_right_recall": 0.20609756097560974,
+    "turn_right_f1": 0.19754529514903565,
+    "macro_f1_route_ci_low": 0.34762239048615595,
+    "macro_f1_route_ci_high": 0.42805096076291543
+  },
+  "baselines": [
+    {
+      "model": "Temporal efficientnet_b0 + telemetry",
+      "samples": 8594,
+      "accuracy": 0.5755178031184547,
+      "balanced_accuracy": 0.39156334041323,
+      "macro_f1": 0.3915542657655546,
+      "weighted_f1": 0.5807008272180106,
+      "log_loss": 1.0841176199931877,
+      "brier_score": 0.5563128377237149,
+      "ece_15_bin": 0.0118174116473228,
+      "p95_latency_ms": 95.37229499110252,
+      "continue_recall": 0.7393577296489918,
+      "continue_f1": 0.7492904446546831,
+      "slow_recall": 0.3543638275499474,
+      "slow_f1": 0.3771684387241186,
+      "stop_recall": 0.3422712933753943,
+      "stop_f1": 0.3580858085808581,
+      "turn_left_recall": 0.3157262905162065,
+      "turn_left_f1": 0.2756813417190775,
+      "turn_right_recall": 0.2060975609756097,
+      "turn_right_f1": 0.1975452951490356,
+      "macro_f1_route_ci_low": 0.3476223904861559,
+      "macro_f1_route_ci_high": 0.4280509607629154
+    },
+    {
+      "model": "Telemetry gradient boosting",
+      "samples": 8594,
+      "accuracy": 0.4147079357691413,
+      "balanced_accuracy": 0.4216633099655398,
+      "macro_f1": 0.341392924380058,
+      "weighted_f1": 0.4530356863806382,
+      "log_loss": 1.2490630281069546,
+      "brier_score": 0.6566714449096165,
+      "ece_15_bin": 0.1216700885218681,
+      "p95_latency_ms": NaN,
+      "continue_recall": 0.4051530993278566,
+      "continue_f1": 0.5513911828230212,
+      "slow_recall": 0.6466876971608833,
+      "slow_f1": 0.3416666666666667,
+      "stop_recall": 0.4889589905362776,
+      "stop_f1": 0.2986512524084778,
+      "turn_left_recall": 0.2797118847539016,
+      "turn_left_f1": 0.2583148558758315,
+      "turn_right_recall": 0.2878048780487804,
+      "turn_right_f1": 0.2569406641262928,
+      "macro_f1_route_ci_low": 0.3149153283503967,
+      "macro_f1_route_ci_high": 0.3493997015446531
+    },
+    {
+      "model": "Vision linear (efficientnet_b0)",
+      "samples": 8594,
+      "accuracy": 0.5751687223644403,
+      "balanced_accuracy": 0.2967510353079744,
+      "macro_f1": 0.305034605073554,
+      "weighted_f1": 0.5344562546214928,
+      "log_loss": 10.24602122531317,
+      "brier_score": 0.8349054717132016,
+      "ece_15_bin": 0.4091336187074714,
+      "p95_latency_ms": NaN,
+      "continue_recall": 0.8269230769230769,
+      "continue_f1": 0.7406354515050168,
+      "slow_recall": 0.1556256572029442,
+      "slow_f1": 0.2148040638606676,
+      "stop_recall": 0.252365930599369,
+      "stop_f1": 0.255591054313099,
+      "turn_left_recall": 0.1500600240096038,
+      "turn_left_f1": 0.1810282404055032,
+      "turn_right_recall": 0.098780487804878,
+      "turn_right_f1": 0.1331142152834839,
+      "macro_f1_route_ci_low": 0.2678056005630319,
+      "macro_f1_route_ci_high": 0.3273143437147999
+    },
+    {
+      "model": "Past-motion rule",
+      "samples": 8594,
+      "accuracy": 0.4549685827321387,
+      "balanced_accuracy": 0.223773826591683,
+      "macro_f1": 0.2235297785953733,
+      "weighted_f1": 0.4532542316647034,
+      "log_loss": 2.058490921490782,
+      "brier_score": 0.966304980218757,
+      "ece_15_bin": 0.4450314172678615,
+      "p95_latency_ms": NaN,
+      "continue_recall": 0.6562733383121733,
+      "continue_f1": 0.6534064504136072,
+      "slow_recall": 0.1019978969505783,
+      "slow_f1": 0.0962779156327543,
+      "stop_recall": 0.0,
+      "stop_f1": 0.0,
+      "turn_left_recall": 0.1776710684273709,
+      "turn_left_f1": 0.1792852816474863,
+      "turn_right_recall": 0.1829268292682926,
+      "turn_right_f1": 0.1886792452830188,
+      "macro_f1_route_ci_low": 0.1873463820585989,
+      "macro_f1_route_ci_high": 0.2345195759071031
+    },
+    {
+      "model": "Telemetry logistic",
+      "samples": 8594,
+      "accuracy": 0.196299744007447,
+      "balanced_accuracy": 0.2871892331606019,
+      "macro_f1": 0.1886698746182765,
+      "weighted_f1": 0.1861407158506876,
+      "log_loss": 1.5832758165931655,
+      "brier_score": 0.8215841116420504,
+      "ece_15_bin": 0.1432688177654283,
+      "p95_latency_ms": NaN,
+      "continue_recall": 0.109596713965646,
+      "continue_f1": 0.1816212871287128,
+      "slow_recall": 0.601472134595163,
+      "slow_f1": 0.2506024096385542,
+      "stop_recall": 0.3706624605678233,
+      "stop_f1": 0.1760959160734357,
+      "turn_left_recall": 0.1956782713085234,
+      "turn_left_f1": 0.1721224920802534,
+      "turn_right_recall": 0.1585365853658536,
+      "turn_right_f1": 0.162907268170426,
+      "macro_f1_route_ci_low": 0.1839562547318565,
+      "macro_f1_route_ci_high": 0.2044359347158287
+    },
+    {
+      "model": "Majority class",
+      "samples": 8594,
+      "accuracy": 0.6232255061670933,
+      "balanced_accuracy": 0.2,
+      "macro_f1": 0.1535770609318996,
+      "weighted_f1": 0.4785657076746885,
+      "log_loss": 1.1837257781136636,
+      "brier_score": 0.5768401032344809,
+      "ece_15_bin": 0.032749657591408,
+      "p95_latency_ms": NaN,
+      "continue_recall": 1.0,
+      "continue_f1": 0.7678853046594982,
+      "slow_recall": 0.0,
+      "slow_f1": 0.0,
+      "stop_recall": 0.0,
+      "stop_f1": 0.0,
+      "turn_left_recall": 0.0,
+      "turn_left_f1": 0.0,
+      "turn_right_recall": 0.0,
+      "turn_right_f1": 0.0,
+      "macro_f1_route_ci_low": 0.1460915032679738,
+      "macro_f1_route_ci_high": 0.1554760284937855
+    }
+  ],
+  "per_class": [
+    {
+      "model": "Temporal efficientnet_b0 + telemetry",
+      "class": "continue",
+      "precision": 0.759493670886076,
+      "recall": 0.7393577296489918,
+      "f1": 0.7492904446546831,
+      "support": 5356
+    },
+    {
+      "model": "Temporal efficientnet_b0 + telemetry",
+      "class": "slow",
+      "precision": 0.40311004784689,
+      "recall": 0.3543638275499474,
+      "f1": 0.3771684387241186,
+      "support": 951
+    },
+    {
+      "model": "Temporal efficientnet_b0 + telemetry",
+      "class": "stop",
+      "precision": 0.3754325259515571,
+      "recall": 0.3422712933753943,
+      "f1": 0.3580858085808581,
+      "support": 634
+    },
+    {
+      "model": "Temporal efficientnet_b0 + telemetry",
+      "class": "turn_left",
+      "precision": 0.2446511627906976,
+      "recall": 0.3157262905162065,
+      "f1": 0.2756813417190775,
+      "support": 833
+    },
+    {
+      "model": "Temporal efficientnet_b0 + telemetry",
+      "class": "turn_right",
+      "precision": 0.1896745230078563,
+      "recall": 0.2060975609756097,
+      "f1": 0.1975452951490356,
+      "support": 820
+    }
+  ],
+  "subsets": [
+    {
+      "model": "Temporal efficientnet_b0 + telemetry",
+      "subset": "all_test",
+      "samples": 8594,
+      "macro_f1": 0.3915542657655546,
+      "balanced_accuracy": 0.39156334041323,
+      "accuracy": 0.5755178031184547
+    },
+    {
+      "model": "Temporal efficientnet_b0 + telemetry",
+      "subset": "unseen_grid_cells",
+      "samples": 1846,
+      "macro_f1": 0.3342824235159851,
+      "balanced_accuracy": 0.3360174291252364,
+      "accuracy": 0.5384615384615384
+    },
+    {
+      "model": "Temporal efficientnet_b0 + telemetry",
+      "subset": "night_utc",
+      "samples": 6930,
+      "macro_f1": 0.4102137401798389,
+      "balanced_accuracy": 0.4080902962744183,
+      "accuracy": 0.5946608946608947
+    },
+    {
+      "model": "Temporal efficientnet_b0 + telemetry",
+      "subset": "gps_accuracy_le_10m",
+      "samples": 8319,
+      "macro_f1": 0.393316436996118,
+      "balanced_accuracy": 0.3936744515708604,
+      "accuracy": 0.5783147012862123
+    }
+  ],
+  "selective_prediction": [
+    {
+      "confidence_threshold": 0.0,
+      "coverage": 1.0,
+      "accepted_examples": 8594,
+      "selective_accuracy": 0.5755178031184547,
+      "selective_macro_f1": 0.3915542657655546
+    },
+    {
+      "confidence_threshold": 0.25,
+      "coverage": 0.9987200372352804,
+      "accepted_examples": 8583,
+      "selective_accuracy": 0.5760223698007689,
+      "selective_macro_f1": 0.3919543899984051
+    },
+    {
+      "confidence_threshold": 0.35,
+      "coverage": 0.9100535257156156,
+      "accepted_examples": 7821,
+      "selective_accuracy": 0.6006904487917146,
+      "selective_macro_f1": 0.4060706565050537
+    },
+    {
+      "confidence_threshold": 0.45,
+      "coverage": 0.6947870607400511,
+      "accepted_examples": 5971,
+      "selective_accuracy": 0.658348685312343,
+      "selective_macro_f1": 0.4327252057994619
+    },
+    {
+      "confidence_threshold": 0.55,
+      "coverage": 0.4717244589248313,
+      "accepted_examples": 4054,
+      "selective_accuracy": 0.7323630981746423,
+      "selective_macro_f1": 0.470227025478709
+    },
+    {
+      "confidence_threshold": 0.65,
+      "coverage": 0.3019548522224808,
+      "accepted_examples": 2595,
+      "selective_accuracy": 0.8092485549132948,
+      "selective_macro_f1": 0.5275062675719726
+    },
+    {
+      "confidence_threshold": 0.75,
+      "coverage": 0.1748894577612287,
+      "accepted_examples": 1503,
+      "selective_accuracy": 0.8782435129740519,
+      "selective_macro_f1": 0.5566727737271018
+    },
+    {
+      "confidence_threshold": 0.85,
+      "coverage": 0.0829648592040958,
+      "accepted_examples": 713,
+      "selective_accuracy": 0.938288920056101,
+      "selective_macro_f1": 0.6097924940442192
+    }
+  ],
+  "calibration": {
+    "method": "temperature scaling",
+    "temperature": 1.106177568435669,
+    "fit_split": "validation after restoring the best checkpoint",
+    "independent_calibration_split": false
+  },
+  "limitations": [
+    "Labels are GPS-derived proxies rather than human safety annotations.",
+    "The test set contains five held-out routes from one operating domain.",
+    "Raw accuracy is distorted by the majority continue class; macro F1 is primary.",
+    "The calibration set is the validation split, not an independent fourth split.",
+    "This evaluation is not evidence of safe vehicle control."
+  ]
+}

model.safetensors

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:241f8335bdbe45e9f88e45640b35e3426e2c1e8fd58b9e0bec774a417988d57c
-size 21384660
+oid sha256:af1a98cd96bb68d642f7a0adeedfacac37b230917afc15f3c87230a99a15b8a2
+size 21384748

recompute_metrics.py

@@ -0,0 +1,120 @@
+"""Recompute YellowCab's public held-out metrics from sanitized predictions."""
+
+from __future__ import annotations
+
+import json
+from pathlib import Path
+
+import numpy as np
+import pandas as pd
+from sklearn.metrics import (
+    accuracy_score,
+    balanced_accuracy_score,
+    brier_score_loss,
+    f1_score,
+    log_loss,
+)
+
+
+CLASSES = ("continue", "slow", "stop", "turn_left", "turn_right")
+SEED = 20260726
+BOOTSTRAP_ITERATIONS = 500
+
+
+def expected_calibration_error(
+    probabilities: np.ndarray,
+    targets: np.ndarray,
+    *,
+    bins: int = 15,
+) -> float:
+    confidence = probabilities.max(axis=1)
+    predictions = probabilities.argmax(axis=1)
+    correct = predictions == targets
+    edges = np.linspace(0.0, 1.0, bins + 1)
+    value = 0.0
+    for index in range(bins):
+        if index == bins - 1:
+            mask = (confidence >= edges[index]) & (confidence <= edges[index + 1])
+        else:
+            mask = (confidence >= edges[index]) & (confidence < edges[index + 1])
+        if mask.any():
+            value += float(mask.mean()) * abs(
+                float(correct[mask].mean()) - float(confidence[mask].mean())
+            )
+    return value
+
+
+def route_bootstrap_macro_f1(
+    frame: pd.DataFrame,
+    targets: np.ndarray,
+    predictions: np.ndarray,
+) -> tuple[float, float]:
+    route_values = frame["route_group"].astype(str).to_numpy()
+    routes = np.unique(route_values)
+    rng = np.random.default_rng(SEED)
+    values: list[float] = []
+    for _ in range(BOOTSTRAP_ITERATIONS):
+        sampled = rng.choice(routes, size=len(routes), replace=True)
+        indices = np.concatenate(
+            [np.flatnonzero(route_values == route) for route in sampled]
+        )
+        values.append(
+            float(
+                f1_score(
+                    targets[indices],
+                    predictions[indices],
+                    labels=np.arange(len(CLASSES)),
+                    average="macro",
+                    zero_division=0,
+                )
+            )
+        )
+    return tuple(float(value) for value in np.quantile(values, [0.025, 0.975]))
+
+
+def main() -> None:
+    root = Path(__file__).resolve().parent
+    frame = pd.read_csv(root / "eval_predictions.csv")
+    class_to_index = {name: index for index, name in enumerate(CLASSES)}
+    targets = frame["label"].map(class_to_index).to_numpy(dtype=np.int64)
+    predictions = frame["prediction"].map(class_to_index).to_numpy(dtype=np.int64)
+    probabilities = frame[
+        [f"prob_{class_name}" for class_name in CLASSES]
+    ].to_numpy(dtype=np.float64)
+    if not np.allclose(probabilities.sum(axis=1), 1.0, atol=1e-6):
+        raise ValueError("Probability rows do not sum to one")
+    one_hot = np.eye(len(CLASSES), dtype=np.float64)[targets]
+    ci_low, ci_high = route_bootstrap_macro_f1(frame, targets, predictions)
+    result = {
+        "samples": int(len(frame)),
+        "routes": int(frame["route_group"].nunique()),
+        "accuracy": float(accuracy_score(targets, predictions)),
+        "balanced_accuracy": float(balanced_accuracy_score(targets, predictions)),
+        "macro_f1": float(
+            f1_score(
+                targets,
+                predictions,
+                labels=np.arange(len(CLASSES)),
+                average="macro",
+                zero_division=0,
+            )
+        ),
+        "weighted_f1": float(
+            f1_score(
+                targets,
+                predictions,
+                labels=np.arange(len(CLASSES)),
+                average="weighted",
+                zero_division=0,
+            )
+        ),
+        "log_loss": float(log_loss(targets, probabilities, labels=range(len(CLASSES)))),
+        "brier_score": float(np.mean(np.sum((probabilities - one_hot) ** 2, axis=1))),
+        "ece_15_bin": expected_calibration_error(probabilities, targets),
+        "macro_f1_route_ci_95": [ci_low, ci_high],
+    }
+    print(json.dumps(result, indent=2))
+
+
+if __name__ == "__main__":
+    main()

selective_metrics.csv

@@ -0,0 +1,9 @@
+confidence_threshold,coverage,accepted_examples,selective_accuracy,selective_macro_f1
+0.0,1.0,8594,0.5755178031184547,0.3915542657655546
+0.25,0.9987200372352805,8583,0.5760223698007689,0.3919543899984051
+0.35,0.9100535257156156,7821,0.6006904487917146,0.4060706565050537
+0.45,0.6947870607400511,5971,0.658348685312343,0.43272520579946194
+0.55,0.4717244589248313,4054,0.7323630981746423,0.470227025478709
+0.65,0.3019548522224808,2595,0.8092485549132948,0.5275062675719726
+0.75,0.17488945776122877,1503,0.8782435129740519,0.5566727737271018
+0.85,0.08296485920409588,713,0.938288920056101,0.6097924940442192

subset_metrics.csv

@@ -0,0 +1,25 @@
+model,subset,samples,macro_f1,balanced_accuracy,accuracy
+Majority class,all_test,8594,0.1535770609318996,0.2,0.6232255061670933
+Majority class,unseen_grid_cells,1846,0.1571052631578947,0.2,0.6468039003250271
+Majority class,night_utc,6930,0.1544077257021351,0.2,0.6287157287157287
+Majority class,gps_accuracy_le_10m,8319,0.153802900266351,0.2,0.6247145089554033
+Past-motion rule,all_test,8594,0.2235297785953733,0.223773826591683,0.4549685827321387
+Past-motion rule,unseen_grid_cells,1846,0.2332335187587507,0.236197410832345,0.4842903575297941
+Past-motion rule,night_utc,6930,0.2274748552099628,0.2276228825290112,0.4647907647907648
+Past-motion rule,gps_accuracy_le_10m,8319,0.2233687790979745,0.2235505984136381,0.4570260848659694
+Telemetry logistic,all_test,8594,0.1886698746182765,0.2871892331606019,0.196299744007447
+Telemetry logistic,unseen_grid_cells,1846,0.1725156222420228,0.256991711558178,0.189599133261105
+Telemetry logistic,night_utc,6930,0.1896834893067224,0.284189614812793,0.1988455988455988
+Telemetry logistic,gps_accuracy_le_10m,8319,0.1864075839388003,0.2847033040497686,0.1962976319269143
+Telemetry gradient boosting,all_test,8594,0.341392924380058,0.4216633099655398,0.4147079357691413
+Telemetry gradient boosting,unseen_grid_cells,1846,0.3136214897768011,0.386078121522199,0.3959913326110509
+Telemetry gradient boosting,night_utc,6930,0.3470049007798196,0.4262284930269192,0.4278499278499278
+Telemetry gradient boosting,gps_accuracy_le_10m,8319,0.3426226516709886,0.4236167573460645,0.4168770284890011
+Vision linear (efficientnet_b0),all_test,8594,0.305034605073554,0.2967510353079744,0.5751687223644403
+Vision linear (efficientnet_b0),unseen_grid_cells,1846,0.2928477852235921,0.2881051221485449,0.5893824485373781
+Vision linear (efficientnet_b0),night_utc,6930,0.3217391911486912,0.3126224230959082,0.5862914862914863
+Vision linear (efficientnet_b0),gps_accuracy_le_10m,8319,0.3060672066310756,0.2972976109580328,0.576631806707537
+Temporal efficientnet_b0 + telemetry,all_test,8594,0.3915542657655546,0.39156334041323,0.5755178031184547
+Temporal efficientnet_b0 + telemetry,unseen_grid_cells,1846,0.3342824235159851,0.3360174291252364,0.5384615384615384
+Temporal efficientnet_b0 + telemetry,night_utc,6930,0.4102137401798389,0.4080902962744183,0.5946608946608947
+Temporal efficientnet_b0 + telemetry,gps_accuracy_le_10m,8319,0.393316436996118,0.3936744515708604,0.5783147012862123