Upload folder using huggingface_hub
Browse files- README.md +124 -0
- adapter/README.md +207 -0
- adapter/adapter_config.json +51 -0
- adapter/adapter_model.safetensors +3 -0
- assay_config.json +7 -0
- assay_head.safetensors +3 -0
- calibration.json +29 -0
- eval-dev-scaled.json +750 -0
- eval-dev.json +750 -0
- eval-holdout-scaled.json +239 -0
- eval-holdout.json +239 -0
- eval-transfer-v4-scaled.json +239 -0
- eval-transfer-v4.json +239 -0
- train_args.json +27 -0
- train_log.jsonl +140 -0
README.md
ADDED
|
@@ -0,0 +1,124 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
---
|
| 2 |
+
license: apache-2.0
|
| 3 |
+
base_model: Qwen/Qwen3.8-27B
|
| 4 |
+
library_name: transformers
|
| 5 |
+
pipeline_tag: text-classification
|
| 6 |
+
tags:
|
| 7 |
+
- assay
|
| 8 |
+
- calibrated
|
| 9 |
+
- decision-model
|
| 10 |
+
- zero-shot-classification
|
| 11 |
+
language:
|
| 12 |
+
- en
|
| 13 |
+
---
|
| 14 |
+
|
| 15 |
+
# assay-27b
|
| 16 |
+
|
| 17 |
+
Calibrated typed decisions from one forward pass. Send a state and named typed questions
|
| 18 |
+
(`bool` yes/no, `choice` over 2..255 described options, `score` over 2..10 ordered levels);
|
| 19 |
+
get a probability distribution per question, a confidence and an evidence score. No text is
|
| 20 |
+
generated, so nothing can come back off-schema.
|
| 21 |
+
|
| 22 |
+
Code, server and training recipe: https://github.com/bgokden/assay
|
| 23 |
+
|
| 24 |
+
## How it is built
|
| 25 |
+
|
| 26 |
+
- Backbone `Qwen/Qwen3.8-27B` with a LoRA adapter (r=16, alpha=32,
|
| 27 |
+
lr=5e-05, 1.0 epoch, batch 4 x 2
|
| 28 |
+
accumulation, 4-bit base (QLoRA)); this repository holds the adapter (`adapter/`) and the evidence head; the base is loaded from `Qwen/Qwen3.8-27B` in 4bit (bitsandbytes) at load time. Loading downloads the base model separately; the 4-bit base needs about 15 GB of GPU memory.
|
| 29 |
+
- The answer is read from the model's own next-token logits over option label tokens at a
|
| 30 |
+
single decision position, so the base model's zero-shot competence is the starting point.
|
| 31 |
+
- Questions are isolated branches over a shared state (block attention mask, restarted
|
| 32 |
+
positions): packed and separate requests agree exactly.
|
| 33 |
+
- Trained with cross-entropy against soft targets: human label distributions where the source
|
| 34 |
+
has them, SORD-smoothed levels for ordinal questions, one-hot otherwise. Choice options are
|
| 35 |
+
shuffled per example.
|
| 36 |
+
- An evidence head (linear on the decision token, `assay_head.safetensors`) predicts whether the
|
| 37 |
+
state supports the question, trained on passage-swapped negatives.
|
| 38 |
+
- Global temperature 1.235 fitted on the calibration split of the training tasks and
|
| 39 |
+
applied unchanged everywhere else.
|
| 40 |
+
|
| 41 |
+
## Evaluation
|
| 42 |
+
|
| 43 |
+
| split | n | accuracy | Brier | NLL | ECE | confident errors |
|
| 44 |
+
|---|---|---|---|---|---|---|
|
| 45 |
+
| seen tasks (dev), raw | 5513 | 0.834 | 0.245 | 0.466 | 0.048 | 0.036 |
|
| 46 |
+
| seen tasks (dev), scaled | 5513 | 0.834 | 0.243 | 0.451 | 0.040 | 0.021 |
|
| 47 |
+
| unseen tasks (holdout), raw | 2020 | 0.842 | 0.220 | 0.392 | 0.031 | 0.015 |
|
| 48 |
+
| unseen tasks (holdout), scaled | 2020 | 0.842 | 0.221 | 0.390 | 0.040 | 0.005 |
|
| 49 |
+
| kev transfer-v4 dev, raw | 764 | 0.842 | 0.234 | 0.460 | 0.065 | 0.047 |
|
| 50 |
+
| kev transfer-v4 dev, scaled | 764 | 0.842 | 0.229 | 0.430 | 0.041 | 0.038 |
|
| 51 |
+
|
| 52 |
+
"Unseen tasks" are eleven datasets never used in training (bbc_news, app_reviews, scitail,
|
| 53 |
+
medical_questions_pairs, tweet_irony, ethos, stance_climate, dream, copa, truthful_qa,
|
| 54 |
+
hh_rlhf). "kev transfer-v4 dev" is the public suite from
|
| 55 |
+
[jaredpalmer/kev-suites](https://huggingface.co/datasets/jaredpalmer/kev-suites) (mmlu,
|
| 56 |
+
emotion, sciq, tweet_offensive, qnli, paws and synthetic rule holdouts); none of its sources
|
| 57 |
+
are in the training data. Brier is the multi-class sum of squared errors (0..2), ECE uses 15
|
| 58 |
+
bins, confident errors are answers with p >= 0.9 that are wrong.
|
| 59 |
+
|
| 60 |
+
| transfer-v4 source | n | accuracy | Brier | ECE |
|
| 61 |
+
|---|---|---|---|---|
|
| 62 |
+
| composition_held_and_or | 32 | 1.000 | 0.002 | 0.012 |
|
| 63 |
+
| composition_held_conditional | 32 | 0.812 | 0.347 | 0.177 |
|
| 64 |
+
| composition_held_or_not | 32 | 0.906 | 0.168 | 0.087 |
|
| 65 |
+
| contrastive_authorization | 40 | 1.000 | 0.000 | 0.003 |
|
| 66 |
+
| contrastive_deadline | 40 | 0.950 | 0.047 | 0.084 |
|
| 67 |
+
| emotion | 116 | 0.647 | 0.478 | 0.077 |
|
| 68 |
+
| mmlu | 116 | 0.784 | 0.333 | 0.119 |
|
| 69 |
+
| paws | 80 | 0.775 | 0.331 | 0.151 |
|
| 70 |
+
| qnli | 80 | 0.963 | 0.062 | 0.055 |
|
| 71 |
+
| sciq | 116 | 0.983 | 0.038 | 0.034 |
|
| 72 |
+
| tweet_offensive | 80 | 0.738 | 0.331 | 0.151 |
|
| 73 |
+
|
| 74 |
+
Latency on one RTX 5090 (bf16, transformers, packed questions over one state versus separate requests):
|
| 75 |
+
|
| 76 |
+
```
|
| 77 |
+
questions packed_ms separate_ms
|
| 78 |
+
1 109.5 109.6
|
| 79 |
+
3 194.0 333.4
|
| 80 |
+
6 250.4 664.7
|
| 81 |
+
12 412.0 1330.3
|
| 82 |
+
24 750.8 2666.8
|
| 83 |
+
```
|
| 84 |
+
|
| 85 |
+
## Usage
|
| 86 |
+
|
| 87 |
+
```python
|
| 88 |
+
from assay.model import AssayModel
|
| 89 |
+
from assay.schema import Question
|
| 90 |
+
|
| 91 |
+
model = AssayModel.from_pretrained("Berk/assay-27b")
|
| 92 |
+
answers = model.answer(
|
| 93 |
+
state="My card was charged twice for order A-104.",
|
| 94 |
+
questions={
|
| 95 |
+
"refund": Question(type="bool", instructions="Does the customer ask for money back?"),
|
| 96 |
+
"team": Question(type="choice", instructions="Which team should handle this?",
|
| 97 |
+
options={"billing": "Charges and refunds", "technical": "Bugs"}),
|
| 98 |
+
},
|
| 99 |
+
)
|
| 100 |
+
print(answers["team"].probabilities, answers["refund"].p_true, answers["refund"].evidence)
|
| 101 |
+
```
|
| 102 |
+
|
| 103 |
+
## Limitations
|
| 104 |
+
|
| 105 |
+
Text only, English training data. No arithmetic, counting, date comparison or multi-hop
|
| 106 |
+
reasoning in one pass; keep those in code. Accuracy drops with unrelated state. The evidence
|
| 107 |
+
head is trained on coarse swapped-passage negatives. Probabilities are calibrated in aggregate
|
| 108 |
+
on the evaluated distributions, which is not a guarantee about any single answer or about your
|
| 109 |
+
data; check calibration on your own labels before acting on thresholds.
|
| 110 |
+
|
| 111 |
+
## Training data
|
| 112 |
+
|
| 113 |
+
Fifty-five public classification, inference, reading-comprehension and preference datasets
|
| 114 |
+
rendered as typed questions with described options, plus a synthetic policy-application
|
| 115 |
+
generator (see `assay/data/tasks.py` in the repository for the rubrics). Each dataset keeps
|
| 116 |
+
its own licence; the per-dataset list is in
|
| 117 |
+
[docs/datasets.md](https://github.com/bgokden/assay/blob/master/docs/datasets.md). Several
|
| 118 |
+
sources carry non-commercial or research-only terms; check them before commercial use.
|
| 119 |
+
|
| 120 |
+
## Relationship to other work
|
| 121 |
+
|
| 122 |
+
Assay is an independent project. Jev and System One are names of TypeSafe AI's products and
|
| 123 |
+
are mentioned only to describe and compare; kev-suites is Jared Palmer's evaluation data.
|
| 124 |
+
Assay is not affiliated with or endorsed by either.
|
adapter/README.md
ADDED
|
@@ -0,0 +1,207 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
---
|
| 2 |
+
base_model: Qwen/Qwen3.8-27B
|
| 3 |
+
library_name: peft
|
| 4 |
+
pipeline_tag: text-generation
|
| 5 |
+
tags:
|
| 6 |
+
- base_model:adapter:Qwen/Qwen3.8-27B
|
| 7 |
+
- lora
|
| 8 |
+
- transformers
|
| 9 |
+
---
|
| 10 |
+
|
| 11 |
+
# Model Card for Model ID
|
| 12 |
+
|
| 13 |
+
<!-- Provide a quick summary of what the model is/does. -->
|
| 14 |
+
|
| 15 |
+
|
| 16 |
+
|
| 17 |
+
## Model Details
|
| 18 |
+
|
| 19 |
+
### Model Description
|
| 20 |
+
|
| 21 |
+
<!-- Provide a longer summary of what this model is. -->
|
| 22 |
+
|
| 23 |
+
|
| 24 |
+
|
| 25 |
+
- **Developed by:** [More Information Needed]
|
| 26 |
+
- **Funded by [optional]:** [More Information Needed]
|
| 27 |
+
- **Shared by [optional]:** [More Information Needed]
|
| 28 |
+
- **Model type:** [More Information Needed]
|
| 29 |
+
- **Language(s) (NLP):** [More Information Needed]
|
| 30 |
+
- **License:** [More Information Needed]
|
| 31 |
+
- **Finetuned from model [optional]:** [More Information Needed]
|
| 32 |
+
|
| 33 |
+
### Model Sources [optional]
|
| 34 |
+
|
| 35 |
+
<!-- Provide the basic links for the model. -->
|
| 36 |
+
|
| 37 |
+
- **Repository:** [More Information Needed]
|
| 38 |
+
- **Paper [optional]:** [More Information Needed]
|
| 39 |
+
- **Demo [optional]:** [More Information Needed]
|
| 40 |
+
|
| 41 |
+
## Uses
|
| 42 |
+
|
| 43 |
+
<!-- Address questions around how the model is intended to be used, including the foreseeable users of the model and those affected by the model. -->
|
| 44 |
+
|
| 45 |
+
### Direct Use
|
| 46 |
+
|
| 47 |
+
<!-- This section is for the model use without fine-tuning or plugging into a larger ecosystem/app. -->
|
| 48 |
+
|
| 49 |
+
[More Information Needed]
|
| 50 |
+
|
| 51 |
+
### Downstream Use [optional]
|
| 52 |
+
|
| 53 |
+
<!-- This section is for the model use when fine-tuned for a task, or when plugged into a larger ecosystem/app -->
|
| 54 |
+
|
| 55 |
+
[More Information Needed]
|
| 56 |
+
|
| 57 |
+
### Out-of-Scope Use
|
| 58 |
+
|
| 59 |
+
<!-- This section addresses misuse, malicious use, and uses that the model will not work well for. -->
|
| 60 |
+
|
| 61 |
+
[More Information Needed]
|
| 62 |
+
|
| 63 |
+
## Bias, Risks, and Limitations
|
| 64 |
+
|
| 65 |
+
<!-- This section is meant to convey both technical and sociotechnical limitations. -->
|
| 66 |
+
|
| 67 |
+
[More Information Needed]
|
| 68 |
+
|
| 69 |
+
### Recommendations
|
| 70 |
+
|
| 71 |
+
<!-- This section is meant to convey recommendations with respect to the bias, risk, and technical limitations. -->
|
| 72 |
+
|
| 73 |
+
Users (both direct and downstream) should be made aware of the risks, biases and limitations of the model. More information needed for further recommendations.
|
| 74 |
+
|
| 75 |
+
## How to Get Started with the Model
|
| 76 |
+
|
| 77 |
+
Use the code below to get started with the model.
|
| 78 |
+
|
| 79 |
+
[More Information Needed]
|
| 80 |
+
|
| 81 |
+
## Training Details
|
| 82 |
+
|
| 83 |
+
### Training Data
|
| 84 |
+
|
| 85 |
+
<!-- This should link to a Dataset Card, perhaps with a short stub of information on what the training data is all about as well as documentation related to data pre-processing or additional filtering. -->
|
| 86 |
+
|
| 87 |
+
[More Information Needed]
|
| 88 |
+
|
| 89 |
+
### Training Procedure
|
| 90 |
+
|
| 91 |
+
<!-- This relates heavily to the Technical Specifications. Content here should link to that section when it is relevant to the training procedure. -->
|
| 92 |
+
|
| 93 |
+
#### Preprocessing [optional]
|
| 94 |
+
|
| 95 |
+
[More Information Needed]
|
| 96 |
+
|
| 97 |
+
|
| 98 |
+
#### Training Hyperparameters
|
| 99 |
+
|
| 100 |
+
- **Training regime:** [More Information Needed] <!--fp32, fp16 mixed precision, bf16 mixed precision, bf16 non-mixed precision, fp16 non-mixed precision, fp8 mixed precision -->
|
| 101 |
+
|
| 102 |
+
#### Speeds, Sizes, Times [optional]
|
| 103 |
+
|
| 104 |
+
<!-- This section provides information about throughput, start/end time, checkpoint size if relevant, etc. -->
|
| 105 |
+
|
| 106 |
+
[More Information Needed]
|
| 107 |
+
|
| 108 |
+
## Evaluation
|
| 109 |
+
|
| 110 |
+
<!-- This section describes the evaluation protocols and provides the results. -->
|
| 111 |
+
|
| 112 |
+
### Testing Data, Factors & Metrics
|
| 113 |
+
|
| 114 |
+
#### Testing Data
|
| 115 |
+
|
| 116 |
+
<!-- This should link to a Dataset Card if possible. -->
|
| 117 |
+
|
| 118 |
+
[More Information Needed]
|
| 119 |
+
|
| 120 |
+
#### Factors
|
| 121 |
+
|
| 122 |
+
<!-- These are the things the evaluation is disaggregating by, e.g., subpopulations or domains. -->
|
| 123 |
+
|
| 124 |
+
[More Information Needed]
|
| 125 |
+
|
| 126 |
+
#### Metrics
|
| 127 |
+
|
| 128 |
+
<!-- These are the evaluation metrics being used, ideally with a description of why. -->
|
| 129 |
+
|
| 130 |
+
[More Information Needed]
|
| 131 |
+
|
| 132 |
+
### Results
|
| 133 |
+
|
| 134 |
+
[More Information Needed]
|
| 135 |
+
|
| 136 |
+
#### Summary
|
| 137 |
+
|
| 138 |
+
|
| 139 |
+
|
| 140 |
+
## Model Examination [optional]
|
| 141 |
+
|
| 142 |
+
<!-- Relevant interpretability work for the model goes here -->
|
| 143 |
+
|
| 144 |
+
[More Information Needed]
|
| 145 |
+
|
| 146 |
+
## Environmental Impact
|
| 147 |
+
|
| 148 |
+
<!-- Total emissions (in grams of CO2eq) and additional considerations, such as electricity usage, go here. Edit the suggested text below accordingly -->
|
| 149 |
+
|
| 150 |
+
Carbon emissions can be estimated using the [Machine Learning Impact calculator](https://mlco2.github.io/impact#compute) presented in [Lacoste et al. (2019)](https://arxiv.org/abs/1910.09700).
|
| 151 |
+
|
| 152 |
+
- **Hardware Type:** [More Information Needed]
|
| 153 |
+
- **Hours used:** [More Information Needed]
|
| 154 |
+
- **Cloud Provider:** [More Information Needed]
|
| 155 |
+
- **Compute Region:** [More Information Needed]
|
| 156 |
+
- **Carbon Emitted:** [More Information Needed]
|
| 157 |
+
|
| 158 |
+
## Technical Specifications [optional]
|
| 159 |
+
|
| 160 |
+
### Model Architecture and Objective
|
| 161 |
+
|
| 162 |
+
[More Information Needed]
|
| 163 |
+
|
| 164 |
+
### Compute Infrastructure
|
| 165 |
+
|
| 166 |
+
[More Information Needed]
|
| 167 |
+
|
| 168 |
+
#### Hardware
|
| 169 |
+
|
| 170 |
+
[More Information Needed]
|
| 171 |
+
|
| 172 |
+
#### Software
|
| 173 |
+
|
| 174 |
+
[More Information Needed]
|
| 175 |
+
|
| 176 |
+
## Citation [optional]
|
| 177 |
+
|
| 178 |
+
<!-- If there is a paper or blog post introducing the model, the APA and Bibtex information for that should go in this section. -->
|
| 179 |
+
|
| 180 |
+
**BibTeX:**
|
| 181 |
+
|
| 182 |
+
[More Information Needed]
|
| 183 |
+
|
| 184 |
+
**APA:**
|
| 185 |
+
|
| 186 |
+
[More Information Needed]
|
| 187 |
+
|
| 188 |
+
## Glossary [optional]
|
| 189 |
+
|
| 190 |
+
<!-- If relevant, include terms and calculations in this section that can help readers understand the model or model card. -->
|
| 191 |
+
|
| 192 |
+
[More Information Needed]
|
| 193 |
+
|
| 194 |
+
## More Information [optional]
|
| 195 |
+
|
| 196 |
+
[More Information Needed]
|
| 197 |
+
|
| 198 |
+
## Model Card Authors [optional]
|
| 199 |
+
|
| 200 |
+
[More Information Needed]
|
| 201 |
+
|
| 202 |
+
## Model Card Contact
|
| 203 |
+
|
| 204 |
+
[More Information Needed]
|
| 205 |
+
### Framework versions
|
| 206 |
+
|
| 207 |
+
- PEFT 0.21.0
|
adapter/adapter_config.json
ADDED
|
@@ -0,0 +1,51 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
{
|
| 2 |
+
"alora_invocation_tokens": null,
|
| 3 |
+
"alpha_pattern": {},
|
| 4 |
+
"arrow_config": null,
|
| 5 |
+
"auto_mapping": null,
|
| 6 |
+
"base_model_name_or_path": "Qwen/Qwen3.8-27B",
|
| 7 |
+
"bias": "none",
|
| 8 |
+
"corda_config": null,
|
| 9 |
+
"ensure_weight_tying": false,
|
| 10 |
+
"eva_config": null,
|
| 11 |
+
"exclude_modules": null,
|
| 12 |
+
"fan_in_fan_out": false,
|
| 13 |
+
"inference_mode": true,
|
| 14 |
+
"init_lora_weights": true,
|
| 15 |
+
"kasa_config": null,
|
| 16 |
+
"layer_replication": null,
|
| 17 |
+
"layers_pattern": null,
|
| 18 |
+
"layers_to_transform": null,
|
| 19 |
+
"loftq_config": {},
|
| 20 |
+
"lora_alpha": 32,
|
| 21 |
+
"lora_bias": false,
|
| 22 |
+
"lora_dropout": 0.05,
|
| 23 |
+
"lora_ga_config": null,
|
| 24 |
+
"megatron_config": null,
|
| 25 |
+
"megatron_core": "megatron.core",
|
| 26 |
+
"modules_to_save": null,
|
| 27 |
+
"monteclora_config": null,
|
| 28 |
+
"peft_type": "LORA",
|
| 29 |
+
"peft_version": "0.21.0",
|
| 30 |
+
"qalora_group_size": 16,
|
| 31 |
+
"r": 16,
|
| 32 |
+
"rank_pattern": {},
|
| 33 |
+
"revision": null,
|
| 34 |
+
"target_modules": [
|
| 35 |
+
"k_proj",
|
| 36 |
+
"down_proj",
|
| 37 |
+
"v_proj",
|
| 38 |
+
"up_proj",
|
| 39 |
+
"o_proj",
|
| 40 |
+
"q_proj",
|
| 41 |
+
"gate_proj"
|
| 42 |
+
],
|
| 43 |
+
"target_parameters": null,
|
| 44 |
+
"task_type": "CAUSAL_LM",
|
| 45 |
+
"trainable_token_indices": null,
|
| 46 |
+
"use_bdlora": null,
|
| 47 |
+
"use_dora": false,
|
| 48 |
+
"use_qalora": false,
|
| 49 |
+
"use_rslora": false,
|
| 50 |
+
"velora_config": null
|
| 51 |
+
}
|
adapter/adapter_model.safetensors
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:a28410bb07763c688913b055a16a819daede42c06379bfa10a63d6f30492e18e
|
| 3 |
+
size 318835672
|
assay_config.json
ADDED
|
@@ -0,0 +1,7 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
{
|
| 2 |
+
"base_model_id": "Qwen/Qwen3.8-27B",
|
| 3 |
+
"temperature": 1.2348757098624759,
|
| 4 |
+
"adapter": "adapter",
|
| 5 |
+
"normalize_evidence_input": true,
|
| 6 |
+
"quantized_base": "4bit"
|
| 7 |
+
}
|
assay_head.safetensors
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:6250ef8cdcefa00d5148e78b7d52f4bc71502c90755efdc5b635fe3046dd6efc
|
| 3 |
+
size 20620
|
calibration.json
ADDED
|
@@ -0,0 +1,29 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
{
|
| 2 |
+
"temperature": 1.2348757098624759,
|
| 3 |
+
"before": {
|
| 4 |
+
"count": 5513,
|
| 5 |
+
"accuracy": 0.8552512243787411,
|
| 6 |
+
"brier": 0.21241614066642384,
|
| 7 |
+
"brier_soft": 0.18826896283299935,
|
| 8 |
+
"nll": 0.4007808281586553,
|
| 9 |
+
"nll_soft": 0.4192940207961012,
|
| 10 |
+
"ece": 0.03507424781716495,
|
| 11 |
+
"mean_confidence": 0.8712977531962971,
|
| 12 |
+
"confident_error_rate": 0.02612007981135498,
|
| 13 |
+
"score_mae": 0.5027471272862926,
|
| 14 |
+
"evidence_auc": 1.0
|
| 15 |
+
},
|
| 16 |
+
"after": {
|
| 17 |
+
"count": 5513,
|
| 18 |
+
"accuracy": 0.8552512243787411,
|
| 19 |
+
"brier": 0.21164036251854193,
|
| 20 |
+
"brier_soft": 0.1859166543100605,
|
| 21 |
+
"nll": 0.39492082617353075,
|
| 22 |
+
"nll_soft": 0.40991277434255546,
|
| 23 |
+
"ece": 0.029256736355713523,
|
| 24 |
+
"mean_confidence": 0.8448527506456408,
|
| 25 |
+
"confident_error_rate": 0.016143660438962454,
|
| 26 |
+
"score_mae": 0.5266111543043351,
|
| 27 |
+
"evidence_auc": 1.0
|
| 28 |
+
}
|
| 29 |
+
}
|
eval-dev-scaled.json
ADDED
|
@@ -0,0 +1,750 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
{
|
| 2 |
+
"overall": {
|
| 3 |
+
"count": 5513,
|
| 4 |
+
"accuracy": 0.8342100489751496,
|
| 5 |
+
"brier": 0.24266185842494856,
|
| 6 |
+
"brier_soft": 0.21710945136112134,
|
| 7 |
+
"nll": 0.45114049174356935,
|
| 8 |
+
"nll_soft": 0.4673940458514404,
|
| 9 |
+
"ece": 0.040077075217092,
|
| 10 |
+
"mean_confidence": 0.8356972425873123,
|
| 11 |
+
"confident_error_rate": 0.020678396517322693,
|
| 12 |
+
"score_mae": 0.49627245924055996,
|
| 13 |
+
"evidence_auc": 0.9999878244320566
|
| 14 |
+
},
|
| 15 |
+
"by_type": {
|
| 16 |
+
"bool": {
|
| 17 |
+
"count": 1620,
|
| 18 |
+
"accuracy": 0.8753086419753087,
|
| 19 |
+
"brier": 0.1834097224750155,
|
| 20 |
+
"brier_soft": 0.17999034550789694,
|
| 21 |
+
"nll": 0.287108752618916,
|
| 22 |
+
"nll_soft": 0.2929012295966247,
|
| 23 |
+
"ece": 0.044949108830161205,
|
| 24 |
+
"mean_confidence": 0.9199214072135956,
|
| 25 |
+
"confident_error_rate": 0.027777777777777776,
|
| 26 |
+
"evidence_auc": 0.9999940298151033
|
| 27 |
+
},
|
| 28 |
+
"choice": {
|
| 29 |
+
"count": 2993,
|
| 30 |
+
"accuracy": 0.8529903107250251,
|
| 31 |
+
"brier": 0.21721720365580133,
|
| 32 |
+
"brier_soft": 0.19167011910193568,
|
| 33 |
+
"nll": 0.4385055987416552,
|
| 34 |
+
"nll_soft": 0.45237298320260255,
|
| 35 |
+
"ece": 0.03450840124038069,
|
| 36 |
+
"mean_confidence": 0.871664572050265,
|
| 37 |
+
"confident_error_rate": 0.023053792181757436,
|
| 38 |
+
"evidence_auc": 0.9999655528763348
|
| 39 |
+
},
|
| 40 |
+
"score": {
|
| 41 |
+
"count": 900,
|
| 42 |
+
"accuracy": 0.6977777777777778,
|
| 43 |
+
"brier": 0.4339333161615588,
|
| 44 |
+
"brier_soft": 0.3685237546210836,
|
| 45 |
+
"nll": 0.7884156607843117,
|
| 46 |
+
"nll_soft": 0.8314344934522995,
|
| 47 |
+
"ece": 0.13329538383179557,
|
| 48 |
+
"mean_confidence": 0.5644823939459822,
|
| 49 |
+
"confident_error_rate": 0.0,
|
| 50 |
+
"score_mae": 0.49627245924055996
|
| 51 |
+
}
|
| 52 |
+
},
|
| 53 |
+
"by_source": {
|
| 54 |
+
"ag_news": {
|
| 55 |
+
"count": 100,
|
| 56 |
+
"accuracy": 0.91,
|
| 57 |
+
"brier": 0.14421606169681028,
|
| 58 |
+
"brier_soft": 0.14421606169681028,
|
| 59 |
+
"nll": 0.27468413989897816,
|
| 60 |
+
"nll_soft": 0.27468413989897816,
|
| 61 |
+
"ece": 0.0614854983919261,
|
| 62 |
+
"mean_confidence": 0.9180683161223662,
|
| 63 |
+
"confident_error_rate": 0.03
|
| 64 |
+
},
|
| 65 |
+
"amazon_polarity": {
|
| 66 |
+
"count": 100,
|
| 67 |
+
"accuracy": 0.95,
|
| 68 |
+
"brier": 0.05888264685815231,
|
| 69 |
+
"brier_soft": 0.05888264685815231,
|
| 70 |
+
"nll": 0.09794057175036192,
|
| 71 |
+
"nll_soft": 0.09794057175036192,
|
| 72 |
+
"ece": 0.01887097873912972,
|
| 73 |
+
"mean_confidence": 0.9638854049157964,
|
| 74 |
+
"confident_error_rate": 0.01
|
| 75 |
+
},
|
| 76 |
+
"amazon_stars": {
|
| 77 |
+
"count": 100,
|
| 78 |
+
"accuracy": 0.69,
|
| 79 |
+
"brier": 0.4871344044281428,
|
| 80 |
+
"brier_soft": 0.4871344044281428,
|
| 81 |
+
"nll": 0.8516013229732908,
|
| 82 |
+
"nll_soft": 0.8516013229732908,
|
| 83 |
+
"ece": 0.17589107107084062,
|
| 84 |
+
"mean_confidence": 0.5336082819156109,
|
| 85 |
+
"confident_error_rate": 0.0,
|
| 86 |
+
"score_mae": 0.4720212013208948
|
| 87 |
+
},
|
| 88 |
+
"anli": {
|
| 89 |
+
"count": 100,
|
| 90 |
+
"accuracy": 0.74,
|
| 91 |
+
"brier": 0.38152495298281364,
|
| 92 |
+
"brier_soft": 0.38152495298281364,
|
| 93 |
+
"nll": 0.661801793051201,
|
| 94 |
+
"nll_soft": 0.661801793051201,
|
| 95 |
+
"ece": 0.10857199956125599,
|
| 96 |
+
"mean_confidence": 0.8285262554356989,
|
| 97 |
+
"confident_error_rate": 0.03
|
| 98 |
+
},
|
| 99 |
+
"arc_challenge": {
|
| 100 |
+
"count": 100,
|
| 101 |
+
"accuracy": 0.98,
|
| 102 |
+
"brier": 0.03293778487998213,
|
| 103 |
+
"brier_soft": 0.03293778487998213,
|
| 104 |
+
"nll": 0.06098433202804499,
|
| 105 |
+
"nll_soft": 0.06098433202804499,
|
| 106 |
+
"ece": 0.010299708559671938,
|
| 107 |
+
"mean_confidence": 0.9801719901814279,
|
| 108 |
+
"confident_error_rate": 0.01
|
| 109 |
+
},
|
| 110 |
+
"arc_easy": {
|
| 111 |
+
"count": 100,
|
| 112 |
+
"accuracy": 0.98,
|
| 113 |
+
"brier": 0.0284256234087672,
|
| 114 |
+
"brier_soft": 0.0284256234087672,
|
| 115 |
+
"nll": 0.0519709495739374,
|
| 116 |
+
"nll_soft": 0.0519709495739374,
|
| 117 |
+
"ece": 0.023087990617371268,
|
| 118 |
+
"mean_confidence": 0.9835592129939819,
|
| 119 |
+
"confident_error_rate": 0.0
|
| 120 |
+
},
|
| 121 |
+
"banking77": {
|
| 122 |
+
"count": 100,
|
| 123 |
+
"accuracy": 0.83,
|
| 124 |
+
"brier": 0.27660070225206373,
|
| 125 |
+
"brier_soft": 0.27660070225206373,
|
| 126 |
+
"nll": 0.712525693075277,
|
| 127 |
+
"nll_soft": 0.712525693075277,
|
| 128 |
+
"ece": 0.10742287422893768,
|
| 129 |
+
"mean_confidence": 0.8310200307491299,
|
| 130 |
+
"confident_error_rate": 0.03
|
| 131 |
+
},
|
| 132 |
+
"boolq": {
|
| 133 |
+
"count": 125,
|
| 134 |
+
"accuracy": 0.72,
|
| 135 |
+
"brier": 0.39824933464474527,
|
| 136 |
+
"brier_soft": 0.39824933464474527,
|
| 137 |
+
"nll": 0.5661039342669112,
|
| 138 |
+
"nll_soft": 0.5661039342669112,
|
| 139 |
+
"ece": 0.19542354013210722,
|
| 140 |
+
"mean_confidence": 0.9012351747266559,
|
| 141 |
+
"confident_error_rate": 0.024,
|
| 142 |
+
"evidence_auc": 1.0
|
| 143 |
+
},
|
| 144 |
+
"cb": {
|
| 145 |
+
"count": 28,
|
| 146 |
+
"accuracy": 0.9642857142857143,
|
| 147 |
+
"brier": 0.0779562215366468,
|
| 148 |
+
"brier_soft": 0.0779562215366468,
|
| 149 |
+
"nll": 0.1411652211306288,
|
| 150 |
+
"nll_soft": 0.1411652211306288,
|
| 151 |
+
"ece": 0.0595834548195279,
|
| 152 |
+
"mean_confidence": 0.9340672209362116,
|
| 153 |
+
"confident_error_rate": 0.0
|
| 154 |
+
},
|
| 155 |
+
"civil_comments_severity": {
|
| 156 |
+
"count": 100,
|
| 157 |
+
"accuracy": 0.85,
|
| 158 |
+
"brier": 0.23714380547223418,
|
| 159 |
+
"brier_soft": 0.15459555443470976,
|
| 160 |
+
"nll": 0.46339746603398696,
|
| 161 |
+
"nll_soft": 0.5929402263240431,
|
| 162 |
+
"ece": 0.14721173331831938,
|
| 163 |
+
"mean_confidence": 0.7201727500927853,
|
| 164 |
+
"confident_error_rate": 0.0,
|
| 165 |
+
"score_mae": 0.34746314098192826
|
| 166 |
+
},
|
| 167 |
+
"civil_comments_toxic": {
|
| 168 |
+
"count": 100,
|
| 169 |
+
"accuracy": 0.96,
|
| 170 |
+
"brier": 0.09337668620304003,
|
| 171 |
+
"brier_soft": 0.037982779335719564,
|
| 172 |
+
"nll": 0.1861470519288842,
|
| 173 |
+
"nll_soft": 0.27998517896776515,
|
| 174 |
+
"ece": 0.09812762637818251,
|
| 175 |
+
"mean_confidence": 0.8618723736218176,
|
| 176 |
+
"confident_error_rate": 0.0
|
| 177 |
+
},
|
| 178 |
+
"clinc150": {
|
| 179 |
+
"count": 100,
|
| 180 |
+
"accuracy": 0.88,
|
| 181 |
+
"brier": 0.18462976225127634,
|
| 182 |
+
"brier_soft": 0.18462976225127634,
|
| 183 |
+
"nll": 0.5128245533780458,
|
| 184 |
+
"nll_soft": 0.5128245533780458,
|
| 185 |
+
"ece": 0.07346104900904875,
|
| 186 |
+
"mean_confidence": 0.9269573388899676,
|
| 187 |
+
"confident_error_rate": 0.03
|
| 188 |
+
},
|
| 189 |
+
"cola": {
|
| 190 |
+
"count": 100,
|
| 191 |
+
"accuracy": 0.82,
|
| 192 |
+
"brier": 0.22831615379946643,
|
| 193 |
+
"brier_soft": 0.22831615379946643,
|
| 194 |
+
"nll": 0.3590869683317728,
|
| 195 |
+
"nll_soft": 0.3590869683317728,
|
| 196 |
+
"ece": 0.07450945927672789,
|
| 197 |
+
"mean_confidence": 0.8555754148249904,
|
| 198 |
+
"confident_error_rate": 0.02
|
| 199 |
+
},
|
| 200 |
+
"commonsense_qa": {
|
| 201 |
+
"count": 100,
|
| 202 |
+
"accuracy": 0.91,
|
| 203 |
+
"brier": 0.15046780462930462,
|
| 204 |
+
"brier_soft": 0.15046780462930462,
|
| 205 |
+
"nll": 0.3438882238808393,
|
| 206 |
+
"nll_soft": 0.3438882238808393,
|
| 207 |
+
"ece": 0.03343498566763066,
|
| 208 |
+
"mean_confidence": 0.9374398583651028,
|
| 209 |
+
"confident_error_rate": 0.04
|
| 210 |
+
},
|
| 211 |
+
"cosmos_qa": {
|
| 212 |
+
"count": 120,
|
| 213 |
+
"accuracy": 0.7166666666666667,
|
| 214 |
+
"brier": 0.44494135470751833,
|
| 215 |
+
"brier_soft": 0.44494135470751833,
|
| 216 |
+
"nll": 0.9113581258768009,
|
| 217 |
+
"nll_soft": 0.9113581258768009,
|
| 218 |
+
"ece": 0.17210482016936163,
|
| 219 |
+
"mean_confidence": 0.868085591538144,
|
| 220 |
+
"confident_error_rate": 0.09166666666666666,
|
| 221 |
+
"evidence_auc": 0.999375
|
| 222 |
+
},
|
| 223 |
+
"dbpedia": {
|
| 224 |
+
"count": 100,
|
| 225 |
+
"accuracy": 1.0,
|
| 226 |
+
"brier": 0.003741281367652323,
|
| 227 |
+
"brier_soft": 0.003741281367652323,
|
| 228 |
+
"nll": 0.00958541989430656,
|
| 229 |
+
"nll_soft": 0.00958541989430656,
|
| 230 |
+
"ece": 0.008060135485155948,
|
| 231 |
+
"mean_confidence": 0.9919398645148441,
|
| 232 |
+
"confident_error_rate": 0.0
|
| 233 |
+
},
|
| 234 |
+
"dynasent": {
|
| 235 |
+
"count": 100,
|
| 236 |
+
"accuracy": 0.73,
|
| 237 |
+
"brier": 0.39455497884064294,
|
| 238 |
+
"brier_soft": 0.27562515443314395,
|
| 239 |
+
"nll": 0.6819105783878199,
|
| 240 |
+
"nll_soft": 0.7407221597727708,
|
| 241 |
+
"ece": 0.17197017219513994,
|
| 242 |
+
"mean_confidence": 0.5922648429725653,
|
| 243 |
+
"confident_error_rate": 0.0,
|
| 244 |
+
"score_mae": 0.4612412387247546
|
| 245 |
+
},
|
| 246 |
+
"enron_spam": {
|
| 247 |
+
"count": 100,
|
| 248 |
+
"accuracy": 0.99,
|
| 249 |
+
"brier": 0.020371589208394592,
|
| 250 |
+
"brier_soft": 0.020371589208394592,
|
| 251 |
+
"nll": 0.04072451183380424,
|
| 252 |
+
"nll_soft": 0.04072451183380424,
|
| 253 |
+
"ece": 0.02126407007058294,
|
| 254 |
+
"mean_confidence": 0.9811057852953994,
|
| 255 |
+
"confident_error_rate": 0.0
|
| 256 |
+
},
|
| 257 |
+
"go_emotions": {
|
| 258 |
+
"count": 100,
|
| 259 |
+
"accuracy": 0.6,
|
| 260 |
+
"brier": 0.6247055007996507,
|
| 261 |
+
"brier_soft": 0.2298631426256989,
|
| 262 |
+
"nll": 1.5618458274135032,
|
| 263 |
+
"nll_soft": 1.9562813612135246,
|
| 264 |
+
"ece": 0.2758994442781854,
|
| 265 |
+
"mean_confidence": 0.33157592066393576,
|
| 266 |
+
"confident_error_rate": 0.0
|
| 267 |
+
},
|
| 268 |
+
"hellaswag": {
|
| 269 |
+
"count": 100,
|
| 270 |
+
"accuracy": 0.97,
|
| 271 |
+
"brier": 0.04435647337572136,
|
| 272 |
+
"brier_soft": 0.04435647337572136,
|
| 273 |
+
"nll": 0.08034530834752719,
|
| 274 |
+
"nll_soft": 0.08034530834752719,
|
| 275 |
+
"ece": 0.03405762729343661,
|
| 276 |
+
"mean_confidence": 0.9562575630567488,
|
| 277 |
+
"confident_error_rate": 0.0
|
| 278 |
+
},
|
| 279 |
+
"imdb": {
|
| 280 |
+
"count": 100,
|
| 281 |
+
"accuracy": 0.98,
|
| 282 |
+
"brier": 0.04711217330218374,
|
| 283 |
+
"brier_soft": 0.04711217330218374,
|
| 284 |
+
"nll": 0.10119131895321937,
|
| 285 |
+
"nll_soft": 0.10119131895321937,
|
| 286 |
+
"ece": 0.032128265447002514,
|
| 287 |
+
"mean_confidence": 0.9619559211052533,
|
| 288 |
+
"confident_error_rate": 0.01
|
| 289 |
+
},
|
| 290 |
+
"ledgar": {
|
| 291 |
+
"count": 100,
|
| 292 |
+
"accuracy": 0.78,
|
| 293 |
+
"brier": 0.29751737785310295,
|
| 294 |
+
"brier_soft": 0.29751737785310295,
|
| 295 |
+
"nll": 0.7476464446710225,
|
| 296 |
+
"nll_soft": 0.7476464446710225,
|
| 297 |
+
"ece": 0.061867458329410076,
|
| 298 |
+
"mean_confidence": 0.8241415790544235,
|
| 299 |
+
"confident_error_rate": 0.03
|
| 300 |
+
},
|
| 301 |
+
"liar": {
|
| 302 |
+
"count": 100,
|
| 303 |
+
"accuracy": 0.39,
|
| 304 |
+
"brier": 0.7232980473841729,
|
| 305 |
+
"brier_soft": 0.7232980473841729,
|
| 306 |
+
"nll": 1.4529365835997112,
|
| 307 |
+
"nll_soft": 1.4529365835997112,
|
| 308 |
+
"ece": 0.09217707472940212,
|
| 309 |
+
"mean_confidence": 0.3114025357085896,
|
| 310 |
+
"confident_error_rate": 0.0,
|
| 311 |
+
"score_mae": 0.9575159525161713
|
| 312 |
+
},
|
| 313 |
+
"measuring_hate_speech": {
|
| 314 |
+
"count": 100,
|
| 315 |
+
"accuracy": 0.84,
|
| 316 |
+
"brier": 0.2793448439064333,
|
| 317 |
+
"brier_soft": 0.1463219404313681,
|
| 318 |
+
"nll": 0.5353710733933518,
|
| 319 |
+
"nll_soft": 0.6702362126220086,
|
| 320 |
+
"ece": 0.16927708181106907,
|
| 321 |
+
"mean_confidence": 0.6707229181889309,
|
| 322 |
+
"confident_error_rate": 0.0,
|
| 323 |
+
"score_mae": 0.5462726519505992
|
| 324 |
+
},
|
| 325 |
+
"medqa": {
|
| 326 |
+
"count": 100,
|
| 327 |
+
"accuracy": 0.88,
|
| 328 |
+
"brier": 0.1503504657387938,
|
| 329 |
+
"brier_soft": 0.1503504657387938,
|
| 330 |
+
"nll": 0.2623507353639012,
|
| 331 |
+
"nll_soft": 0.2623507353639012,
|
| 332 |
+
"ece": 0.08144816899443122,
|
| 333 |
+
"mean_confidence": 0.8978548497595163,
|
| 334 |
+
"confident_error_rate": 0.0
|
| 335 |
+
},
|
| 336 |
+
"mnli": {
|
| 337 |
+
"count": 100,
|
| 338 |
+
"accuracy": 0.97,
|
| 339 |
+
"brier": 0.06228021631952076,
|
| 340 |
+
"brier_soft": 0.06228021631952076,
|
| 341 |
+
"nll": 0.13270113391968807,
|
| 342 |
+
"nll_soft": 0.13270113391968807,
|
| 343 |
+
"ece": 0.08238346171353027,
|
| 344 |
+
"mean_confidence": 0.9110402983650409,
|
| 345 |
+
"confident_error_rate": 0.0
|
| 346 |
+
},
|
| 347 |
+
"mrpc": {
|
| 348 |
+
"count": 100,
|
| 349 |
+
"accuracy": 0.9,
|
| 350 |
+
"brier": 0.16585409895732134,
|
| 351 |
+
"brier_soft": 0.16585409895732134,
|
| 352 |
+
"nll": 0.2645067375959682,
|
| 353 |
+
"nll_soft": 0.2645067375959682,
|
| 354 |
+
"ece": 0.061724041454668824,
|
| 355 |
+
"mean_confidence": 0.885295208394654,
|
| 356 |
+
"confident_error_rate": 0.01
|
| 357 |
+
},
|
| 358 |
+
"multirc": {
|
| 359 |
+
"count": 125,
|
| 360 |
+
"accuracy": 0.768,
|
| 361 |
+
"brier": 0.39100223311737586,
|
| 362 |
+
"brier_soft": 0.39100223311737586,
|
| 363 |
+
"nll": 0.61268910167795,
|
| 364 |
+
"nll_soft": 0.61268910167795,
|
| 365 |
+
"ece": 0.18123682958658455,
|
| 366 |
+
"mean_confidence": 0.9092147247418939,
|
| 367 |
+
"confident_error_rate": 0.128,
|
| 368 |
+
"evidence_auc": 0.9997333333333334
|
| 369 |
+
},
|
| 370 |
+
"openbookqa": {
|
| 371 |
+
"count": 100,
|
| 372 |
+
"accuracy": 0.98,
|
| 373 |
+
"brier": 0.036709257253909985,
|
| 374 |
+
"brier_soft": 0.036709257253909985,
|
| 375 |
+
"nll": 0.07752743679878692,
|
| 376 |
+
"nll_soft": 0.07752743679878692,
|
| 377 |
+
"ece": 0.016796433783453687,
|
| 378 |
+
"mean_confidence": 0.9650003552691371,
|
| 379 |
+
"confident_error_rate": 0.0
|
| 380 |
+
},
|
| 381 |
+
"piqa": {
|
| 382 |
+
"count": 100,
|
| 383 |
+
"accuracy": 0.91,
|
| 384 |
+
"brier": 0.12286978168514663,
|
| 385 |
+
"brier_soft": 0.12286978168514663,
|
| 386 |
+
"nll": 0.2169208854703175,
|
| 387 |
+
"nll_soft": 0.2169208854703175,
|
| 388 |
+
"ece": 0.07068212570085167,
|
| 389 |
+
"mean_confidence": 0.9379244741154052,
|
| 390 |
+
"confident_error_rate": 0.03
|
| 391 |
+
},
|
| 392 |
+
"policy_application": {
|
| 393 |
+
"count": 150,
|
| 394 |
+
"accuracy": 0.9466666666666667,
|
| 395 |
+
"brier": 0.07417797153585332,
|
| 396 |
+
"brier_soft": 0.07417797153585332,
|
| 397 |
+
"nll": 0.11845208966989801,
|
| 398 |
+
"nll_soft": 0.11845208966989801,
|
| 399 |
+
"ece": 0.044110458372438946,
|
| 400 |
+
"mean_confidence": 0.9800457791058929,
|
| 401 |
+
"confident_error_rate": 0.03333333333333333,
|
| 402 |
+
"evidence_auc": 1.0
|
| 403 |
+
},
|
| 404 |
+
"qqp": {
|
| 405 |
+
"count": 100,
|
| 406 |
+
"accuracy": 0.92,
|
| 407 |
+
"brier": 0.13311880331318848,
|
| 408 |
+
"brier_soft": 0.13311880331318848,
|
| 409 |
+
"nll": 0.2320502253844424,
|
| 410 |
+
"nll_soft": 0.2320502253844424,
|
| 411 |
+
"ece": 0.07355407226380711,
|
| 412 |
+
"mean_confidence": 0.8951777070215553,
|
| 413 |
+
"confident_error_rate": 0.02
|
| 414 |
+
},
|
| 415 |
+
"race": {
|
| 416 |
+
"count": 125,
|
| 417 |
+
"accuracy": 0.792,
|
| 418 |
+
"brier": 0.2674134484204704,
|
| 419 |
+
"brier_soft": 0.2674134484204704,
|
| 420 |
+
"nll": 0.513583670296167,
|
| 421 |
+
"nll_soft": 0.513583670296167,
|
| 422 |
+
"ece": 0.07423129886353177,
|
| 423 |
+
"mean_confidence": 0.8306386892918173,
|
| 424 |
+
"confident_error_rate": 0.016,
|
| 425 |
+
"evidence_auc": 1.0
|
| 426 |
+
},
|
| 427 |
+
"rotten_tomatoes": {
|
| 428 |
+
"count": 100,
|
| 429 |
+
"accuracy": 0.98,
|
| 430 |
+
"brier": 0.03743266007384592,
|
| 431 |
+
"brier_soft": 0.03743266007384592,
|
| 432 |
+
"nll": 0.08081407315197316,
|
| 433 |
+
"nll_soft": 0.08081407315197316,
|
| 434 |
+
"ece": 0.0357435013308515,
|
| 435 |
+
"mean_confidence": 0.9567344445787098,
|
| 436 |
+
"confident_error_rate": 0.01
|
| 437 |
+
},
|
| 438 |
+
"rte": {
|
| 439 |
+
"count": 100,
|
| 440 |
+
"accuracy": 0.93,
|
| 441 |
+
"brier": 0.11037080326966946,
|
| 442 |
+
"brier_soft": 0.11037080326966946,
|
| 443 |
+
"nll": 0.18424526282282977,
|
| 444 |
+
"nll_soft": 0.18424526282282977,
|
| 445 |
+
"ece": 0.03139877402899702,
|
| 446 |
+
"mean_confidence": 0.9613987740289969,
|
| 447 |
+
"confident_error_rate": 0.02
|
| 448 |
+
},
|
| 449 |
+
"scotus": {
|
| 450 |
+
"count": 100,
|
| 451 |
+
"accuracy": 0.61,
|
| 452 |
+
"brier": 0.5896053540662425,
|
| 453 |
+
"brier_soft": 0.5896053540662425,
|
| 454 |
+
"nll": 1.4407866367191966,
|
| 455 |
+
"nll_soft": 1.4407866367191966,
|
| 456 |
+
"ece": 0.17632924711827827,
|
| 457 |
+
"mean_confidence": 0.7863292471182782,
|
| 458 |
+
"confident_error_rate": 0.04
|
| 459 |
+
},
|
| 460 |
+
"shp": {
|
| 461 |
+
"count": 100,
|
| 462 |
+
"accuracy": 0.65,
|
| 463 |
+
"brier": 0.47029834003111476,
|
| 464 |
+
"brier_soft": 0.10051645750786714,
|
| 465 |
+
"nll": 0.6632380214745927,
|
| 466 |
+
"nll_soft": 0.6838533045907272,
|
| 467 |
+
"ece": 0.11777214652077381,
|
| 468 |
+
"mean_confidence": 0.5322278534792262,
|
| 469 |
+
"confident_error_rate": 0.0
|
| 470 |
+
},
|
| 471 |
+
"sms_spam": {
|
| 472 |
+
"count": 100,
|
| 473 |
+
"accuracy": 0.99,
|
| 474 |
+
"brier": 0.01162780117821516,
|
| 475 |
+
"brier_soft": 0.01162780117821516,
|
| 476 |
+
"nll": 0.02331012246933196,
|
| 477 |
+
"nll_soft": 0.02331012246933196,
|
| 478 |
+
"ece": 0.016788442416964124,
|
| 479 |
+
"mean_confidence": 0.988267077668862,
|
| 480 |
+
"confident_error_rate": 0.0
|
| 481 |
+
},
|
| 482 |
+
"snli": {
|
| 483 |
+
"count": 100,
|
| 484 |
+
"accuracy": 0.92,
|
| 485 |
+
"brier": 0.12711939635362396,
|
| 486 |
+
"brier_soft": 0.12711939635362396,
|
| 487 |
+
"nll": 0.22879034755833566,
|
| 488 |
+
"nll_soft": 0.22879034755833566,
|
| 489 |
+
"ece": 0.0444196020597182,
|
| 490 |
+
"mean_confidence": 0.9308260670907695,
|
| 491 |
+
"confident_error_rate": 0.03
|
| 492 |
+
},
|
| 493 |
+
"social_iqa": {
|
| 494 |
+
"count": 100,
|
| 495 |
+
"accuracy": 0.84,
|
| 496 |
+
"brier": 0.2560615224983722,
|
| 497 |
+
"brier_soft": 0.2560615224983722,
|
| 498 |
+
"nll": 0.4826577509065636,
|
| 499 |
+
"nll_soft": 0.4826577509065636,
|
| 500 |
+
"ece": 0.07618387345343328,
|
| 501 |
+
"mean_confidence": 0.8898566355070517,
|
| 502 |
+
"confident_error_rate": 0.04
|
| 503 |
+
},
|
| 504 |
+
"sst2": {
|
| 505 |
+
"count": 100,
|
| 506 |
+
"accuracy": 0.98,
|
| 507 |
+
"brier": 0.04105489560088457,
|
| 508 |
+
"brier_soft": 0.04105489560088457,
|
| 509 |
+
"nll": 0.10883305537700429,
|
| 510 |
+
"nll_soft": 0.10883305537700429,
|
| 511 |
+
"ece": 0.011059711327280758,
|
| 512 |
+
"mean_confidence": 0.9782684225353241,
|
| 513 |
+
"confident_error_rate": 0.02
|
| 514 |
+
},
|
| 515 |
+
"sst5": {
|
| 516 |
+
"count": 100,
|
| 517 |
+
"accuracy": 0.66,
|
| 518 |
+
"brier": 0.497079796594402,
|
| 519 |
+
"brier_soft": 0.497079796594402,
|
| 520 |
+
"nll": 0.8742453682539675,
|
| 521 |
+
"nll_soft": 0.8742453682539675,
|
| 522 |
+
"ece": 0.16584622118782563,
|
| 523 |
+
"mean_confidence": 0.5079338519681397,
|
| 524 |
+
"confident_error_rate": 0.0,
|
| 525 |
+
"score_mae": 0.449198760613678
|
| 526 |
+
},
|
| 527 |
+
"stance_abortion": {
|
| 528 |
+
"count": 60,
|
| 529 |
+
"accuracy": 0.65,
|
| 530 |
+
"brier": 0.5170044623404539,
|
| 531 |
+
"brier_soft": 0.5170044623404539,
|
| 532 |
+
"nll": 0.8217929893106105,
|
| 533 |
+
"nll_soft": 0.8217929893106105,
|
| 534 |
+
"ece": 0.19912084090351673,
|
| 535 |
+
"mean_confidence": 0.8303765398473246,
|
| 536 |
+
"confident_error_rate": 0.06666666666666667
|
| 537 |
+
},
|
| 538 |
+
"stance_atheism": {
|
| 539 |
+
"count": 60,
|
| 540 |
+
"accuracy": 0.7833333333333333,
|
| 541 |
+
"brier": 0.3001419959341259,
|
| 542 |
+
"brier_soft": 0.3001419959341259,
|
| 543 |
+
"nll": 0.47098576663159153,
|
| 544 |
+
"nll_soft": 0.47098576663159153,
|
| 545 |
+
"ece": 0.14294118305587333,
|
| 546 |
+
"mean_confidence": 0.8563605992498861,
|
| 547 |
+
"confident_error_rate": 0.016666666666666666
|
| 548 |
+
},
|
| 549 |
+
"stance_feminist": {
|
| 550 |
+
"count": 60,
|
| 551 |
+
"accuracy": 0.7833333333333333,
|
| 552 |
+
"brier": 0.2920330154544224,
|
| 553 |
+
"brier_soft": 0.2920330154544224,
|
| 554 |
+
"nll": 0.49062532038555723,
|
| 555 |
+
"nll_soft": 0.49062532038555723,
|
| 556 |
+
"ece": 0.06563462797049141,
|
| 557 |
+
"mean_confidence": 0.8395306016627094,
|
| 558 |
+
"confident_error_rate": 0.016666666666666666
|
| 559 |
+
},
|
| 560 |
+
"stance_hillary": {
|
| 561 |
+
"count": 60,
|
| 562 |
+
"accuracy": 0.7166666666666667,
|
| 563 |
+
"brier": 0.36920640328234444,
|
| 564 |
+
"brier_soft": 0.36920640328234444,
|
| 565 |
+
"nll": 0.5975554352281234,
|
| 566 |
+
"nll_soft": 0.5975554352281234,
|
| 567 |
+
"ece": 0.08165786929705794,
|
| 568 |
+
"mean_confidence": 0.7783727969746086,
|
| 569 |
+
"confident_error_rate": 0.03333333333333333
|
| 570 |
+
},
|
| 571 |
+
"stsb": {
|
| 572 |
+
"count": 100,
|
| 573 |
+
"accuracy": 0.65,
|
| 574 |
+
"brier": 0.4598476608257509,
|
| 575 |
+
"brier_soft": 0.20566258588156333,
|
| 576 |
+
"nll": 0.8174855825848395,
|
| 577 |
+
"nll_soft": 0.8814355956930671,
|
| 578 |
+
"ece": 0.12916616422774185,
|
| 579 |
+
"mean_confidence": 0.5881235918154974,
|
| 580 |
+
"confident_error_rate": 0.0,
|
| 581 |
+
"score_mae": 0.4591854752360724
|
| 582 |
+
},
|
| 583 |
+
"subjectivity": {
|
| 584 |
+
"count": 100,
|
| 585 |
+
"accuracy": 0.99,
|
| 586 |
+
"brier": 0.025245447097925453,
|
| 587 |
+
"brier_soft": 0.025245447097925453,
|
| 588 |
+
"nll": 0.06443650373540663,
|
| 589 |
+
"nll_soft": 0.06443650373540663,
|
| 590 |
+
"ece": 0.05522051201824588,
|
| 591 |
+
"mean_confidence": 0.9472574338913156,
|
| 592 |
+
"confident_error_rate": 0.0
|
| 593 |
+
},
|
| 594 |
+
"swag": {
|
| 595 |
+
"count": 100,
|
| 596 |
+
"accuracy": 0.82,
|
| 597 |
+
"brier": 0.24280604965542157,
|
| 598 |
+
"brier_soft": 0.24280604965542157,
|
| 599 |
+
"nll": 0.43879576779754104,
|
| 600 |
+
"nll_soft": 0.43879576779754104,
|
| 601 |
+
"ece": 0.07809132699403978,
|
| 602 |
+
"mean_confidence": 0.8768457190873439,
|
| 603 |
+
"confident_error_rate": 0.02
|
| 604 |
+
},
|
| 605 |
+
"trec": {
|
| 606 |
+
"count": 100,
|
| 607 |
+
"accuracy": 0.97,
|
| 608 |
+
"brier": 0.044109563870200826,
|
| 609 |
+
"brier_soft": 0.044109563870200826,
|
| 610 |
+
"nll": 0.1119764191390441,
|
| 611 |
+
"nll_soft": 0.1119764191390441,
|
| 612 |
+
"ece": 0.03157924188503147,
|
| 613 |
+
"mean_confidence": 0.9651043539300282,
|
| 614 |
+
"confident_error_rate": 0.01
|
| 615 |
+
},
|
| 616 |
+
"tweet_hate": {
|
| 617 |
+
"count": 100,
|
| 618 |
+
"accuracy": 0.5,
|
| 619 |
+
"brier": 0.6361912887410113,
|
| 620 |
+
"brier_soft": 0.6361912887410113,
|
| 621 |
+
"nll": 0.87589744678152,
|
| 622 |
+
"nll_soft": 0.87589744678152,
|
| 623 |
+
"ece": 0.30966203984896934,
|
| 624 |
+
"mean_confidence": 0.7996620398489693,
|
| 625 |
+
"confident_error_rate": 0.05
|
| 626 |
+
},
|
| 627 |
+
"tweet_sentiment": {
|
| 628 |
+
"count": 100,
|
| 629 |
+
"accuracy": 0.74,
|
| 630 |
+
"brier": 0.4046324049729258,
|
| 631 |
+
"brier_soft": 0.4046324049729258,
|
| 632 |
+
"nll": 0.6904489665258657,
|
| 633 |
+
"nll_soft": 0.6904489665258657,
|
| 634 |
+
"ece": 0.17709361051488434,
|
| 635 |
+
"mean_confidence": 0.5807963141231423,
|
| 636 |
+
"confident_error_rate": 0.0,
|
| 637 |
+
"score_mae": 0.3678134887808973
|
| 638 |
+
},
|
| 639 |
+
"vitaminc": {
|
| 640 |
+
"count": 100,
|
| 641 |
+
"accuracy": 0.84,
|
| 642 |
+
"brier": 0.22414852755241488,
|
| 643 |
+
"brier_soft": 0.22414852755241488,
|
| 644 |
+
"nll": 0.4106686914678594,
|
| 645 |
+
"nll_soft": 0.4106686914678594,
|
| 646 |
+
"ece": 0.05939190838622676,
|
| 647 |
+
"mean_confidence": 0.8887477138644124,
|
| 648 |
+
"confident_error_rate": 0.03
|
| 649 |
+
},
|
| 650 |
+
"wic": {
|
| 651 |
+
"count": 100,
|
| 652 |
+
"accuracy": 0.78,
|
| 653 |
+
"brier": 0.32325615773124416,
|
| 654 |
+
"brier_soft": 0.32325615773124416,
|
| 655 |
+
"nll": 0.49691056539654466,
|
| 656 |
+
"nll_soft": 0.49691056539654466,
|
| 657 |
+
"ece": 0.11144833190894102,
|
| 658 |
+
"mean_confidence": 0.8764042284964352,
|
| 659 |
+
"confident_error_rate": 0.06
|
| 660 |
+
},
|
| 661 |
+
"winogrande": {
|
| 662 |
+
"count": 100,
|
| 663 |
+
"accuracy": 0.91,
|
| 664 |
+
"brier": 0.1699682822388243,
|
| 665 |
+
"brier_soft": 0.1699682822388243,
|
| 666 |
+
"nll": 0.32014269821420394,
|
| 667 |
+
"nll_soft": 0.32014269821420394,
|
| 668 |
+
"ece": 0.09579456110528843,
|
| 669 |
+
"mean_confidence": 0.9299164455248426,
|
| 670 |
+
"confident_error_rate": 0.06
|
| 671 |
+
},
|
| 672 |
+
"yelp": {
|
| 673 |
+
"count": 100,
|
| 674 |
+
"accuracy": 0.73,
|
| 675 |
+
"brier": 0.4223639030293232,
|
| 676 |
+
"brier_soft": 0.4223639030293232,
|
| 677 |
+
"nll": 0.7283440053059708,
|
| 678 |
+
"nll_soft": 0.7283440053059708,
|
| 679 |
+
"ece": 0.1725769529348749,
|
| 680 |
+
"mean_confidence": 0.5753164587285784,
|
| 681 |
+
"confident_error_rate": 0.0,
|
| 682 |
+
"score_mae": 0.40574022304004387
|
| 683 |
+
}
|
| 684 |
+
},
|
| 685 |
+
"reliability": [
|
| 686 |
+
{
|
| 687 |
+
"bin": "0.0-0.1",
|
| 688 |
+
"count": 0,
|
| 689 |
+
"confidence": NaN,
|
| 690 |
+
"accuracy": NaN
|
| 691 |
+
},
|
| 692 |
+
{
|
| 693 |
+
"bin": "0.1-0.2",
|
| 694 |
+
"count": 14,
|
| 695 |
+
"confidence": 0.1695548747932898,
|
| 696 |
+
"accuracy": 0.14285714285714285
|
| 697 |
+
},
|
| 698 |
+
{
|
| 699 |
+
"bin": "0.2-0.3",
|
| 700 |
+
"count": 94,
|
| 701 |
+
"confidence": 0.250037071579069,
|
| 702 |
+
"accuracy": 0.35106382978723405
|
| 703 |
+
},
|
| 704 |
+
{
|
| 705 |
+
"bin": "0.3-0.4",
|
| 706 |
+
"count": 120,
|
| 707 |
+
"confidence": 0.3580054189178388,
|
| 708 |
+
"accuracy": 0.475
|
| 709 |
+
},
|
| 710 |
+
{
|
| 711 |
+
"bin": "0.4-0.5",
|
| 712 |
+
"count": 317,
|
| 713 |
+
"confidence": 0.45936274075655936,
|
| 714 |
+
"accuracy": 0.5394321766561514
|
| 715 |
+
},
|
| 716 |
+
{
|
| 717 |
+
"bin": "0.5-0.6",
|
| 718 |
+
"count": 462,
|
| 719 |
+
"confidence": 0.5473715655718976,
|
| 720 |
+
"accuracy": 0.6363636363636364
|
| 721 |
+
},
|
| 722 |
+
{
|
| 723 |
+
"bin": "0.6-0.7",
|
| 724 |
+
"count": 311,
|
| 725 |
+
"confidence": 0.6482043563078829,
|
| 726 |
+
"accuracy": 0.7009646302250804
|
| 727 |
+
},
|
| 728 |
+
{
|
| 729 |
+
"bin": "0.7-0.8",
|
| 730 |
+
"count": 373,
|
| 731 |
+
"confidence": 0.7494447313036959,
|
| 732 |
+
"accuracy": 0.7372654155495979
|
| 733 |
+
},
|
| 734 |
+
{
|
| 735 |
+
"bin": "0.8-0.9",
|
| 736 |
+
"count": 603,
|
| 737 |
+
"confidence": 0.8560699667307652,
|
| 738 |
+
"accuracy": 0.736318407960199
|
| 739 |
+
},
|
| 740 |
+
{
|
| 741 |
+
"bin": "0.9-1.0",
|
| 742 |
+
"count": 3219,
|
| 743 |
+
"confidence": 0.976238805062891,
|
| 744 |
+
"accuracy": 0.9645852749301025
|
| 745 |
+
}
|
| 746 |
+
],
|
| 747 |
+
"seconds": 387.77550768852234,
|
| 748 |
+
"model": "runs/assay-27b",
|
| 749 |
+
"data": "data/v2/dev.jsonl"
|
| 750 |
+
}
|
eval-dev.json
ADDED
|
@@ -0,0 +1,750 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
{
|
| 2 |
+
"overall": {
|
| 3 |
+
"count": 5513,
|
| 4 |
+
"accuracy": 0.8342100489751496,
|
| 5 |
+
"brier": 0.24499939603673204,
|
| 6 |
+
"brier_soft": 0.22122350305548508,
|
| 7 |
+
"nll": 0.4655553015844398,
|
| 8 |
+
"nll_soft": 0.48562642001788464,
|
| 9 |
+
"ece": 0.04843755672557183,
|
| 10 |
+
"mean_confidence": 0.8631851509248467,
|
| 11 |
+
"confident_error_rate": 0.0359151097406131,
|
| 12 |
+
"score_mae": 0.4705968118889738,
|
| 13 |
+
"evidence_auc": 0.9999878244320566
|
| 14 |
+
},
|
| 15 |
+
"by_type": {
|
| 16 |
+
"bool": {
|
| 17 |
+
"count": 1620,
|
| 18 |
+
"accuracy": 0.8753086419753087,
|
| 19 |
+
"brier": 0.19217866542576822,
|
| 20 |
+
"brier_soft": 0.18932657502523315,
|
| 21 |
+
"nll": 0.3103588304174498,
|
| 22 |
+
"nll_soft": 0.3175118193167458,
|
| 23 |
+
"ece": 0.06666441711019516,
|
| 24 |
+
"mean_confidence": 0.9396361920258015,
|
| 25 |
+
"confident_error_rate": 0.043209876543209874,
|
| 26 |
+
"evidence_auc": 0.9999940298151033
|
| 27 |
+
},
|
| 28 |
+
"choice": {
|
| 29 |
+
"count": 2993,
|
| 30 |
+
"accuracy": 0.8529903107250251,
|
| 31 |
+
"brier": 0.2219172131885553,
|
| 32 |
+
"brier_soft": 0.19782538508715383,
|
| 33 |
+
"nll": 0.4634021404898978,
|
| 34 |
+
"nll_soft": 0.48052663665615447,
|
| 35 |
+
"ece": 0.05453548012022721,
|
| 36 |
+
"mean_confidence": 0.8982264049258726,
|
| 37 |
+
"confident_error_rate": 0.04276645506181089,
|
| 38 |
+
"evidence_auc": 0.9999655528763348
|
| 39 |
+
},
|
| 40 |
+
"score": {
|
| 41 |
+
"count": 900,
|
| 42 |
+
"accuracy": 0.6977777777777778,
|
| 43 |
+
"brier": 0.41683779254157016,
|
| 44 |
+
"brier_soft": 0.356449714709067,
|
| 45 |
+
"nll": 0.7520694065249816,
|
| 46 |
+
"nll_soft": 0.8051923141706663,
|
| 47 |
+
"ece": 0.09215386988560845,
|
| 48 |
+
"mean_confidence": 0.6090416400263835,
|
| 49 |
+
"confident_error_rate": 0.0,
|
| 50 |
+
"score_mae": 0.4705968118889738
|
| 51 |
+
}
|
| 52 |
+
},
|
| 53 |
+
"by_source": {
|
| 54 |
+
"ag_news": {
|
| 55 |
+
"count": 100,
|
| 56 |
+
"accuracy": 0.91,
|
| 57 |
+
"brier": 0.1465319479149763,
|
| 58 |
+
"brier_soft": 0.1465319479149763,
|
| 59 |
+
"nll": 0.2904767555416907,
|
| 60 |
+
"nll_soft": 0.2904767555416907,
|
| 61 |
+
"ece": 0.06696793567024037,
|
| 62 |
+
"mean_confidence": 0.9411689492273619,
|
| 63 |
+
"confident_error_rate": 0.05
|
| 64 |
+
},
|
| 65 |
+
"amazon_polarity": {
|
| 66 |
+
"count": 100,
|
| 67 |
+
"accuracy": 0.95,
|
| 68 |
+
"brier": 0.06128869305634376,
|
| 69 |
+
"brier_soft": 0.06128869305634376,
|
| 70 |
+
"nll": 0.10096164621475562,
|
| 71 |
+
"nll_soft": 0.10096164621475562,
|
| 72 |
+
"ece": 0.023046591891905697,
|
| 73 |
+
"mean_confidence": 0.9730465918919056,
|
| 74 |
+
"confident_error_rate": 0.01
|
| 75 |
+
},
|
| 76 |
+
"amazon_stars": {
|
| 77 |
+
"count": 100,
|
| 78 |
+
"accuracy": 0.69,
|
| 79 |
+
"brier": 0.4731412386648261,
|
| 80 |
+
"brier_soft": 0.4731412386648261,
|
| 81 |
+
"nll": 0.8151410149736901,
|
| 82 |
+
"nll_soft": 0.8151410149736901,
|
| 83 |
+
"ece": 0.1366735053126959,
|
| 84 |
+
"mean_confidence": 0.579111282045835,
|
| 85 |
+
"confident_error_rate": 0.0,
|
| 86 |
+
"score_mae": 0.4509802678028108
|
| 87 |
+
},
|
| 88 |
+
"anli": {
|
| 89 |
+
"count": 100,
|
| 90 |
+
"accuracy": 0.74,
|
| 91 |
+
"brier": 0.3976344462707522,
|
| 92 |
+
"brier_soft": 0.3976344462707522,
|
| 93 |
+
"nll": 0.7224163582147134,
|
| 94 |
+
"nll_soft": 0.7224163582147134,
|
| 95 |
+
"ece": 0.14788199940722557,
|
| 96 |
+
"mean_confidence": 0.8672915745774106,
|
| 97 |
+
"confident_error_rate": 0.08
|
| 98 |
+
},
|
| 99 |
+
"arc_challenge": {
|
| 100 |
+
"count": 100,
|
| 101 |
+
"accuracy": 0.98,
|
| 102 |
+
"brier": 0.03366315261301923,
|
| 103 |
+
"brier_soft": 0.03366315261301923,
|
| 104 |
+
"nll": 0.060837047229714225,
|
| 105 |
+
"nll_soft": 0.060837047229714225,
|
| 106 |
+
"ece": 0.011432103687330048,
|
| 107 |
+
"mean_confidence": 0.9886713474416573,
|
| 108 |
+
"confident_error_rate": 0.01
|
| 109 |
+
},
|
| 110 |
+
"arc_easy": {
|
| 111 |
+
"count": 100,
|
| 112 |
+
"accuracy": 0.98,
|
| 113 |
+
"brier": 0.029996236913866735,
|
| 114 |
+
"brier_soft": 0.029996236913866735,
|
| 115 |
+
"nll": 0.05253505337779216,
|
| 116 |
+
"nll_soft": 0.05253505337779216,
|
| 117 |
+
"ece": 0.018028940214597153,
|
| 118 |
+
"mean_confidence": 0.990060141873229,
|
| 119 |
+
"confident_error_rate": 0.01
|
| 120 |
+
},
|
| 121 |
+
"banking77": {
|
| 122 |
+
"count": 100,
|
| 123 |
+
"accuracy": 0.83,
|
| 124 |
+
"brier": 0.28099173269454264,
|
| 125 |
+
"brier_soft": 0.28099173269454264,
|
| 126 |
+
"nll": 0.7554653567994057,
|
| 127 |
+
"nll_soft": 0.7554653567994057,
|
| 128 |
+
"ece": 0.11505295474931794,
|
| 129 |
+
"mean_confidence": 0.8828400651254894,
|
| 130 |
+
"confident_error_rate": 0.07
|
| 131 |
+
},
|
| 132 |
+
"boolq": {
|
| 133 |
+
"count": 125,
|
| 134 |
+
"accuracy": 0.72,
|
| 135 |
+
"brier": 0.43268232797354933,
|
| 136 |
+
"brier_soft": 0.43268232797354933,
|
| 137 |
+
"nll": 0.6422496879755143,
|
| 138 |
+
"nll_soft": 0.6422496879755143,
|
| 139 |
+
"ece": 0.22190986014776565,
|
| 140 |
+
"mean_confidence": 0.9281403357012442,
|
| 141 |
+
"confident_error_rate": 0.072,
|
| 142 |
+
"evidence_auc": 1.0
|
| 143 |
+
},
|
| 144 |
+
"cb": {
|
| 145 |
+
"count": 28,
|
| 146 |
+
"accuracy": 0.9642857142857143,
|
| 147 |
+
"brier": 0.07835667610715101,
|
| 148 |
+
"brier_soft": 0.07835667610715101,
|
| 149 |
+
"nll": 0.13341829300103206,
|
| 150 |
+
"nll_soft": 0.13341829300103206,
|
| 151 |
+
"ece": 0.05279191576259608,
|
| 152 |
+
"mean_confidence": 0.954691373765687,
|
| 153 |
+
"confident_error_rate": 0.03571428571428571
|
| 154 |
+
},
|
| 155 |
+
"civil_comments_severity": {
|
| 156 |
+
"count": 100,
|
| 157 |
+
"accuracy": 0.85,
|
| 158 |
+
"brier": 0.21990075062806394,
|
| 159 |
+
"brier_soft": 0.15256546144647498,
|
| 160 |
+
"nll": 0.42595271409351193,
|
| 161 |
+
"nll_soft": 0.5859219055739261,
|
| 162 |
+
"ece": 0.1029050524606589,
|
| 163 |
+
"mean_confidence": 0.7619406731562297,
|
| 164 |
+
"confident_error_rate": 0.0,
|
| 165 |
+
"score_mae": 0.3013470975763519
|
| 166 |
+
},
|
| 167 |
+
"civil_comments_toxic": {
|
| 168 |
+
"count": 100,
|
| 169 |
+
"accuracy": 0.96,
|
| 170 |
+
"brier": 0.08184155550658603,
|
| 171 |
+
"brier_soft": 0.035637691017917736,
|
| 172 |
+
"nll": 0.15796842110857123,
|
| 173 |
+
"nll_soft": 0.2738468412771658,
|
| 174 |
+
"ece": 0.07203162905008202,
|
| 175 |
+
"mean_confidence": 0.8907618087052214,
|
| 176 |
+
"confident_error_rate": 0.0
|
| 177 |
+
},
|
| 178 |
+
"clinc150": {
|
| 179 |
+
"count": 100,
|
| 180 |
+
"accuracy": 0.88,
|
| 181 |
+
"brier": 0.19125913762163368,
|
| 182 |
+
"brier_soft": 0.19125913762163368,
|
| 183 |
+
"nll": 0.5776643056115763,
|
| 184 |
+
"nll_soft": 0.5776643056115763,
|
| 185 |
+
"ece": 0.09283844266870002,
|
| 186 |
+
"mean_confidence": 0.9496720060766602,
|
| 187 |
+
"confident_error_rate": 0.06
|
| 188 |
+
},
|
| 189 |
+
"cola": {
|
| 190 |
+
"count": 100,
|
| 191 |
+
"accuracy": 0.82,
|
| 192 |
+
"brier": 0.23386052759583184,
|
| 193 |
+
"brier_soft": 0.23386052759583184,
|
| 194 |
+
"nll": 0.37037031376428564,
|
| 195 |
+
"nll_soft": 0.37037031376428564,
|
| 196 |
+
"ece": 0.09766561804689824,
|
| 197 |
+
"mean_confidence": 0.8833423357882064,
|
| 198 |
+
"confident_error_rate": 0.03
|
| 199 |
+
},
|
| 200 |
+
"commonsense_qa": {
|
| 201 |
+
"count": 100,
|
| 202 |
+
"accuracy": 0.91,
|
| 203 |
+
"brier": 0.15377198586333088,
|
| 204 |
+
"brier_soft": 0.15377198586333088,
|
| 205 |
+
"nll": 0.3819044445089102,
|
| 206 |
+
"nll_soft": 0.3819044445089102,
|
| 207 |
+
"ece": 0.05144685490055159,
|
| 208 |
+
"mean_confidence": 0.9596385183475031,
|
| 209 |
+
"confident_error_rate": 0.05
|
| 210 |
+
},
|
| 211 |
+
"cosmos_qa": {
|
| 212 |
+
"count": 120,
|
| 213 |
+
"accuracy": 0.7166666666666667,
|
| 214 |
+
"brier": 0.4672354484316241,
|
| 215 |
+
"brier_soft": 0.4672354484316241,
|
| 216 |
+
"nll": 1.0482225583041105,
|
| 217 |
+
"nll_soft": 1.0482225583041105,
|
| 218 |
+
"ece": 0.19858953692704095,
|
| 219 |
+
"mean_confidence": 0.8962658679831853,
|
| 220 |
+
"confident_error_rate": 0.13333333333333333,
|
| 221 |
+
"evidence_auc": 0.999375
|
| 222 |
+
},
|
| 223 |
+
"dbpedia": {
|
| 224 |
+
"count": 100,
|
| 225 |
+
"accuracy": 1.0,
|
| 226 |
+
"brier": 0.0033691323280231667,
|
| 227 |
+
"brier_soft": 0.0033691323280231667,
|
| 228 |
+
"nll": 0.00628770684289355,
|
| 229 |
+
"nll_soft": 0.00628770684289355,
|
| 230 |
+
"ece": 0.0050215623881869585,
|
| 231 |
+
"mean_confidence": 0.994978437611813,
|
| 232 |
+
"confident_error_rate": 0.0
|
| 233 |
+
},
|
| 234 |
+
"dynasent": {
|
| 235 |
+
"count": 100,
|
| 236 |
+
"accuracy": 0.73,
|
| 237 |
+
"brier": 0.37323258779066765,
|
| 238 |
+
"brier_soft": 0.2605018455661368,
|
| 239 |
+
"nll": 0.6422479980426224,
|
| 240 |
+
"nll_soft": 0.7148729964174031,
|
| 241 |
+
"ece": 0.15595018764547483,
|
| 242 |
+
"mean_confidence": 0.6386131537774967,
|
| 243 |
+
"confident_error_rate": 0.0,
|
| 244 |
+
"score_mae": 0.42700063346629696
|
| 245 |
+
},
|
| 246 |
+
"enron_spam": {
|
| 247 |
+
"count": 100,
|
| 248 |
+
"accuracy": 0.99,
|
| 249 |
+
"brier": 0.020538910131957966,
|
| 250 |
+
"brier_soft": 0.020538910131957966,
|
| 251 |
+
"nll": 0.037125773454637925,
|
| 252 |
+
"nll_soft": 0.037125773454637925,
|
| 253 |
+
"ece": 0.018135869575440167,
|
| 254 |
+
"mean_confidence": 0.9889960329597918,
|
| 255 |
+
"confident_error_rate": 0.01
|
| 256 |
+
},
|
| 257 |
+
"go_emotions": {
|
| 258 |
+
"count": 100,
|
| 259 |
+
"accuracy": 0.6,
|
| 260 |
+
"brier": 0.5706133279962924,
|
| 261 |
+
"brier_soft": 0.2146223457949257,
|
| 262 |
+
"nll": 1.4243747447457458,
|
| 263 |
+
"nll_soft": 1.9114536152236135,
|
| 264 |
+
"ece": 0.2153529377728328,
|
| 265 |
+
"mean_confidence": 0.4180754880267198,
|
| 266 |
+
"confident_error_rate": 0.0
|
| 267 |
+
},
|
| 268 |
+
"hellaswag": {
|
| 269 |
+
"count": 100,
|
| 270 |
+
"accuracy": 0.97,
|
| 271 |
+
"brier": 0.04398586313752212,
|
| 272 |
+
"brier_soft": 0.04398586313752212,
|
| 273 |
+
"nll": 0.07282392796652562,
|
| 274 |
+
"nll_soft": 0.07282392796652562,
|
| 275 |
+
"ece": 0.03281997526145711,
|
| 276 |
+
"mean_confidence": 0.967702411105094,
|
| 277 |
+
"confident_error_rate": 0.0
|
| 278 |
+
},
|
| 279 |
+
"imdb": {
|
| 280 |
+
"count": 100,
|
| 281 |
+
"accuracy": 0.98,
|
| 282 |
+
"brier": 0.04503053337307919,
|
| 283 |
+
"brier_soft": 0.04503053337307919,
|
| 284 |
+
"nll": 0.10202723595434093,
|
| 285 |
+
"nll_soft": 0.10202723595434093,
|
| 286 |
+
"ece": 0.02799987491297936,
|
| 287 |
+
"mean_confidence": 0.9735376597268748,
|
| 288 |
+
"confident_error_rate": 0.01
|
| 289 |
+
},
|
| 290 |
+
"ledgar": {
|
| 291 |
+
"count": 100,
|
| 292 |
+
"accuracy": 0.78,
|
| 293 |
+
"brier": 0.31022259775593414,
|
| 294 |
+
"brier_soft": 0.31022259775593414,
|
| 295 |
+
"nll": 0.7916681164712028,
|
| 296 |
+
"nll_soft": 0.7916681164712028,
|
| 297 |
+
"ece": 0.10685236902591677,
|
| 298 |
+
"mean_confidence": 0.8791648234039952,
|
| 299 |
+
"confident_error_rate": 0.05
|
| 300 |
+
},
|
| 301 |
+
"liar": {
|
| 302 |
+
"count": 100,
|
| 303 |
+
"accuracy": 0.39,
|
| 304 |
+
"brier": 0.7168465726369471,
|
| 305 |
+
"brier_soft": 0.7168465726369471,
|
| 306 |
+
"nll": 1.430973024742663,
|
| 307 |
+
"nll_soft": 1.430973024742663,
|
| 308 |
+
"ece": 0.07922166544125636,
|
| 309 |
+
"mean_confidence": 0.33914869030092576,
|
| 310 |
+
"confident_error_rate": 0.0,
|
| 311 |
+
"score_mae": 0.9383143759928696
|
| 312 |
+
},
|
| 313 |
+
"measuring_hate_speech": {
|
| 314 |
+
"count": 100,
|
| 315 |
+
"accuracy": 0.84,
|
| 316 |
+
"brier": 0.2523259794626622,
|
| 317 |
+
"brier_soft": 0.1350772440937763,
|
| 318 |
+
"nll": 0.48511890737634117,
|
| 319 |
+
"nll_soft": 0.6516605742694233,
|
| 320 |
+
"ece": 0.1276134336558651,
|
| 321 |
+
"mean_confidence": 0.7243118123610279,
|
| 322 |
+
"confident_error_rate": 0.0,
|
| 323 |
+
"score_mae": 0.49301893575302613
|
| 324 |
+
},
|
| 325 |
+
"medqa": {
|
| 326 |
+
"count": 100,
|
| 327 |
+
"accuracy": 0.88,
|
| 328 |
+
"brier": 0.15509559657024174,
|
| 329 |
+
"brier_soft": 0.15509559657024174,
|
| 330 |
+
"nll": 0.2630871286801779,
|
| 331 |
+
"nll_soft": 0.2630871286801779,
|
| 332 |
+
"ece": 0.07094760936712856,
|
| 333 |
+
"mean_confidence": 0.9227796946829281,
|
| 334 |
+
"confident_error_rate": 0.02
|
| 335 |
+
},
|
| 336 |
+
"mnli": {
|
| 337 |
+
"count": 100,
|
| 338 |
+
"accuracy": 0.97,
|
| 339 |
+
"brier": 0.05622277315175722,
|
| 340 |
+
"brier_soft": 0.05622277315175722,
|
| 341 |
+
"nll": 0.11140887517203295,
|
| 342 |
+
"nll_soft": 0.11140887517203295,
|
| 343 |
+
"ece": 0.06498924366805267,
|
| 344 |
+
"mean_confidence": 0.9358105562109016,
|
| 345 |
+
"confident_error_rate": 0.01
|
| 346 |
+
},
|
| 347 |
+
"mrpc": {
|
| 348 |
+
"count": 100,
|
| 349 |
+
"accuracy": 0.9,
|
| 350 |
+
"brier": 0.16885290914097817,
|
| 351 |
+
"brier_soft": 0.16885290914097817,
|
| 352 |
+
"nll": 0.26678509569101083,
|
| 353 |
+
"nll_soft": 0.26678509569101083,
|
| 354 |
+
"ece": 0.056738491245727135,
|
| 355 |
+
"mean_confidence": 0.9094846375493769,
|
| 356 |
+
"confident_error_rate": 0.02
|
| 357 |
+
},
|
| 358 |
+
"multirc": {
|
| 359 |
+
"count": 125,
|
| 360 |
+
"accuracy": 0.768,
|
| 361 |
+
"brier": 0.4125322538820213,
|
| 362 |
+
"brier_soft": 0.4125322538820213,
|
| 363 |
+
"nll": 0.702692751642681,
|
| 364 |
+
"nll_soft": 0.702692751642681,
|
| 365 |
+
"ece": 0.2040484715924632,
|
| 366 |
+
"mean_confidence": 0.9360141123782398,
|
| 367 |
+
"confident_error_rate": 0.152,
|
| 368 |
+
"evidence_auc": 0.9997333333333334
|
| 369 |
+
},
|
| 370 |
+
"openbookqa": {
|
| 371 |
+
"count": 100,
|
| 372 |
+
"accuracy": 0.98,
|
| 373 |
+
"brier": 0.036469910012703805,
|
| 374 |
+
"brier_soft": 0.036469910012703805,
|
| 375 |
+
"nll": 0.07196625626831986,
|
| 376 |
+
"nll_soft": 0.07196625626831986,
|
| 377 |
+
"ece": 0.02228237729925685,
|
| 378 |
+
"mean_confidence": 0.977701842636366,
|
| 379 |
+
"confident_error_rate": 0.0
|
| 380 |
+
},
|
| 381 |
+
"piqa": {
|
| 382 |
+
"count": 100,
|
| 383 |
+
"accuracy": 0.91,
|
| 384 |
+
"brier": 0.12451330680810697,
|
| 385 |
+
"brier_soft": 0.12451330680810697,
|
| 386 |
+
"nll": 0.23446545298444263,
|
| 387 |
+
"nll_soft": 0.23446545298444263,
|
| 388 |
+
"ece": 0.06753078707740053,
|
| 389 |
+
"mean_confidence": 0.9522836212325586,
|
| 390 |
+
"confident_error_rate": 0.03
|
| 391 |
+
},
|
| 392 |
+
"policy_application": {
|
| 393 |
+
"count": 150,
|
| 394 |
+
"accuracy": 0.9466666666666667,
|
| 395 |
+
"brier": 0.0777876053884264,
|
| 396 |
+
"brier_soft": 0.0777876053884264,
|
| 397 |
+
"nll": 0.13521520720550428,
|
| 398 |
+
"nll_soft": 0.13521520720550428,
|
| 399 |
+
"ece": 0.04668170693287413,
|
| 400 |
+
"mean_confidence": 0.985035205551867,
|
| 401 |
+
"confident_error_rate": 0.03333333333333333,
|
| 402 |
+
"evidence_auc": 1.0
|
| 403 |
+
},
|
| 404 |
+
"qqp": {
|
| 405 |
+
"count": 100,
|
| 406 |
+
"accuracy": 0.92,
|
| 407 |
+
"brier": 0.13043954738084676,
|
| 408 |
+
"brier_soft": 0.13043954738084676,
|
| 409 |
+
"nll": 0.22957888778440388,
|
| 410 |
+
"nll_soft": 0.22957888778440388,
|
| 411 |
+
"ece": 0.03919742572811998,
|
| 412 |
+
"mean_confidence": 0.9198058774243847,
|
| 413 |
+
"confident_error_rate": 0.04
|
| 414 |
+
},
|
| 415 |
+
"race": {
|
| 416 |
+
"count": 125,
|
| 417 |
+
"accuracy": 0.792,
|
| 418 |
+
"brier": 0.273025071208486,
|
| 419 |
+
"brier_soft": 0.273025071208486,
|
| 420 |
+
"nll": 0.5401223012436509,
|
| 421 |
+
"nll_soft": 0.5401223012436509,
|
| 422 |
+
"ece": 0.09241534466772917,
|
| 423 |
+
"mean_confidence": 0.8570207713461646,
|
| 424 |
+
"confident_error_rate": 0.04,
|
| 425 |
+
"evidence_auc": 1.0
|
| 426 |
+
},
|
| 427 |
+
"rotten_tomatoes": {
|
| 428 |
+
"count": 100,
|
| 429 |
+
"accuracy": 0.98,
|
| 430 |
+
"brier": 0.035290060552569,
|
| 431 |
+
"brier_soft": 0.035290060552569,
|
| 432 |
+
"nll": 0.07276457767313435,
|
| 433 |
+
"nll_soft": 0.07276457767313435,
|
| 434 |
+
"ece": 0.017489686473322937,
|
| 435 |
+
"mean_confidence": 0.9711417797012452,
|
| 436 |
+
"confident_error_rate": 0.01
|
| 437 |
+
},
|
| 438 |
+
"rte": {
|
| 439 |
+
"count": 100,
|
| 440 |
+
"accuracy": 0.93,
|
| 441 |
+
"brier": 0.11679704186673533,
|
| 442 |
+
"brier_soft": 0.11679704186673533,
|
| 443 |
+
"nll": 0.20138234100382452,
|
| 444 |
+
"nll_soft": 0.20138234100382452,
|
| 445 |
+
"ece": 0.04667671248922903,
|
| 446 |
+
"mean_confidence": 0.976676712489229,
|
| 447 |
+
"confident_error_rate": 0.03
|
| 448 |
+
},
|
| 449 |
+
"scotus": {
|
| 450 |
+
"count": 100,
|
| 451 |
+
"accuracy": 0.61,
|
| 452 |
+
"brier": 0.6341498051249195,
|
| 453 |
+
"brier_soft": 0.6341498051249195,
|
| 454 |
+
"nll": 1.6277115865464133,
|
| 455 |
+
"nll_soft": 1.6277115865464133,
|
| 456 |
+
"ece": 0.25533107042012765,
|
| 457 |
+
"mean_confidence": 0.8527349247542704,
|
| 458 |
+
"confident_error_rate": 0.13
|
| 459 |
+
},
|
| 460 |
+
"shp": {
|
| 461 |
+
"count": 100,
|
| 462 |
+
"accuracy": 0.65,
|
| 463 |
+
"brier": 0.46443770760873426,
|
| 464 |
+
"brier_soft": 0.09936027473515505,
|
| 465 |
+
"nll": 0.6572154778957793,
|
| 466 |
+
"nll_soft": 0.6826727776739735,
|
| 467 |
+
"ece": 0.1103068097721516,
|
| 468 |
+
"mean_confidence": 0.5396931902278483,
|
| 469 |
+
"confident_error_rate": 0.0
|
| 470 |
+
},
|
| 471 |
+
"sms_spam": {
|
| 472 |
+
"count": 100,
|
| 473 |
+
"accuracy": 0.99,
|
| 474 |
+
"brier": 0.01279388922890178,
|
| 475 |
+
"brier_soft": 0.01279388922890178,
|
| 476 |
+
"nll": 0.019404364213720597,
|
| 477 |
+
"nll_soft": 0.019404364213720597,
|
| 478 |
+
"ece": 0.011369026874576384,
|
| 479 |
+
"mean_confidence": 0.994594708705868,
|
| 480 |
+
"confident_error_rate": 0.0
|
| 481 |
+
},
|
| 482 |
+
"snli": {
|
| 483 |
+
"count": 100,
|
| 484 |
+
"accuracy": 0.92,
|
| 485 |
+
"brier": 0.1303060757325426,
|
| 486 |
+
"brier_soft": 0.1303060757325426,
|
| 487 |
+
"nll": 0.240589231743456,
|
| 488 |
+
"nll_soft": 0.240589231743456,
|
| 489 |
+
"ece": 0.04670265643035693,
|
| 490 |
+
"mean_confidence": 0.9518659140843424,
|
| 491 |
+
"confident_error_rate": 0.05
|
| 492 |
+
},
|
| 493 |
+
"social_iqa": {
|
| 494 |
+
"count": 100,
|
| 495 |
+
"accuracy": 0.84,
|
| 496 |
+
"brier": 0.2641777007342555,
|
| 497 |
+
"brier_soft": 0.2641777007342555,
|
| 498 |
+
"nll": 0.5338294339120516,
|
| 499 |
+
"nll_soft": 0.5338294339120516,
|
| 500 |
+
"ece": 0.09934345700863399,
|
| 501 |
+
"mean_confidence": 0.9154886411744081,
|
| 502 |
+
"confident_error_rate": 0.06
|
| 503 |
+
},
|
| 504 |
+
"sst2": {
|
| 505 |
+
"count": 100,
|
| 506 |
+
"accuracy": 0.98,
|
| 507 |
+
"brier": 0.04030628615298806,
|
| 508 |
+
"brier_soft": 0.04030628615298806,
|
| 509 |
+
"nll": 0.11747284260512826,
|
| 510 |
+
"nll_soft": 0.11747284260512826,
|
| 511 |
+
"ece": 0.012949483786838481,
|
| 512 |
+
"mean_confidence": 0.9895809775228284,
|
| 513 |
+
"confident_error_rate": 0.02
|
| 514 |
+
},
|
| 515 |
+
"sst5": {
|
| 516 |
+
"count": 100,
|
| 517 |
+
"accuracy": 0.66,
|
| 518 |
+
"brier": 0.47858939401943057,
|
| 519 |
+
"brier_soft": 0.47858939401943057,
|
| 520 |
+
"nll": 0.8312380624917151,
|
| 521 |
+
"nll_soft": 0.8312380624917151,
|
| 522 |
+
"ece": 0.14479568835917578,
|
| 523 |
+
"mean_confidence": 0.5538169983846644,
|
| 524 |
+
"confident_error_rate": 0.0,
|
| 525 |
+
"score_mae": 0.4382710876978508
|
| 526 |
+
},
|
| 527 |
+
"stance_abortion": {
|
| 528 |
+
"count": 60,
|
| 529 |
+
"accuracy": 0.65,
|
| 530 |
+
"brier": 0.5483838340126799,
|
| 531 |
+
"brier_soft": 0.5483838340126799,
|
| 532 |
+
"nll": 0.916795129103024,
|
| 533 |
+
"nll_soft": 0.916795129103024,
|
| 534 |
+
"ece": 0.2470612492514964,
|
| 535 |
+
"mean_confidence": 0.8717795686329541,
|
| 536 |
+
"confident_error_rate": 0.15
|
| 537 |
+
},
|
| 538 |
+
"stance_atheism": {
|
| 539 |
+
"count": 60,
|
| 540 |
+
"accuracy": 0.7833333333333333,
|
| 541 |
+
"brier": 0.31917383626334284,
|
| 542 |
+
"brier_soft": 0.31917383626334284,
|
| 543 |
+
"nll": 0.49926359046026203,
|
| 544 |
+
"nll_soft": 0.49926359046026203,
|
| 545 |
+
"ece": 0.15642293053711553,
|
| 546 |
+
"mean_confidence": 0.889754045572878,
|
| 547 |
+
"confident_error_rate": 0.03333333333333333
|
| 548 |
+
},
|
| 549 |
+
"stance_feminist": {
|
| 550 |
+
"count": 60,
|
| 551 |
+
"accuracy": 0.7833333333333333,
|
| 552 |
+
"brier": 0.3049816554075578,
|
| 553 |
+
"brier_soft": 0.3049816554075578,
|
| 554 |
+
"nll": 0.5154249440169435,
|
| 555 |
+
"nll_soft": 0.5154249440169435,
|
| 556 |
+
"ece": 0.10461663391963241,
|
| 557 |
+
"mean_confidence": 0.8737570622243047,
|
| 558 |
+
"confident_error_rate": 0.05
|
| 559 |
+
},
|
| 560 |
+
"stance_hillary": {
|
| 561 |
+
"count": 60,
|
| 562 |
+
"accuracy": 0.7166666666666667,
|
| 563 |
+
"brier": 0.3812783742364739,
|
| 564 |
+
"brier_soft": 0.3812783742364739,
|
| 565 |
+
"nll": 0.6242170771095513,
|
| 566 |
+
"nll_soft": 0.6242170771095513,
|
| 567 |
+
"ece": 0.11280688269490617,
|
| 568 |
+
"mean_confidence": 0.8143798914819186,
|
| 569 |
+
"confident_error_rate": 0.05
|
| 570 |
+
},
|
| 571 |
+
"stsb": {
|
| 572 |
+
"count": 100,
|
| 573 |
+
"accuracy": 0.65,
|
| 574 |
+
"brier": 0.45170767908012466,
|
| 575 |
+
"brier_soft": 0.20552974536260238,
|
| 576 |
+
"nll": 0.8029710252702897,
|
| 577 |
+
"nll_soft": 0.8819413373331731,
|
| 578 |
+
"ece": 0.10983419549350473,
|
| 579 |
+
"mean_confidence": 0.6357630237891139,
|
| 580 |
+
"confident_error_rate": 0.0,
|
| 581 |
+
"score_mae": 0.44494913340968734
|
| 582 |
+
},
|
| 583 |
+
"subjectivity": {
|
| 584 |
+
"count": 100,
|
| 585 |
+
"accuracy": 0.99,
|
| 586 |
+
"brier": 0.02143104125551823,
|
| 587 |
+
"brier_soft": 0.02143104125551823,
|
| 588 |
+
"nll": 0.047739931072467476,
|
| 589 |
+
"nll_soft": 0.047739931072467476,
|
| 590 |
+
"ece": 0.039431940261527555,
|
| 591 |
+
"mean_confidence": 0.9635951571408603,
|
| 592 |
+
"confident_error_rate": 0.0
|
| 593 |
+
},
|
| 594 |
+
"swag": {
|
| 595 |
+
"count": 100,
|
| 596 |
+
"accuracy": 0.82,
|
| 597 |
+
"brier": 0.2537048639515909,
|
| 598 |
+
"brier_soft": 0.2537048639515909,
|
| 599 |
+
"nll": 0.4672971887646461,
|
| 600 |
+
"nll_soft": 0.4672971887646461,
|
| 601 |
+
"ece": 0.08815823139371007,
|
| 602 |
+
"mean_confidence": 0.9081582313937099,
|
| 603 |
+
"confident_error_rate": 0.05
|
| 604 |
+
},
|
| 605 |
+
"trec": {
|
| 606 |
+
"count": 100,
|
| 607 |
+
"accuracy": 0.97,
|
| 608 |
+
"brier": 0.04378629853100501,
|
| 609 |
+
"brier_soft": 0.04378629853100501,
|
| 610 |
+
"nll": 0.11549823864485118,
|
| 611 |
+
"nll_soft": 0.11549823864485118,
|
| 612 |
+
"ece": 0.025471181892042873,
|
| 613 |
+
"mean_confidence": 0.9775477901155262,
|
| 614 |
+
"confident_error_rate": 0.01
|
| 615 |
+
},
|
| 616 |
+
"tweet_hate": {
|
| 617 |
+
"count": 100,
|
| 618 |
+
"accuracy": 0.5,
|
| 619 |
+
"brier": 0.6883996904854307,
|
| 620 |
+
"brier_soft": 0.6883996904854307,
|
| 621 |
+
"nll": 0.9833215742341621,
|
| 622 |
+
"nll_soft": 0.9833215742341621,
|
| 623 |
+
"ece": 0.3459218953511739,
|
| 624 |
+
"mean_confidence": 0.8344252768252641,
|
| 625 |
+
"confident_error_rate": 0.11
|
| 626 |
+
},
|
| 627 |
+
"tweet_sentiment": {
|
| 628 |
+
"count": 100,
|
| 629 |
+
"accuracy": 0.74,
|
| 630 |
+
"brier": 0.3831706610576699,
|
| 631 |
+
"brier_soft": 0.3831706610576699,
|
| 632 |
+
"nll": 0.6497370609790863,
|
| 633 |
+
"nll_soft": 0.6497370609790863,
|
| 634 |
+
"ece": 0.13208543978054657,
|
| 635 |
+
"mean_confidence": 0.6252989072735136,
|
| 636 |
+
"confident_error_rate": 0.0,
|
| 637 |
+
"score_mae": 0.3541000877573742
|
| 638 |
+
},
|
| 639 |
+
"vitaminc": {
|
| 640 |
+
"count": 100,
|
| 641 |
+
"accuracy": 0.84,
|
| 642 |
+
"brier": 0.23269831680007946,
|
| 643 |
+
"brier_soft": 0.23269831680007946,
|
| 644 |
+
"nll": 0.4394585245066372,
|
| 645 |
+
"nll_soft": 0.4394585245066372,
|
| 646 |
+
"ece": 0.09072915887713841,
|
| 647 |
+
"mean_confidence": 0.9197737336036776,
|
| 648 |
+
"confident_error_rate": 0.06
|
| 649 |
+
},
|
| 650 |
+
"wic": {
|
| 651 |
+
"count": 100,
|
| 652 |
+
"accuracy": 0.78,
|
| 653 |
+
"brier": 0.34073352692700504,
|
| 654 |
+
"brier_soft": 0.34073352692700504,
|
| 655 |
+
"nll": 0.5452054678565941,
|
| 656 |
+
"nll_soft": 0.5452054678565941,
|
| 657 |
+
"ece": 0.13825743108455368,
|
| 658 |
+
"mean_confidence": 0.9071892135345369,
|
| 659 |
+
"confident_error_rate": 0.1
|
| 660 |
+
},
|
| 661 |
+
"winogrande": {
|
| 662 |
+
"count": 100,
|
| 663 |
+
"accuracy": 0.91,
|
| 664 |
+
"brier": 0.17067644004760918,
|
| 665 |
+
"brier_soft": 0.17067644004760918,
|
| 666 |
+
"nll": 0.35855105926065917,
|
| 667 |
+
"nll_soft": 0.35855105926065917,
|
| 668 |
+
"ece": 0.08624812639081393,
|
| 669 |
+
"mean_confidence": 0.9450851377589773,
|
| 670 |
+
"confident_error_rate": 0.07
|
| 671 |
+
},
|
| 672 |
+
"yelp": {
|
| 673 |
+
"count": 100,
|
| 674 |
+
"accuracy": 0.73,
|
| 675 |
+
"brier": 0.402625269533739,
|
| 676 |
+
"brier_soft": 0.402625269533739,
|
| 677 |
+
"nll": 0.6852448507549156,
|
| 678 |
+
"nll_soft": 0.6852448507549156,
|
| 679 |
+
"ece": 0.16558757141209074,
|
| 680 |
+
"mean_confidence": 0.6233702191486455,
|
| 681 |
+
"confident_error_rate": 0.0,
|
| 682 |
+
"score_mae": 0.3873896875444972
|
| 683 |
+
}
|
| 684 |
+
},
|
| 685 |
+
"reliability": [
|
| 686 |
+
{
|
| 687 |
+
"bin": "0.0-0.1",
|
| 688 |
+
"count": 0,
|
| 689 |
+
"confidence": NaN,
|
| 690 |
+
"accuracy": NaN
|
| 691 |
+
},
|
| 692 |
+
{
|
| 693 |
+
"bin": "0.1-0.2",
|
| 694 |
+
"count": 5,
|
| 695 |
+
"confidence": 0.1795541536525354,
|
| 696 |
+
"accuracy": 0.0
|
| 697 |
+
},
|
| 698 |
+
{
|
| 699 |
+
"bin": "0.2-0.3",
|
| 700 |
+
"count": 67,
|
| 701 |
+
"confidence": 0.251950177472451,
|
| 702 |
+
"accuracy": 0.29850746268656714
|
| 703 |
+
},
|
| 704 |
+
{
|
| 705 |
+
"bin": "0.3-0.4",
|
| 706 |
+
"count": 83,
|
| 707 |
+
"confidence": 0.35303291214353666,
|
| 708 |
+
"accuracy": 0.3493975903614458
|
| 709 |
+
},
|
| 710 |
+
{
|
| 711 |
+
"bin": "0.4-0.5",
|
| 712 |
+
"count": 271,
|
| 713 |
+
"confidence": 0.46348321355010486,
|
| 714 |
+
"accuracy": 0.5129151291512916
|
| 715 |
+
},
|
| 716 |
+
{
|
| 717 |
+
"bin": "0.5-0.6",
|
| 718 |
+
"count": 405,
|
| 719 |
+
"confidence": 0.5506984297677288,
|
| 720 |
+
"accuracy": 0.6148148148148148
|
| 721 |
+
},
|
| 722 |
+
{
|
| 723 |
+
"bin": "0.6-0.7",
|
| 724 |
+
"count": 314,
|
| 725 |
+
"confidence": 0.6475632613940633,
|
| 726 |
+
"accuracy": 0.6719745222929936
|
| 727 |
+
},
|
| 728 |
+
{
|
| 729 |
+
"bin": "0.7-0.8",
|
| 730 |
+
"count": 328,
|
| 731 |
+
"confidence": 0.7527084045456743,
|
| 732 |
+
"accuracy": 0.725609756097561
|
| 733 |
+
},
|
| 734 |
+
{
|
| 735 |
+
"bin": "0.8-0.9",
|
| 736 |
+
"count": 442,
|
| 737 |
+
"confidence": 0.8559694673474576,
|
| 738 |
+
"accuracy": 0.7081447963800905
|
| 739 |
+
},
|
| 740 |
+
{
|
| 741 |
+
"bin": "0.9-1.0",
|
| 742 |
+
"count": 3598,
|
| 743 |
+
"confidence": 0.9823404759359501,
|
| 744 |
+
"accuracy": 0.9449694274596998
|
| 745 |
+
}
|
| 746 |
+
],
|
| 747 |
+
"seconds": 387.9096157550812,
|
| 748 |
+
"model": "runs/assay-27b",
|
| 749 |
+
"data": "data/v2/dev.jsonl"
|
| 750 |
+
}
|
eval-holdout-scaled.json
ADDED
|
@@ -0,0 +1,239 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
{
|
| 2 |
+
"overall": {
|
| 3 |
+
"count": 2020,
|
| 4 |
+
"accuracy": 0.8415841584158416,
|
| 5 |
+
"brier": 0.220799111831009,
|
| 6 |
+
"brier_soft": 0.220799111831009,
|
| 7 |
+
"nll": 0.39014477942134185,
|
| 8 |
+
"nll_soft": 0.39014477942134185,
|
| 9 |
+
"ece": 0.03993673549801924,
|
| 10 |
+
"mean_confidence": 0.8201206323505023,
|
| 11 |
+
"confident_error_rate": 0.0049504950495049506,
|
| 12 |
+
"score_mae": 0.7803430055728247
|
| 13 |
+
},
|
| 14 |
+
"by_type": {
|
| 15 |
+
"bool": {
|
| 16 |
+
"count": 800,
|
| 17 |
+
"accuracy": 0.855,
|
| 18 |
+
"brier": 0.1998546341034784,
|
| 19 |
+
"brier_soft": 0.1998546341034784,
|
| 20 |
+
"nll": 0.3135695750039896,
|
| 21 |
+
"nll_soft": 0.3135695750039896,
|
| 22 |
+
"ece": 0.041050193204409134,
|
| 23 |
+
"mean_confidence": 0.8587700991805063,
|
| 24 |
+
"confident_error_rate": 0.00875
|
| 25 |
+
},
|
| 26 |
+
"choice": {
|
| 27 |
+
"count": 1020,
|
| 28 |
+
"accuracy": 0.8676470588235294,
|
| 29 |
+
"brier": 0.18523493798833834,
|
| 30 |
+
"brier_soft": 0.18523493798833834,
|
| 31 |
+
"nll": 0.33966287329231043,
|
| 32 |
+
"nll_soft": 0.33966287329231043,
|
| 33 |
+
"ece": 0.03498720217459626,
|
| 34 |
+
"mean_confidence": 0.8462023584334264,
|
| 35 |
+
"confident_error_rate": 0.0029411764705882353
|
| 36 |
+
},
|
| 37 |
+
"score": {
|
| 38 |
+
"count": 200,
|
| 39 |
+
"accuracy": 0.655,
|
| 40 |
+
"brier": 0.4859543093387518,
|
| 41 |
+
"brier_soft": 0.4859543093387518,
|
| 42 |
+
"nll": 0.9539033183488113,
|
| 43 |
+
"nll_soft": 0.9539033183488113,
|
| 44 |
+
"ece": 0.19196724371336882,
|
| 45 |
+
"mean_confidence": 0.5325059620075736,
|
| 46 |
+
"confident_error_rate": 0.0,
|
| 47 |
+
"score_mae": 0.7803430055728247
|
| 48 |
+
}
|
| 49 |
+
},
|
| 50 |
+
"by_source": {
|
| 51 |
+
"app_reviews": {
|
| 52 |
+
"count": 200,
|
| 53 |
+
"accuracy": 0.655,
|
| 54 |
+
"brier": 0.4859543093387518,
|
| 55 |
+
"brier_soft": 0.4859543093387518,
|
| 56 |
+
"nll": 0.9539033183488113,
|
| 57 |
+
"nll_soft": 0.9539033183488113,
|
| 58 |
+
"ece": 0.19196724371336882,
|
| 59 |
+
"mean_confidence": 0.5325059620075736,
|
| 60 |
+
"confident_error_rate": 0.0,
|
| 61 |
+
"score_mae": 0.7803430055728247
|
| 62 |
+
},
|
| 63 |
+
"bbc_news": {
|
| 64 |
+
"count": 200,
|
| 65 |
+
"accuracy": 0.99,
|
| 66 |
+
"brier": 0.028912731212050385,
|
| 67 |
+
"brier_soft": 0.028912731212050385,
|
| 68 |
+
"nll": 0.07044915532346424,
|
| 69 |
+
"nll_soft": 0.07044915532346424,
|
| 70 |
+
"ece": 0.051107039480480426,
|
| 71 |
+
"mean_confidence": 0.9438197171243428,
|
| 72 |
+
"confident_error_rate": 0.0
|
| 73 |
+
},
|
| 74 |
+
"copa": {
|
| 75 |
+
"count": 100,
|
| 76 |
+
"accuracy": 0.99,
|
| 77 |
+
"brier": 0.01453758967734365,
|
| 78 |
+
"brier_soft": 0.01453758967734365,
|
| 79 |
+
"nll": 0.029948603826919724,
|
| 80 |
+
"nll_soft": 0.029948603826919724,
|
| 81 |
+
"ece": 0.022917495301317405,
|
| 82 |
+
"mean_confidence": 0.9817517021787172,
|
| 83 |
+
"confident_error_rate": 0.0
|
| 84 |
+
},
|
| 85 |
+
"dream": {
|
| 86 |
+
"count": 200,
|
| 87 |
+
"accuracy": 0.985,
|
| 88 |
+
"brier": 0.029222745427828708,
|
| 89 |
+
"brier_soft": 0.029222745427828708,
|
| 90 |
+
"nll": 0.08134387315614394,
|
| 91 |
+
"nll_soft": 0.08134387315614394,
|
| 92 |
+
"ece": 0.012822483843521316,
|
| 93 |
+
"mean_confidence": 0.9815427931884518,
|
| 94 |
+
"confident_error_rate": 0.005
|
| 95 |
+
},
|
| 96 |
+
"ethos": {
|
| 97 |
+
"count": 200,
|
| 98 |
+
"accuracy": 0.785,
|
| 99 |
+
"brier": 0.2789473107906072,
|
| 100 |
+
"brier_soft": 0.2789473107906072,
|
| 101 |
+
"nll": 0.4138867544823541,
|
| 102 |
+
"nll_soft": 0.4138867544823541,
|
| 103 |
+
"ece": 0.079336313886892,
|
| 104 |
+
"mean_confidence": 0.8364448087371015,
|
| 105 |
+
"confident_error_rate": 0.02
|
| 106 |
+
},
|
| 107 |
+
"hh_rlhf": {
|
| 108 |
+
"count": 200,
|
| 109 |
+
"accuracy": 0.645,
|
| 110 |
+
"brier": 0.4451497617751254,
|
| 111 |
+
"brier_soft": 0.4451497617751254,
|
| 112 |
+
"nll": 0.6353554945088988,
|
| 113 |
+
"nll_soft": 0.6353554945088988,
|
| 114 |
+
"ece": 0.0625498708907104,
|
| 115 |
+
"mean_confidence": 0.596423357696465,
|
| 116 |
+
"confident_error_rate": 0.0
|
| 117 |
+
},
|
| 118 |
+
"medical_questions_pairs": {
|
| 119 |
+
"count": 200,
|
| 120 |
+
"accuracy": 0.895,
|
| 121 |
+
"brier": 0.13672426710472746,
|
| 122 |
+
"brier_soft": 0.13672426710472746,
|
| 123 |
+
"nll": 0.22289993483247128,
|
| 124 |
+
"nll_soft": 0.22289993483247128,
|
| 125 |
+
"ece": 0.06834734948342853,
|
| 126 |
+
"mean_confidence": 0.8813892348533718,
|
| 127 |
+
"confident_error_rate": 0.0
|
| 128 |
+
},
|
| 129 |
+
"scitail": {
|
| 130 |
+
"count": 200,
|
| 131 |
+
"accuracy": 0.91,
|
| 132 |
+
"brier": 0.1300524297562204,
|
| 133 |
+
"brier_soft": 0.1300524297562204,
|
| 134 |
+
"nll": 0.2165815487842471,
|
| 135 |
+
"nll_soft": 0.2165815487842471,
|
| 136 |
+
"ece": 0.02789948287264498,
|
| 137 |
+
"mean_confidence": 0.9132580009454067,
|
| 138 |
+
"confident_error_rate": 0.015
|
| 139 |
+
},
|
| 140 |
+
"stance_climate": {
|
| 141 |
+
"count": 120,
|
| 142 |
+
"accuracy": 0.8833333333333333,
|
| 143 |
+
"brier": 0.19515038307691712,
|
| 144 |
+
"brier_soft": 0.19515038307691712,
|
| 145 |
+
"nll": 0.3915492223860603,
|
| 146 |
+
"nll_soft": 0.3915492223860603,
|
| 147 |
+
"ece": 0.15516476801357199,
|
| 148 |
+
"mean_confidence": 0.7656094020870176,
|
| 149 |
+
"confident_error_rate": 0.0
|
| 150 |
+
},
|
| 151 |
+
"truthful_qa": {
|
| 152 |
+
"count": 200,
|
| 153 |
+
"accuracy": 0.78,
|
| 154 |
+
"brier": 0.317053920640699,
|
| 155 |
+
"brier_soft": 0.317053920640699,
|
| 156 |
+
"nll": 0.6952282954571797,
|
| 157 |
+
"nll_soft": 0.6952282954571797,
|
| 158 |
+
"ece": 0.09467769715035201,
|
| 159 |
+
"mean_confidence": 0.8436046676596461,
|
| 160 |
+
"confident_error_rate": 0.01
|
| 161 |
+
},
|
| 162 |
+
"tweet_irony": {
|
| 163 |
+
"count": 200,
|
| 164 |
+
"accuracy": 0.83,
|
| 165 |
+
"brier": 0.25369452876235854,
|
| 166 |
+
"brier_soft": 0.25369452876235854,
|
| 167 |
+
"nll": 0.4009100619168858,
|
| 168 |
+
"nll_soft": 0.4009100619168858,
|
| 169 |
+
"ece": 0.05814862800170818,
|
| 170 |
+
"mean_confidence": 0.8039883521861452,
|
| 171 |
+
"confident_error_rate": 0.0
|
| 172 |
+
}
|
| 173 |
+
},
|
| 174 |
+
"reliability": [
|
| 175 |
+
{
|
| 176 |
+
"bin": "0.0-0.1",
|
| 177 |
+
"count": 0,
|
| 178 |
+
"confidence": NaN,
|
| 179 |
+
"accuracy": NaN
|
| 180 |
+
},
|
| 181 |
+
{
|
| 182 |
+
"bin": "0.1-0.2",
|
| 183 |
+
"count": 0,
|
| 184 |
+
"confidence": NaN,
|
| 185 |
+
"accuracy": NaN
|
| 186 |
+
},
|
| 187 |
+
{
|
| 188 |
+
"bin": "0.2-0.3",
|
| 189 |
+
"count": 7,
|
| 190 |
+
"confidence": 0.2744105925520648,
|
| 191 |
+
"accuracy": 0.14285714285714285
|
| 192 |
+
},
|
| 193 |
+
{
|
| 194 |
+
"bin": "0.3-0.4",
|
| 195 |
+
"count": 43,
|
| 196 |
+
"confidence": 0.35869775450764224,
|
| 197 |
+
"accuracy": 0.4418604651162791
|
| 198 |
+
},
|
| 199 |
+
{
|
| 200 |
+
"bin": "0.4-0.5",
|
| 201 |
+
"count": 104,
|
| 202 |
+
"confidence": 0.4636549146792239,
|
| 203 |
+
"accuracy": 0.4423076923076923
|
| 204 |
+
},
|
| 205 |
+
{
|
| 206 |
+
"bin": "0.5-0.6",
|
| 207 |
+
"count": 236,
|
| 208 |
+
"confidence": 0.5565186483404105,
|
| 209 |
+
"accuracy": 0.6483050847457628
|
| 210 |
+
},
|
| 211 |
+
{
|
| 212 |
+
"bin": "0.6-0.7",
|
| 213 |
+
"count": 172,
|
| 214 |
+
"confidence": 0.6513175851364689,
|
| 215 |
+
"accuracy": 0.6627906976744186
|
| 216 |
+
},
|
| 217 |
+
{
|
| 218 |
+
"bin": "0.7-0.8",
|
| 219 |
+
"count": 181,
|
| 220 |
+
"confidence": 0.7479925616462004,
|
| 221 |
+
"accuracy": 0.7679558011049724
|
| 222 |
+
},
|
| 223 |
+
{
|
| 224 |
+
"bin": "0.8-0.9",
|
| 225 |
+
"count": 261,
|
| 226 |
+
"confidence": 0.8562325391696354,
|
| 227 |
+
"accuracy": 0.8505747126436781
|
| 228 |
+
},
|
| 229 |
+
{
|
| 230 |
+
"bin": "0.9-1.0",
|
| 231 |
+
"count": 1016,
|
| 232 |
+
"confidence": 0.97327787066598,
|
| 233 |
+
"accuracy": 0.9901574803149606
|
| 234 |
+
}
|
| 235 |
+
],
|
| 236 |
+
"seconds": 123.60580730438232,
|
| 237 |
+
"model": "runs/assay-27b",
|
| 238 |
+
"data": "data/v2/holdout.jsonl"
|
| 239 |
+
}
|
eval-holdout.json
ADDED
|
@@ -0,0 +1,239 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
{
|
| 2 |
+
"overall": {
|
| 3 |
+
"count": 2020,
|
| 4 |
+
"accuracy": 0.8415841584158416,
|
| 5 |
+
"brier": 0.2199661750237191,
|
| 6 |
+
"brier_soft": 0.2199661750237191,
|
| 7 |
+
"nll": 0.39249676153765684,
|
| 8 |
+
"nll_soft": 0.39249676153765684,
|
| 9 |
+
"ece": 0.0305830851696879,
|
| 10 |
+
"mean_confidence": 0.8462516664578095,
|
| 11 |
+
"confident_error_rate": 0.01485148514851485,
|
| 12 |
+
"score_mae": 0.734591401760134
|
| 13 |
+
},
|
| 14 |
+
"by_type": {
|
| 15 |
+
"bool": {
|
| 16 |
+
"count": 800,
|
| 17 |
+
"accuracy": 0.855,
|
| 18 |
+
"brier": 0.20325044236015188,
|
| 19 |
+
"brier_soft": 0.20325044236015188,
|
| 20 |
+
"nll": 0.31629045427560526,
|
| 21 |
+
"nll_soft": 0.31629045427560526,
|
| 22 |
+
"ece": 0.03714657739278206,
|
| 23 |
+
"mean_confidence": 0.8853094374262506,
|
| 24 |
+
"confident_error_rate": 0.0225
|
| 25 |
+
},
|
| 26 |
+
"choice": {
|
| 27 |
+
"count": 1020,
|
| 28 |
+
"accuracy": 0.8676470588235294,
|
| 29 |
+
"brier": 0.18443775278710198,
|
| 30 |
+
"brier_soft": 0.18443775278710198,
|
| 31 |
+
"nll": 0.34555459237851266,
|
| 32 |
+
"nll_soft": 0.34555459237851266,
|
| 33 |
+
"ece": 0.017727219065961402,
|
| 34 |
+
"mean_confidence": 0.8679584693166184,
|
| 35 |
+
"confident_error_rate": 0.011764705882352941
|
| 36 |
+
},
|
| 37 |
+
"score": {
|
| 38 |
+
"count": 200,
|
| 39 |
+
"accuracy": 0.655,
|
| 40 |
+
"brier": 0.4680240590847353,
|
| 41 |
+
"brier_soft": 0.4680240590847353,
|
| 42 |
+
"nll": 0.9367270532974982,
|
| 43 |
+
"nll_soft": 0.9367270532974982,
|
| 44 |
+
"ece": 0.14919123973044776,
|
| 45 |
+
"mean_confidence": 0.5793158880041201,
|
| 46 |
+
"confident_error_rate": 0.0,
|
| 47 |
+
"score_mae": 0.734591401760134
|
| 48 |
+
}
|
| 49 |
+
},
|
| 50 |
+
"by_source": {
|
| 51 |
+
"app_reviews": {
|
| 52 |
+
"count": 200,
|
| 53 |
+
"accuracy": 0.655,
|
| 54 |
+
"brier": 0.4680240590847353,
|
| 55 |
+
"brier_soft": 0.4680240590847353,
|
| 56 |
+
"nll": 0.9367270532974982,
|
| 57 |
+
"nll_soft": 0.9367270532974982,
|
| 58 |
+
"ece": 0.14919123973044776,
|
| 59 |
+
"mean_confidence": 0.5793158880041201,
|
| 60 |
+
"confident_error_rate": 0.0,
|
| 61 |
+
"score_mae": 0.734591401760134
|
| 62 |
+
},
|
| 63 |
+
"bbc_news": {
|
| 64 |
+
"count": 200,
|
| 65 |
+
"accuracy": 0.99,
|
| 66 |
+
"brier": 0.02485312024968503,
|
| 67 |
+
"brier_soft": 0.02485312024968503,
|
| 68 |
+
"nll": 0.05229038920662954,
|
| 69 |
+
"nll_soft": 0.05229038920662954,
|
| 70 |
+
"ece": 0.03289109205025749,
|
| 71 |
+
"mean_confidence": 0.9612872383267218,
|
| 72 |
+
"confident_error_rate": 0.0
|
| 73 |
+
},
|
| 74 |
+
"copa": {
|
| 75 |
+
"count": 100,
|
| 76 |
+
"accuracy": 0.99,
|
| 77 |
+
"brier": 0.014510678299897408,
|
| 78 |
+
"brier_soft": 0.014510678299897408,
|
| 79 |
+
"nll": 0.02477687623355849,
|
| 80 |
+
"nll_soft": 0.02477687623355849,
|
| 81 |
+
"ece": 0.016801298912001265,
|
| 82 |
+
"mean_confidence": 0.9887446982443209,
|
| 83 |
+
"confident_error_rate": 0.0
|
| 84 |
+
},
|
| 85 |
+
"dream": {
|
| 86 |
+
"count": 200,
|
| 87 |
+
"accuracy": 0.985,
|
| 88 |
+
"brier": 0.02921864351778212,
|
| 89 |
+
"brier_soft": 0.02921864351778212,
|
| 90 |
+
"nll": 0.08814026262617411,
|
| 91 |
+
"nll_soft": 0.08814026262617411,
|
| 92 |
+
"ece": 0.01292344301454983,
|
| 93 |
+
"mean_confidence": 0.9882869129162648,
|
| 94 |
+
"confident_error_rate": 0.01
|
| 95 |
+
},
|
| 96 |
+
"ethos": {
|
| 97 |
+
"count": 200,
|
| 98 |
+
"accuracy": 0.785,
|
| 99 |
+
"brier": 0.28935880012627835,
|
| 100 |
+
"brier_soft": 0.28935880012627835,
|
| 101 |
+
"nll": 0.43330864098692673,
|
| 102 |
+
"nll_soft": 0.43330864098692673,
|
| 103 |
+
"ece": 0.0902209003025014,
|
| 104 |
+
"mean_confidence": 0.8620058140147207,
|
| 105 |
+
"confident_error_rate": 0.04
|
| 106 |
+
},
|
| 107 |
+
"hh_rlhf": {
|
| 108 |
+
"count": 200,
|
| 109 |
+
"accuracy": 0.645,
|
| 110 |
+
"brier": 0.44173160731312494,
|
| 111 |
+
"brier_soft": 0.44173160731312494,
|
| 112 |
+
"nll": 0.6311810092807139,
|
| 113 |
+
"nll_soft": 0.6311810092807139,
|
| 114 |
+
"ece": 0.03958897205020696,
|
| 115 |
+
"mean_confidence": 0.6160584099275223,
|
| 116 |
+
"confident_error_rate": 0.0
|
| 117 |
+
},
|
| 118 |
+
"medical_questions_pairs": {
|
| 119 |
+
"count": 200,
|
| 120 |
+
"accuracy": 0.895,
|
| 121 |
+
"brier": 0.13780739465855052,
|
| 122 |
+
"brier_soft": 0.13780739465855052,
|
| 123 |
+
"nll": 0.21259324888263684,
|
| 124 |
+
"nll_soft": 0.21259324888263684,
|
| 125 |
+
"ece": 0.051844561382003836,
|
| 126 |
+
"mean_confidence": 0.9072524506750079,
|
| 127 |
+
"confident_error_rate": 0.005
|
| 128 |
+
},
|
| 129 |
+
"scitail": {
|
| 130 |
+
"count": 200,
|
| 131 |
+
"accuracy": 0.91,
|
| 132 |
+
"brier": 0.13208857504118351,
|
| 133 |
+
"brier_soft": 0.13208857504118351,
|
| 134 |
+
"nll": 0.22125867806726154,
|
| 135 |
+
"nll_soft": 0.22125867806726154,
|
| 136 |
+
"ece": 0.04096846269343061,
|
| 137 |
+
"mean_confidence": 0.9329582954663271,
|
| 138 |
+
"confident_error_rate": 0.025
|
| 139 |
+
},
|
| 140 |
+
"stance_climate": {
|
| 141 |
+
"count": 120,
|
| 142 |
+
"accuracy": 0.8833333333333333,
|
| 143 |
+
"brier": 0.1806405349295063,
|
| 144 |
+
"brier_soft": 0.1806405349295063,
|
| 145 |
+
"nll": 0.35196629973303767,
|
| 146 |
+
"nll_soft": 0.35196629973303767,
|
| 147 |
+
"ece": 0.10556056521146755,
|
| 148 |
+
"mean_confidence": 0.8141962073468899,
|
| 149 |
+
"confident_error_rate": 0.0
|
| 150 |
+
},
|
| 151 |
+
"truthful_qa": {
|
| 152 |
+
"count": 200,
|
| 153 |
+
"accuracy": 0.78,
|
| 154 |
+
"brier": 0.3291895080259756,
|
| 155 |
+
"brier_soft": 0.3291895080259756,
|
| 156 |
+
"nll": 0.7671485420602951,
|
| 157 |
+
"nll_soft": 0.7671485420602951,
|
| 158 |
+
"ece": 0.09806555881395083,
|
| 159 |
+
"mean_confidence": 0.8780655588139509,
|
| 160 |
+
"confident_error_rate": 0.05
|
| 161 |
+
},
|
| 162 |
+
"tweet_irony": {
|
| 163 |
+
"count": 200,
|
| 164 |
+
"accuracy": 0.83,
|
| 165 |
+
"brier": 0.2537469996145951,
|
| 166 |
+
"brier_soft": 0.2537469996145951,
|
| 167 |
+
"nll": 0.39800124916559576,
|
| 168 |
+
"nll_soft": 0.39800124916559576,
|
| 169 |
+
"ece": 0.06288414121493499,
|
| 170 |
+
"mean_confidence": 0.8390211895489462,
|
| 171 |
+
"confident_error_rate": 0.02
|
| 172 |
+
}
|
| 173 |
+
},
|
| 174 |
+
"reliability": [
|
| 175 |
+
{
|
| 176 |
+
"bin": "0.0-0.1",
|
| 177 |
+
"count": 0,
|
| 178 |
+
"confidence": NaN,
|
| 179 |
+
"accuracy": NaN
|
| 180 |
+
},
|
| 181 |
+
{
|
| 182 |
+
"bin": "0.1-0.2",
|
| 183 |
+
"count": 0,
|
| 184 |
+
"confidence": NaN,
|
| 185 |
+
"accuracy": NaN
|
| 186 |
+
},
|
| 187 |
+
{
|
| 188 |
+
"bin": "0.2-0.3",
|
| 189 |
+
"count": 4,
|
| 190 |
+
"confidence": 0.27240217902249636,
|
| 191 |
+
"accuracy": 0.0
|
| 192 |
+
},
|
| 193 |
+
{
|
| 194 |
+
"bin": "0.3-0.4",
|
| 195 |
+
"count": 32,
|
| 196 |
+
"confidence": 0.3696628771674506,
|
| 197 |
+
"accuracy": 0.46875
|
| 198 |
+
},
|
| 199 |
+
{
|
| 200 |
+
"bin": "0.4-0.5",
|
| 201 |
+
"count": 92,
|
| 202 |
+
"confidence": 0.4695831445360151,
|
| 203 |
+
"accuracy": 0.391304347826087
|
| 204 |
+
},
|
| 205 |
+
{
|
| 206 |
+
"bin": "0.5-0.6",
|
| 207 |
+
"count": 202,
|
| 208 |
+
"confidence": 0.5557999645214033,
|
| 209 |
+
"accuracy": 0.6138613861386139
|
| 210 |
+
},
|
| 211 |
+
{
|
| 212 |
+
"bin": "0.6-0.7",
|
| 213 |
+
"count": 147,
|
| 214 |
+
"confidence": 0.651499808775641,
|
| 215 |
+
"accuracy": 0.6938775510204082
|
| 216 |
+
},
|
| 217 |
+
{
|
| 218 |
+
"bin": "0.7-0.8",
|
| 219 |
+
"count": 176,
|
| 220 |
+
"confidence": 0.749253377021942,
|
| 221 |
+
"accuracy": 0.6647727272727273
|
| 222 |
+
},
|
| 223 |
+
{
|
| 224 |
+
"bin": "0.8-0.9",
|
| 225 |
+
"count": 200,
|
| 226 |
+
"confidence": 0.8559249921875981,
|
| 227 |
+
"accuracy": 0.845
|
| 228 |
+
},
|
| 229 |
+
{
|
| 230 |
+
"bin": "0.9-1.0",
|
| 231 |
+
"count": 1167,
|
| 232 |
+
"confidence": 0.9787594161484913,
|
| 233 |
+
"accuracy": 0.974293059125964
|
| 234 |
+
}
|
| 235 |
+
],
|
| 236 |
+
"seconds": 123.55612516403198,
|
| 237 |
+
"model": "runs/assay-27b",
|
| 238 |
+
"data": "data/v2/holdout.jsonl"
|
| 239 |
+
}
|
eval-transfer-v4-scaled.json
ADDED
|
@@ -0,0 +1,239 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
{
|
| 2 |
+
"overall": {
|
| 3 |
+
"count": 764,
|
| 4 |
+
"accuracy": 0.8416230366492147,
|
| 5 |
+
"brier": 0.228823902196938,
|
| 6 |
+
"brier_soft": 0.228823902196938,
|
| 7 |
+
"nll": 0.4303391753036349,
|
| 8 |
+
"nll_soft": 0.4303391753036349,
|
| 9 |
+
"ece": 0.041427708936302335,
|
| 10 |
+
"mean_confidence": 0.8759641062345186,
|
| 11 |
+
"confident_error_rate": 0.03795811518324607,
|
| 12 |
+
"score_mae": 0.08815105918628001
|
| 13 |
+
},
|
| 14 |
+
"by_type": {
|
| 15 |
+
"bool": {
|
| 16 |
+
"count": 280,
|
| 17 |
+
"accuracy": 0.85,
|
| 18 |
+
"brier": 0.20686426156907256,
|
| 19 |
+
"brier_soft": 0.20686426156907256,
|
| 20 |
+
"nll": 0.32621192520165093,
|
| 21 |
+
"nll_soft": 0.32621192520165093,
|
| 22 |
+
"ece": 0.06418264691914997,
|
| 23 |
+
"mean_confidence": 0.9034955801491882,
|
| 24 |
+
"confident_error_rate": 0.039285714285714285
|
| 25 |
+
},
|
| 26 |
+
"choice": {
|
| 27 |
+
"count": 444,
|
| 28 |
+
"accuracy": 0.8265765765765766,
|
| 29 |
+
"brier": 0.2590440644829713,
|
| 30 |
+
"brier_soft": 0.2590440644829713,
|
| 31 |
+
"nll": 0.5251947192633012,
|
| 32 |
+
"nll_soft": 0.5251947192633012,
|
| 33 |
+
"ece": 0.04839628894718921,
|
| 34 |
+
"mean_confidence": 0.854881037572032,
|
| 35 |
+
"confident_error_rate": 0.04054054054054054
|
| 36 |
+
},
|
| 37 |
+
"score": {
|
| 38 |
+
"count": 40,
|
| 39 |
+
"accuracy": 0.95,
|
| 40 |
+
"brier": 0.04709758521702534,
|
| 41 |
+
"brier_soft": 0.04709758521702534,
|
| 42 |
+
"nll": 0.10633338806522816,
|
| 43 |
+
"nll_soft": 0.10633338806522816,
|
| 44 |
+
"ece": 0.083569772392317,
|
| 45 |
+
"mean_confidence": 0.9172658509854346,
|
| 46 |
+
"confident_error_rate": 0.0,
|
| 47 |
+
"score_mae": 0.08815105918628001
|
| 48 |
+
}
|
| 49 |
+
},
|
| 50 |
+
"by_source": {
|
| 51 |
+
"composition_held_and_or": {
|
| 52 |
+
"count": 32,
|
| 53 |
+
"accuracy": 1.0,
|
| 54 |
+
"brier": 0.0024598475826333784,
|
| 55 |
+
"brier_soft": 0.0024598475826333784,
|
| 56 |
+
"nll": 0.012256508487456694,
|
| 57 |
+
"nll_soft": 0.012256508487456694,
|
| 58 |
+
"ece": 0.011590125121385776,
|
| 59 |
+
"mean_confidence": 0.9884098748786143,
|
| 60 |
+
"confident_error_rate": 0.0
|
| 61 |
+
},
|
| 62 |
+
"composition_held_conditional": {
|
| 63 |
+
"count": 32,
|
| 64 |
+
"accuracy": 0.8125,
|
| 65 |
+
"brier": 0.34715965230721574,
|
| 66 |
+
"brier_soft": 0.34715965230721574,
|
| 67 |
+
"nll": 0.6216016756942785,
|
| 68 |
+
"nll_soft": 0.6216016756942785,
|
| 69 |
+
"ece": 0.17664373260319421,
|
| 70 |
+
"mean_confidence": 0.9769288116292272,
|
| 71 |
+
"confident_error_rate": 0.1875
|
| 72 |
+
},
|
| 73 |
+
"composition_held_or_not": {
|
| 74 |
+
"count": 32,
|
| 75 |
+
"accuracy": 0.90625,
|
| 76 |
+
"brier": 0.168494398454657,
|
| 77 |
+
"brier_soft": 0.168494398454657,
|
| 78 |
+
"nll": 0.27925353580760964,
|
| 79 |
+
"nll_soft": 0.27925353580760964,
|
| 80 |
+
"ece": 0.08669154028980341,
|
| 81 |
+
"mean_confidence": 0.9929415402898034,
|
| 82 |
+
"confident_error_rate": 0.09375
|
| 83 |
+
},
|
| 84 |
+
"contrastive_authorization": {
|
| 85 |
+
"count": 40,
|
| 86 |
+
"accuracy": 1.0,
|
| 87 |
+
"brier": 1.9275568936614584e-05,
|
| 88 |
+
"brier_soft": 1.9275568936614584e-05,
|
| 89 |
+
"nll": 0.0030790094590994214,
|
| 90 |
+
"nll_soft": 0.0030790094590994214,
|
| 91 |
+
"ece": 0.0030741802684094433,
|
| 92 |
+
"mean_confidence": 0.9969258197315906,
|
| 93 |
+
"confident_error_rate": 0.0
|
| 94 |
+
},
|
| 95 |
+
"contrastive_deadline": {
|
| 96 |
+
"count": 40,
|
| 97 |
+
"accuracy": 0.95,
|
| 98 |
+
"brier": 0.04709758521702534,
|
| 99 |
+
"brier_soft": 0.04709758521702534,
|
| 100 |
+
"nll": 0.10633338806522816,
|
| 101 |
+
"nll_soft": 0.10633338806522816,
|
| 102 |
+
"ece": 0.083569772392317,
|
| 103 |
+
"mean_confidence": 0.9172658509854346,
|
| 104 |
+
"confident_error_rate": 0.0,
|
| 105 |
+
"score_mae": 0.08815105918628001
|
| 106 |
+
},
|
| 107 |
+
"emotion": {
|
| 108 |
+
"count": 116,
|
| 109 |
+
"accuracy": 0.646551724137931,
|
| 110 |
+
"brier": 0.47781526614581615,
|
| 111 |
+
"brier_soft": 0.47781526614581615,
|
| 112 |
+
"nll": 0.919468529423018,
|
| 113 |
+
"nll_soft": 0.919468529423018,
|
| 114 |
+
"ece": 0.07703300800806251,
|
| 115 |
+
"mean_confidence": 0.599839918781091,
|
| 116 |
+
"confident_error_rate": 0.008620689655172414
|
| 117 |
+
},
|
| 118 |
+
"mmlu": {
|
| 119 |
+
"count": 116,
|
| 120 |
+
"accuracy": 0.7844827586206896,
|
| 121 |
+
"brier": 0.33259283968426123,
|
| 122 |
+
"brier_soft": 0.33259283968426123,
|
| 123 |
+
"nll": 0.7725269861278372,
|
| 124 |
+
"nll_soft": 0.7725269861278372,
|
| 125 |
+
"ece": 0.11853164362557794,
|
| 126 |
+
"mean_confidence": 0.8894832273685209,
|
| 127 |
+
"confident_error_rate": 0.06896551724137931
|
| 128 |
+
},
|
| 129 |
+
"paws": {
|
| 130 |
+
"count": 80,
|
| 131 |
+
"accuracy": 0.775,
|
| 132 |
+
"brier": 0.3309925042360514,
|
| 133 |
+
"brier_soft": 0.3309925042360514,
|
| 134 |
+
"nll": 0.541684319155395,
|
| 135 |
+
"nll_soft": 0.541684319155395,
|
| 136 |
+
"ece": 0.15122859544283523,
|
| 137 |
+
"mean_confidence": 0.904908261708847,
|
| 138 |
+
"confident_error_rate": 0.1125
|
| 139 |
+
},
|
| 140 |
+
"qnli": {
|
| 141 |
+
"count": 80,
|
| 142 |
+
"accuracy": 0.9625,
|
| 143 |
+
"brier": 0.06201026023343834,
|
| 144 |
+
"brier_soft": 0.06201026023343834,
|
| 145 |
+
"nll": 0.11861526966281896,
|
| 146 |
+
"nll_soft": 0.11861526966281896,
|
| 147 |
+
"ece": 0.0549367479773906,
|
| 148 |
+
"mean_confidence": 0.9167776334313775,
|
| 149 |
+
"confident_error_rate": 0.0
|
| 150 |
+
},
|
| 151 |
+
"sciq": {
|
| 152 |
+
"count": 116,
|
| 153 |
+
"accuracy": 0.9827586206896551,
|
| 154 |
+
"brier": 0.038177410406259445,
|
| 155 |
+
"brier_soft": 0.038177410406259445,
|
| 156 |
+
"nll": 0.06633965956334043,
|
| 157 |
+
"nll_soft": 0.06633965956334043,
|
| 158 |
+
"ece": 0.03355711460065191,
|
| 159 |
+
"mean_confidence": 0.9667304178543323,
|
| 160 |
+
"confident_error_rate": 0.0
|
| 161 |
+
},
|
| 162 |
+
"tweet_offensive": {
|
| 163 |
+
"count": 80,
|
| 164 |
+
"accuracy": 0.7375,
|
| 165 |
+
"brier": 0.331012513237796,
|
| 166 |
+
"brier_soft": 0.331012513237796,
|
| 167 |
+
"nll": 0.4799026446580147,
|
| 168 |
+
"nll_soft": 0.4799026446580147,
|
| 169 |
+
"ece": 0.15083210595826038,
|
| 170 |
+
"mean_confidence": 0.8420857255161387,
|
| 171 |
+
"confident_error_rate": 0.025
|
| 172 |
+
}
|
| 173 |
+
},
|
| 174 |
+
"reliability": [
|
| 175 |
+
{
|
| 176 |
+
"bin": "0.0-0.1",
|
| 177 |
+
"count": 0,
|
| 178 |
+
"confidence": NaN,
|
| 179 |
+
"accuracy": NaN
|
| 180 |
+
},
|
| 181 |
+
{
|
| 182 |
+
"bin": "0.1-0.2",
|
| 183 |
+
"count": 0,
|
| 184 |
+
"confidence": NaN,
|
| 185 |
+
"accuracy": NaN
|
| 186 |
+
},
|
| 187 |
+
{
|
| 188 |
+
"bin": "0.2-0.3",
|
| 189 |
+
"count": 1,
|
| 190 |
+
"confidence": 0.2863755673514975,
|
| 191 |
+
"accuracy": 0.0
|
| 192 |
+
},
|
| 193 |
+
{
|
| 194 |
+
"bin": "0.3-0.4",
|
| 195 |
+
"count": 12,
|
| 196 |
+
"confidence": 0.3548456743596838,
|
| 197 |
+
"accuracy": 0.5833333333333334
|
| 198 |
+
},
|
| 199 |
+
{
|
| 200 |
+
"bin": "0.4-0.5",
|
| 201 |
+
"count": 30,
|
| 202 |
+
"confidence": 0.4537790778486413,
|
| 203 |
+
"accuracy": 0.4666666666666667
|
| 204 |
+
},
|
| 205 |
+
{
|
| 206 |
+
"bin": "0.5-0.6",
|
| 207 |
+
"count": 53,
|
| 208 |
+
"confidence": 0.5532438982939633,
|
| 209 |
+
"accuracy": 0.5283018867924528
|
| 210 |
+
},
|
| 211 |
+
{
|
| 212 |
+
"bin": "0.6-0.7",
|
| 213 |
+
"count": 40,
|
| 214 |
+
"confidence": 0.6630960879374065,
|
| 215 |
+
"accuracy": 0.575
|
| 216 |
+
},
|
| 217 |
+
{
|
| 218 |
+
"bin": "0.7-0.8",
|
| 219 |
+
"count": 50,
|
| 220 |
+
"confidence": 0.749056549882067,
|
| 221 |
+
"accuracy": 0.7
|
| 222 |
+
},
|
| 223 |
+
{
|
| 224 |
+
"bin": "0.8-0.9",
|
| 225 |
+
"count": 59,
|
| 226 |
+
"confidence": 0.8574375075804922,
|
| 227 |
+
"accuracy": 0.7796610169491526
|
| 228 |
+
},
|
| 229 |
+
{
|
| 230 |
+
"bin": "0.9-1.0",
|
| 231 |
+
"count": 519,
|
| 232 |
+
"confidence": 0.9772471495175656,
|
| 233 |
+
"accuracy": 0.9441233140655106
|
| 234 |
+
}
|
| 235 |
+
],
|
| 236 |
+
"seconds": 40.363425970077515,
|
| 237 |
+
"model": "runs/assay-27b",
|
| 238 |
+
"data": "data/suites/kev-transfer-v4-dev.jsonl"
|
| 239 |
+
}
|
eval-transfer-v4.json
ADDED
|
@@ -0,0 +1,239 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
{
|
| 2 |
+
"overall": {
|
| 3 |
+
"count": 764,
|
| 4 |
+
"accuracy": 0.8416230366492147,
|
| 5 |
+
"brier": 0.23366786332576134,
|
| 6 |
+
"brier_soft": 0.23366786332576134,
|
| 7 |
+
"nll": 0.45999593150161044,
|
| 8 |
+
"nll_soft": 0.45999593150161044,
|
| 9 |
+
"ece": 0.06523326879131716,
|
| 10 |
+
"mean_confidence": 0.9013523079735777,
|
| 11 |
+
"confident_error_rate": 0.04712041884816754,
|
| 12 |
+
"score_mae": 0.06568647111125461
|
| 13 |
+
},
|
| 14 |
+
"by_type": {
|
| 15 |
+
"bool": {
|
| 16 |
+
"count": 280,
|
| 17 |
+
"accuracy": 0.85,
|
| 18 |
+
"brier": 0.21610131477479474,
|
| 19 |
+
"brier_soft": 0.21610131477479474,
|
| 20 |
+
"nll": 0.3515884562588345,
|
| 21 |
+
"nll_soft": 0.3515884562588345,
|
| 22 |
+
"ece": 0.08110998832325675,
|
| 23 |
+
"mean_confidence": 0.9250817518080554,
|
| 24 |
+
"confident_error_rate": 0.05714285714285714
|
| 25 |
+
},
|
| 26 |
+
"choice": {
|
| 27 |
+
"count": 444,
|
| 28 |
+
"accuracy": 0.8265765765765766,
|
| 29 |
+
"brier": 0.2619705373448098,
|
| 30 |
+
"brier_soft": 0.2619705373448098,
|
| 31 |
+
"nll": 0.5623854204187092,
|
| 32 |
+
"nll_soft": 0.5623854204187092,
|
| 33 |
+
"ece": 0.07234439122776523,
|
| 34 |
+
"mean_confidence": 0.8829180473388407,
|
| 35 |
+
"confident_error_rate": 0.04504504504504504
|
| 36 |
+
},
|
| 37 |
+
"score": {
|
| 38 |
+
"count": 40,
|
| 39 |
+
"accuracy": 0.95,
|
| 40 |
+
"brier": 0.04247402157108923,
|
| 41 |
+
"brier_soft": 0.04247402157108923,
|
| 42 |
+
"nll": 0.08232493122124444,
|
| 43 |
+
"nll_soft": 0.08232493122124444,
|
| 44 |
+
"ece": 0.06295530532303786,
|
| 45 |
+
"mean_confidence": 0.9398664941778165,
|
| 46 |
+
"confident_error_rate": 0.0,
|
| 47 |
+
"score_mae": 0.06568647111125461
|
| 48 |
+
}
|
| 49 |
+
},
|
| 50 |
+
"by_source": {
|
| 51 |
+
"composition_held_and_or": {
|
| 52 |
+
"count": 32,
|
| 53 |
+
"accuracy": 1.0,
|
| 54 |
+
"brier": 0.0010265396372586473,
|
| 55 |
+
"brier_soft": 0.0010265396372586473,
|
| 56 |
+
"nll": 0.00744750174425,
|
| 57 |
+
"nll_soft": 0.00744750174425,
|
| 58 |
+
"ece": 0.007177238579851596,
|
| 59 |
+
"mean_confidence": 0.9928227614201484,
|
| 60 |
+
"confident_error_rate": 0.0
|
| 61 |
+
},
|
| 62 |
+
"composition_held_conditional": {
|
| 63 |
+
"count": 32,
|
| 64 |
+
"accuracy": 0.8125,
|
| 65 |
+
"brier": 0.3609123301012935,
|
| 66 |
+
"brier_soft": 0.3609123301012935,
|
| 67 |
+
"nll": 0.750269028687234,
|
| 68 |
+
"nll_soft": 0.750269028687234,
|
| 69 |
+
"ece": 0.17994145932455827,
|
| 70 |
+
"mean_confidence": 0.9882197907318171,
|
| 71 |
+
"confident_error_rate": 0.1875
|
| 72 |
+
},
|
| 73 |
+
"composition_held_or_not": {
|
| 74 |
+
"count": 32,
|
| 75 |
+
"accuracy": 0.90625,
|
| 76 |
+
"brier": 0.17745481693255763,
|
| 77 |
+
"brier_soft": 0.17745481693255763,
|
| 78 |
+
"nll": 0.33929718078098925,
|
| 79 |
+
"nll_soft": 0.33929718078098925,
|
| 80 |
+
"ece": 0.0904297754653719,
|
| 81 |
+
"mean_confidence": 0.9966797754653719,
|
| 82 |
+
"confident_error_rate": 0.09375
|
| 83 |
+
},
|
| 84 |
+
"contrastive_authorization": {
|
| 85 |
+
"count": 40,
|
| 86 |
+
"accuracy": 1.0,
|
| 87 |
+
"brier": 1.301873731350463e-06,
|
| 88 |
+
"brier_soft": 1.301873731350463e-06,
|
| 89 |
+
"nll": 0.0007950641143637151,
|
| 90 |
+
"nll_soft": 0.0007950641143637151,
|
| 91 |
+
"ece": 0.0007947384626161291,
|
| 92 |
+
"mean_confidence": 0.9992052615373839,
|
| 93 |
+
"confident_error_rate": 0.0
|
| 94 |
+
},
|
| 95 |
+
"contrastive_deadline": {
|
| 96 |
+
"count": 40,
|
| 97 |
+
"accuracy": 0.95,
|
| 98 |
+
"brier": 0.04247402157108923,
|
| 99 |
+
"brier_soft": 0.04247402157108923,
|
| 100 |
+
"nll": 0.08232493122124444,
|
| 101 |
+
"nll_soft": 0.08232493122124444,
|
| 102 |
+
"ece": 0.06295530532303786,
|
| 103 |
+
"mean_confidence": 0.9398664941778165,
|
| 104 |
+
"confident_error_rate": 0.0,
|
| 105 |
+
"score_mae": 0.06568647111125461
|
| 106 |
+
},
|
| 107 |
+
"emotion": {
|
| 108 |
+
"count": 116,
|
| 109 |
+
"accuracy": 0.646551724137931,
|
| 110 |
+
"brier": 0.47142674905119664,
|
| 111 |
+
"brier_soft": 0.47142674905119664,
|
| 112 |
+
"nll": 0.9009041198087586,
|
| 113 |
+
"nll_soft": 0.9009041198087586,
|
| 114 |
+
"ece": 0.0875629354087716,
|
| 115 |
+
"mean_confidence": 0.6661347203870895,
|
| 116 |
+
"confident_error_rate": 0.017241379310344827
|
| 117 |
+
},
|
| 118 |
+
"mmlu": {
|
| 119 |
+
"count": 116,
|
| 120 |
+
"accuracy": 0.7844827586206896,
|
| 121 |
+
"brier": 0.34407690962026555,
|
| 122 |
+
"brier_soft": 0.34407690962026555,
|
| 123 |
+
"nll": 0.8865278702283488,
|
| 124 |
+
"nll_soft": 0.8865278702283488,
|
| 125 |
+
"ece": 0.14835926317316953,
|
| 126 |
+
"mean_confidence": 0.917652102352729,
|
| 127 |
+
"confident_error_rate": 0.07758620689655173
|
| 128 |
+
},
|
| 129 |
+
"paws": {
|
| 130 |
+
"count": 80,
|
| 131 |
+
"accuracy": 0.775,
|
| 132 |
+
"brier": 0.34762263713242636,
|
| 133 |
+
"brier_soft": 0.34762263713242636,
|
| 134 |
+
"nll": 0.6147711331747864,
|
| 135 |
+
"nll_soft": 0.6147711331747864,
|
| 136 |
+
"ece": 0.15929430289975277,
|
| 137 |
+
"mean_confidence": 0.9307858966627347,
|
| 138 |
+
"confident_error_rate": 0.15
|
| 139 |
+
},
|
| 140 |
+
"qnli": {
|
| 141 |
+
"count": 80,
|
| 142 |
+
"accuracy": 0.9625,
|
| 143 |
+
"brier": 0.05830143324049297,
|
| 144 |
+
"brier_soft": 0.05830143324049297,
|
| 145 |
+
"nll": 0.10144160569398128,
|
| 146 |
+
"nll_soft": 0.10144160569398128,
|
| 147 |
+
"ece": 0.05166224596877384,
|
| 148 |
+
"mean_confidence": 0.9363518495172769,
|
| 149 |
+
"confident_error_rate": 0.0
|
| 150 |
+
},
|
| 151 |
+
"sciq": {
|
| 152 |
+
"count": 116,
|
| 153 |
+
"accuracy": 0.9827586206896551,
|
| 154 |
+
"brier": 0.03841289829077971,
|
| 155 |
+
"brier_soft": 0.03841289829077971,
|
| 156 |
+
"nll": 0.06252221605864929,
|
| 157 |
+
"nll_soft": 0.06252221605864929,
|
| 158 |
+
"ece": 0.03045631037515725,
|
| 159 |
+
"mean_confidence": 0.9742174749730302,
|
| 160 |
+
"confident_error_rate": 0.0
|
| 161 |
+
},
|
| 162 |
+
"tweet_offensive": {
|
| 163 |
+
"count": 80,
|
| 164 |
+
"accuracy": 0.7375,
|
| 165 |
+
"brier": 0.3504298804019966,
|
| 166 |
+
"brier_soft": 0.3504298804019966,
|
| 167 |
+
"nll": 0.5139493259799714,
|
| 168 |
+
"nll_soft": 0.5139493259799714,
|
| 169 |
+
"ece": 0.15698528550077231,
|
| 170 |
+
"mean_confidence": 0.8710457543794903,
|
| 171 |
+
"confident_error_rate": 0.05
|
| 172 |
+
}
|
| 173 |
+
},
|
| 174 |
+
"reliability": [
|
| 175 |
+
{
|
| 176 |
+
"bin": "0.0-0.1",
|
| 177 |
+
"count": 0,
|
| 178 |
+
"confidence": NaN,
|
| 179 |
+
"accuracy": NaN
|
| 180 |
+
},
|
| 181 |
+
{
|
| 182 |
+
"bin": "0.1-0.2",
|
| 183 |
+
"count": 0,
|
| 184 |
+
"confidence": NaN,
|
| 185 |
+
"accuracy": NaN
|
| 186 |
+
},
|
| 187 |
+
{
|
| 188 |
+
"bin": "0.2-0.3",
|
| 189 |
+
"count": 0,
|
| 190 |
+
"confidence": NaN,
|
| 191 |
+
"accuracy": NaN
|
| 192 |
+
},
|
| 193 |
+
{
|
| 194 |
+
"bin": "0.3-0.4",
|
| 195 |
+
"count": 9,
|
| 196 |
+
"confidence": 0.3614360119116289,
|
| 197 |
+
"accuracy": 0.5555555555555556
|
| 198 |
+
},
|
| 199 |
+
{
|
| 200 |
+
"bin": "0.4-0.5",
|
| 201 |
+
"count": 19,
|
| 202 |
+
"confidence": 0.4628657382665659,
|
| 203 |
+
"accuracy": 0.47368421052631576
|
| 204 |
+
},
|
| 205 |
+
{
|
| 206 |
+
"bin": "0.5-0.6",
|
| 207 |
+
"count": 39,
|
| 208 |
+
"confidence": 0.5494683944332894,
|
| 209 |
+
"accuracy": 0.46153846153846156
|
| 210 |
+
},
|
| 211 |
+
{
|
| 212 |
+
"bin": "0.6-0.7",
|
| 213 |
+
"count": 36,
|
| 214 |
+
"confidence": 0.6379403082210656,
|
| 215 |
+
"accuracy": 0.5555555555555556
|
| 216 |
+
},
|
| 217 |
+
{
|
| 218 |
+
"bin": "0.7-0.8",
|
| 219 |
+
"count": 48,
|
| 220 |
+
"confidence": 0.7524977237611029,
|
| 221 |
+
"accuracy": 0.6041666666666666
|
| 222 |
+
},
|
| 223 |
+
{
|
| 224 |
+
"bin": "0.8-0.9",
|
| 225 |
+
"count": 61,
|
| 226 |
+
"confidence": 0.855432037802958,
|
| 227 |
+
"accuracy": 0.7540983606557377
|
| 228 |
+
},
|
| 229 |
+
{
|
| 230 |
+
"bin": "0.9-1.0",
|
| 231 |
+
"count": 552,
|
| 232 |
+
"confidence": 0.9853069323046632,
|
| 233 |
+
"accuracy": 0.9347826086956522
|
| 234 |
+
}
|
| 235 |
+
],
|
| 236 |
+
"seconds": 40.56191968917847,
|
| 237 |
+
"model": "runs/assay-27b",
|
| 238 |
+
"data": "data/suites/kev-transfer-v4-dev.jsonl"
|
| 239 |
+
}
|
train_args.json
ADDED
|
@@ -0,0 +1,27 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
{
|
| 2 |
+
"base": "Qwen/Qwen3.8-27B",
|
| 3 |
+
"data": "data/v2",
|
| 4 |
+
"out": "runs/assay-27b",
|
| 5 |
+
"lora_r": 16,
|
| 6 |
+
"lora_alpha": 32,
|
| 7 |
+
"lr": 5e-05,
|
| 8 |
+
"head_lr": 0.001,
|
| 9 |
+
"epochs": 1.0,
|
| 10 |
+
"batch_size": 4,
|
| 11 |
+
"grad_accum": 2,
|
| 12 |
+
"warmup": 0.03,
|
| 13 |
+
"weight_decay": 0.0,
|
| 14 |
+
"evidence_weight": 0.5,
|
| 15 |
+
"score_sigma": 0.5,
|
| 16 |
+
"hard_targets": false,
|
| 17 |
+
"max_state_tokens": 1024,
|
| 18 |
+
"limit": null,
|
| 19 |
+
"eval_every": 0,
|
| 20 |
+
"log_every": 100,
|
| 21 |
+
"seed": 0,
|
| 22 |
+
"no_gradient_checkpointing": false,
|
| 23 |
+
"checkpoint_every": 500,
|
| 24 |
+
"resume": true,
|
| 25 |
+
"quant": "4bit",
|
| 26 |
+
"eval_batch_size": 4
|
| 27 |
+
}
|
train_log.jsonl
ADDED
|
@@ -0,0 +1,140 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
{"step": 100, "update": 50, "epoch": 0, "loss": 1.0466632970422507, "answer_loss": 0.9093208994437009, "evidence_loss": 0.2746847915649414, "lr": 1.2561576354679805e-05, "elapsed": 154.3350830078125}
|
| 2 |
+
{"step": 200, "update": 100, "epoch": 0, "loss": 0.7416581735294312, "answer_loss": 0.6910765176918358, "evidence_loss": 0.10116331100463867, "lr": 2.4876847290640397e-05, "elapsed": 257.4809904098511}
|
| 3 |
+
{"step": 300, "update": 150, "epoch": 0, "loss": 0.6601098832546267, "answer_loss": 0.6098437773936893, "evidence_loss": 0.10053220888599754, "lr": 3.719211822660099e-05, "elapsed": 371.6660888195038}
|
| 4 |
+
{"step": 400, "update": 200, "epoch": 0, "loss": 0.7968255157326348, "answer_loss": 0.7370119733747561, "evidence_loss": 0.11962707860395312, "lr": 4.950738916256158e-05, "elapsed": 487.11128091812134}
|
| 5 |
+
{"step": 500, "update": 250, "epoch": 0, "loss": 0.5190051701292395, "answer_loss": 0.5003233546391129, "evidence_loss": 0.03736363284289837, "lr": 4.999372686422346e-05, "elapsed": 602.2875165939331}
|
| 6 |
+
{"step": 600, "update": 300, "epoch": 0, "loss": 0.7238669737521559, "answer_loss": 0.7120701452810317, "evidence_loss": 0.02359365880023688, "lr": 4.997328388910923e-05, "elapsed": 721.729451417923}
|
| 7 |
+
{"step": 700, "update": 350, "epoch": 0, "loss": 0.5110658522252924, "answer_loss": 0.501890649611596, "evidence_loss": 0.018350402312353253, "lr": 4.993865712608182e-05, "elapsed": 826.3627226352692}
|
| 8 |
+
{"step": 800, "update": 400, "epoch": 0, "loss": 0.707598408163758, "answer_loss": 0.6938729958058684, "evidence_loss": 0.027450826987624168, "lr": 4.9889866241704306e-05, "elapsed": 933.359840631485}
|
| 9 |
+
{"step": 900, "update": 450, "epoch": 0, "loss": 0.5563830675184727, "answer_loss": 0.5391353450086899, "evidence_loss": 0.03449544358138155, "lr": 4.982693894717237e-05, "elapsed": 1040.5720059871674}
|
| 10 |
+
{"step": 1000, "update": 500, "epoch": 0, "loss": 0.5940256051276811, "answer_loss": 0.5801869906159118, "evidence_loss": 0.02767722815130867, "lr": 4.97499109825754e-05, "elapsed": 1145.7536494731903}
|
| 11 |
+
{"step": 1100, "update": 550, "epoch": 0, "loss": 0.5344747617281973, "answer_loss": 0.513588702082634, "evidence_loss": 0.04177212506803585, "lr": 4.965882609659774e-05, "elapsed": 1275.479078054428}
|
| 12 |
+
{"step": 1200, "update": 600, "epoch": 0, "loss": 0.5738527677254751, "answer_loss": 0.572868535448797, "evidence_loss": 0.0019684605054499115, "lr": 4.955373602167117e-05, "elapsed": 1361.5137095451355}
|
| 13 |
+
{"step": 1300, "update": 650, "epoch": 0, "loss": 0.6228927255561575, "answer_loss": 0.6063712534727529, "evidence_loss": 0.033042942728825436, "lr": 4.943470044459301e-05, "elapsed": 1467.5046110153198}
|
| 14 |
+
{"step": 1400, "update": 700, "epoch": 0, "loss": 0.5342604766273871, "answer_loss": 0.5312385475146584, "evidence_loss": 0.0060438554310348995, "lr": 4.930178697262657e-05, "elapsed": 1567.4875328540802}
|
| 15 |
+
{"step": 1500, "update": 750, "epoch": 0, "loss": 0.48719982599723155, "answer_loss": 0.47769132474437354, "evidence_loss": 0.019017004827378513, "lr": 4.9155071095103e-05, "elapsed": 1676.7254557609558}
|
| 16 |
+
{"step": 1600, "update": 800, "epoch": 0, "loss": 0.5404521024005953, "answer_loss": 0.528679445082089, "evidence_loss": 0.02354531695116606, "lr": 4.899463614054647e-05, "elapsed": 1792.2951774597168}
|
| 17 |
+
{"step": 1700, "update": 850, "epoch": 0, "loss": 0.5006524209631606, "answer_loss": 0.5006182217225432, "evidence_loss": 6.839833200501744e-05, "lr": 4.882057322934704e-05, "elapsed": 1886.2667684555054}
|
| 18 |
+
{"step": 1800, "update": 900, "epoch": 0, "loss": 0.5758555604936555, "answer_loss": 0.539689148021862, "evidence_loss": 0.0723328263185147, "lr": 4.863298122200809e-05, "elapsed": 2002.5272870063782}
|
| 19 |
+
{"step": 1900, "update": 950, "epoch": 0, "loss": 0.49814286177512257, "answer_loss": 0.4793539487721864, "evidence_loss": 0.03757782667705669, "lr": 4.843196666299762e-05, "elapsed": 2123.8212616443634}
|
| 20 |
+
{"step": 2000, "update": 1000, "epoch": 0, "loss": 0.5743265111767687, "answer_loss": 0.5593323490326293, "evidence_loss": 0.02998832721196475, "lr": 4.821764372023544e-05, "elapsed": 2229.363822221756}
|
| 21 |
+
{"step": 2100, "update": 1050, "epoch": 0, "loss": 0.4775368913402781, "answer_loss": 0.4539963689164142, "evidence_loss": 0.04708105071846186, "lr": 4.799013412025052e-05, "elapsed": 2349.06672334671}
|
| 22 |
+
{"step": 2200, "update": 1100, "epoch": 0, "loss": 0.5363413573545404, "answer_loss": 0.5363412640371825, "evidence_loss": 1.952083284395556e-07, "lr": 4.7749567079045424e-05, "elapsed": 2445.444249868393}
|
| 23 |
+
{"step": 2300, "update": 1150, "epoch": 0, "loss": 0.4746944914071355, "answer_loss": 0.4746943751501385, "evidence_loss": 2.3482379049477586e-07, "lr": 4.74960792287069e-05, "elapsed": 2552.2553861141205}
|
| 24 |
+
{"step": 2400, "update": 1200, "epoch": 0, "loss": 0.5232260901574045, "answer_loss": 0.5228329078108073, "evidence_loss": 0.0007863650698963553, "lr": 4.722981453980462e-05, "elapsed": 2657.0607981681824}
|
| 25 |
+
{"step": 2500, "update": 1250, "epoch": 0, "loss": 0.5513794144941494, "answer_loss": 0.533010982836131, "evidence_loss": 0.0367368643773321, "lr": 4.695092423962183e-05, "elapsed": 2767.573856115341}
|
| 26 |
+
{"step": 2600, "update": 1300, "epoch": 0, "loss": 0.5735468521388247, "answer_loss": 0.5634960023453459, "evidence_loss": 0.020101699072139424, "lr": 4.6659566726264524e-05, "elapsed": 2877.4249551296234}
|
| 27 |
+
{"step": 2700, "update": 1350, "epoch": 0, "loss": 0.42455267066659874, "answer_loss": 0.40903420170943716, "evidence_loss": 0.031036934792302347, "lr": 4.635590747869793e-05, "elapsed": 2994.5280821323395}
|
| 28 |
+
{"step": 2800, "update": 1400, "epoch": 0, "loss": 0.48469938746362456, "answer_loss": 0.48457985530549197, "evidence_loss": 0.00023906831981600261, "lr": 4.604011896276135e-05, "elapsed": 3103.987872838974}
|
| 29 |
+
{"step": 2900, "update": 1450, "epoch": 0, "loss": 0.5344389405541006, "answer_loss": 0.530350395317364, "evidence_loss": 0.008177093630389045, "lr": 4.571238053321464e-05, "elapsed": 3224.394252538681}
|
| 30 |
+
{"step": 3000, "update": 1500, "epoch": 0, "loss": 0.5819819614558946, "answer_loss": 0.5376607599260751, "evidence_loss": 0.08864240093117996, "lr": 4.5372878331872246e-05, "elapsed": 3350.2840735912323}
|
| 31 |
+
{"step": 3100, "update": 1550, "epoch": 0, "loss": 0.4642395470815245, "answer_loss": 0.46400510797160677, "evidence_loss": 0.00046887970986347227, "lr": 4.502180518188236e-05, "elapsed": 3472.488645553589}
|
| 32 |
+
{"step": 3200, "update": 1600, "epoch": 0, "loss": 0.5830743358284235, "answer_loss": 0.5706681804778054, "evidence_loss": 0.02481231747850984, "lr": 4.4659360478211355e-05, "elapsed": 3590.0721051692963}
|
| 33 |
+
{"step": 3300, "update": 1650, "epoch": 0, "loss": 0.5431933886697515, "answer_loss": 0.5277707322640345, "evidence_loss": 0.03084531450773291, "lr": 4.428575007439575e-05, "elapsed": 3691.8125829696655}
|
| 34 |
+
{"step": 3400, "update": 1700, "epoch": 0, "loss": 0.5290522611211054, "answer_loss": 0.5236124264192767, "evidence_loss": 0.010879670253632314, "lr": 4.390118616562595e-05, "elapsed": 3807.3116776943207}
|
| 35 |
+
{"step": 3500, "update": 1750, "epoch": 0, "loss": 0.48495657499530354, "answer_loss": 0.4821242574881762, "evidence_loss": 0.005664636454030691, "lr": 4.35058871682282e-05, "elapsed": 3912.8884415626526}
|
| 36 |
+
{"step": 3600, "update": 1800, "epoch": 0, "loss": 0.4668302389397286, "answer_loss": 0.46674671555054376, "evidence_loss": 0.00016704751175979028, "lr": 4.3100077595613146e-05, "elapsed": 4032.690903902054}
|
| 37 |
+
{"step": 3700, "update": 1850, "epoch": 0, "loss": 0.5461908026458695, "answer_loss": 0.5188855572417378, "evidence_loss": 0.054610492039893, "lr": 4.2683987930761596e-05, "elapsed": 4135.709332227707}
|
| 38 |
+
{"step": 3800, "update": 1900, "epoch": 0, "loss": 0.46654491690685973, "answer_loss": 0.4664767611282878, "evidence_loss": 0.0001363158306709411, "lr": 4.225785449531973e-05, "elapsed": 4234.659541130066}
|
| 39 |
+
{"step": 3900, "update": 1950, "epoch": 0, "loss": 0.42730378168082095, "answer_loss": 0.4273031175887445, "evidence_loss": 1.3314065834246768e-06, "lr": 4.182191931537822e-05, "elapsed": 4346.67816734314}
|
| 40 |
+
{"step": 4000, "update": 2000, "epoch": 0, "loss": 0.5631801752815955, "answer_loss": 0.5631604929215973, "evidence_loss": 3.936643363275705e-05, "lr": 4.137642998401147e-05, "elapsed": 4453.173367977142}
|
| 41 |
+
{"step": 4100, "update": 2050, "epoch": 0, "loss": 0.47845682796556505, "answer_loss": 0.45774981057271363, "evidence_loss": 0.041414034293519364, "lr": 4.0921639520655043e-05, "elapsed": 4547.15971159935}
|
| 42 |
+
{"step": 4200, "update": 2100, "epoch": 0, "loss": 0.5083158849657048, "answer_loss": 0.47591323538741565, "evidence_loss": 0.06480530041098546, "lr": 4.045780622740113e-05, "elapsed": 4666.14689540863}
|
| 43 |
+
{"step": 4300, "update": 2150, "epoch": 0, "loss": 0.492032068799017, "answer_loss": 0.46687227254151364, "evidence_loss": 0.050319592165840844, "lr": 3.998519354229365e-05, "elapsed": 4782.390250205994}
|
| 44 |
+
{"step": 4100, "update": 2050, "epoch": 0, "loss": 0.4950320903561078, "answer_loss": 0.4731320877978578, "evidence_loss": 0.04380001190301136, "lr": 4.0921639520655043e-05, "elapsed": 93.81770753860474}
|
| 45 |
+
{"step": 4200, "update": 2100, "epoch": 0, "loss": 0.5207801886694505, "answer_loss": 0.48821464135311543, "evidence_loss": 0.06513109242844621, "lr": 4.045780622740113e-05, "elapsed": 212.7313916683197}
|
| 46 |
+
{"step": 4300, "update": 2150, "epoch": 0, "loss": 0.491773782215314, "answer_loss": 0.4684547888091765, "evidence_loss": 0.04663798563404666, "lr": 3.998519354229365e-05, "elapsed": 328.43280816078186}
|
| 47 |
+
{"step": 4400, "update": 2200, "epoch": 0, "loss": 0.4974437666492304, "answer_loss": 0.46550733276759276, "evidence_loss": 0.06387287114079772, "lr": 3.9504069889706476e-05, "elapsed": 430.45074915885925}
|
| 48 |
+
{"step": 4500, "update": 2250, "epoch": 0, "loss": 0.5081180370456423, "answer_loss": 0.5022541567738518, "evidence_loss": 0.011727759505515536, "lr": 3.901470852788942e-05, "elapsed": 530.8654174804688}
|
| 49 |
+
{"step": 4600, "update": 2300, "epoch": 0, "loss": 0.4814008385059424, "answer_loss": 0.4805939413566375, "evidence_loss": 0.0016137976962623667, "lr": 3.8517387393769e-05, "elapsed": 641.8180623054504}
|
| 50 |
+
{"step": 4700, "update": 2350, "epoch": 0, "loss": 0.5245338723540772, "answer_loss": 0.5204225685761776, "evidence_loss": 0.008222610500095881, "lr": 3.801238894509176e-05, "elapsed": 753.5041408538818}
|
| 51 |
+
{"step": 4800, "update": 2400, "epoch": 0, "loss": 0.3832768932898762, "answer_loss": 0.37741137452132534, "evidence_loss": 0.01173103900319043, "lr": 3.7500000000000003e-05, "elapsed": 861.6717767715454}
|
| 52 |
+
{"step": 4900, "update": 2450, "epoch": 0, "loss": 0.5356292374519398, "answer_loss": 0.5253163638827391, "evidence_loss": 0.0206257526912173, "lr": 3.6980511574131034e-05, "elapsed": 979.6903760433197}
|
| 53 |
+
{"step": 5000, "update": 2500, "epoch": 0, "loss": 0.520200138433138, "answer_loss": 0.5201998082955833, "evidence_loss": 6.633352515539314e-07, "lr": 3.6454218715332324e-05, "elapsed": 1098.0372767448425}
|
| 54 |
+
{"step": 5100, "update": 2550, "epoch": 0, "loss": 0.45275376982346643, "answer_loss": 0.452752989862347, "evidence_loss": 1.5633659917235755e-06, "lr": 3.5921420336086606e-05, "elapsed": 1204.1972362995148}
|
| 55 |
+
{"step": 5200, "update": 2600, "epoch": 0, "loss": 0.5308002051792573, "answer_loss": 0.5307994494203012, "evidence_loss": 1.517018775767376e-06, "lr": 3.538241904374201e-05, "elapsed": 1316.0419623851776}
|
| 56 |
+
{"step": 5300, "update": 2650, "epoch": 0, "loss": 0.4377178295957856, "answer_loss": 0.4377175888605416, "evidence_loss": 4.800104222102419e-07, "lr": 3.4837520968643644e-05, "elapsed": 1429.3595159053802}
|
| 57 |
+
{"step": 5400, "update": 2700, "epoch": 0, "loss": 0.439339493070147, "answer_loss": 0.439101385162212, "evidence_loss": 0.0004762143806309371, "lr": 3.42870355902643e-05, "elapsed": 1560.911366224289}
|
| 58 |
+
{"step": 5500, "update": 2750, "epoch": 0, "loss": 0.46321893111336976, "answer_loss": 0.44960031494498254, "evidence_loss": 0.027237233082020085, "lr": 3.373127556143291e-05, "elapsed": 1670.3537316322327}
|
| 59 |
+
{"step": 5600, "update": 2800, "epoch": 0, "loss": 0.5332666349224746, "answer_loss": 0.5150763697223738, "evidence_loss": 0.03638052615694885, "lr": 3.317055653076081e-05, "elapsed": 1776.308336019516}
|
| 60 |
+
{"step": 5700, "update": 2850, "epoch": 0, "loss": 0.5509440461546182, "answer_loss": 0.5439056775625795, "evidence_loss": 0.014076729759511686, "lr": 3.260519696336636e-05, "elapsed": 1882.9945969581604}
|
| 61 |
+
{"step": 5800, "update": 2900, "epoch": 0, "loss": 0.3639013960887678, "answer_loss": 0.36380102701368744, "evidence_loss": 0.0002007373871879281, "lr": 3.2035517960000046e-05, "elapsed": 1991.515107870102}
|
| 62 |
+
{"step": 5900, "update": 2950, "epoch": 0, "loss": 0.5100994391855784, "answer_loss": 0.4940486961381976, "evidence_loss": 0.032101496025152304, "lr": 3.146184307467255e-05, "elapsed": 2096.060045480728}
|
| 63 |
+
{"step": 6000, "update": 3000, "epoch": 0, "loss": 0.48002396800206043, "answer_loss": 0.48002270987955853, "evidence_loss": 2.5178889558929996e-06, "lr": 3.088449813088953e-05, "elapsed": 2205.22669005394}
|
| 64 |
+
{"step": 6100, "update": 3050, "epoch": 0, "loss": 0.37645776741206644, "answer_loss": 0.3764565281197429, "evidence_loss": 2.4761888549562626e-06, "lr": 3.0303811036597396e-05, "elapsed": 2311.468476295471}
|
| 65 |
+
{"step": 6200, "update": 3100, "epoch": 0, "loss": 0.45311510397470556, "answer_loss": 0.4516035089973593, "evidence_loss": 0.0030231872583493667, "lr": 2.9720111597945237e-05, "elapsed": 2417.656094312668}
|
| 66 |
+
{"step": 6300, "update": 3150, "epoch": 0, "loss": 0.4857915040678927, "answer_loss": 0.48259961118747013, "evidence_loss": 0.006383783599609565, "lr": 2.9133731331968607e-05, "elapsed": 2534.446584224701}
|
| 67 |
+
{"step": 6400, "update": 3200, "epoch": 0, "loss": 0.44749429628136567, "answer_loss": 0.43921083775814623, "evidence_loss": 0.01656691852699405, "lr": 2.854500327830162e-05, "elapsed": 2637.2639179229736}
|
| 68 |
+
{"step": 6500, "update": 3250, "epoch": 0, "loss": 0.4559358202840667, "answer_loss": 0.4494398599502165, "evidence_loss": 0.0129919221403469, "lr": 2.795426181002427e-05, "elapsed": 2757.463133096695}
|
| 69 |
+
{"step": 6600, "update": 3300, "epoch": 0, "loss": 0.5957308709388599, "answer_loss": 0.5911648769211024, "evidence_loss": 0.00913198922353331, "lr": 2.7361842443752427e-05, "elapsed": 2893.4360072612762}
|
| 70 |
+
{"step": 6700, "update": 3350, "epoch": 0, "loss": 0.4670684744877508, "answer_loss": 0.4642935997445602, "evidence_loss": 0.005549743893116101, "lr": 2.676808164907833e-05, "elapsed": 3005.8959035873413}
|
| 71 |
+
{"step": 6800, "update": 3400, "epoch": 0, "loss": 0.4658524791576201, "answer_loss": 0.46540585774695503, "evidence_loss": 0.0008932487317431992, "lr": 2.6173316657469816e-05, "elapsed": 3110.553952217102}
|
| 72 |
+
{"step": 6900, "update": 3450, "epoch": 0, "loss": 0.47041740787768505, "answer_loss": 0.47037568698084214, "evidence_loss": 8.344542271885303e-05, "lr": 2.557788527073688e-05, "elapsed": 3246.5727021694183}
|
| 73 |
+
{"step": 7000, "update": 3500, "epoch": 0, "loss": 0.4430269662756473, "answer_loss": 0.4333769900747575, "evidence_loss": 0.019299950632438865, "lr": 2.4982125669174254e-05, "elapsed": 3357.106659889221}
|
| 74 |
+
{"step": 7100, "update": 3550, "epoch": 0, "loss": 0.53851259538671, "answer_loss": 0.5313033811445348, "evidence_loss": 0.014418437511594391, "lr": 2.4386376219489085e-05, "elapsed": 3474.9814581871033}
|
| 75 |
+
{"step": 7200, "update": 3600, "epoch": 0, "loss": 0.47839031941490245, "answer_loss": 0.47835108564584516, "evidence_loss": 7.846973450837514e-05, "lr": 2.3790975282622648e-05, "elapsed": 3595.1623787879944}
|
| 76 |
+
{"step": 7300, "update": 3650, "epoch": 0, "loss": 0.43931732105556875, "answer_loss": 0.4079691893584095, "evidence_loss": 0.06269626595791976, "lr": 2.3196261021575427e-05, "elapsed": 3720.810885667801}
|
| 77 |
+
{"step": 7400, "update": 3700, "epoch": 0, "loss": 0.4365746343682986, "answer_loss": 0.4256263518787455, "evidence_loss": 0.0218965731957978, "lr": 2.2602571209344562e-05, "elapsed": 3834.521643638611}
|
| 78 |
+
{"step": 7500, "update": 3750, "epoch": 0, "loss": 0.4500409300159663, "answer_loss": 0.422346715615131, "evidence_loss": 0.055388428679780194, "lr": 2.2010243037082813e-05, "elapsed": 3940.470145702362}
|
| 79 |
+
{"step": 7600, "update": 3800, "epoch": 0, "loss": 0.4599592524394393, "answer_loss": 0.45990150287747383, "evidence_loss": 0.0001155024600024035, "lr": 2.1419612922588012e-05, "elapsed": 4059.6920573711395}
|
| 80 |
+
{"step": 7700, "update": 3850, "epoch": 0, "loss": 0.48828520355047655, "answer_loss": 0.48828473751433193, "evidence_loss": 9.283837837870123e-07, "lr": 2.0831016319231703e-05, "elapsed": 4151.278962612152}
|
| 81 |
+
{"step": 7800, "update": 3900, "epoch": 0, "loss": 0.45586519574339035, "answer_loss": 0.4558648662210908, "evidence_loss": 6.68125817355758e-07, "lr": 2.024478752543556e-05, "elapsed": 4266.632747888565}
|
| 82 |
+
{"step": 7900, "update": 3950, "epoch": 0, "loss": 0.48138918802374975, "answer_loss": 0.4782767493254505, "evidence_loss": 0.006224881735236352, "lr": 1.9661259494803795e-05, "elapsed": 4369.711484909058}
|
| 83 |
+
{"step": 8000, "update": 4000, "epoch": 0, "loss": 0.5006899367499864, "answer_loss": 0.5003588348929771, "evidence_loss": 0.0006622073367475423, "lr": 1.9080763647019328e-05, "elapsed": 4471.893168449402}
|
| 84 |
+
{"step": 8100, "update": 4050, "epoch": 0, "loss": 0.4319746668660082, "answer_loss": 0.4280566773447208, "evidence_loss": 0.007835977174427172, "lr": 1.850362967961117e-05, "elapsed": 4577.780429840088}
|
| 85 |
+
{"step": 8200, "update": 4100, "epoch": 0, "loss": 0.4673368737427518, "answer_loss": 0.46149286096915604, "evidence_loss": 0.011688025794315991, "lr": 1.7930185380699906e-05, "elapsed": 4687.738959312439}
|
| 86 |
+
{"step": 8300, "update": 4150, "epoch": 0, "loss": 0.4473612277809298, "answer_loss": 0.4473604839979089, "evidence_loss": 1.4891167899122592e-06, "lr": 1.73607564428277e-05, "elapsed": 4798.392721891403}
|
| 87 |
+
{"step": 8400, "update": 4200, "epoch": 0, "loss": 0.46840745707973835, "answer_loss": 0.46764759002951906, "evidence_loss": 0.001519733650734949, "lr": 1.6795666277978432e-05, "elapsed": 4900.386907577515}
|
| 88 |
+
{"step": 8500, "update": 4250, "epoch": 0, "loss": 0.45989898687432285, "answer_loss": 0.4355156029490172, "evidence_loss": 0.04876676511669871, "lr": 1.6235235833893107e-05, "elapsed": 5008.159985780716}
|
| 89 |
+
{"step": 8600, "update": 4300, "epoch": 0, "loss": 0.4570891419821419, "answer_loss": 0.4568088326486759, "evidence_loss": 0.0005606203279592137, "lr": 1.567978341178492e-05, "elapsed": 5109.400854349136}
|
| 90 |
+
{"step": 8700, "update": 4350, "epoch": 0, "loss": 0.48315355408005417, "answer_loss": 0.4775139547570143, "evidence_loss": 0.01127920375422434, "lr": 1.5129624485557331e-05, "elapsed": 5215.7930335998535}
|
| 91 |
+
{"step": 8800, "update": 4400, "epoch": 0, "loss": 0.4190238815994235, "answer_loss": 0.4189879939588718, "evidence_loss": 7.177452474024904e-05, "lr": 1.4585071522628044e-05, "elapsed": 5321.05663394928}
|
| 92 |
+
{"step": 8900, "update": 4450, "epoch": 0, "loss": 0.4748892635566881, "answer_loss": 0.4730017786214012, "evidence_loss": 0.0037749671035356867, "lr": 1.4046433806460445e-05, "elapsed": 5432.626954078674}
|
| 93 |
+
{"step": 9000, "update": 4500, "epoch": 0, "loss": 0.4353832123952452, "answer_loss": 0.4353829197387677, "evidence_loss": 5.880843872319019e-07, "lr": 1.3514017260903495e-05, "elapsed": 5535.756450653076}
|
| 94 |
+
{"step": 9100, "update": 4550, "epoch": 0, "loss": 0.39594513215357435, "answer_loss": 0.3925364017987158, "evidence_loss": 0.006817455651212754, "lr": 1.2988124276439632e-05, "elapsed": 5648.987658023834}
|
| 95 |
+
{"step": 9200, "update": 4600, "epoch": 0, "loss": 0.38374086744734087, "answer_loss": 0.3753158838243689, "evidence_loss": 0.016849961100750762, "lr": 1.2469053538439599e-05, "elapsed": 5755.371156454086}
|
| 96 |
+
{"step": 9300, "update": 4650, "epoch": 0, "loss": 0.47940689367242156, "answer_loss": 0.4711623780394439, "evidence_loss": 0.016489021650122594, "lr": 1.1957099857521489e-05, "elapsed": 5870.985414505005}
|
| 97 |
+
{"step": 9400, "update": 4700, "epoch": 0, "loss": 0.4299167326278985, "answer_loss": 0.42968212099513037, "evidence_loss": 0.0004692193736286754, "lr": 1.1452554002110647e-05, "elapsed": 5981.759161233902}
|
| 98 |
+
{"step": 9500, "update": 4750, "epoch": 0, "loss": 0.4164541179564549, "answer_loss": 0.4056672558869468, "evidence_loss": 0.021573721798752103, "lr": 1.0955702533295189e-05, "elapsed": 6094.58864569664}
|
| 99 |
+
{"step": 9600, "update": 4800, "epoch": 0, "loss": 0.45831689791113606, "answer_loss": 0.4583165735419607, "evidence_loss": 6.530154345796291e-07, "lr": 1.0466827642071293e-05, "elapsed": 6202.668437004089}
|
| 100 |
+
{"step": 9700, "update": 4850, "epoch": 0, "loss": 0.4710958126705373, "answer_loss": 0.4710936766676605, "evidence_loss": 4.27520848830909e-06, "lr": 9.986206989070355e-06, "elapsed": 6322.259366512299}
|
| 101 |
+
{"step": 9800, "update": 4900, "epoch": 0, "loss": 0.40285806550004055, "answer_loss": 0.3935272289440036, "evidence_loss": 0.018661671176353138, "lr": 9.514113546859365e-06, "elapsed": 6437.950893163681}
|
| 102 |
+
{"step": 9900, "update": 4950, "epoch": 0, "loss": 0.4606573904794641, "answer_loss": 0.4605170532001648, "evidence_loss": 0.0002806783346022712, "lr": 9.050815444903748e-06, "elapsed": 6561.037968873978}
|
| 103 |
+
{"step": 10000, "update": 5000, "epoch": 0, "loss": 0.48335507344920187, "answer_loss": 0.48319887116551397, "evidence_loss": 0.000312406980083324, "lr": 8.596575817281035e-06, "elapsed": 6677.962034702301}
|
| 104 |
+
{"step": 10100, "update": 5050, "epoch": 0, "loss": 0.4223300266102888, "answer_loss": 0.4126726179290563, "evidence_loss": 0.0193148145511077, "lr": 8.151652653231584e-06, "elapsed": 6796.059580564499}
|
| 105 |
+
{"step": 10200, "update": 5100, "epoch": 0, "loss": 0.43294058976927774, "answer_loss": 0.43288003307650796, "evidence_loss": 0.0001211141562089324, "lr": 7.716298650631428e-06, "elapsed": 6908.10114979744}
|
| 106 |
+
{"step": 10300, "update": 5150, "epoch": 0, "loss": 0.48472219418210444, "answer_loss": 0.48419279763125816, "evidence_loss": 0.001058796302504561, "lr": 7.2907610724704006e-06, "elapsed": 7023.115696668625}
|
| 107 |
+
{"step": 10400, "update": 5200, "epoch": 0, "loss": 0.46715727597591467, "answer_loss": 0.4556182489270577, "evidence_loss": 0.023078054624129436, "lr": 6.87528160641698e-06, "elapsed": 7139.109053373337}
|
| 108 |
+
{"step": 10500, "update": 5250, "epoch": 0, "loss": 0.4871681500226259, "answer_loss": 0.4812744154036045, "evidence_loss": 0.011787464185908902, "lr": 6.4700962275497854e-06, "elapsed": 7250.013733863831}
|
| 109 |
+
{"step": 10600, "update": 5300, "epoch": 0, "loss": 0.43944938538712447, "answer_loss": 0.43879598681931387, "evidence_loss": 0.001306796344651957, "lr": 6.075435064333479e-06, "elapsed": 7359.430778264999}
|
| 110 |
+
{"step": 10700, "update": 5350, "epoch": 0, "loss": 0.37098298177617833, "answer_loss": 0.37096054804947926, "evidence_loss": 4.486462179386308e-05, "lr": 5.6915222679154165e-06, "elapsed": 7463.829654216766}
|
| 111 |
+
{"step": 10800, "update": 5400, "epoch": 0, "loss": 0.35547495214559605, "answer_loss": 0.35547234656638466, "evidence_loss": 5.2090198801124645e-06, "lr": 5.3185758848170535e-06, "elapsed": 7575.687963962555}
|
| 112 |
+
{"step": 10900, "update": 5450, "epoch": 0, "loss": 0.4352072715136455, "answer_loss": 0.43379306112299676, "evidence_loss": 0.0028284223711148113, "lr": 4.956807733092617e-06, "elapsed": 7683.106428861618}
|
| 113 |
+
{"step": 11000, "update": 5500, "epoch": 0, "loss": 0.39414913783228256, "answer_loss": 0.3941442339553032, "evidence_loss": 9.807227767169025e-06, "lr": 4.606423282025213e-06, "elapsed": 7782.823413610458}
|
| 114 |
+
{"step": 11100, "update": 5550, "epoch": 0, "loss": 0.43862153622700134, "answer_loss": 0.43857135064637986, "evidence_loss": 0.00010037092405127623, "lr": 4.2676215354288565e-06, "elapsed": 7881.112507343292}
|
| 115 |
+
{"step": 11200, "update": 5600, "epoch": 0, "loss": 0.40736669033183714, "answer_loss": 0.40225677982729396, "evidence_loss": 0.010219817206588004, "lr": 3.940594918622512e-06, "elapsed": 7995.154633760452}
|
| 116 |
+
{"step": 11300, "update": 5650, "epoch": 0, "loss": 0.49603209488268474, "answer_loss": 0.4827433528547408, "evidence_loss": 0.026577481231875026, "lr": 3.6255291691405652e-06, "elapsed": 8103.613470315933}
|
| 117 |
+
{"step": 11400, "update": 5700, "epoch": 0, "loss": 0.46784877389494794, "answer_loss": 0.4678220012877137, "evidence_loss": 5.354465382424678e-05, "lr": 3.3226032312415612e-06, "elapsed": 8202.745464324951}
|
| 118 |
+
{"step": 11500, "update": 5750, "epoch": 0, "loss": 0.5164436005242169, "answer_loss": 0.4999234300921671, "evidence_loss": 0.03304033864309986, "lr": 3.031989154275361e-06, "elapsed": 8335.71618938446}
|
| 119 |
+
{"step": 11600, "update": 5800, "epoch": 0, "loss": 0.45249480980506634, "answer_loss": 0.4464464603882516, "evidence_loss": 0.01209669656710048, "lr": 2.753851994966189e-06, "elapsed": 8436.460706472397}
|
| 120 |
+
{"step": 11700, "update": 5850, "epoch": 0, "loss": 0.45089312612020876, "answer_loss": 0.45075863058737015, "evidence_loss": 0.0002689919732142698, "lr": 2.4883497236673353e-06, "elapsed": 8541.418382883072}
|
| 121 |
+
{"step": 11800, "update": 5900, "epoch": 0, "loss": 0.4452268042968353, "answer_loss": 0.44424556556536116, "evidence_loss": 0.0019624787477732753, "lr": 2.235633134640497e-06, "elapsed": 8648.319731712341}
|
| 122 |
+
{"step": 11900, "update": 5950, "epoch": 0, "loss": 0.4153186084679328, "answer_loss": 0.41129743786295875, "evidence_loss": 0.008042340509709901, "lr": 1.9958457604109584e-06, "elapsed": 8752.258842468262}
|
| 123 |
+
{"step": 12000, "update": 6000, "epoch": 0, "loss": 0.42718260430934607, "answer_loss": 0.42684502567804883, "evidence_loss": 0.0006751604532109922, "lr": 1.7691237902470236e-06, "elapsed": 8851.503152132034}
|
| 124 |
+
{"step": 12100, "update": 6050, "epoch": 0, "loss": 0.47669057034196155, "answer_loss": 0.45829702616814755, "evidence_loss": 0.036787092402428244, "lr": 1.5555959928102188e-06, "elapsed": 8965.566213130951}
|
| 125 |
+
{"step": 12200, "update": 6100, "epoch": 0, "loss": 0.4425456356949871, "answer_loss": 0.4425402174919145, "evidence_loss": 1.0831030594999902e-05, "lr": 1.355383643020003e-06, "elapsed": 9064.531921386719}
|
| 126 |
+
{"step": 12300, "update": 6150, "epoch": 0, "loss": 0.44247193973104004, "answer_loss": 0.43773227051162394, "evidence_loss": 0.009479338799953893, "lr": 1.1686004531746757e-06, "elapsed": 9169.78522515297}
|
| 127 |
+
{"step": 12400, "update": 6200, "epoch": 0, "loss": 0.405267085271189, "answer_loss": 0.40471490265685134, "evidence_loss": 0.0011043679019212506, "lr": 9.953525083674947e-07, "elapsed": 9274.789207458496}
|
| 128 |
+
{"step": 12500, "update": 6250, "epoch": 0, "loss": 0.45154568844736787, "answer_loss": 0.44815644546149996, "evidence_loss": 0.006778485635009019, "lr": 8.357382062347819e-07, "elapsed": 9369.492041826248}
|
| 129 |
+
{"step": 12600, "update": 6300, "epoch": 0, "loss": 0.4115633034714847, "answer_loss": 0.4115076944499742, "evidence_loss": 0.00011121496921987273, "lr": 6.898482010701035e-07, "elapsed": 9493.970655679703}
|
| 130 |
+
{"step": 12700, "update": 6350, "epoch": 0, "loss": 0.46379801131319254, "answer_loss": 0.4637708245939575, "evidence_loss": 5.4374564861063846e-05, "lr": 5.577653523364479e-07, "elapsed": 9608.941450834274}
|
| 131 |
+
{"step": 12800, "update": 6400, "epoch": 0, "loss": 0.35798254798399287, "answer_loss": 0.35261169414268806, "evidence_loss": 0.010741709093840654, "lr": 4.3956467760545573e-07, "elapsed": 9717.863548517227}
|
| 132 |
+
{"step": 12900, "update": 6450, "epoch": 0, "loss": 0.4216876322880853, "answer_loss": 0.421679410393117, "evidence_loss": 1.6444868206235695e-05, "lr": 3.3531330995058287e-07, "elapsed": 9828.310291528702}
|
| 133 |
+
{"step": 13000, "update": 6500, "epoch": 0, "loss": 0.4379342442762572, "answer_loss": 0.43724123114021496, "evidence_loss": 0.0013860227765007949, "lr": 2.450704598182779e-07, "elapsed": 9935.698092222214}
|
| 134 |
+
{"step": 13100, "update": 6550, "epoch": 0, "loss": 0.43254806531302165, "answer_loss": 0.43075663063908, "evidence_loss": 0.0035828758762909275, "lr": 1.6888738139892724e-07, "elapsed": 10034.31571650505}
|
| 135 |
+
{"step": 13200, "update": 6600, "epoch": 0, "loss": 0.37071704588306603, "answer_loss": 0.37065767009044065, "evidence_loss": 0.00011874930562044028, "lr": 1.0680734351659061e-07, "elapsed": 10144.50322151184}
|
| 136 |
+
{"step": 13300, "update": 6650, "epoch": 0, "loss": 0.4850760158908088, "answer_loss": 0.48505614486406556, "evidence_loss": 3.974013048264169e-05, "lr": 5.886560505412031e-08, "elapsed": 10258.301747083664}
|
| 137 |
+
{"step": 13400, "update": 6700, "epoch": 0, "loss": 0.48753603306831794, "answer_loss": 0.48752045711735265, "evidence_loss": 3.115377083759085e-05, "lr": 2.5089394927563235e-08, "elapsed": 10363.920886039734}
|
| 138 |
+
{"step": 13500, "update": 6750, "epoch": 0, "loss": 0.47188873108942064, "answer_loss": 0.4717700101213995, "evidence_loss": 0.00023744011210965254, "lr": 5.497896621281528e-09, "elapsed": 10465.92549085617}
|
| 139 |
+
{"step": 13588, "dev": {"count": 5513, "accuracy": 0.8345728278614185, "brier": 0.24521264721151587, "brier_soft": 0.22135841773342682, "nll": 0.46613226108284944, "nll_soft": 0.48607260387495255, "ece": 0.04855302549823719, "mean_confidence": 0.8630526785971036, "confident_error_rate": 0.036277888626881916, "score_mae": 0.471010453508593, "evidence_auc": 0.9999887610142061}, "dev_by_source": {"ag_news": {"count": 100, "accuracy": 0.91, "brier": 0.14675387157720124, "brier_soft": 0.14675387157720124, "nll": 0.2912302939848356, "nll_soft": 0.2912302939848356, "ece": 0.06685722156976967, "mean_confidence": 0.941055656674617, "confident_error_rate": 0.05}, "amazon_polarity": {"count": 100, "accuracy": 0.95, "brier": 0.06195212131140546, "brier_soft": 0.06195212131140546, "nll": 0.10053970676292343, "nll_soft": 0.10053970676292343, "ece": 0.03339375320732605, "mean_confidence": 0.9740179407588133, "confident_error_rate": 0.01}, "amazon_stars": {"count": 100, "accuracy": 0.67, "brier": 0.47443850133431553, "brier_soft": 0.47443850133431553, "nll": 0.8165046029997326, "nll_soft": 0.8165046029997326, "ece": 0.13569209388350084, "mean_confidence": 0.5784805667887004, "confident_error_rate": 0.0, "score_mae": 0.4510158879717489}, "anli": {"count": 100, "accuracy": 0.74, "brier": 0.39964040280539254, "brier_soft": 0.39964040280539254, "nll": 0.7264094626418607, "nll_soft": 0.7264094626418607, "ece": 0.15583201403785235, "mean_confidence": 0.8675408405401412, "confident_error_rate": 0.08}, "arc_challenge": {"count": 100, "accuracy": 0.98, "brier": 0.0338466710496252, "brier_soft": 0.0338466710496252, "nll": 0.06223066351037851, "nll_soft": 0.06223066351037851, "ece": 0.013031628765155607, "mean_confidence": 0.9885343841116019, "confident_error_rate": 0.01}, "arc_easy": {"count": 100, "accuracy": 0.98, "brier": 0.02898907365294522, "brier_soft": 0.02898907365294522, "nll": 0.049804731538844765, "nll_soft": 0.049804731538844765, "ece": 0.0203574919070956, "mean_confidence": 0.9893880594062854, "confident_error_rate": 0.01}, "banking77": {"count": 100, "accuracy": 0.83, "brier": 0.28368283342030337, "brier_soft": 0.28368283342030337, "nll": 0.757150864558469, "nll_soft": 0.757150864558469, "ece": 0.10648182762315579, "mean_confidence": 0.8818236425754715, "confident_error_rate": 0.07}, "boolq": {"count": 125, "accuracy": 0.72, "brier": 0.4320747195048023, "brier_soft": 0.4320747195048023, "nll": 0.6419028511365726, "nll_soft": 0.6419028511365726, "ece": 0.22197986221059174, "mean_confidence": 0.9277224964550744, "confident_error_rate": 0.08, "evidence_auc": 1.0}, "cb": {"count": 28, "accuracy": 0.9642857142857143, "brier": 0.08009602886981086, "brier_soft": 0.08009602886981086, "nll": 0.1383170388054993, "nll_soft": 0.1383170388054993, "ece": 0.0539652842052121, "mean_confidence": 0.9543658374733184, "confident_error_rate": 0.03571428571428571}, "civil_comments_severity": {"count": 100, "accuracy": 0.86, "brier": 0.22068328925164493, "brier_soft": 0.1529327563945061, "nll": 0.4278688922326199, "nll_soft": 0.5868835533574466, "ece": 0.11533257179634501, "mean_confidence": 0.7616463548019397, "confident_error_rate": 0.01, "score_mae": 0.30176027449639936}, "civil_comments_toxic": {"count": 100, "accuracy": 0.96, "brier": 0.08335265266916769, "brier_soft": 0.035325646018993, "nll": 0.15999763596847744, "nll_soft": 0.273538273309447, "ece": 0.07025124166293703, "mean_confidence": 0.8897487583370629, "confident_error_rate": 0.0}, "clinc150": {"count": 100, "accuracy": 0.88, "brier": 0.1914855487934617, "brier_soft": 0.1914855487934617, "nll": 0.5818385912529987, "nll_soft": 0.5818385912529987, "ece": 0.07644133988403692, "mean_confidence": 0.9504764250442571, "confident_error_rate": 0.05}, "cola": {"count": 100, "accuracy": 0.82, "brier": 0.23936409929410002, "brier_soft": 0.23936409929410002, "nll": 0.3789224756239914, "nll_soft": 0.3789224756239914, "ece": 0.11088711147702807, "mean_confidence": 0.8847813277631773, "confident_error_rate": 0.05}, "commonsense_qa": {"count": 100, "accuracy": 0.91, "brier": 0.15279201504656054, "brier_soft": 0.15279201504656054, "nll": 0.38053510799344925, "nll_soft": 0.38053510799344925, "ece": 0.055310114097351794, "mean_confidence": 0.9592039264777928, "confident_error_rate": 0.05}, "cosmos_qa": {"count": 120, "accuracy": 0.7166666666666667, "brier": 0.46689377122498227, "brier_soft": 0.46689377122498227, "nll": 1.0493689192831135, "nll_soft": 1.0493689192831135, "ece": 0.1842691185339397, "mean_confidence": 0.8969067516393356, "confident_error_rate": 0.13333333333333333, "evidence_auc": 0.9996875}, "dbpedia": {"count": 100, "accuracy": 1.0, "brier": 0.003369142699290202, "brier_soft": 0.003369142699290202, "nll": 0.0062849907399302775, "nll_soft": 0.0062849907399302775, "ece": 0.005018755002197648, "mean_confidence": 0.9949812449978023, "confident_error_rate": 0.0}, "dynasent": {"count": 100, "accuracy": 0.74, "brier": 0.3740545426955645, "brier_soft": 0.26081181440449636, "nll": 0.6428067650335194, "nll_soft": 0.7148067644257405, "ece": 0.15895209598817536, "mean_confidence": 0.6365691849311554, "confident_error_rate": 0.0, "score_mae": 0.42895381500154295}, "enron_spam": {"count": 100, "accuracy": 0.99, "brier": 0.020026272903107403, "brier_soft": 0.020026272903107403, "nll": 0.03562782185316014, "nll_soft": 0.03562782185316014, "ece": 0.019098380748180314, "mean_confidence": 0.9891996384225809, "confident_error_rate": 0.01}, "go_emotions": {"count": 100, "accuracy": 0.61, "brier": 0.5741589899965046, "brier_soft": 0.21588675035041727, "nll": 1.4307033727614344, "nll_soft": 1.91358506717411, "ece": 0.20796112646172915, "mean_confidence": 0.4163597675175832, "confident_error_rate": 0.0}, "hellaswag": {"count": 100, "accuracy": 0.97, "brier": 0.04486725792629876, "brier_soft": 0.04486725792629876, "nll": 0.0745400472799033, "nll_soft": 0.0745400472799033, "ece": 0.03637399457536667, "mean_confidence": 0.9683134231198267, "confident_error_rate": 0.0}, "imdb": {"count": 100, "accuracy": 0.98, "brier": 0.04568085346299085, "brier_soft": 0.04568085346299085, "nll": 0.10292772526304866, "nll_soft": 0.10292772526304866, "ece": 0.028451957696295264, "mean_confidence": 0.9728709039056583, "confident_error_rate": 0.01}, "ledgar": {"count": 100, "accuracy": 0.77, "brier": 0.3097771675840078, "brier_soft": 0.3097771675840078, "nll": 0.7872336976006159, "nll_soft": 0.7872336976006159, "ece": 0.1081009333709956, "mean_confidence": 0.8781009333709956, "confident_error_rate": 0.05}, "liar": {"count": 100, "accuracy": 0.4, "brier": 0.7189634047428461, "brier_soft": 0.7189634047428461, "nll": 1.436837871988536, "nll_soft": 1.436837871988536, "ece": 0.06865019845085027, "mean_confidence": 0.3381281030587656, "confident_error_rate": 0.0, "score_mae": 0.9441482667310059}, "measuring_hate_speech": {"count": 100, "accuracy": 0.84, "brier": 0.25256207725930363, "brier_soft": 0.13580843340392565, "nll": 0.4850187830942446, "nll_soft": 0.6520812834706986, "ece": 0.1169304401504889, "mean_confidence": 0.7261185069578411, "confident_error_rate": 0.0, "score_mae": 0.49203690882532636}, "medqa": {"count": 100, "accuracy": 0.89, "brier": 0.1535992569978112, "brier_soft": 0.1535992569978112, "nll": 0.2625025810302494, "nll_soft": 0.2625025810302494, "ece": 0.07181873065356893, "mean_confidence": 0.9229456023995973, "confident_error_rate": 0.02}, "mnli": {"count": 100, "accuracy": 0.97, "brier": 0.05573973735423007, "brier_soft": 0.05573973735423007, "nll": 0.1098216209157549, "nll_soft": 0.1098216209157549, "ece": 0.06763109111270017, "mean_confidence": 0.9361391609146414, "confident_error_rate": 0.01}, "mrpc": {"count": 100, "accuracy": 0.9, "brier": 0.1668976595442053, "brier_soft": 0.1668976595442053, "nll": 0.26565708797200693, "nll_soft": 0.26565708797200693, "ece": 0.04940636901408266, "mean_confidence": 0.9098964097140114, "confident_error_rate": 0.02}, "multirc": {"count": 125, "accuracy": 0.768, "brier": 0.41389630382118325, "brier_soft": 0.41389630382118325, "nll": 0.7069573124860199, "nll_soft": 0.7069573124860199, "ece": 0.20462945216784828, "mean_confidence": 0.9358565258357632, "confident_error_rate": 0.152, "evidence_auc": 0.9997333333333334}, "openbookqa": {"count": 100, "accuracy": 0.98, "brier": 0.03658790275205525, "brier_soft": 0.03658790275205525, "nll": 0.0717812851783545, "nll_soft": 0.0717812851783545, "ece": 0.020032410400518365, "mean_confidence": 0.9769089288843115, "confident_error_rate": 0.0}, "piqa": {"count": 100, "accuracy": 0.91, "brier": 0.12530473877108808, "brier_soft": 0.12530473877108808, "nll": 0.23578200111355627, "nll_soft": 0.23578200111355627, "ece": 0.05405603140631721, "mean_confidence": 0.9513254109878784, "confident_error_rate": 0.03}, "policy_application": {"count": 150, "accuracy": 0.9466666666666667, "brier": 0.07846287806722692, "brier_soft": 0.07846287806722692, "nll": 0.13571846060017997, "nll_soft": 0.13571846060017997, "ece": 0.046881348646941226, "mean_confidence": 0.9855395932885163, "confident_error_rate": 0.03333333333333333, "evidence_auc": 1.0}, "qqp": {"count": 100, "accuracy": 0.92, "brier": 0.1322711368798063, "brier_soft": 0.1322711368798063, "nll": 0.23146400397635053, "nll_soft": 0.23146400397635053, "ece": 0.039428147172002034, "mean_confidence": 0.9206436247298535, "confident_error_rate": 0.04}, "race": {"count": 125, "accuracy": 0.792, "brier": 0.27270894946390484, "brier_soft": 0.27270894946390484, "nll": 0.5369188120492517, "nll_soft": 0.5369188120492517, "ece": 0.09056234813224388, "mean_confidence": 0.8570818045914835, "confident_error_rate": 0.04, "evidence_auc": 1.0}, "rotten_tomatoes": {"count": 100, "accuracy": 0.98, "brier": 0.034796008989876, "brier_soft": 0.034796008989876, "nll": 0.07214729080189161, "nll_soft": 0.07214729080189161, "ece": 0.016962863770244453, "mean_confidence": 0.9715926379457167, "confident_error_rate": 0.01}, "rte": {"count": 100, "accuracy": 0.93, "brier": 0.1188184574788011, "brier_soft": 0.1188184574788011, "nll": 0.2052937800968413, "nll_soft": 0.2052937800968413, "ece": 0.04706417933618942, "mean_confidence": 0.9770641793361895, "confident_error_rate": 0.03}, "scotus": {"count": 100, "accuracy": 0.61, "brier": 0.6325394621335682, "brier_soft": 0.6325394621335682, "nll": 1.6277428172003265, "nll_soft": 1.6277428172003265, "ece": 0.25235176671516, "mean_confidence": 0.8520386087095022, "confident_error_rate": 0.13}, "shp": {"count": 100, "accuracy": 0.63, "brier": 0.46616116956928333, "brier_soft": 0.10030172097143464, "nll": 0.6589587286619418, "nll_soft": 0.683599739803445, "ece": 0.10805766937270722, "mean_confidence": 0.5400307704442183, "confident_error_rate": 0.0}, "sms_spam": {"count": 100, "accuracy": 0.99, "brier": 0.012794386799121387, "brier_soft": 0.012794386799121387, "nll": 0.01940308157677538, "nll_soft": 0.01940308157677538, "ece": 0.011367616270430043, "mean_confidence": 0.9945961193100143, "confident_error_rate": 0.0}, "snli": {"count": 100, "accuracy": 0.92, "brier": 0.1289873154832004, "brier_soft": 0.1289873154832004, "nll": 0.23811747993428342, "nll_soft": 0.23811747993428342, "ece": 0.06210566404106858, "mean_confidence": 0.9526758552214877, "confident_error_rate": 0.05}, "social_iqa": {"count": 100, "accuracy": 0.83, "brier": 0.26525446472548825, "brier_soft": 0.26525446472548825, "nll": 0.5398700586730137, "nll_soft": 0.5398700586730137, "ece": 0.09454909352171337, "mean_confidence": 0.9155281247457637, "confident_error_rate": 0.06}, "sst2": {"count": 100, "accuracy": 0.98, "brier": 0.040267340327601445, "brier_soft": 0.040267340327601445, "nll": 0.11672427288562982, "nll_soft": 0.11672427288562982, "ece": 0.01171578851442261, "mean_confidence": 0.9896982162000899, "confident_error_rate": 0.02}, "sst5": {"count": 100, "accuracy": 0.67, "brier": 0.4781693492848172, "brier_soft": 0.4781693492848172, "nll": 0.829577053110301, "nll_soft": 0.829577053110301, "ece": 0.14951808706248798, "mean_confidence": 0.554350865405928, "confident_error_rate": 0.0, "score_mae": 0.4378708779667829}, "stance_abortion": {"count": 60, "accuracy": 0.65, "brier": 0.5461631051578539, "brier_soft": 0.5461631051578539, "nll": 0.9158908383826021, "nll_soft": 0.9158908383826021, "ece": 0.24628237403169523, "mean_confidence": 0.870929660288412, "confident_error_rate": 0.15}, "stance_atheism": {"count": 60, "accuracy": 0.8, "brier": 0.31998745599311396, "brier_soft": 0.31998745599311396, "nll": 0.5000077747926376, "nll_soft": 0.5000077747926376, "ece": 0.14067486996116663, "mean_confidence": 0.8896575243678433, "confident_error_rate": 0.03333333333333333}, "stance_feminist": {"count": 60, "accuracy": 0.7833333333333333, "brier": 0.30915546723565146, "brier_soft": 0.30915546723565146, "nll": 0.5200861811211221, "nll_soft": 0.5200861811211221, "ece": 0.09158266666502275, "mean_confidence": 0.8748122325072478, "confident_error_rate": 0.05}, "stance_hillary": {"count": 60, "accuracy": 0.7166666666666667, "brier": 0.3835906907389575, "brier_soft": 0.3835906907389575, "nll": 0.6306440958758969, "nll_soft": 0.6306440958758969, "ece": 0.12429301714888963, "mean_confidence": 0.8135901747968072, "confident_error_rate": 0.05}, "stsb": {"count": 100, "accuracy": 0.65, "brier": 0.4467653588401074, "brier_soft": 0.20158728761075526, "nll": 0.7971364231767178, "nll_soft": 0.8773070175167166, "ece": 0.12983348156631902, "mean_confidence": 0.6330772411677799, "confident_error_rate": 0.0, "score_mae": 0.4435737806984108}, "subjectivity": {"count": 100, "accuracy": 0.99, "brier": 0.021267046464386992, "brier_soft": 0.021267046464386992, "nll": 0.04746715932690836, "nll_soft": 0.04746715932690836, "ece": 0.03920972397831717, "mean_confidence": 0.9638173734240708, "confident_error_rate": 0.0}, "swag": {"count": 100, "accuracy": 0.82, "brier": 0.25416755318707396, "brier_soft": 0.25416755318707396, "nll": 0.4673240743680174, "nll_soft": 0.4673240743680174, "ece": 0.08878300453136384, "mean_confidence": 0.9073698161052748, "confident_error_rate": 0.05}, "trec": {"count": 100, "accuracy": 0.97, "brier": 0.042537389155980695, "brier_soft": 0.042537389155980695, "nll": 0.11417283998567929, "nll_soft": 0.11417283998567929, "ece": 0.02627913171850392, "mean_confidence": 0.9776115812099822, "confident_error_rate": 0.01}, "tweet_hate": {"count": 100, "accuracy": 0.5, "brier": 0.6857332540214516, "brier_soft": 0.6857332540214516, "nll": 0.981097446252855, "nll_soft": 0.981097446252855, "ece": 0.33351503217194933, "mean_confidence": 0.833515032171949, "confident_error_rate": 0.11}, "tweet_sentiment": {"count": 100, "accuracy": 0.74, "brier": 0.38173636485792045, "brier_soft": 0.38173636485792045, "nll": 0.648052632172017, "nll_soft": 0.648052632172017, "ece": 0.11421575179540841, "mean_confidence": 0.6257842482045916, "confident_error_rate": 0.0, "score_mae": 0.35166887086126225}, "vitaminc": {"count": 100, "accuracy": 0.85, "brier": 0.23076445057087602, "brier_soft": 0.23076445057087602, "nll": 0.4375848144910949, "nll_soft": 0.4375848144910949, "ece": 0.09955280097416959, "mean_confidence": 0.91871404204557, "confident_error_rate": 0.06}, "wic": {"count": 100, "accuracy": 0.78, "brier": 0.3413490724528473, "brier_soft": 0.3413490724528473, "nll": 0.5457579942139619, "nll_soft": 0.5457579942139619, "ece": 0.16134131925967396, "mean_confidence": 0.9073427942234525, "confident_error_rate": 0.1}, "winogrande": {"count": 100, "accuracy": 0.91, "brier": 0.16856567514663653, "brier_soft": 0.16856567514663653, "nll": 0.3547808080133929, "nll_soft": 0.3547808080133929, "ece": 0.07623049848240572, "mean_confidence": 0.945613134694146, "confident_error_rate": 0.06}, "yelp": {"count": 100, "accuracy": 0.73, "brier": 0.40297617138952174, "brier_soft": 0.40297617138952174, "nll": 0.685889895087886, "nll_soft": 0.685889895087886, "ece": 0.1806122500426958, "mean_confidence": 0.622435088128629, "confident_error_rate": 0.0, "score_mae": 0.38806539902485776}}}
|
| 140 |
+
{"step": 13588, "dev": {"count": 5513, "accuracy": 0.8342100489751496, "brier": 0.24499939603673204, "brier_soft": 0.22122350305548508, "nll": 0.4655553015844398, "nll_soft": 0.48562642001788464, "ece": 0.04843755672557183, "mean_confidence": 0.8631851509248467, "confident_error_rate": 0.0359151097406131, "score_mae": 0.4705968118889738, "evidence_auc": 0.9999878244320566}, "dev_by_source": {"ag_news": {"count": 100, "accuracy": 0.91, "brier": 0.1465319479149763, "brier_soft": 0.1465319479149763, "nll": 0.2904767555416907, "nll_soft": 0.2904767555416907, "ece": 0.06696793567024037, "mean_confidence": 0.9411689492273619, "confident_error_rate": 0.05}, "amazon_polarity": {"count": 100, "accuracy": 0.95, "brier": 0.06128869305634376, "brier_soft": 0.06128869305634376, "nll": 0.10096164621475562, "nll_soft": 0.10096164621475562, "ece": 0.023046591891905697, "mean_confidence": 0.9730465918919056, "confident_error_rate": 0.01}, "amazon_stars": {"count": 100, "accuracy": 0.69, "brier": 0.4731412386648261, "brier_soft": 0.4731412386648261, "nll": 0.8151410149736901, "nll_soft": 0.8151410149736901, "ece": 0.1366735053126959, "mean_confidence": 0.579111282045835, "confident_error_rate": 0.0, "score_mae": 0.4509802678028108}, "anli": {"count": 100, "accuracy": 0.74, "brier": 0.3976344462707522, "brier_soft": 0.3976344462707522, "nll": 0.7224163582147134, "nll_soft": 0.7224163582147134, "ece": 0.14788199940722557, "mean_confidence": 0.8672915745774106, "confident_error_rate": 0.08}, "arc_challenge": {"count": 100, "accuracy": 0.98, "brier": 0.03366315261301923, "brier_soft": 0.03366315261301923, "nll": 0.060837047229714225, "nll_soft": 0.060837047229714225, "ece": 0.011432103687330048, "mean_confidence": 0.9886713474416573, "confident_error_rate": 0.01}, "arc_easy": {"count": 100, "accuracy": 0.98, "brier": 0.029996236913866735, "brier_soft": 0.029996236913866735, "nll": 0.05253505337779216, "nll_soft": 0.05253505337779216, "ece": 0.018028940214597153, "mean_confidence": 0.990060141873229, "confident_error_rate": 0.01}, "banking77": {"count": 100, "accuracy": 0.83, "brier": 0.28099173269454264, "brier_soft": 0.28099173269454264, "nll": 0.7554653567994057, "nll_soft": 0.7554653567994057, "ece": 0.11505295474931794, "mean_confidence": 0.8828400651254894, "confident_error_rate": 0.07}, "boolq": {"count": 125, "accuracy": 0.72, "brier": 0.43268232797354933, "brier_soft": 0.43268232797354933, "nll": 0.6422496879755143, "nll_soft": 0.6422496879755143, "ece": 0.22190986014776565, "mean_confidence": 0.9281403357012442, "confident_error_rate": 0.072, "evidence_auc": 1.0}, "cb": {"count": 28, "accuracy": 0.9642857142857143, "brier": 0.07835667610715101, "brier_soft": 0.07835667610715101, "nll": 0.13341829300103206, "nll_soft": 0.13341829300103206, "ece": 0.05279191576259608, "mean_confidence": 0.954691373765687, "confident_error_rate": 0.03571428571428571}, "civil_comments_severity": {"count": 100, "accuracy": 0.85, "brier": 0.21990075062806394, "brier_soft": 0.15256546144647498, "nll": 0.42595271409351193, "nll_soft": 0.5859219055739261, "ece": 0.1029050524606589, "mean_confidence": 0.7619406731562297, "confident_error_rate": 0.0, "score_mae": 0.3013470975763519}, "civil_comments_toxic": {"count": 100, "accuracy": 0.96, "brier": 0.08184155550658603, "brier_soft": 0.035637691017917736, "nll": 0.15796842110857123, "nll_soft": 0.2738468412771658, "ece": 0.07203162905008202, "mean_confidence": 0.8907618087052214, "confident_error_rate": 0.0}, "clinc150": {"count": 100, "accuracy": 0.88, "brier": 0.19125913762163368, "brier_soft": 0.19125913762163368, "nll": 0.5776643056115763, "nll_soft": 0.5776643056115763, "ece": 0.09283844266870002, "mean_confidence": 0.9496720060766602, "confident_error_rate": 0.06}, "cola": {"count": 100, "accuracy": 0.82, "brier": 0.23386052759583184, "brier_soft": 0.23386052759583184, "nll": 0.37037031376428564, "nll_soft": 0.37037031376428564, "ece": 0.09766561804689824, "mean_confidence": 0.8833423357882064, "confident_error_rate": 0.03}, "commonsense_qa": {"count": 100, "accuracy": 0.91, "brier": 0.15377198586333088, "brier_soft": 0.15377198586333088, "nll": 0.3819044445089102, "nll_soft": 0.3819044445089102, "ece": 0.05144685490055159, "mean_confidence": 0.9596385183475031, "confident_error_rate": 0.05}, "cosmos_qa": {"count": 120, "accuracy": 0.7166666666666667, "brier": 0.4672354484316241, "brier_soft": 0.4672354484316241, "nll": 1.0482225583041105, "nll_soft": 1.0482225583041105, "ece": 0.19858953692704095, "mean_confidence": 0.8962658679831853, "confident_error_rate": 0.13333333333333333, "evidence_auc": 0.999375}, "dbpedia": {"count": 100, "accuracy": 1.0, "brier": 0.0033691323280231667, "brier_soft": 0.0033691323280231667, "nll": 0.00628770684289355, "nll_soft": 0.00628770684289355, "ece": 0.0050215623881869585, "mean_confidence": 0.994978437611813, "confident_error_rate": 0.0}, "dynasent": {"count": 100, "accuracy": 0.73, "brier": 0.37323258779066765, "brier_soft": 0.2605018455661368, "nll": 0.6422479980426224, "nll_soft": 0.7148729964174031, "ece": 0.15595018764547483, "mean_confidence": 0.6386131537774967, "confident_error_rate": 0.0, "score_mae": 0.42700063346629696}, "enron_spam": {"count": 100, "accuracy": 0.99, "brier": 0.020538910131957966, "brier_soft": 0.020538910131957966, "nll": 0.037125773454637925, "nll_soft": 0.037125773454637925, "ece": 0.018135869575440167, "mean_confidence": 0.9889960329597918, "confident_error_rate": 0.01}, "go_emotions": {"count": 100, "accuracy": 0.6, "brier": 0.5706133279962924, "brier_soft": 0.2146223457949257, "nll": 1.4243747447457458, "nll_soft": 1.9114536152236135, "ece": 0.2153529377728328, "mean_confidence": 0.4180754880267198, "confident_error_rate": 0.0}, "hellaswag": {"count": 100, "accuracy": 0.97, "brier": 0.04398586313752212, "brier_soft": 0.04398586313752212, "nll": 0.07282392796652562, "nll_soft": 0.07282392796652562, "ece": 0.03281997526145711, "mean_confidence": 0.967702411105094, "confident_error_rate": 0.0}, "imdb": {"count": 100, "accuracy": 0.98, "brier": 0.04503053337307919, "brier_soft": 0.04503053337307919, "nll": 0.10202723595434093, "nll_soft": 0.10202723595434093, "ece": 0.02799987491297936, "mean_confidence": 0.9735376597268748, "confident_error_rate": 0.01}, "ledgar": {"count": 100, "accuracy": 0.78, "brier": 0.31022259775593414, "brier_soft": 0.31022259775593414, "nll": 0.7916681164712028, "nll_soft": 0.7916681164712028, "ece": 0.10685236902591677, "mean_confidence": 0.8791648234039952, "confident_error_rate": 0.05}, "liar": {"count": 100, "accuracy": 0.39, "brier": 0.7168465726369471, "brier_soft": 0.7168465726369471, "nll": 1.430973024742663, "nll_soft": 1.430973024742663, "ece": 0.07922166544125636, "mean_confidence": 0.33914869030092576, "confident_error_rate": 0.0, "score_mae": 0.9383143759928696}, "measuring_hate_speech": {"count": 100, "accuracy": 0.84, "brier": 0.2523259794626622, "brier_soft": 0.1350772440937763, "nll": 0.48511890737634117, "nll_soft": 0.6516605742694233, "ece": 0.1276134336558651, "mean_confidence": 0.7243118123610279, "confident_error_rate": 0.0, "score_mae": 0.49301893575302613}, "medqa": {"count": 100, "accuracy": 0.88, "brier": 0.15509559657024174, "brier_soft": 0.15509559657024174, "nll": 0.2630871286801779, "nll_soft": 0.2630871286801779, "ece": 0.07094760936712856, "mean_confidence": 0.9227796946829281, "confident_error_rate": 0.02}, "mnli": {"count": 100, "accuracy": 0.97, "brier": 0.05622277315175722, "brier_soft": 0.05622277315175722, "nll": 0.11140887517203295, "nll_soft": 0.11140887517203295, "ece": 0.06498924366805267, "mean_confidence": 0.9358105562109016, "confident_error_rate": 0.01}, "mrpc": {"count": 100, "accuracy": 0.9, "brier": 0.16885290914097817, "brier_soft": 0.16885290914097817, "nll": 0.26678509569101083, "nll_soft": 0.26678509569101083, "ece": 0.056738491245727135, "mean_confidence": 0.9094846375493769, "confident_error_rate": 0.02}, "multirc": {"count": 125, "accuracy": 0.768, "brier": 0.4125322538820213, "brier_soft": 0.4125322538820213, "nll": 0.702692751642681, "nll_soft": 0.702692751642681, "ece": 0.2040484715924632, "mean_confidence": 0.9360141123782398, "confident_error_rate": 0.152, "evidence_auc": 0.9997333333333334}, "openbookqa": {"count": 100, "accuracy": 0.98, "brier": 0.036469910012703805, "brier_soft": 0.036469910012703805, "nll": 0.07196625626831986, "nll_soft": 0.07196625626831986, "ece": 0.02228237729925685, "mean_confidence": 0.977701842636366, "confident_error_rate": 0.0}, "piqa": {"count": 100, "accuracy": 0.91, "brier": 0.12451330680810697, "brier_soft": 0.12451330680810697, "nll": 0.23446545298444263, "nll_soft": 0.23446545298444263, "ece": 0.06753078707740053, "mean_confidence": 0.9522836212325586, "confident_error_rate": 0.03}, "policy_application": {"count": 150, "accuracy": 0.9466666666666667, "brier": 0.0777876053884264, "brier_soft": 0.0777876053884264, "nll": 0.13521520720550428, "nll_soft": 0.13521520720550428, "ece": 0.04668170693287413, "mean_confidence": 0.985035205551867, "confident_error_rate": 0.03333333333333333, "evidence_auc": 1.0}, "qqp": {"count": 100, "accuracy": 0.92, "brier": 0.13043954738084676, "brier_soft": 0.13043954738084676, "nll": 0.22957888778440388, "nll_soft": 0.22957888778440388, "ece": 0.03919742572811998, "mean_confidence": 0.9198058774243847, "confident_error_rate": 0.04}, "race": {"count": 125, "accuracy": 0.792, "brier": 0.273025071208486, "brier_soft": 0.273025071208486, "nll": 0.5401223012436509, "nll_soft": 0.5401223012436509, "ece": 0.09241534466772917, "mean_confidence": 0.8570207713461646, "confident_error_rate": 0.04, "evidence_auc": 1.0}, "rotten_tomatoes": {"count": 100, "accuracy": 0.98, "brier": 0.035290060552569, "brier_soft": 0.035290060552569, "nll": 0.07276457767313435, "nll_soft": 0.07276457767313435, "ece": 0.017489686473322937, "mean_confidence": 0.9711417797012452, "confident_error_rate": 0.01}, "rte": {"count": 100, "accuracy": 0.93, "brier": 0.11679704186673533, "brier_soft": 0.11679704186673533, "nll": 0.20138234100382452, "nll_soft": 0.20138234100382452, "ece": 0.04667671248922903, "mean_confidence": 0.976676712489229, "confident_error_rate": 0.03}, "scotus": {"count": 100, "accuracy": 0.61, "brier": 0.6341498051249195, "brier_soft": 0.6341498051249195, "nll": 1.6277115865464133, "nll_soft": 1.6277115865464133, "ece": 0.25533107042012765, "mean_confidence": 0.8527349247542704, "confident_error_rate": 0.13}, "shp": {"count": 100, "accuracy": 0.65, "brier": 0.46443770760873426, "brier_soft": 0.09936027473515505, "nll": 0.6572154778957793, "nll_soft": 0.6826727776739735, "ece": 0.1103068097721516, "mean_confidence": 0.5396931902278483, "confident_error_rate": 0.0}, "sms_spam": {"count": 100, "accuracy": 0.99, "brier": 0.01279388922890178, "brier_soft": 0.01279388922890178, "nll": 0.019404364213720597, "nll_soft": 0.019404364213720597, "ece": 0.011369026874576384, "mean_confidence": 0.994594708705868, "confident_error_rate": 0.0}, "snli": {"count": 100, "accuracy": 0.92, "brier": 0.1303060757325426, "brier_soft": 0.1303060757325426, "nll": 0.240589231743456, "nll_soft": 0.240589231743456, "ece": 0.04670265643035693, "mean_confidence": 0.9518659140843424, "confident_error_rate": 0.05}, "social_iqa": {"count": 100, "accuracy": 0.84, "brier": 0.2641777007342555, "brier_soft": 0.2641777007342555, "nll": 0.5338294339120516, "nll_soft": 0.5338294339120516, "ece": 0.09934345700863399, "mean_confidence": 0.9154886411744081, "confident_error_rate": 0.06}, "sst2": {"count": 100, "accuracy": 0.98, "brier": 0.04030628615298806, "brier_soft": 0.04030628615298806, "nll": 0.11747284260512826, "nll_soft": 0.11747284260512826, "ece": 0.012949483786838481, "mean_confidence": 0.9895809775228284, "confident_error_rate": 0.02}, "sst5": {"count": 100, "accuracy": 0.66, "brier": 0.47858939401943057, "brier_soft": 0.47858939401943057, "nll": 0.8312380624917151, "nll_soft": 0.8312380624917151, "ece": 0.14479568835917578, "mean_confidence": 0.5538169983846644, "confident_error_rate": 0.0, "score_mae": 0.4382710876978508}, "stance_abortion": {"count": 60, "accuracy": 0.65, "brier": 0.5483838340126799, "brier_soft": 0.5483838340126799, "nll": 0.916795129103024, "nll_soft": 0.916795129103024, "ece": 0.2470612492514964, "mean_confidence": 0.8717795686329541, "confident_error_rate": 0.15}, "stance_atheism": {"count": 60, "accuracy": 0.7833333333333333, "brier": 0.31917383626334284, "brier_soft": 0.31917383626334284, "nll": 0.49926359046026203, "nll_soft": 0.49926359046026203, "ece": 0.15642293053711553, "mean_confidence": 0.889754045572878, "confident_error_rate": 0.03333333333333333}, "stance_feminist": {"count": 60, "accuracy": 0.7833333333333333, "brier": 0.3049816554075578, "brier_soft": 0.3049816554075578, "nll": 0.5154249440169435, "nll_soft": 0.5154249440169435, "ece": 0.10461663391963241, "mean_confidence": 0.8737570622243047, "confident_error_rate": 0.05}, "stance_hillary": {"count": 60, "accuracy": 0.7166666666666667, "brier": 0.3812783742364739, "brier_soft": 0.3812783742364739, "nll": 0.6242170771095513, "nll_soft": 0.6242170771095513, "ece": 0.11280688269490617, "mean_confidence": 0.8143798914819186, "confident_error_rate": 0.05}, "stsb": {"count": 100, "accuracy": 0.65, "brier": 0.45170767908012466, "brier_soft": 0.20552974536260238, "nll": 0.8029710252702897, "nll_soft": 0.8819413373331731, "ece": 0.10983419549350473, "mean_confidence": 0.6357630237891139, "confident_error_rate": 0.0, "score_mae": 0.44494913340968734}, "subjectivity": {"count": 100, "accuracy": 0.99, "brier": 0.02143104125551823, "brier_soft": 0.02143104125551823, "nll": 0.047739931072467476, "nll_soft": 0.047739931072467476, "ece": 0.039431940261527555, "mean_confidence": 0.9635951571408603, "confident_error_rate": 0.0}, "swag": {"count": 100, "accuracy": 0.82, "brier": 0.2537048639515909, "brier_soft": 0.2537048639515909, "nll": 0.4672971887646461, "nll_soft": 0.4672971887646461, "ece": 0.08815823139371007, "mean_confidence": 0.9081582313937099, "confident_error_rate": 0.05}, "trec": {"count": 100, "accuracy": 0.97, "brier": 0.04378629853100501, "brier_soft": 0.04378629853100501, "nll": 0.11549823864485118, "nll_soft": 0.11549823864485118, "ece": 0.025471181892042873, "mean_confidence": 0.9775477901155262, "confident_error_rate": 0.01}, "tweet_hate": {"count": 100, "accuracy": 0.5, "brier": 0.6883996904854307, "brier_soft": 0.6883996904854307, "nll": 0.9833215742341621, "nll_soft": 0.9833215742341621, "ece": 0.3459218953511739, "mean_confidence": 0.8344252768252641, "confident_error_rate": 0.11}, "tweet_sentiment": {"count": 100, "accuracy": 0.74, "brier": 0.3831706610576699, "brier_soft": 0.3831706610576699, "nll": 0.6497370609790863, "nll_soft": 0.6497370609790863, "ece": 0.13208543978054657, "mean_confidence": 0.6252989072735136, "confident_error_rate": 0.0, "score_mae": 0.3541000877573742}, "vitaminc": {"count": 100, "accuracy": 0.84, "brier": 0.23269831680007946, "brier_soft": 0.23269831680007946, "nll": 0.4394585245066372, "nll_soft": 0.4394585245066372, "ece": 0.09072915887713841, "mean_confidence": 0.9197737336036776, "confident_error_rate": 0.06}, "wic": {"count": 100, "accuracy": 0.78, "brier": 0.34073352692700504, "brier_soft": 0.34073352692700504, "nll": 0.5452054678565941, "nll_soft": 0.5452054678565941, "ece": 0.13825743108455368, "mean_confidence": 0.9071892135345369, "confident_error_rate": 0.1}, "winogrande": {"count": 100, "accuracy": 0.91, "brier": 0.17067644004760918, "brier_soft": 0.17067644004760918, "nll": 0.35855105926065917, "nll_soft": 0.35855105926065917, "ece": 0.08624812639081393, "mean_confidence": 0.9450851377589773, "confident_error_rate": 0.07}, "yelp": {"count": 100, "accuracy": 0.73, "brier": 0.402625269533739, "brier_soft": 0.402625269533739, "nll": 0.6852448507549156, "nll_soft": 0.6852448507549156, "ece": 0.16558757141209074, "mean_confidence": 0.6233702191486455, "confident_error_rate": 0.0, "score_mae": 0.3873896875444972}}}
|