hf ({'pretrained': '/home/mose/Projects/llm/Hy3-REAP-96-rp2048', 'tp_plan': 'auto'}), gen_kwargs: ({}), limit: None, num_fewshot: None, batch_size: 32
|                 Tasks                 |Version|     Filter     |n-shot|  Metric   |   |Value |   |Stderr|
|---------------------------------------|------:|----------------|-----:|-----------|---|-----:|---|-----:|
|mmlu                                   |      2|none            |      |acc        |   |0.7620|±  |0.0034|
| - humanities                          |      2|none            |     0|acc        |↑  |0.6723|±  |0.0066|
|  - formal_logic                       |      1|none            |     0|acc        |↑  |0.6746|±  |0.0419|
|  - high_school_european_history       |      1|none            |     0|acc        |↑  |0.7939|±  |0.0316|
|  - high_school_us_history             |      1|none            |     0|acc        |↑  |0.8333|±  |0.0262|
|  - high_school_world_history          |      1|none            |     0|acc        |↑  |0.8565|±  |0.0228|
|  - international_law                  |      1|none            |     0|acc        |↑  |0.8182|±  |0.0352|
|  - jurisprudence                      |      1|none            |     0|acc        |↑  |0.8611|±  |0.0334|
|  - logical_fallacies                  |      1|none            |     0|acc        |↑  |0.8344|±  |0.0292|
|  - moral_disputes                     |      1|none            |     0|acc        |↑  |0.7832|±  |0.0222|
|  - moral_scenarios                    |      1|none            |     0|acc        |↑  |0.5754|±  |0.0165|
|  - philosophy                         |      1|none            |     0|acc        |↑  |0.8006|±  |0.0227|
|  - prehistory                         |      1|none            |     0|acc        |↑  |0.7407|±  |0.0244|
|  - professional_law                   |      1|none            |     0|acc        |↑  |0.5606|±  |0.0127|
|  - world_religions                    |      1|none            |     0|acc        |↑  |0.6491|±  |0.0366|
| - other                               |      2|none            |     0|acc        |↑  |0.7789|±  |0.0072|
|  - business_ethics                    |      1|none            |     0|acc        |↑  |0.8500|±  |0.0359|
|  - clinical_knowledge                 |      1|none            |     0|acc        |↑  |0.8340|±  |0.0229|
|  - college_medicine                   |      1|none            |     0|acc        |↑  |0.7630|±  |0.0324|
|  - global_facts                       |      1|none            |     0|acc        |↑  |0.6000|±  |0.0492|
|  - human_aging                        |      1|none            |     0|acc        |↑  |0.7578|±  |0.0288|
|  - management                         |      1|none            |     0|acc        |↑  |0.8447|±  |0.0359|
|  - marketing                          |      1|none            |     0|acc        |↑  |0.9103|±  |0.0187|
|  - medical_genetics                   |      1|none            |     0|acc        |↑  |0.8900|±  |0.0314|
|  - miscellaneous                      |      1|none            |     0|acc        |↑  |0.7995|±  |0.0143|
|  - nutrition                          |      1|none            |     0|acc        |↑  |0.8039|±  |0.0227|
|  - professional_accounting            |      1|none            |     0|acc        |↑  |0.6312|±  |0.0288|
|  - professional_medicine              |      1|none            |     0|acc        |↑  |0.8456|±  |0.0220|
|  - virology                           |      1|none            |     0|acc        |↑  |0.5060|±  |0.0389|
| - social sciences                     |      2|none            |     0|acc        |↑  |0.8570|±  |0.0062|
|  - econometrics                       |      1|none            |     0|acc        |↑  |0.7018|±  |0.0430|
|  - high_school_geography              |      1|none            |     0|acc        |↑  |0.9091|±  |0.0205|
|  - high_school_government_and_politics|      1|none            |     0|acc        |↑  |0.8756|±  |0.0238|
|  - high_school_macroeconomics         |      1|none            |     0|acc        |↑  |0.7821|±  |0.0209|
|  - high_school_microeconomics         |      1|none            |     0|acc        |↑  |0.9118|±  |0.0184|
|  - high_school_psychology             |      1|none            |     0|acc        |↑  |0.9413|±  |0.0101|
|  - human_sexuality                    |      1|none            |     0|acc        |↑  |0.8855|±  |0.0279|
|  - professional_psychology            |      1|none            |     0|acc        |↑  |0.8301|±  |0.0152|
|  - public_relations                   |      1|none            |     0|acc        |↑  |0.7545|±  |0.0412|
|  - security_studies                   |      1|none            |     0|acc        |↑  |0.8327|±  |0.0239|
|  - sociology                          |      1|none            |     0|acc        |↑  |0.8905|±  |0.0221|
|  - us_foreign_policy                  |      1|none            |     0|acc        |↑  |0.8300|±  |0.0378|
| - stem                                |      2|none            |     0|acc        |↑  |0.7866|±  |0.0070|
|  - abstract_algebra                   |      1|none            |     0|acc        |↑  |0.5500|±  |0.0500|
|  - anatomy                            |      1|none            |     0|acc        |↑  |0.7556|±  |0.0371|
|  - astronomy                          |      1|none            |     0|acc        |↑  |0.9145|±  |0.0228|
|  - college_biology                    |      1|none            |     0|acc        |↑  |0.9028|±  |0.0248|
|  - college_chemistry                  |      1|none            |     0|acc        |↑  |0.5400|±  |0.0501|
|  - college_computer_science           |      1|none            |     0|acc        |↑  |0.7200|±  |0.0451|
|  - college_mathematics                |      1|none            |     0|acc        |↑  |0.6600|±  |0.0476|
|  - college_physics                    |      1|none            |     0|acc        |↑  |0.7941|±  |0.0402|
|  - computer_security                  |      1|none            |     0|acc        |↑  |0.6800|±  |0.0469|
|  - conceptual_physics                 |      1|none            |     0|acc        |↑  |0.9106|±  |0.0186|
|  - electrical_engineering             |      1|none            |     0|acc        |↑  |0.7517|±  |0.0360|
|  - elementary_mathematics             |      1|none            |     0|acc        |↑  |0.8757|±  |0.0170|
|  - high_school_biology                |      1|none            |     0|acc        |↑  |0.9065|±  |0.0166|
|  - high_school_chemistry              |      1|none            |     0|acc        |↑  |0.6601|±  |0.0333|
|  - high_school_computer_science       |      1|none            |     0|acc        |↑  |0.9300|±  |0.0256|
|  - high_school_mathematics            |      1|none            |     0|acc        |↑  |0.6444|±  |0.0292|
|  - high_school_physics                |      1|none            |     0|acc        |↑  |0.7947|±  |0.0330|
|  - high_school_statistics             |      1|none            |     0|acc        |↑  |0.8287|±  |0.0257|
|  - machine_learning                   |      1|none            |     0|acc        |↑  |0.6964|±  |0.0436|
|gsm8k                                  |      3|flexible-extract|     5|exact_match|↑  |0.9401|±  |0.0065|
|                                       |       |strict-match    |     5|exact_match|↑  |0.9378|±  |0.0067|

|      Groups      |Version|Filter|n-shot|Metric|   |Value |   |Stderr|
|------------------|------:|------|-----:|------|---|-----:|---|-----:|
|mmlu              |      2|none  |      |acc   |   |0.7620|±  |0.0034|
| - humanities     |      2|none  |     0|acc   |↑  |0.6723|±  |0.0066|
| - other          |      2|none  |     0|acc   |↑  |0.7789|±  |0.0072|
| - social sciences|      2|none  |     0|acc   |↑  |0.8570|±  |0.0062|
| - stem           |      2|none  |     0|acc   |↑  |0.7866|±  |0.0070|