| --- |
| library_name: sklearn |
| license: mit |
| pipeline_tag: text-classification |
| tags: |
| - artefactual |
| - skops |
| - sklearn |
| - hallucination-detection |
| - uncertainty-estimation |
| --- |
| |
|
|
| # Model description |
|
|
| **Target model:** [`microsoft/phi-4`](https://huggingface.co/microsoft/phi-4) -- this detector scores responses produced by that model. It is not a fine-tune of it and contains none of its weights. |
|
|
| A calibrated WEPR hallucination detector for responses generated by [`microsoft/phi-4`](https://huggingface.co/microsoft/phi-4). |
|
|
| **WEPR (Weighted EPR)** keeps the ranks separate, giving the calibration one coefficient per rank (mean and max over the token axis, so `2k` features). It reads strictly more of the distribution than EPR at the same calibration cost. |
|
|
| The artifact is the fitted `LogisticRegression` alone. The feature extraction that feeds it -- parsing top-`15` log-probabilities out of a completion response and reducing them to entropy features -- lives in the [artefactual](https://github.com/artefactory/artefactual) library, so this file contains no custom classes and loads with an empty `trusted` list. |
|
|
| Introduced in [Learned Hallucination Detection in Black-Box LLMs Using Token-Level Entropy Production Rate](https://doi.org/10.1007/978-3-032-21289-4_8) (ECIR 2026); the preprint is [arXiv:2509.04492](https://arxiv.org/abs/2509.04492). |
|
|
| ## Intended uses & limitations |
|
|
| Scores a response on `[0, 1]`, where 1 is the hallucination class. |
|
|
| - **Tied to `microsoft/phi-4`.** The coefficients are fit against that model's output distribution. Scoring another model's responses with them is not meaningful, even though nothing in the file prevents it. |
| - **Fixed at k=15.** Responses must be generated with `logprobs=True` and `top_logprobs=15`. Fewer ranks are rejected rather than zero-filled, because the missing ranks are unfetched rather than absent and padding them would score the response as more confident than it was. |
| - **No published operating point.** The paper reports ROC-AUC and PR-AUC, both threshold-free, so no decision threshold is published. Choose one on your own labelled data. |
|
|
| ## Evaluation Results |
|
|
| See [the paper](https://arxiv.org/abs/2509.04492). It reports ROC-AUC and PR-AUC across the evaluated models; no figures are restated here so that this card cannot drift from the published results. |
|
|
| # How to Get Started with the Model |
|
|
| ```python |
| from artefactual.scoring import WEPR |
| |
| detector = WEPR.from_pretrained("artefactory/wepr-phi4") |
| scores = detector.predict_proba(response)[:, 1] |
| ``` |
|
|
| `response` is an OpenAI-compatible chat completion or responses payload carrying `top_logprobs=15`. |
|
|
| Requires `artefactual>=2026.9`, where the detector is the `WEPR` class. Up to 2026.08.1 the same weights were loaded with the lowercase `wepr()` factory. |
|
|
| # Model Card Authors |
|
|
| Artefact Research Center |
|
|
| # Model Card Contact |
|
|
| https://github.com/artefactory/artefactual/issues |
|
|
| # Citation |
|
|
| ```bibtex |
| @inproceedings{moslonka2026learned, |
| title = {Learned Hallucination Detection in Black-Box LLMs Using Token-Level Entropy Production Rate}, |
| author = {Moslonka, Charles and Randrianarivo, Hicham and Garnier, Arthur and Malherbe, Emmanuel}, |
| booktitle = {Advances in Information Retrieval}, |
| series = {Lecture Notes in Computer Science}, |
| volume = {16483}, |
| pages = {115--130}, |
| publisher = {Springer, Cham}, |
| year = {2026}, |
| doi = {10.1007/978-3-032-21289-4_8}, |
| } |
| ``` |
|
|
