README.md

---
library_name: sklearn
license: mit
pipeline_tag: text-classification
tags:
- artefactual
- skops
- sklearn
- hallucination-detection
- uncertainty-estimation
---


# Model description

**Target model:** [`microsoft/phi-4`](https://huggingface.co/microsoft/phi-4) -- this detector scores responses produced by that model. It is not a fine-tune of it and contains none of its weights.

A calibrated WEPR hallucination detector for responses generated by [`microsoft/phi-4`](https://huggingface.co/microsoft/phi-4).

**WEPR (Weighted EPR)** keeps the ranks separate, giving the calibration one coefficient per rank (mean and max over the token axis, so `2k` features). It reads strictly more of the distribution than EPR at the same calibration cost.

The artifact is the fitted `LogisticRegression` alone. The feature extraction that feeds it -- parsing top-`15` log-probabilities out of a completion response and reducing them to entropy features -- lives in the [artefactual](https://github.com/artefactory/artefactual) library, so this file contains no custom classes and loads with an empty `trusted` list.

Introduced in [Learned Hallucination Detection in Black-Box LLMs Using Token-Level Entropy Production Rate](https://doi.org/10.1007/978-3-032-21289-4_8) (ECIR 2026); the preprint is [arXiv:2509.04492](https://arxiv.org/abs/2509.04492).

## Intended uses & limitations

Scores a response on `[0, 1]`, where 1 is the hallucination class.

- **Tied to `microsoft/phi-4`.** The coefficients are fit against that model's output distribution. Scoring another model's responses with them is not meaningful, even though nothing in the file prevents it.
- **Fixed at k=15.** Responses must be generated with `logprobs=True` and `top_logprobs=15`. Fewer ranks are rejected rather than zero-filled, because the missing ranks are unfetched rather than absent and padding them would score the response as more confident than it was.
- **No published operating point.** The paper reports ROC-AUC and PR-AUC, both threshold-free, so no decision threshold is published. Choose one on your own labelled data.

## Evaluation Results

See [the paper](https://arxiv.org/abs/2509.04492). It reports ROC-AUC and PR-AUC across the evaluated models; no figures are restated here so that this card cannot drift from the published results.

# How to Get Started with the Model

```python
from artefactual.scoring import WEPR

detector = WEPR.from_pretrained("artefactory/wepr-phi4")
scores = detector.predict_proba(response)[:, 1]
```

`response` is an OpenAI-compatible chat completion or responses payload carrying `top_logprobs=15`.

Requires `artefactual>=2026.9`, where the detector is the `WEPR` class. Up to 2026.08.1 the same weights were loaded with the lowercase `wepr()` factory.

# Model Card Authors

Artefact Research Center

# Model Card Contact

https://github.com/artefactory/artefactual/issues

# Citation

```bibtex
@inproceedings{moslonka2026learned,
  title     = {Learned Hallucination Detection in Black-Box LLMs Using Token-Level Entropy Production Rate},
  author    = {Moslonka, Charles and Randrianarivo, Hicham and Garnier, Arthur and Malherbe, Emmanuel},
  booktitle = {Advances in Information Retrieval},
  series    = {Lecture Notes in Computer Science},
  volume    = {16483},
  pages     = {115--130},
  publisher = {Springer, Cham},
  year      = {2026},
  doi       = {10.1007/978-3-032-21289-4_8},
}
```