Hugging Face's logo

bibbbu
/
multilingual-ai-human-detector_xlm-roberta-base

Text Classification
Transformers
Safetensors
English
Chinese
Vietnamese
xlm-roberta
ai-text-detection
ai-content-forensics
multilingual
Eval Results (legacy)
text-embeddings-inference
Model card Files
xet
 Community
multilingual-ai-human-detector_xlm-roberta-base / README.md
bibbbu's picture
bibbbu
Update README.md
b4aa2e0 verified
preview code
|
Raw
History Blame Contribute Delete
5.59 kB
metadata 
license: mit
base_model: FacebookAI/xlm-roberta-base
library_name: transformers
pipeline_tag: text-classification
language:
  - en
  - zh
  - vi
tags:
  - text-classification
  - ai-text-detection
  - ai-content-forensics
  - xlm-roberta
  - multilingual
metrics:
  - f1
model-index:
  - name: multilingual-ai-human-detector_xlm-roberta-base
    results:
      - task:
          type: text-classification
          name: AI-generated vs human-written text detection
        dataset:
          name: Multilingual QA corpus (HC3 / HC3-Chinese / Vietnamese Reddit)
          type: custom
        metrics:
          - type: f1
            value: 0.971
            name: F1 (overall, EN+ZH+VI test set)
          - type: f1
            value: 0.989
            name: F1 (English)
          - type: f1
            value: 0.9462
            name: F1 (Chinese)
          - type: f1
            value: 0.9783
            name: F1 (Vietnamese)

Multilingual AI-vs-Human Text Detector — XLM-RoBERTa-base

A binary classifier that detects whether a passage was written by a human or generated by an AI model, fine-tuned from XLM-RoBERTa-base to work across English, Chinese, and Vietnamese with a single checkpoint.

Built as an AI content forensics research project: the full system — data pipeline, statistical baselines, per-language evaluation, REST API, and web demo — lives in the companion repository.

Why a multilingual detector?

Monolingual detectors fail badly outside their pre-training language: in the same experimental setup, an English-only DistilBERT drops to F1 0.784 on Chinese (vs 0.977 on English) because Chinese characters tokenise into meaningless subword fragments. This model uses multilingual pre-training to hold F1 0.946–0.989 across all three languages — evidence that cross-lingual AI-content detection requires multilingual representations, not per-language models.

Evaluation

Test-set F1 (70/15/15 prompt-level split, seed 42; 90 test samples per language, 270 total):

Language F1
English 0.9890
Vietnamese 0.9783
Chinese 0.9462
Overall 0.9710

Comparison against baselines trained on the same data (full table, confusion matrices, and analysis in the GitHub README):

Model Overall F1
Logistic Regression (TF-IDF) 0.9776
XLM-RoBERTa (this model) 0.9710
Multinomial NB (TF-IDF) 0.9181
DistilBERT 0.8960

Note that the TF-IDF logistic regression baseline is competitive in-domain — the value of this transformer model is expected in robustness to paraphrase and vocabulary shift, which n-gram features cannot capture (see Limitations).

Training

  • Base model: FacebookAI/xlm-roberta-base (~279M params), sequence classification head, F32.
  • Data: 900 QA pairs (300 per language), each with one human and one AI answer:
    • English — HC3 (reddit_eli5)
    • Chinese — HC3-Chinese (open_qa)
    • Vietnamese — crawled from Vietnamese Reddit communities
    • AI answers generated by Qwen2.5-1.5B-Instruct
  • Setup: fine-tuned 3 epochs, learning rate 2e-5, max_length=256, prompt-level 70/15/15 train/val/test split (seed 42) so no question appears in both train and test.
  • Reproduce: python scripts/train_transformer.py --model-name xlm-roberta-base in the GitHub repo.

Usage 

from transformers import pipeline

detector = pipeline(
    "text-classification",
    model="bibbbu/multilingual-ai-human-detector_xlm-roberta-base",
)

texts = [
    "Honestly I just left it overnight and it worked fine, no idea why lol",
    "There are several important factors to consider when addressing this question.",
]
print(detector(texts))
# [{'label': 'human', 'score': ...}, {'label': 'ai', 'score': ...}]

For batch inference, a FastAPI server, and a Streamlit demo, see the GitHub repository.

Intended use & limitations

Intended use: research on cross-lingual AI-text detection; educational and portfolio use; experimentation with content-authenticity pipelines.

Limitations — read before relying on predictions:

  • ⚠️ Do not use this model alone for high-stakes decisions (academic-misconduct accusations, content moderation enforcement, hiring). AI-text detectors produce false positives, and non-native writers are a known false-positive risk for detectors in general.
  • Generator specificity: AI-labelled training text comes from a single generator (Qwen2.5-1.5B-Instruct). Detection of text from other models (GPT-4-class, Claude, Gemini) is untested and likely weaker.
  • Domain shift: trained on QA-style forum answers; performance on news, legal, academic, or social-media text is untested.
  • Small evaluation set: 270 test samples total; strong scores should be confirmed on larger, held-out domains before any production use.
  • Language coverage: English, Chinese, Vietnamese only.

Citation 

@misc{vu2024multilingual,
  title  = {Multilingual AI-Human Text Detection},
  author = {Vu, Tuong Vy},
  year   = {2024},
  url    = {https://github.com/vutuongvy101/multilingual-ai-human-text-detection}
}