README.md

---
language:
  - ca
license: apache-2.0
library_name: transformers
pipeline_tag: text-classification
base_model: PlanTL-GOB-ES/roberta-base-ca
tags:
  - eamt-2026
  - catalan
  - valencian
  - text-classification
  - translationese
  - reward-model
---

# HT/MT Classifier for Catalan Translation Naturalness

## Model Summary

`guerreropaula/ht_mt_classifier_best` is a binary text classifier fine-tuned from `PlanTL-GOB-ES/roberta-base-ca` to distinguish human translations from machine translations in Catalan. In this project, it is used as an auxiliary reward model for GRPOv1.

The classifier predicts whether a Catalan sentence is more likely to be:

- `HT`: human translation
- `MT`: machine translation

Its output probability `P(HT | text)` is used as a naturalness-oriented reward signal during reinforcement learning.

## Model Details

- Model ID: `guerreropaula/ht_mt_classifier_best`
- Collection: `guerreropaula/spanish-valencian-mt-rl`
- Developed by: Paula Guerrero Castello
- Base model: `PlanTL-GOB-ES/roberta-base-ca`
- Task: binary text classification
- Labels: `0 = MT`, `1 = HT`
- License for model weights: Apache 2.0

## Intended Use

This model is intended for:

- research on translationese detection in Catalan
- reward shaping for GRPOv1
- analysis of machine-like versus human-like translation style

It is not intended for:

- quality estimation as a standalone substitute for MT evaluation
- detecting factual errors, hallucinations, or adequacy issues
- making sociolinguistic claims about Valencian or Catalan varieties

## Training Data

The classifier is trained from public parallel corpora in the Softcatala Parallel Corpus. Human Catalan references are paired with synthetic MT outputs generated from the same Spanish source sentences.

Corpora used:

- `TildeMODEL.es-ca`
- `dogc-es-ca`
- `europarl.es-ca`

Synthetic MT side:

- `Helsinki-NLP/opus-mt-es-ca`
- `facebook/nllb-200-distilled-600M`

Dataset construction notes:

- sentences shorter than 20 characters or longer than 500 characters are filtered out
- each corpus is capped at 20,000 parallel pairs before MT generation
- MT systems are assigned with a 50/50 split
- the final classifier dataset is shuffled and split 90/10 into train and validation

## Training Procedure

The model is fine-tuned with the Hugging Face `Trainer` API.

- Base model: `PlanTL-GOB-ES/roberta-base-ca`
- Max input length: 128
- Epochs: 5
- Learning rate: `2e-5`
- Batch size: 32
- Warmup ratio: 0.06
- Weight decay: 0.01
- Early stopping patience: 4
- Best checkpoint selected by: macro F1 on the validation split

Tracked validation metrics:

- accuracy
- macro F1
- macro precision
- macro recall

## How To Use

```python
from transformers import AutoTokenizer, AutoModelForSequenceClassification
import torch.nn.functional as F

model_id = "guerreropaula/ht_mt_classifier_best"
tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForSequenceClassification.from_pretrained(model_id)

text = "La seua participacio en el projecte va ser decisiva."
inputs = tokenizer(text, return_tensors="pt", truncation=True, max_length=256)
probs = F.softmax(model(**inputs).logits, dim=-1)
print({"MT": float(probs[0, 0]), "HT": float(probs[0, 1])})
```

## Evaluation

The training code evaluates the classifier on a held-out validation split using accuracy, macro F1, macro precision, and macro recall. This repository snapshot does not include a standalone exported metrics file for the final checkpoint, so the model card reports the training protocol and downstream use transparently rather than inventing absent numbers.

In the project, the classifier's practical impact is observed indirectly through GRPOv1, where its probabilities are used as a reward component.

## Limitations

- The classifier measures style similarity to human translations, not translation correctness.
- The negative class is synthetic MT generated by two specific systems, so the detector may learn system-specific artifacts.
- The model is trained on Catalan data and then used to reward Valencian-oriented outputs.
- A high `P(HT | text)` score should not be interpreted as proof of adequacy or dialectal authenticity.

## License

This classifier is distributed under the Apache 2.0 license inherited from `PlanTL-GOB-ES/roberta-base-ca`. Users should also review the licenses of the source corpora and MT systems used to build the training data.

## Citation

```bibtex
@inproceedings{guerrero-2026-enhancing,
  title     = {Enhancing LLM Translation Performance for Spanish-Valencian through Supervised Fine-tuning and Reinforcement Learning},
  author    = {Guerrero Castello, Paula},
  booktitle = {Proceedings of the 25th Annual Conference of the European Association for Machine Translation},
  year      = {2026}
}
```