nirnit-13/muril-supports-refutes

MuRIL Fine-Tuned for Claim-Evidence Classification (SUPPORTS / REFUTES)

Fine-tuned google/muril-base-cased for binary claim-evidence fact verification on code-mixed Hindi–Bengali–English text. Given a claim and a piece of evidence, the model predicts whether the evidence SUPPORTS or REFUTES the claim.

How to Use

from transformers import AutoTokenizer, AutoModelForSequenceClassification
import torch

MODEL_NAME = "nirnit-13/muril-supports-refutes"

tokenizer = AutoTokenizer.from_pretrained(MODEL_NAME)
model = AutoModelForSequenceClassification.from_pretrained(MODEL_NAME)
model.eval()

claim = "भारत दुनिया में दूध का सबसे बड़ा उत्पादक देश है।"
evidence = "भारत विश्व में दुग्ध उत्पादन में प्रथम स्थान पर है।"

inputs = tokenizer(claim, evidence, truncation=True, max_length=192, padding="max_length", return_tensors="pt")

with torch.no_grad():
    outputs = model(**inputs)
    probs = torch.softmax(outputs.logits, dim=-1)
    pred_id = torch.argmax(probs, dim=-1).item()

id2label = {0: "SUPPORTS", 1: "REFUTES"}
print(id2label[pred_id], "| confidence:", probs[0][pred_id].item())

Performance (Dev Set)

Metric	Value
Accuracy	0.822
Macro F1	0.801
F1 (SUPPORTS)	0.865
F1 (REFUTES)	0.738

Limitations

• Trained on informal, code-mixed, social-media-style claims. Performance degrades on out-of-domain, formally-written text or differently-structured evidence sources (see "Domain Generalization Caveat" below).
• Best used with max_length=192 tokenization to match training conditions.

---

Training & Tuning Details

The sections below document the hyperparameter tuning methodology used to produce this model.

Overview

This document describes the hyperparameter tuning process used to fine-tune google/muril-base-cased for a binary claim-evidence classification task (SUPPORTS vs. REFUTES) on code-mixed Hindi–Bengali–English data.

• Task: Given a claim and a piece of evidence, classify whether the evidence SUPPORTS or REFUTES the claim.
• Model: google/muril-base-cased (encoder-only, fine-tuned via AutoModelForSequenceClassification)
• Framework: HuggingFace transformers + datasets, run on Google Colab (free-tier T4 GPU)
• Search library: Optuna (TPE sampler), integrated via Trainer.hyperparameter_search()

Dataset

Split	File	Size	Notes
Train	train_subtask1.json	4,536	Fields: ID, Text, Evidence, Label
Dev	dev_subtask1.json	1,134	Same structure; used for validation during tuning
Label distribution (train)	—	SUPPORTS: 3,040 / REFUTES: 1,496	~2:1 class imbalance

Why Hyperparameter Tuning Was Needed

Initial manually-configured training runs plateaued around Macro F1 ≈ 0.79–0.80, with a persistent gap between SUPPORTS and REFUTES F1 scores (minority class underperforming). Manual tuning (adjusting one variable at a time — dropout, label smoothing, class weights) was slow and occasionally made results worse by stacking multiple untested changes at once. Optuna was introduced to systematically and efficiently search the hyperparameter space using smarter sampling (TPE) and early pruning of poor trials.

Search Setup

Tunable Hyperparameters and Search Space

def optuna_hp_space(trial):
    return {
        "learning_rate": trial.suggest_float("learning_rate", 1e-5, 5e-5, log=True),
        "per_device_train_batch_size": trial.suggest_categorical("per_device_train_batch_size", [4, 8, 16, 32]),
        "num_train_epochs": trial.suggest_int("num_train_epochs", 3, 16),
        "weight_decay": trial.suggest_float("weight_decay", 0.0, 0.1),
        "warmup_ratio": trial.suggest_float("warmup_ratio", 0.0, 0.2),
        "max_grad_norm": trial.suggest_float("max_grad_norm", 0.5, 2.0),
    }

Fixed Settings (not tuned)

• max_length = 192 (tokenization)
• fp16 = True (mixed precision, required for free-tier Colab GPU memory limits)
• eval_strategy = "epoch", save_strategy = "epoch"
• save_total_limit = 1 (only the best/most recent checkpoint retained)
• metric_for_best_model = "f1" (macro F1 used as the trial objective)
• Class-weighted CrossEntropyLoss (weights computed automatically from train label frequencies) to address the ~2:1 SUPPORTS:REFUTES imbalance

Persistence

Since Colab's free tier disconnects after a time/usage limit, the Optuna study was configured with persistent SQLite storage so the search could resume across sessions without losing trial history:

storage_path = "sqlite:////content/drive/MyDrive/muril_ckpt/optuna_study.db"

best_trial = trainer.hyperparameter_search(
    direction="maximize",
    backend="optuna",
    hp_space=optuna_hp_space,
    compute_objective=lambda metrics: metrics["eval_f1"],
    n_trials=15,
    study_name="muril_hp_search",
    storage=storage_path,
    load_if_exists=True,
)

• Only the .db file is required to resume a search across sessions/accounts — model checkpoints are not needed for this, since each trial trains a fresh model via model_init().
• n_trials is a cumulative budget across all resumed sessions, not per-session.

Results

Total trials run: 15 (several pruned early by Optuna due to clearly poor intermediate performance, e.g. high learning rates that failed to converge).

Best Trial

Hyperparameter	Value
learning_rate	1.7991465194475932e-05
per_device_train_batch_size	32
num_train_epochs	16
weight_decay	0.0764751706565559
warmup_ratio	0.026509480810828625
max_grad_norm	1.2004964752527867

Dev set performance (best trial):

Metric	Value
Accuracy	0.822
Macro F1	0.801
F1 (SUPPORTS)	0.865
F1 (REFUTES)	0.738

Key Findings from the Search

• Higher batch size (32) + moderate-high learning rate (~1.5–1.8e-5) + more epochs (10–16) consistently produced the best results.
• Very high learning rates (>3e-5) combined with small batch sizes caused training to fail to converge (loss stuck near the random-guessing baseline of ~1.386) — these trials were pruned or scored near F1 ≈ 0.39–0.50.
• Smaller batch sizes (4–8) produced noisier, less reliable gradients and were more prone to early pruning.

Known Limitation: Overfitting at High Epoch Counts

A full training run at the best trial's settings (16 epochs) showed a classic overfitting pattern when loss curves were inspected in detail:

• Training loss decreased continuously throughout all 16 epochs.
• Validation loss reached its minimum around epoch 5–6 and increased steadily afterward.
• Despite this, Macro F1 continued to creep upward slightly through epoch 16 — meaning metric_for_best_model="f1" may have selected an already-overfit checkpoint rather than the true best-generalizing one (epoch 5–6, by validation loss).

Recommendation for future runs: either cap num_train_epochs closer to 6–8, or switch metric_for_best_model to "eval_loss" (with greater_is_better=False) to select the checkpoint with the best generalization rather than the highest (potentially overfit) F1.

Domain Generalization Caveat

The tuned model performs well on data matching the training distribution (informal, code-mixed social-media-style claims) but was observed to generalize poorly to:

1. Clean, formally-written factual claims (out-of-domain test set) — near-total prediction collapse toward REFUTES.
2. A structurally different evidence-retrieval pipeline (news-article snippets with fusion-ranked retrieval scores, from a separate subtask) — Macro F1 dropped to ~0.60, with SUPPORTS recall collapsing to 0.45.

This indicates the tuned hyperparameters optimize well for the training/dev distribution, but do not by themselves solve domain-shift generalization — a separate concern from hyperparameter tuning.

Reproducing This Search

1. Mount Google Drive and place train_subtask1.json / dev_subtask1.json in the configured folder.
2. Run the Optuna search cell — it will create (or resume, if optuna_study.db already exists at the given path) the study and run trials up to n_trials.
3. Retrieve best_trial.hyperparameters and plug them into a final TrainingArguments for a full training run.
4. Save the resulting model (trainer.save_model(...)) for downstream inference.