njand/wav2vec2-xls-r-latin

Wav2Vec2 XLS-R 300M - Latin ASR (Restored Classical Pronunciation)

Fine-tuned version of facebook/wav2vec2-xls-r-300m for Latin Automatic Speech Recognition (ASR), trained on reading passages from njand/llpsi-speech-dataset.

• Live Demo: Gradio Interface
• Source Code: GitHub Repository
• Base Model: facebook/wav2vec2-xls-r-300m
• Dataset: njand/llpsi-speech-dataset (currently private)
• Pronunciation Standard: Restored Classical Pronunciation
• Orthography: Macrons (ā, ē, ī, ō, ū, ȳ); consonantal j/v

---

🛠️ Model Variants & Optimization

To facilitate production deployment on CPU-based infrastructure, this repository provides the model in three formats:

Format	Precision	File Size	Recommended Use Case
PyTorch	FP32	1.26 GB	Training, fine-tuning, and research
ONNX	FP32	1.26 GB	Cross-platform inference
ONNX Quantized	INT8	355 MB	Production, Edge, & CPU Inference

Why Quantized? The INT8 version offers ~3x faster inference speed on standard CPUs compared to the original FP32 PyTorch weights while retaining near-identical transcription accuracy.

---

🚀 Quickstart

Option 1: Standard Inference (PyTorch)

from transformers import pipeline

transcriber = pipeline("automatic-speech-recognition", model="njand/wav2vec2-xls-r-latin")
result = transcriber("sample.wav")
print(result["text"])

Option 2: Optimized Inference (Optimum ONNX Runtime)

For CPU inference, I recommend using the INT8 quantized model. Install dependencies: pip install "optimum[onnxruntime]"

from optimum.onnxruntime import ORTModelForCTC
from transformers import AutoProcessor

# Load the quantized model
model = ORTModelForCTC.from_pretrained(
    "njand/wav2vec2-xls-r-latin", 
    file_name="model_quantized.onnx"
)
processor = AutoProcessor.from_pretrained("njand/wav2vec2-xls-r-latin")

# Inference pipeline
inputs = processor("sample.wav", return_tensors="pt")
outputs = model(**inputs)
# ... decode outputs

---

🔤 Orthography & Text Normalization

This model transcribes audio using the following orthographic conventions:

• Vowel Quantity: Macronizes long vowels (ā, ē, ī, ō, ū, ȳ).
• Consonantal vs. Vocalic Glides: Distinguishes consonantal j and v from vocalic i and u (e.g., ējiciō rather than eicio, vīvus rather than uiuus).
• Casing & Punctuation: Transcribes lowercase text with no punctuation.

---

📊 Cross-Validation Performance

Evaluated across 5-fold cross-validation on njand/llpsi-speech-dataset:

Fold	Strict WER	Strict CER	Norm WER	Norm CER
Fold 0	27.09%	13.06%	25.26%	12.48%
Fold 1	26.33%	12.92%	24.61%	12.33%
Fold 2	26.27%	12.74%	24.59%	12.18%
Fold 3	27.21%	12.91%	25.26%	12.27%
Fold 4	28.16%	13.12%	26.59%	12.58%
Average (CV)	27.01%	12.95%	25.26%	12.37%

Note: Normalized WER/CER measure core word recognition (macrons stripped, j/v → i/u), while Strict WER/CER enforce exact macron placement and j/v orthography.

---

⚡ Environmental Impact & Compute

• Hardware: NVIDIA L4 GPU (24GB VRAM) via Modal
• Total Training Time: 5.87 hours
• Estimated Carbon Emissions: 0.3135 kg CO2eq

---

⚠️ Limitations & Out-of-Scope Use

• Pronunciation Variances: Performance will drop on audio using Ecclesiastical / Italianate pronunciation (e.g., pronouncing c before e/i as /tʃ/ rather than /k/).
• Audio Conditions: Optimized for clear, single-speaker reading. Background noise or overlapping speakers will reduce accuracy significantly.