SINAI/ALIA-MrBERT-es-cultural-heritage-embeddings

ALIA MrBERT Spanish Cultural and Heritage Embeddings Model

This repository contains ALIA MrBERT Spanish Cultural and Heritage Embeddings, a Spanish cultural heritage domain bi-encoder model for semantic similarity and information retrieval tasks. It is built upon MrBERT-es, a bilingual (Spanish–English) foundational language model based on the ModernBERT architecture, and fine-tuned on domain-specific cultural heritage data using a Curriculum Learning strategy.

DISCLAIMER: This model is a domain-specific proof-of-concept designed to demonstrate retrieval capabilities in the Spanish cultural heritage domain. While optimized for this domain, results should be verified against official cultural heritage sources and expert consultation. The model may fail in out-of-domain or adversarial inputs.

---

Model Details

Model Lineage

ModernBERT (architecture)
       ↓
  MrBERT-es (BSC-LT)
  Bilingual ES/EN encoder
  150M parameters
       ↓
  ALIA-MrBERT-es-cultural-heritage-embeddings (SINAI)
  Cultural heritage domain fine-tuning
  Curriculum Learning + Hard Negatives

Key Features

• 🔍 Domain: Spanish cultural and heritage texts
• 📐 Architecture: ModernBERT with Mean Pooling (bi-encoder)
• 📏 Long context: up to 8,192 tokens
• 🎓 Training strategy: Curriculum Learning (easy → medium → hard)
• ⚙️ Negative mining: Positive-Aware Hard Negative Mining (NVIDIA approach)

Architecture

This model uses the same base architecture as MrBERT-es, extended with a Mean Pooling layer for sentence-level embeddings:

Base Architecture	ModernBERT
Total Parameters	~150M
Hidden size	768
Intermediate size	1,152
Attention heads	12
Hidden layers	22
Context length	8,192 tokens
Vocabulary size	51,200
Precision	bfloat16
Positional encoding	RoPE
Activation function	GeLU
Attention type	Mixed (global every 3 layers + sliding window)
Pooling strategy	Mean Pooling

---

Training

Training Strategy: Curriculum Learning

The model was fine-tuned using a two-phase Curriculum Learning strategy and progressively increasing the difficulty of training examples thanks to SINAI/ALIA-es-cultural-heritage-triplets/train

Phase	Epochs	Negative Type	Difficulty Progression
Phase 1	6	Random negatives	Easy → Medium → Hard
Phase 2	3	Hard negatives (mined)	Easy → Medium → Hard
Total	9	—	—

Phase 1 – Contrastive Learning with Random Negatives: Training uses triplets {query, relevant_doc, [irrelevant_docs]} with in-batch negatives. Examples are sorted by difficulty across 3 sub-phases (2 epochs each).

Phase 2 – Advanced Refinement with Hard Negatives: Refinement using mined hard negatives with Positive-Aware Mining (NVIDIA approach) to avoid false negatives. A candidate is only considered a negative if:

score < score_positive - margin   (margin = 0.05)

Hyperparameter Optimization

Before training, hyperparameter search was conducted using Optuna (20 trials, subsets of 5,000 examples):

• Sampler: TPESampler (Tree-structured Parzen Estimator)
• Pruner: MedianPruner
• Storage: SQLite for trial persistence

Hyperparameter	Search Space
Learning Rate	[1×10⁻⁶, 5×10⁻⁵] (log-uniform)
Warmup Ratio	[0.05, 0.2]
Weight Decay	[0.0, 0.1]
Mini Batch Size	{1, 4, 8, 12}

Final Training Hyperparameters

Hyperparameter	Value	Description
Learning Rate	4,7×10⁻⁵	Nominal learning rate
Batch Size	32	Global batch size
Cache Mini-Batch	4	For CachedMultipleNegativesRankingLoss
Warmup Ratio	0.1978	Linear LR warmup at start of each phase
Weight Decay	0.0078	L2 regularization
Optimizer	AdamW	Standard HuggingFace Trainer optimizer
Precision	bf16	Bfloat16 for Ampere+ architectures
Max Sequence Length	8,192	Maximum tokens processed

Loss Function

• CachedMultipleNegativesRankingLoss: Enables training with large batches (256) without VRAM overflow, by recalculating embeddings in smaller sub-batches (cache size: 4).

Training Framework

Component	Details
Library	sentence-transformers
Distributed	DDP (Distributed Data Parallel) via torchrun
Memory optimization	Gradient Checkpointing
Logging	WandB (offline mode)

---

Intended Use

Direct Use

This model is designed for semantic similarity and information retrieval tasks in the Spanish cultural heritage domain. Primary use cases include:

• Semantic search: Finding relevant cultural heritage documents from a query
• RAG pipelines: Generating context-enriched answers about cultural heritage using retrieval-augmented generation
• Heritage document clustering: Grouping similar cultural texts by semantic content
• Duplicate detection: Identifying semantically similar cultural heritage descriptions or passages

Out-of-Scope Use

• General-domain retrieval (model is specialized for cultural heritage Spanish)
• Cross-lingual retrieval beyond Spanish
• Use as a generative model (this is an encoder-only model)
• Authoritative assessments of cultural significance without expert validation

---

How to Use

With sentence-transformers

from sentence_transformers import SentenceTransformer

model = SentenceTransformer("SINAI/ALIA-MrBERT-es-cultural-heritage-embeddings")

queries = ["¿Qué información existe sobre el patrimonio inmaterial de los pueblos de Andalucía?"]
documents = [
    "El Toque Manual de Campanas es un patrimonio inmaterial que ha sido declarado por la UNESCO como expresión de la identidad cultural española...",
    "Los bienes culturales inmateriales incluyen tradiciones, rituales, conocimientos orales y expresiones que se transmiten de generación en generación...",
]

query_embeddings = model.encode(queries, prompt_name="query")
doc_embeddings = model.encode(documents)

scores = model.similarity(query_embeddings, doc_embeddings)
print(scores)

With transformers (manual)

import torch
from transformers import AutoTokenizer, AutoModel

model_name = "SINAI/ALIA-MrBERT-es-cultural-heritage-embeddings"
tokenizer = AutoTokenizer.from_pretrained(model_name, trust_remote_code=True)
model = AutoModel.from_pretrained(model_name, trust_remote_code=True)

def mean_pooling(model_output, attention_mask):
    token_embeddings = model_output[0]
    input_mask_expanded = attention_mask.unsqueeze(-1).expand(token_embeddings.size()).float()
    return torch.sum(token_embeddings * input_mask_expanded, 1) / torch.clamp(input_mask_expanded.sum(1), min=1e-9)

texts = ["¿Cuáles son los monumentos históricos más importantes de la España medieval?"]
encoded = tokenizer(texts, padding=True, truncation=True, return_tensors="pt")

with torch.no_grad():
    outputs = model(**encoded)

embeddings = mean_pooling(outputs, encoded["attention_mask"])
print(embeddings.shape)  # (1, 768)

---

Evaluation

The model was evaluated using the MTEB (Massive Text Embedding Benchmark) framework, adapted for the legal domain. The main reported metric is NDCG@10 (Normalized Discounted Cumulative Gain at k=10), which is the standard metric used in retrieval leaderboards and aligns with the metric reported in the MrBERT family.

An additional evaluation was made thanks to ragas evaluation framework and MiniMaxAI/MiniMax-M2.5 language model. These metrics are calculated by averaging on each pair puntuation from particular subsets of some of the following datasets.

Evaluation Datasets

Dataset	Category	Description
QA	Retrieval	Spanish subset of the MIRACL dataset in MTEB format (jinaai/miracl-es)
STS	STS	Combination of three Spanish STS datasets: PAWS-X (google-research-datasets/paws-x), STS22 (mteb/sts22-crosslingual-sts), and SemRel2024 (SemRel/SemRel2024)
EVAU	Retrieval	Subset consisting of manually curated questions and answers aligned with the Spanish Evaluación para el Acceso a la Universidad (EVAU) (ESQAD/EvAU_QA)
ices	Retrieval	Iberian characters information retrieval dataset built from somosnlp-hackathon-2025/ibero-characters-es
andalucia_org	Retrieval	Set of synthetic pairs based on informative texts about the Autonomous Community of Andalusia in andalucia.org
pairs1940	Retrieval	Subset of 1,940 cultural evaluation pairs (query + passage) located in SINAI/ALIA-es-cultural-heritage-triplets/test.
pairs7275	Retrieval	Subset of 7,275 cultural evaluation pairs (query + passage) located in SINAI/ALIA-es-cultural-heritage-triplets/test.

Results (NDCG@10)

Model	QA	STS	EVAU	ices	andalucia_org	pairs1940	pairs7275
BAAI/bge-m3	0.9839	0.4629	0.1476	0.9415	0.9761	0.9797	0.9708
Qwen/Qwen3-Embedding-0.6B	0.9869	0.5033	0.1944	0.9528	0.9816	0.9933	0.9889
sentence-transformers/paraphrase-multilingual-mpnet-base-v2	0.9419	0.4632	0.0651	0.7601	0.5749	0.4095	0.3509
ALIA-MrBERT-es-cultural-heritage-embeddings (ours)	0.9756	0.5145	0.1759	0.9554	0.9869	0.9935	0.9912

BAAI/bge-m3 and Qwen/Qwen3-Embedding-0.6B are SOTA models with a size x4 larger than ours. Yet, our model shows comparable results.

Results (ragas)

Subset: triplets_queries1724_contexts7947

Model	Context Precision (avg)	Context Utilization (avg)	Context Relevance (avg)
Qwen/Qwen3-Embedding-0.6B	0.9738	0.9731	0.9555
ALIA-MrBERT-es-cultural-heritage-embeddings (ours)	0.9837	0.9844	0.9581

Subset: EVAU_queries1200_contexts1200

Model	Context Relevance (avg)
Qwen/Qwen3-Embedding-0.6B	0.587
ALIA-MrBERT-es-cultural-heritage-embeddings (ours)	0.4503

Subset: ices_queries72_contexts72

Model	Context Relevance (avg)
Qwen/Qwen3-Embedding-0.6B	0.9583
ALIA-MrBERT-es-cultural-heritage-embeddings (ours)	0.9618

Note: Each evaluation subset is named following the pattern {dataset}_queries{N}_contexts{M}, where N is the number of queries evaluated against M contexts taken from the datasets.

---

Limitations and Biases

Known Limitations

• Domain specificity: The model is optimized for Spanish cultural heritage texts. Performance may degrade significantly on general-domain or other specialized texts.
• Language: Although MrBERT-es supports Spanish and English, this fine-tuned model focuses on Spanish cultural heritage content.
• Cultural expertise: Semantic similarity does not guarantee accurate cultural assessment or historical correctness. Retrieved documents should always be verified by qualified cultural heritage experts.
• Context length: Despite supporting up to 8,192 tokens, very long documents may require chunking strategies for optimal retrieval performance.

Biases

• The model may reflect biases present in the Spanish cultural heritage corpus used for training.
• It may underperform on cultural heritage texts from underrepresented regions or minority communities, as training focused on well-documented Spanish national heritage.

---

Additional Information

License

Apache License, Version 2.0

Citation

If you use this model in your research, please cite:

@misc{ALIA-MrBERT-es-cultural-heritage-embeddings,
  title        = {ALIA MrBERT Spanish Cultural and Heritage Embeddings Model},
  author       = {SINAI Research Group, Universidad de Jaén},
  year         = {2026},
  publisher    = {HuggingFace},
  howpublished = {\url{https://huggingface.co/SINAI/ALIA-MrBERT-es-cultural-heritage-embeddings}}
}

Please also cite the base model:

@misc{tamayo2026mrbertmodernmultilingualencoders,
      title={MrBERT: Modern Multilingual Encoders via Vocabulary, Domain, and Dimensional Adaptation}, 
      author={Daniel Tamayo and Iñaki Lacunza and Paula Rivera-Hidalgo and Severino Da Dalt and Javier Aula-Blasco and Aitor Gonzalez-Agirre and Marta Villegas},
      year={2026},
      eprint={2602.21379},
      archivePrefix={arXiv},
      primaryClass={cs.CL},
      url={https://arxiv.org/abs/2602.21379}, 
}

Funding

This work is funded by the Ministerio para la Transformación Digital y de la Función Pública - Funded by EU – NextGenerationEU within the framework of the project ALIA.

Acknowledgments

This dataset has been generated thanks to CEATIC ( Centro de Estudios Avanzados en Tecnologías de la Información y de la Comunicación) – UJA (Universidad de Jaén) which provided the needed computational resources on its clusters.

---

Contact: ALIA Project - SINAI Research Group - Universidad de Jaén

More Information: SINAI Research Group | ALIA-UJA Project