---
license: mit
pipeline_tag: multimodal-rating-prediction
tags:
- multimodal
- efficiency
- distilbert
- dinov2
- lora
- cvpr2026
---

# LoViF: LatentQuery Fusion Baseline Checkpoint (DistilBERT + DINOv2)

This repository contains the official baseline checkpoint for the **LoViF 2026 Efficiency Track** multimodal rating prediction task.

## Model Description

The model implements a **LatentQuery Fusion** architecture, designed for high-efficiency multimodal inference. It fuses visual features from DINOv2 and textual features from DistilBERT using a cross-attention based latent query mechanism.

- **Vision Backbone:** `facebook/dinov2-base` (LoRA adapter)
- **Text Backbone:** `distilbert/distilbert-base-uncased` (LoRA adapter)
- **Fusion Method:** LatentQuery Fusion (LQF)
- **Total Parameters:** 186.09M
- **Trainable Parameters (LoRA):** 33.15M

## Performance Metrics

Based on our validation set (SetA validation split):

- **PLCC (Pearson Linear Correlation Coefficient):** 0.2627
- **MSE (Mean Squared Error):** 0.5823
- **FLOPs per sample (k=2):** 116.79G
- **FLOPs per sample (k=1):** 70.58G

## Usage

To use this checkpoint, clone the [LoViF repository](https://github.com/nwpuluka/lovif) and follow the reproduction steps in the README.

### Download via CLI

```bash
huggingface-cli download nwpuluka/lovif-paper-submission-checkpoint best.ckpt --local-dir .
```

### Loading in Python

```python
import torch
checkpoint = torch.load("best.ckpt", map_location="cpu")
model_state = checkpoint["model_state"]
# Load model_state into your LoViF model instance
```

## Reproduction

This checkpoint was trained using the `baseline_lqf_distilbert_dinov2_train.yaml` configuration for one round of optimization with modality dropout enabled.

## Citation

If you use this model in your research, please cite our LoViF 2026 paper.