---
language:
- en
tags:
- clip
- vision-language
- probabilistic
- uncertainty
license: mit
base_model: laion/CLIP-ViT-B-32-laion2B-s34B-b79K
library_name: transformers
---

# BayesVLM-Base

Project: https://aaltoml.github.io/BayesVLM/  
Paper: https://arxiv.org/abs/2412.06014  
GitHub: https://github.com/AaltoML/BayesVLM

## Model summary
BayesVLM-Base is a post-hoc probabilistic version of CLIP ViT-B/32. It augments the standard CLIP image and text embeddings with estimated covariance, and returns mean, variance, and standard deviation for embeddings and logits. The base CLIP weights are unchanged; only additional covariance buffers are added.

## Model details
- Architecture: CLIP ViT-B/32 with projection dimension 512 and BayesVLM covariance buffers for text and vision projections.
- Base model: `laion/CLIP-ViT-B-32-laion2B-s34B-b79K`.
- Outputs: `text_embeds`, `image_embeds`, `logits_per_image`, plus corresponding variance and standard deviation fields.
- Library: `transformers` with custom model code.

## How it works
BayesVLM uses a post-hoc Laplace-style approximation around the projection layers of the CLIP model. The covariance is represented with Kronecker-factorized terms (stored as `A_inv` and `B_inv`), and inference uses these buffers to estimate uncertainty over embedding and logit outputs.

## Intended use
- Zero-shot image classification and retrieval with uncertainty estimates.
- Uncertainty-aware downstream pipelines that consume CLIP embeddings.

## Training data
This model is derived from the CLIP base model `laion/CLIP-ViT-B-32-laion2B-s34B-b79K`. BayesVLM itself is post-hoc and does not retrain CLIP weights. Covariance buffers are computed from Hessian-based approximations as described in the BayesVLM paper.

## Evaluation
Evaluation results are reported in the BayesVLM paper. This model card does not include additional benchmarking.

## Usage
```python
from transformers import AutoModel, CLIPProcessor
import torch

model = AutoModel.from_pretrained("BayesVLM-Base", trust_remote_code=True)
processor = CLIPProcessor.from_pretrained("BayesVLM-Base")

inputs = processor(
    text=["a photo of a dog", "a photo of a cat"],
    images=[...],
    return_tensors="pt",
    padding=True,
)

with torch.no_grad():
    outputs = model(**inputs, return_dict=True)

logits = outputs.logits_per_image
logits_var = outputs.logits_per_image_var
```

## License
MIT.

## Citation
```bibtex
@inproceedings{baumann2026bayesvlm,
  title     = {Post-hoc Probabilistic Vision-Language Models},
  author    = {Baumann, Anton and Li, Rui and Klasson, Marcus and Mentu, Santeri and Karthik, Shyamgopal and Akata, Zeynep and Solin, Arno and Trapp, Martin},
  booktitle = {International Conference on Learning Representations {(ICLR)}},
  year      = {2026},
}
```