---
pipeline_tag: image-to-text
tags:
- openvision2
- image-captioning
- vision-encoder
library_name: open_clip
---

# OpenVision2 · ViT-H/14 @224 — vision encoder + caption decoder

This repo now contains the **full OpenVision2 generative model** at ViT-H/14, 224px:
the **vision encoder** (originally released here) **plus the caption text decoder** it
was jointly trained with. Together they map an image → a descriptive caption.

The encoder files are unchanged; only the decoder (and this card) were added.

## Files
| file | role |
|---|---|
| `open_clip_pytorch_model.bin`, `open_clip_config.json` | ViT-H/14 vision encoder (open_clip format) |
| `caption_decoder.safetensors` | caption text decoder weights |
| `text_decoder_config.json` | decoder architecture config |
| `modeling_openvision2_decoder.py` | standalone PyTorch decoder + `generate()` |
| `bert_base_vocab_bos_eos.txt` | tokenizer vocab (`[PAD]=0 [bos]=1 [eos]=2`) |
| `caption_example.py` | end-to-end image → caption demo |

## Decoder architecture
A **concat / prefix-LM** autoregressive transformer (not a CoCa cross-attention decoder):
the ViT patch tokens are linearly projected and **prepended as a bidirectional prefix**,
and text is generated **causally** while attending to all image tokens.

`24 layers · width 1024 · 16 heads · mlp 4096 · vocab 32000 · pre-LN · gelu(tanh) ·
LayerNorm eps 1e-6 · no positional embedding on the text stream.`
It consumes the encoder's pre-final-norm patch tokens (open_clip `output_tokens=True`).

## Usage (image → caption)
Needs the patched `open_clip` that provides `create_vision_encoder_and_transforms`
(from https://github.com/UCSC-VLAA/OpenVision), plus `torch`, `safetensors`, `pillow`.

```python
import json, numpy as np, torch
from PIL import Image
from huggingface_hub import hf_hub_download
from open_clip.factory import create_vision_encoder_and_transforms
from modeling_openvision2_decoder import OpenVision2TextDecoder, OpenVision2TextDecoderConfig
from safetensors.torch import load_file

repo = "UCSC-VLAA/openvision2-vit-huge-patch14-224-vision-only"
enc = create_vision_encoder_and_transforms(model_name=f"hf-hub:{repo}").eval()

cfg = json.load(open(hf_hub_download(repo, "text_decoder_config.json")))
dec = OpenVision2TextDecoder(OpenVision2TextDecoderConfig(
    width=cfg["width"], depth=cfg["depth"], num_heads=cfg["num_heads"], mlp_dim=cfg["mlp_dim"],
    vocab_size=cfg["vocab_size"], vision_width=cfg["vision_width"]))
dec.load_state_dict(load_file(hf_hub_download(repo, "caption_decoder.safetensors"))); dec.eval()
vocab = [l.rstrip("\n") for l in open(hf_hub_download(repo, "bert_base_vocab_bos_eos.txt"))]

def preprocess(path, res=224):
    im = Image.open(path).convert("RGB"); w, h = im.size; s = res / min(w, h)
    im = im.resize((round(w*s), round(h*s)), Image.BILINEAR); w, h = im.size
    l, t = (w-res)//2, (h-res)//2; im = im.crop((l, t, l+res, t+res))
    x = (np.asarray(im, np.float32) - np.array([.485,.456,.406])*255) / (np.array([.229,.224,.225])*255)
    return torch.tensor(x.transpose(2,0,1)[None], dtype=torch.float32)

with torch.no_grad():
    _, tokens = enc(preprocess("image.jpg"))
    ids = dec.generate(tokens, max_len=64, bos_id=1, eos_id=2)[0].tolist()

words = []
for i in ids:
    if i == 2: break              # eos
    if i in (0, 1, 2): continue   # pad / bos / eos
    tk = vocab[i]
    if tk.startswith("##"): words[-1] = words[-1] + tk[2:] if words else tk[2:]
    else: words.append(tk)
print(" ".join(words))
```

See `caption_example.py` for a runnable script.

## Notes
- The encoder and decoder are a **matched pair** exported from the same training checkpoint.
- Captions are LLaVA-style **dense** descriptions (the decoder was trained on synthetic
  dense captions), so outputs are detailed and multi-sentence.