README.md

---
license: mit
language:
- en
tags:
- remote-sensing
- earth-observation
- vision
- feature-extraction
- croma
- sentinel-1
- sentinel-2
- multimodal
library_name: transformers
pipeline_tag: feature-extraction
---

# CROMA Transformers Models

Self-contained HuggingFace model checkpoints for [CROMA](https://arxiv.org/pdf/2311.00566.pdf). Each subfolder ships its own model, image processor, and custom pipeline code — no external `croma` package required.

CROMA expects dual-modality Sentinel-1 (2-channel SAR) and Sentinel-2 (12-channel optical) inputs at 120×120 px by default. Per-channel mean ± 2σ normalization is applied by the bundled image processor.

## Available checkpoints

| Folder | Hidden size | Optical layers | Heads |
|--------|-------------|----------------|-------|
| `croma-base-patch8-120/` | 768 | 12 | 16 |
| `croma-large-patch8-120/` | 1024 | 24 | 16 |

## Usage

Processors default to **`do_resize: false`**. CROMA expects SAR `(2, H, W)` and optical `(12, H, W)` at the same native spatial size (120×120 at pretraining; larger sizes work without forced resize).

```python
from transformers import pipeline

MODEL = "/path/to/CROMA-transformers/croma-base-patch8-120"

pipe = pipeline(
    task="croma-feature-extraction",
    model=MODEL,
    trust_remote_code=True,
)

import numpy as np

sar = np.random.randn(2, 256, 256).astype(np.float32)
optical = np.random.randn(12, 256, 256).astype(np.float32)

joint_gap = pipe(
    sar_images=sar,
    optical_images=optical,
    use_8_bit=True,
    pool=True,
    return_tensors=True,
)

sar_tokens = pipe(
    sar_images=sar,
    output_mode="sar",
    pool=False,
    return_tensors=True,
)
```

Opt in to 120×120 resize:

```python
joint_gap = pipe(
    sar_images=sar,
    optical_images=optical,
    use_8_bit=True,
    pool=True,
    return_tensors=True,
    image_processor_kwargs={"do_resize": True},
)
```

Or load components directly:

```python
from transformers import AutoModel, AutoImageProcessor

model = AutoModel.from_pretrained(MODEL, trust_remote_code=True)
processor = AutoImageProcessor.from_pretrained(MODEL, trust_remote_code=True)

batch = processor(sar_images=sar, optical_images=optical, return_tensors="pt")
outputs = model(**batch)
print(outputs.pooler_output.shape)  # joint GAP
```

## Dependencies

- `transformers`
- `torch`
- `einops`
- `numpy`
- `opencv-python` (only when resizing non-RGB inputs)

## Rebuild from raw checkpoints

```bash
conda activate rsgen
python /path/to/projects/CROMA/scripts/build_transformers_repos.py
```