See our collection for all versions of BEiT.

Run BEiT with Keras 3: JAX, PyTorch, or TensorFlow

zeromodels/beit-base-finetuned-ade-640-640

Paper: BEiT: BERT Pre-Training of Image Transformers (arXiv:2106.08254) · HF Papers

BEiT is a ViT-family vision transformer with a per-layer relative position bias, a learnable layer scale on each residual branch, and mean pooling of the patch tokens. Base backbone with a UPerNet head fine-tuned on ADE20K (150 classes).

For more details on the model, please go to Microsoft's original model card.

Pure-Keras 3 conversion of microsoft/beit-base-finetuned-ade-640-640 for zeromodels. One implementation runs unmodified on TensorFlow / Torch / JAX.

This is a semantic segmentation checkpoint (BeitSemanticSegment).

✨ Quick start

import os
os.environ["KERAS_BACKEND"] = "torch"  # or "jax" / "tensorflow"

from PIL import Image
from zeromodels.models.beit import BeitSemanticSegment, BeitImageProcessor

model = BeitSemanticSegment.from_weights("zeromodels/beit-base-finetuned-ade-640-640")
processor = BeitImageProcessor.from_weights("zeromodels/beit-base-finetuned-ade-640-640")  # resize 640 + post-processing

image = Image.open("scene.jpg").convert("RGB")
pixels = processor(image)               # (1, 640, 640, 3), raw [0, 255]; the model normalizes
logits = model(pixels, training=False)  # (1, H/4, W/4, 150), channels_last
seg = processor.post_process_semantic_segmentation(
    logits, target_sizes=[image.size[::-1]]  # (height, width)
)[0]                                     # per-pixel ADE20K label map at full resolution

Load the segmentation checkpoints the same way with from_weights("zeromodels/<variant>") (the classification variants are in the collection):

Variant	Hub	Task
`beit-base-finetuned-ade-640-640`	`zeromodels/beit-base-finetuned-ade-640-640`	semantic segmentation
`beit-large-finetuned-ade-640-640`	`zeromodels/beit-large-finetuned-ade-640-640`	semantic segmentation

Tips

Set KERAS_BACKEND before importing Keras / zeromodels.
Normalization (0.5/0.5) is baked into the model, so pass raw [0, 255] pixels; BeitImageProcessor handles the resize and does not normalize.
BeitSemanticSegment returns logits at a quarter of the input resolution. BeitImageProcessor.post_process_semantic_segmentation(logits, target_sizes=[(H, W)]) upsamples them to your image size (bilinear, then argmax) and returns a per-pixel label map.
BeitModel.from_weights(..., as_backbone=True) returns the per-block token sequences for feature extraction.
See BEiT docs and Loading Weights.
Community / upstream safetensors still work via the hf: prefix, e.g. BeitSemanticSegment.from_weights("hf:microsoft/beit-base-finetuned-ade-640-640").