File size: 2,381 Bytes
49c9a09 | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 | ---
tags:
- vision-transformer
- image-classification
- rope
- imagenet100
- pytorch
license: apache-2.0
datasets:
- imagenet100
metrics:
- accuracy
---
# Rope Vit - IMAGENET100
This model was trained using the [vit-analysis](https://github.com/your-repo/vit-analysis) framework for analyzing Vision Transformer positional encoding methods.
## Model Details
| Property | Value |
|----------|-------|
| **Model Type** | ROPE Vision Transformer |
| **Dataset** | imagenet100 |
| **Best Accuracy** | 77.30% |
| **Image Size** | 224 |
| **Patch Size** | 16 |
| **Hidden Dim** | 192 |
| **Depth** | 12 |
| **Num Heads** | 3 |
| **MLP Dim** | 768 |
| **Num Classes** | 100 |
## Model Description
This is a Vision Transformer with **Rotary Position Embeddings (RoPE)**.
RoPE encodes position information directly into the attention mechanism, enabling better
generalization to different sequence lengths and improved extrapolation capabilities.
- **RoPE Theta:** 10.0
## Usage
```python
import torch
from models import RoPESimpleVisionTransformer
# Initialize model
model = RoPESimpleVisionTransformer(
image_size=224,
patch_size=16,
num_layers=12,
num_heads=3,
hidden_dim=192,
mlp_dim=768,
num_classes=100,
)
# Load checkpoint
checkpoint = torch.load('rope_vit_imagenet100_best.pth', map_location='cpu')
state_dict = checkpoint['state_dict']
# Remove 'module.' prefix if present (from DDP training)
state_dict = {k.replace('module.', ''): v for k, v in state_dict.items()}
model.load_state_dict(state_dict)
model.eval()
# Inference
from torchvision import transforms
from PIL import Image
transform = transforms.Compose([
transforms.Resize(256),
transforms.CenterCrop(224),
transforms.ToTensor(),
transforms.Normalize(mean=[0.485, 0.456, 0.406], std=[0.229, 0.224, 0.225]),
])
image = Image.open('your_image.jpg').convert('RGB')
input_tensor = transform(image).unsqueeze(0)
with torch.no_grad():
output = model(input_tensor)
prediction = output.argmax(dim=1)
```
## Training
This model was trained with:
- **Framework:** PyTorch
- **Optimizer:** AdamW
- **Mixed Precision:** Enabled
## Citation
If you use this model, please cite:
```bibtex
@misc{vit-analysis,
title={Vision Transformer Position Encoding Analysis},
year={2024},
url={https://github.com/your-repo/vit-analysis}
}
```
## License
Apache 2.0
|