---
language: en
tags:
- attnres
- image-classification
- imagenet100
- resnet-50
datasets:
- imagenet-1m
metrics:
- accuracy
- macro-f1
- macro-auc
---

# AttnRes-50: 2D Attention Residuals for ResNet-50

## Model Description
Try to extends the original 1D AttnRes (https://arxiv.org/abs/2603.15031) to 2D spatial feature maps for computer vision. 
**However it seems not work on Classification :(**

| Mode | Description |
|:---|:---|
| **none** | Standard ResNet-50 (baseline)
| **stage** | Intra-stage AttnRes — each block attends to prior block outputs within the same stage |
| **global** | Cross-stage AttnRes — each block attends to all prior block outputs across stages, with spatial/channel alignment via interpolation and 1×1 conv projections | 

## Dataset: ImageNet-100

ImageNet-100 is a 100-class subset of ImageNet-1K (ILSVRC2012). This study uses the split defined by [Tian et al. (2020)](https://github.com/HobbitLong/CMC).

| Split | Samples |
|:---|---:|
| Total | 126,689 |
| Training (80%) | 101,351 |
| Validation (20%) | 25,338 |

The train/val split is deterministic (`torch.manual_seed(42)` with `randperm`).

## Training Details

### Hyperparameters

| Parameter | Value |
|:---|---|
| Optimizer | SGD with momentum (µ=0.9) |
| Learning rate | 0.1 (cosine annealing to 0 over 90 epochs) |
| Weight decay | 1×10⁻⁴ |
| Batch size | 32 |
| Epochs | 90 |
| Input size | 224×224 |
| GPU | NVIDIA A100 (80GB) — single GPU |

### Data Augmentation

- **Training**: RandomResizedCrop(224), RandomHorizontalFlip, ToTensor, Normalize(mean=[0.485,0.456,0.406], std=[0.229,0.224,0.225])
- **Validation**: Resize(256), CenterCrop(224), ToTensor, Normalize (same stats)

## Results

| Mode | Acc@1 | Acc@5 | Val Loss | Macro F1 | Macro AUC | Params |
|:---|---:|:---:|:---:|:---:|:---:|---:|
| **none** (Baseline) | **86.86%** | **96.83%** | **0.5034** | **86.65%** | **99.75%** | 23.77M |
| stage (Intra-Stage) | 85.80% | 96.62% | 0.5268 | 85.59% | 99.74% | 23.77M |
| global (Cross-Stage) | 85.45% | 96.57% | 0.5380 | 85.23% | 99.72% | 28.36M |

- Both attention variants **underperform the standard ResNet-50 baseline** on ImageNet-100 (~1–1.4% lower Acc@1).

## Usage

```python
import torch
from attnres_resnet_demo import AttnResNet50

# Load baseline (standard ResNet-50)
model = AttnResNet50(mode='none', num_classes=100)
ckpt = torch.load('result/best_none.pth', map_location='cpu')
model.load_state_dict(ckpt['model_state_dict'])
model.eval()

# Inference
from PIL import Image
import torchvision.transforms as T

transform = T.Compose([
    T.Resize(256),
    T.CenterCrop(224),
    T.ToTensor(),
    T.Normalize(mean=[0.485, 0.456, 0.406],
                std=[0.229, 0.224, 0.225]),
])

img = Image.open('example.jpg')
x = transform(img).unsqueeze(0)
with torch.no_grad():
    logits = model(x)
    pred = logits.argmax(dim=1)
```

## Checkpoints

| Mode | File | Size |
|:---|---:|---:|
| none | `best_none.pth` | 182 MB |
| stage | `best_stage.pth` | 182 MB |
| global | `best_global.pth` | 200 MB |

Each checkpoint contains:
```python
{
    'epoch': int,
    'mode': str,            # 'none' | 'stage' | 'global'
    'model_state_dict': OrderedDict,
    'optimizer_state_dict': OrderedDict,
    'best_acc1': float,
    'num_classes': int,      # 100
}
```

> **Note for global mode**: The model uses a lazy-initialized `ModuleDict` for cross-stage channel projections. When loading the checkpoint, run a dummy forward pass first to populate the projection layers before calling `load_state_dict`.

## Limitations

- **Small-scale study**: ImageNet-100 is only 100 classes; conclusions may not transfer to full ImageNet-1K.
- **No dual mode results**: The dual-path variant was not trained due to negative findings from stage and global.
- **Single training run per mode**: No repeated runs with different seeds to measure variance.
- **Standard training recipe**: Uses a basic SGD + cosine annealing schedule without modern tricks (warmup, label smoothing, mixup, EMA).

## Citation

If you use this code or findings, please cite the original AttnRes work:

```bibtex
@misc{chen2026attnres,
  title         = {Attention Residuals},
  author        = {Kimi Team  and Chen, Guangyu  and Zhang, Yu  and Su, Jianlin  and Xu, Weixin  and Pan, Siyuan  and Wang, Yaoyu  and Wang, Yucheng  and Chen, Guanduo  and Yin, Bohong  and Chen, Yutian  and Yan, Junjie  and Wei, Ming  and Zhang, Y.  and Meng, Fanqing  and Hong, Chao  and Xie, Xiaotong  and Liu, Shaowei  and Lu, Enzhe  and Tai, Yunpeng  and Chen, Yanru  and Men, Xin  and Guo, Haiqing  and Charles, Y.  and Lu, Haoyu  and Sui, Lin  and Zhu, Jinguo  and Zhou, Zaida  and He, Weiran  and Huang, Weixiao  and Xu, Xinran  and Wang, Yuzhi  and Lai, Guokun  and Du, Yulun  and Wu, Yuxin  and Yang, Zhilin  and Zhou, Xinyu},
  year          = {2026},
  archiveprefix = {arXiv},
  eprint        = {2603.15031},
  primaryclass  = {cs.CL}
}
```