--- language: - en license: unknown library_name: mmdetection pipeline_tag: object-detection inference: false datasets: - lvis tags: - object-detection - co-detr - co-dino - detr - coco - swin - mmdetection - pytorch - iccv2023 - lvis - objects365 model-index: - name: co_dino_5scale_lsj_swin_large_16e_o365tolvis results: - task: type: object-detection dataset: name: LVIS v1 val type: lvis split: val metrics: - type: box AP value: 64.5 name: box AP --- # Co-DINO (Swin-L, 16 epochs (Objects365 pretrain, LVIS fine-tune), LVIS)

## Disclaimer This is **not** an official release. All credit for the method, the architecture, and the trained weights belongs entirely to the original authors: **Zhuofan Zong, Guanglu Song, and Yu Liu** (SenseTime X-Lab). I claim no contribution to the underlying research or training. I host the *same* checkpoint the authors published to Google Drive, with a model card and a copy of its config file. ## Mirror rationale The original non-ViT Co-DETR checkpoints are distributed through a single shared Google Drive folder, which is inconvenient for automated and reproducible downloads (no per-file addressing, checksums, or versioning). The authors published their ViT-L checkpoints to the Hub but not the ResNet-50 / Swin ones. I mirror this checkpoint to provide stable artifact hosting, an explicit file checksum (see [Provenance](#provenance)), and versioned Hub access. I mirror only the pretrained weights; I do not redistribute any training or evaluation data. I will transfer or remove this mirror at the original authors' request. Please cite the original paper (see [Citation](#citation)) and use the [official repository](https://github.com/Sense-X/Co-DETR) or the maintained fork [`dronefreak/Co-DETR`](https://github.com/dronefreak/Co-DETR) for code. --- # Quickstart > This is **not** a plain-PyTorch / `transformers` model. Inference needs the > Co-DETR project code and the OpenMMLab 1.x stack (MMDetection 2.25.3, > MMCV-full 1.5.0, PyTorch 1.11). The maintained fork ships a one-command > environment setup for exactly this. ```bash # 1. Get the code + a validated environment git clone https://github.com/dronefreak/Co-DETR && cd Co-DETR bash tools/setup_codetr_env.sh # builds the `codetr` conda env conda activate codetr # 2. Pull this checkpoint + its config from the Hub pip install -U "huggingface_hub[cli]" hf download dronefreak/co-dino-5scale-lsj-swin-l-o365-lvis \ co_dino_5scale_lsj_swin_large_16e_o365tolvis.pth co_dino_5scale_lsj_swin_large_16e_o365tolvis.py \ --local-dir checkpoints/ # 3. Run detection on an image / folder / video / webcam python tools/inference.py \ --config checkpoints/co_dino_5scale_lsj_swin_large_16e_o365tolvis.py \ --checkpoint checkpoints/co_dino_5scale_lsj_swin_large_16e_o365tolvis.pth \ --input demo/demo.jpg --out-dir outputs/ --save-json ``` The bundled config is identical to `projects/configs/co_dino/co_dino_5scale_lsj_swin_large_16e_o365tolvis.py` in the repo. --- # What is Co-DETR / this checkpoint **Co-DETR** ("DETRs with Collaborative Hybrid Assignments Training") is a training scheme, not a new architecture. It attaches auxiliary heads that use *one-to-many* label assignment (an ATSS head and a Faster-R-CNN-style RoI head) alongside the DETR decoder's *one-to-one* Hungarian matching, and feeds the positive proposals those heads produce back into the decoder as extra queries. This makes the encoder's features more discriminative and speeds up convergence. The auxiliary heads are used **only during training**: this checkpoint's config sets `eval_module='detr'`, so at inference only the Co-DINO query head runs and the RPN / RoI / ATSS heads add no inference-time computation to the deployed detector. **This checkpoint** is Co-DINO (5 feature scales, large-scale jitter (LSJ)) with a **Swin-L** backbone: | | | |---|---| | Detector | `CoDETR` meta-model; deployed head is `CoDINOHead` | | Backbone | Swin-L, pretrained on ImageNet-22k | | Object queries | 900 | | Aux heads in state-dict | `RPNHead` + `CoStandardRoIHead`, `CoATSSHead`: present in the weights but **not executed at inference** (`eval_module='detr'`) | | Parameters | 221.5 M executed at inference (backbone + neck + query head). The checkpoint additionally stores the training-only RPN / RoI / ATSS heads (246.1 M total), which inference never touches. | | Classes | 1203 (LVIS v1 categories, federated/long-tailed evaluation) | | Training schedule | Objects365 pretrain -> LVIS v1 fine-tune, 16 epochs (Objects365 pretrain, LVIS fine-tune) | | Test-time input | single scale, no flip / no TTA (the config's test pipeline; see [Measured inference footprint](#measured-inference-footprint) for the actual resized size on the sample image) | - **Paper**: [DETRs with Collaborative Hybrid Assignments Training](https://arxiv.org/abs/2211.12860), ICCV 2023 - **Config**: [`co_dino_5scale_lsj_swin_large_16e_o365tolvis.py`](co_dino_5scale_lsj_swin_large_16e_o365tolvis.py) (bundled; identical to the Co-DETR repo copy) --- # Intended use & limitations **Intended use.** Research and engineering work that needs an object detector in the Co-DETR family trained on LVIS v1 `val`: as a baseline, a feature/detection backbone, or a starting point for fine-tuning on your own dataset. **Limitations.** - Trained and evaluated on **LVIS v1**; predicts **1203 long-tailed categories**, not the 80 COCO classes. LVIS uses federated evaluation (not every category is exhaustively annotated in every image). Read the [LVIS paper](https://arxiv.org/abs/1908.03195) before comparing its AP to a COCO-80 number. - Accuracy can degrade substantially on domain-shifted imagery (aerial, medical, document, non-natural images, etc.). - The reported **64.5 box AP is the authors' number; I have not reproduced it** (see [Evaluation](#evaluation-as-reported-by-the-original-authors)). - **No real-time claim.** See [measured latency](#measured-inference-footprint) for the actual timings I got. - Runs only on the **legacy OpenMMLab 1.x stack** (MMDetection 2.25.3 / MMCV-full 1.5.0 / PyTorch 1.11); no `transformers` / `pipeline()` support, and `inference: false` on the Hub. - The auxiliary training heads in the checkpoint are **not exposed as inference outputs**; only the DETR query head's detections are returned. --- # What I checked Integrity and "does it load and run" checks only. This is **not** an accuracy reproduction: - It loads in the `codetr` env (Python 3.8 / torch 1.11.0+cu113 / mmcv-full 1.5.0 / mmdet 2.25.3): **not** a full state-dict key match: unexpected keys: 3 `*.fed_loss_weight` (a training-only LVIS federated-loss buffer), none affecting inference. - I ran it end-to-end through `tools/inference.py` on the two dashcam clips shown in the banner above (GPU, `cuda:0`); it produced valid annotated outputs. - I did **not** re-measure LVIS v1 `val` AP; the 64.5 box AP below is the authors' number. ## Measured inference footprint **Method.** I built the model with `mmdet.apis.init_detector`, then ran `inference_detector()` on a single image, `device='cuda:0'`, **batch size 1, FP32** (no AMP), `cudnn.benchmark=False`: one warm-up call, then `torch.cuda.reset_peak_memory_stats()` and 10 timed calls with `torch.cuda.synchronize()`. `max_memory_allocated` includes the resident model weights plus activation buffers; it **excludes** the CUDA context. | Item | Value | |---|---| | GPU | NVIDIA GeForce RTX 4070 SUPER, 12 GB, driver 580.173.02 | | Stack | Python 3.8, torch 1.11.0+cu113, mmcv-full 1.5.0, mmdet 2.25.3 | | Precision / batch | FP32 / 1 | | Params executed at inference | **221.5 M** | | Params stored in the checkpoint | 246.1 M | | Test image | `demo/demo.jpg` -> 1536 x 1536 fed to backbone | | Peak `max_memory_allocated` | 4065 MiB (4.26 GB) | | Peak `max_memory_reserved` | 5910 MiB (6.20 GB) | | Latency / image (warm) | mean 695 ms (693-699, n=10) | CPU inference (`--device cpu`) also works and needs no GPU; it is slower. --- # Evaluation (as reported by the original authors) | Benchmark | Metric | Value | Source | Reproduced here? | |---|---|---|---|---| | LVIS v1 `val` | box AP | 64.5 | Co-DETR paper (arXiv:2211.12860) / official model zoo | **No** | Schedule: 16 epochs (Objects365 pretrain, LVIS fine-tune), 900 object queries, single-scale test, no TTA. I have not independently re-measured these numbers. --- # Training data - **Objects365** pretraining: ~2M images, 365 categories. Shao et al., *Objects365: A Large-Scale, High-Quality Dataset for Object Detection*, ICCV 2019 ([paper](https://openaccess.thecvf.com/content_ICCV_2019/papers/Shao_Objects365_A_Large-Scale_High-Quality_Dataset_for_Object_Detection_ICCV_2019_paper.pdf)). **Objects365's terms restrict use to non-commercial research**; see [License status](#license-status). - **LVIS v1** fine-tuning: 1203 categories on the COCO 2017 image set with different (federated, long-tailed) annotations. Gupta, Dollar & Girshick, *LVIS: A Dataset for Large Vocabulary Instance Segmentation*, CVPR 2019 ([arXiv:1908.03195](https://arxiv.org/abs/1908.03195)). - Backbone initialization: Swin-L pretrained on ImageNet-22k. I do **not** redistribute LVIS or COCO here. Get LVIS from [lvisdataset.org](https://www.lvisdataset.org/) (it reuses COCO 2017 images; see cocodataset.org for those) and review both datasets' own terms yourself. This checkpoint's backbone/detector **was** pretrained on Objects365 before the fine-tune above. Unlike the plain-COCO checkpoints in this zoo, Objects365's non-commercial-research terms are a real constraint here, not a hypothetical one. --- # License status The Co-DETR **source code** is MIT-licensed (© 2022 SenseTime X-Lab, [LICENSE](https://github.com/Sense-X/Co-DETR/blob/main/LICENSE)). The upstream repository does **not** provide a separate, explicit license for this checkpoint's weights. The original authors have published *other* Co-DETR checkpoints (the ViT-L family) on Hugging Face with MIT metadata, but I do **not** treat that as definitive evidence that these weights are independently licensed under MIT. Because the applicable rights are undetermined, I set the Hub metadata for this repo to `license: unknown`. **Verify the applicable rights yourself before redistribution or commercial use.** ⚠️ **Objects365 pretrain.** This checkpoint's backbone/detector was pretrained on the Objects365 dataset before being fine-tuned on LVIS. Objects365's own terms restrict it to **non-commercial research use** ([Objects365 terms](http://www.objects365.org/download.html)); that restriction plausibly carries through to weights derived from it. Treat this checkpoint as research-only unless you independently confirm otherwise; this is a stronger caveat than the general `license: unknown` above, not a substitute for it. If you are one of the original authors and want this mirror removed or transferred, open an issue on this repo or contact [dronefreak](https://github.com/dronefreak) and I will action it. --- # Citation ```bibtex @inproceedings{zong2023detrs, title={DETRs with Collaborative Hybrid Assignments Training}, author={Zong, Zhuofan and Song, Guanglu and Liu, Yu}, booktitle={Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV)}, pages={6748--6758}, year={2023} } ``` The underlying detector and backbone: ```bibtex @inproceedings{zhu2021deformable, title={Deformable DETR: Deformable Transformers for End-to-End Object Detection}, author={Zhu, Xizhou and Su, Weijie and Lu, Lewei and Li, Bin and Wang, Xiaogang and Dai, Jifeng}, booktitle={International Conference on Learning Representations (ICLR)}, year={2021} } @inproceedings{liu2021swin, title={Swin Transformer: Hierarchical Vision Transformer using Shifted Windows}, author={Liu, Ze and Lin, Yutong and Cao, Yue and Hu, Han and Wei, Yixuan and Zhang, Zheng and Lin, Stephen and Guo, Baining}, booktitle={Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV)}, year={2021} } @inproceedings{shao2019objects365, title={Objects365: A Large-Scale, High-Quality Dataset for Object Detection}, author={Shao, Shuai and Li, Zeming and Zhang, Tianyuan and Peng, Chao and Yu, Gang and Zhang, Xiangyu and Li, Jing and Sun, Jian}, booktitle={Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV)}, year={2019} } @inproceedings{gupta2019lvis, title={LVIS: A Dataset for Large Vocabulary Instance Segmentation}, author={Gupta, Agrim and Dollar, Piotr and Girshick, Ross}, booktitle={Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)}, year={2019} } ``` Built with [MMDetection](https://github.com/open-mmlab/mmdetection) / [MMCV](https://github.com/open-mmlab/mmcv) (OpenMMLab). --- # Acknowledgements I sincerely thank **Zhuofan Zong, Guanglu Song, and Yu Liu** for developing Co-DETR and releasing the pretrained weights, and the OpenMMLab team for MMDetection and MMCV. This mirror only makes one of their existing checkpoints easier and more reliable to obtain. --- # Provenance `co_dino_5scale_lsj_swin_large_16e_o365tolvis.pth`, 2,958,206,635 bytes, SHA-256 `0dd664cb2373e4475d8f9ebc5f37a55c3cbead2d1ab076abd35e792af232f3a2`. I copied it byte-for-byte and unmodified from the authors' Google Drive folder `1nAXOkzqrEgz-YnXxIEs4d5j9li_kmrnv`. Repo files: `co_dino_5scale_lsj_swin_large_16e_o365tolvis.pth` (weights), `co_dino_5scale_lsj_swin_large_16e_o365tolvis.py` (the MMDetection config needed to build the model), `config.json` (metadata summary; also what the Hub uses to count downloads), plus `assets/demo_banner.mp4` / `assets/demo_banner_poster.jpg` for the card. `config.json` is descriptive only, `inference: false`, and is not a `transformers` config.