| --- |
| license: apache-2.0 |
| language: |
| - en |
| metrics: |
| - accuracy |
| library_name: transformers |
| tags: |
| - misinformation |
| - fake news |
| - vlm |
| - mllm |
| - llm |
| --- |
| # Model Card |
|
|
| <!-- Provide a quick summary of what the model is/does. --> |
| SNIFFER is a multimodal large language model specifically engineered for Out-Of-Context misinformation detection and explanation. |
| It employs two-stage instruction tuning on [InstructBLIP](https://huggingface.co/Salesforce/instructblip-vicuna-13b), including news-domain alignment and task-specific tuning. |
|
|
| The whole model is composed of three parts: 1) _internal checking_ that analyzes the consistency of the image and text content; 2) _external checking_ that analyzes the relevance between the context of the retrieved image and the provided text, and 3) _composed reasoning_ that combines the two-pronged analysis to arrive at a final judgment and explanation. |
|
|
| Here the checkpoint is used for the _internal checking_ part. |
|
|
|
|
| ## Model Sources |
|
|
| <!-- Provide the basic links for the model. --> |
| - **Paper:** https://arxiv.org/abs/2403.03170 (to be appear in CVPR 2024) |
| - **Project:** https://pengqi.site/Sniffer/ |
| - **Repository:** https://github.com/MischaQI/Sniffer |
|
|
|
|
| ## Results |
|
|
| Dataset: [NewsCLIPpings](https://github.com/g-luo/news_clippings) |
|
|
| <div align="center"> |
| </div> |
|
|
| | Model | All | Fake | Real | |
| | :-------------------- | :----| :----| :----| |
| | SAFE | 52.8 | 54.8 | 52.0 | |
| | EANN | 58.1 | 61.8 | 56.2 | |
| | VisualBERT | 58.6 | 38.9 | 78.4 | |
| | CLIP | 66.0 | 64.3 | 67.7| |
| | DT-Transformer | 77.1 | 78.6 | 75.6 | |
| | CCN | 84.7 | 84.8 | 84.5 | |
| | Neu-Sym detector | 68.2 | - | - | |
| | **SNIFFER (ours)** | **88.4** | **86.9** | **91.8** | |
|
|
|
|
| ## Citation |
|
|
| <!-- If there is a paper or blog post introducing the model, the APA and Bibtex information for that should go in this section. --> |
| ``` |
| @inproceedings{qi2023sniffer, |
| author = {Qi, Peng and Yan, Zehong and Hsu, Wynne and Lee, Mong Li}, |
| title = {SNIFFER: Multimodal Large Language Model for Explainable Out-of-Context Misinformation Detection}, |
| booktitle = {Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition}, |
| year = {2024} |
| } |
| ``` |
|
|
