Add Factor Graph Attention weights (VisDial v1.0, MRR 66.01)

README.md

@@ -0,0 +1,138 @@
+---
+license: mit
+library_name: transformers
+pipeline_tag: visual-question-answering
+tags:
+  - visual-dialog
+  - visdial
+  - multimodal
+  - attention
+  - factor-graph-attention
+  - ndcg
+datasets:
+  - Idan/visdial-fga-preprocessed
+metrics:
+  - recall
+  - mrr
+  - ndcg
+model-index:
+  - name: fga-ndcg
+    results:
+      - task:
+          type: visual-question-answering
+          name: Visual Dialog
+        dataset:
+          type: visdial
+          name: VisDial v1.0 val
+        metrics:
+          - type: ndcg
+            value: 69.07
+            name: NDCG
+          - type: mrr
+            value: 49.03
+            name: MRR
+          - type: recall
+            value: 34.27
+            name: R@1
+          - type: recall
+            value: 66.15
+            name: R@5
+          - type: recall
+            value: 80.13
+            name: R@10
+---
+
+# Factor Graph Attention — NDCG variant
+
+* A general multimodal attention approach inspired by probabilistic graphical models.
+* Achieves a state-of-the-art performance (MRR) on visual dialog task.
+
+This is [Factor Graph Attention](https://arxiv.org/abs/1904.05880) (CVPR'19) finetuned
+on the **dense relevance annotations**, which optimizes NDCG. Code:
+[idansc/fga](https://github.com/idansc/fga).
+
+For the MRR-oriented model, see [`Idan/fga`](https://huggingface.co/Idan/fga).
+
+## Why there are two models
+
+The sparse training label calls one of the 100 candidates correct and the other 99
+equally wrong. That is what MRR measures. But for "is it daytime?" the candidate list
+contains *yes*, *yeah*, *it is* and *yes it is*, and the label picks one arbitrarily.
+NDCG instead scores against the graded relevance five annotators assigned to every
+candidate.
+
+Training on the graded signal moves NDCG a long way and costs MRR, because the two
+metrics genuinely disagree about what a good ranking is. That tension is the subject of
+the 2020 challenge submission, https://github.com/idansc/mrr-ndcg.
+
+## Results
+
+On VisDial v1.0 val. The baseline is the MRR model this was finetuned from:
+
+| Model | NDCG | MRR | R@1 | R@5 | R@10 | Mean rank |
+| --- | --- | --- | --- | --- | --- | --- |
+| [`Idan/fga`](https://huggingface.co/Idan/fga) (sparse) | 56.46 | **66.01** | 52.46 | 82.95 | 90.97 | 3.92 |
+| **This model** (dense only) | **69.07** | 49.03 | 34.27 | 66.15 | 80.13 | 6.68 |
+| dense + sparse, weight 0.5 | 68.00 | 61.93 | 49.55 | 76.59 | 86.14 | 5.07 |
+| dense + sparse, weight 1.0 | 66.61 | 63.27 | 50.69 | 78.33 | 87.38 | 4.78 |
+
+This checkpoint is the pure-dense end of that trade-off: the best NDCG, and the worst
+MRR. If you want one model that is respectable at both, the `sparse_weight=1.0` row
+gives up 2.5 NDCG to recover 14 MRR, and is reproducible with `--sparse_weight 1.0`.
+
+Trained for 5 epochs at lr 1e-4 on the 2,000-image dense subset of **train**. The val
+annotations are never trained on — they are the evaluation set.
+
+### On the objective
+
+The loss is a soft-label cross entropy: the relevance vector is normalized into a
+distribution and matched against the model's log-softmax over the candidates,
+
+```
+loss = -sum_i p_i log q_i,     p = relevance / sum(relevance)
+```
+
+which is the cross entropy the sparse loss already computes, generalized from a one-hot
+target to a graded one.
+
+A smooth approximation of NDCG itself (ApproxNDCG, Qin et al.) was also tried and is
+implemented in the repo. It is worse on this task — +6.0 NDCG against +12.6, while
+losing more MRR — most likely because it concentrates its gradient on the few positions
+the discount rewards, whereas the soft cross entropy draws signal from all 100
+candidates, which matters when there are only 2,000 training examples.
+
+## Usage
+
+```python
+from fga import FGAForVisualDialog
+
+model = FGAForVisualDialog.from_pretrained("Idan/fga-ndcg")
+outputs = model(**batch)
+ranking = outputs.logits.argsort(dim=-1, descending=True)
+```
+
+Install with `pip install git+https://github.com/idansc/fga.git`.
+
+Reproduce with:
+
+```bash
+python scripts/finetune_dense.py \
+    --model_name_or_path <sparse checkpoint> \
+    --train_dense_path data/visdial_1.0_train_dense_annotations.json \
+    --output_dir models/fga-ndcg \
+    --loss soft_ce --sparse_weight 0.0 \
+    --learning_rate 1e-4 --num_train_epochs 5
+```
+
+## Citation
+
+Please cite Factor Graph Attention if you use this work in your research:
+```
+@inproceedings{schwartz2019factor,
+  title={Factor graph attention},
+  author={Schwartz, Idan and Yu, Seunghak and Hazan, Tamir and Schwing, Alexander G},
+  booktitle={Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition},
+  pages={2039--2048},
+  year={2019}
+}
+```

config.json

@@ -0,0 +1,56 @@
+{
+  "architectures": [
+    "FGAForVisualDialog"
+  ],
+  "classifier_dropout": 0.5,
+  "dtype": "float32",
+  "empty_token_id": 11321,
+  "hidden_ans_dim": 512,
+  "hidden_cap_dim": 128,
+  "hidden_hist_dim": 128,
+  "hidden_img_dim": 2048,
+  "hidden_ques_dim": 512,
+  "initializer_type": "he",
+  "legacy_unary_dropout": false,
+  "lstm_initializer_type": "he",
+  "model_type": "fga",
+  "num_history_rounds": 9,
+  "num_options": 100,
+  "pad_token_id": 0,
+  "sharing_factor_weights": {
+    "4": [
+      9,
+      [
+        0,
+        1
+      ]
+    ],
+    "5": [
+      9,
+      [
+        0,
+        1
+      ]
+    ]
+  },
+  "size_force": false,
+  "stop_token_id": 11320,
+  "text_encoder_type": "lstm",
+  "transformers_version": "5.14.1",
+  "unary_dropout": 0.5,
+  "use_cache": false,
+  "use_pairwise": true,
+  "use_prior": true,
+  "use_self": true,
+  "use_unary": true,
+  "utility_sizes": [
+    100,
+    21,
+    41,
+    37,
+    21,
+    21
+  ],
+  "vocab_size": 11322,
+  "word_embed_dim": 200
+}

model.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:c0005e59451850afbed219efd39c8b462822cca4f55c4dbf270d76f4a41fa66e
+size 209755804