Idan/fga-vqa-trainval

Open-ended VQA, trained on train2014 + val2014

The protocol behind VQA v1's published test-dev numbers: train on both annotated splits, evaluate on test-dev. Built on the attention layer of Factor Graph Attention (CVPR 2019).

Code: github.com/idansc/fga.

No score, by construction

VQA v1's test answers were never released — visualqa.org distributes annotations for train2014 and val2014 only — so this model has no number attached to it. One exists only after predictions are submitted to the evaluation server.

For a model with a verifiable score, see Idan/fga-vqa: the same architecture trained on train2014 alone, scoring 61.97 on val2014 under the official metric.

Training

351,596 questions (230,084 train + 121,512 val), 36 bottom-up region features, 20 epochs, batch 512, soft_ce against the VQA score each answer earns from the ten annotators — worth about a point over a single label, and about 0.8 over the sigmoid-and-binary-cross-entropy form the 2017 challenge writeup recommends.

Usage

from fga.tasks.vqa import OpenEndedVQAModel

model = OpenEndedVQAModel.from_pretrained("Idan/fga-vqa-trainval")
out = model(question_input_ids=q, image_features=v)

To produce a submission file for the evaluation server:

python scripts/predict_vqa_test.py --model Idan/fga-vqa-trainval --vqa_dir vqa \
    --questions vqa/raw/OpenEnded_mscoco_test2015_questions.json \
    --features vqa/test_features.h5 --output test2015_results.json

That script tokenizes with the training vocabulary and applies the same per-region feature normalization used during training; getting either wrong quietly corrupts a submission.

Citation

@inproceedings{schwartz2019factor,
  title={Factor graph attention},
  author={Schwartz, Idan and Yu, Seunghak and Hazan, Tamir and Schwing, Alexander G},
  booktitle={Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition},
  pages={2039--2048},
  year={2019}
}

@inproceedings{schwartz2017high,
  title={High-Order Attention Models for Visual Question Answering},
  author={Schwartz, Idan and Schwing, Alexander G and Hazan, Tamir},
  booktitle={Advances in Neural Information Processing Systems},
  year={2017}
}