| --- |
| license: mit |
| language: |
| - en |
| tags: |
| - vqa |
| - visual-question-answering |
| - knowledge-guided |
| - vit |
| - flan-t5 |
| - ok-vqa |
| - multimodal |
| - pytorch |
| datasets: |
| - ok-vqa |
| pipeline_tag: visual-question-answering |
| library_name: pytorch |
| --- |
| |
| # Knowledge-Guided VQA Models |
|
|
| This repository contains trained model weights for **Knowledge-Guided Visual Question Answering** on the [OK-VQA](https://okvqa.allenai.org/) dataset. |
|
|
| These models explore different strategies for injecting external LLM-derived knowledge (visual focus cues and semantic knowledge) into a frozen Vision Transformer (ViT) + Flan-T5 pipeline for open-ended VQA. |
|
|
| ## 🏗️ Architecture |
|
|
| | Component | Model | |
| |:---|:---| |
| | Vision Encoder | `google/vit-base-patch16-224` | |
| | Text Encoder | CLIP Text (`openai/clip-vit-base-patch16`) | |
| | Decoder (Answer Generator) | `google/flan-t5-small` | |
| | Dataset | OK-VQA (train + val split) | |
|
|
| ## 📦 Available Models |
|
|
| ### 1. `best_baseline_model.pth` (309 MB) |
| - **Strategy:** No external knowledge. Standard ViT → CLS projection → T5 decoder. |
| - **Purpose:** Serves as the baseline for comparison. |
|
|
| ### 2. `best_fusion_model.pth` (313 MB) |
| - **Strategy:** Late Fusion. Concatenates ViT CLS token with CLIP-encoded guidance vector, then projects to T5. |
| - **Knowledge Source:** LLM-generated visual focus + semantic knowledge via CLIP text encoder. |
|
|
| ### 3. `best_attention_model.pth` (325 MB) |
| - **Strategy:** Attention Modification. Injects per-layer guidance projections into the top 6 ViT self-attention layers. |
| - **Knowledge Source:** LLM-generated visual focus + semantic knowledge via CLIP text encoder. |
|
|
| ## 🔧 Usage |
|
|
| ```python |
| import torch |
| from transformers import ViTModel, AutoTokenizer, AutoModelForSeq2SeqLM |
| |
| device = torch.device("cuda" if torch.cuda.is_available() else "cpu") |
| |
| # Load base models |
| t5_tokenizer = AutoTokenizer.from_pretrained("google/flan-t5-small") |
| t5 = AutoModelForSeq2SeqLM.from_pretrained("google/flan-t5-small").to(device) |
| vit = ViTModel.from_pretrained("google/vit-base-patch16-224").to(device) |
| |
| # Define your model class (see notebook for full definitions) |
| # Then load weights: |
| model = YourModelClass().to(device) |
| model.load_state_dict(torch.load("best_baseline_model.pth", map_location=device)) |
| ``` |
|
|
| ## 📊 Training Details |
|
|
| - **Optimizer:** Adam (lr=3e-5) |
| - **Early Stopping:** Patience = 3-4 epochs |
| - **Max Epochs:** 15 |
| - **Batch Size:** 8 |
| - **Evaluation Metrics:** Exact Match, Edit Similarity, Token F1, BERTScore |
|
|
| ## 📄 Citation |
|
|
| If you use these models in your research, please cite: |
|
|
| ```bibtex |
| @misc{knowledge-guided-vqa-2026, |
| title={Knowledge-Guided Visual Question Answering with Adaptive Vision-Language Integration}, |
| author={Aneerban}, |
| year={2026}, |
| publisher={Hugging Face}, |
| url={https://huggingface.co/rishii100/knowledge-guided-vqa-models} |
| } |
| ``` |
|
|
| ## 📜 License |
|
|
| MIT License |
|
|
