haripra1112001/llava-vqa-tta

LLaVA-VQA with Test-Time Augmentation (3-Model Ensemble)

A multimodal multiple-choice Visual Question Answering (VQA) system built on the LLaVA framework: a LongCLIP ViT-B/32 vision encoder connected to a Qwen3-0.6B language model through a trained projector, fine-tuned with LoRA.

Inference uses Test-Time option Shuffling (TTA) — each question is answered several times with the A/B/C/D options in different orders, and the scores are averaged back to cancel positional bias. Three independently trained models are combined in a weighted ensemble.

📄 Full technical write-up: llava_vqa_report.pdf — detailed methodology, per-experiment configurations, training dynamics, and error analysis.

Method

Starting from a projector-only baseline, the model was improved through a series of experiments across four areas:

1. Prompt engineering — a structured "analyze the image, then answer" instruction (kept for all later models).
2. LoRA fine-tuning — low-rank adapters injected into the Qwen3 attention layers, which gave the single largest gain.
3. Architecture tweaks — (a) a 4-class classification head on the final hidden state, and (b) a token-logit objective that reads the LM-head logits for the A/B/C/D tokens directly.
4. Regularization — a fresh, regularized LoRA run at rank 16.

At inference, test-time option shuffling averages predictions over multiple option orderings (4 fast / 24 thorough passes), and a weighted ensemble of the three strongest models (exp4 0.35, exp3 0.15, exp5b 0.50) produces the final answer.

Results

Validation accuracy across the progression:

Stage	Val Acc
Baseline (projector only)	46.61%
LoRA r=8 + cosine LR	59.09%
Classification head (exp3)	66.64%
Token-logit + LoRA r=16 (exp4)	68.85%
Fresh regularized LoRA r=16 (exp5b)	69.03%
TTA option shuffling (24 passes, single model)	70.30%
3-model ensemble + TTA (24 passes)	72.15%

See the full report for the complete experiment table, confusion-matrix error analysis, and the option-shuffling debiasing study.

Base models (downloaded automatically, not stored here)

• Vision encoder: AlpachinoNLP/LongCLIP-ViT-B-32
• Language model: Qwen/Qwen3-0.6B

Training data

Trained on the following dataset: juntaoyu/ECS7001P_Assignment2_Data.

wget 'https://huggingface.co/datasets/juntaoyu/ECS7001P_Assignment2_Data/resolve/main/ECS7001P_Assignment2_data.zip'
unzip 'ECS7001P_Assignment2_data.zip' -x '__MACOSX/*'

What this repo contains (the trained artifacts)

Each experiment folder holds a trained projector (best.pt) and a LoRA adapter (lora_best/):

Key	Folder	Answer mechanism	LoRA rank
exp4	experiment_4_token_logit_r16_try3	LM-head logits for A/B/C/D	r=16
exp3	experiment_3_cls_head_try_3	separate classification head	r=8
exp5b	experiment_5b_fresh_regularized_r16_try3	LM-head logits for A/B/C/D	r=16

Ensemble mixing weights (tuned on the validation set):

exp4 = 0.35    exp3 = 0.15    exp5b = 0.50

Installation

pip install -r requirements.txt

Usage

Download the repo, then run single-sample inference:

hf download haripra1112001/llava-vqa-tta --local-dir ./llava-vqa-tta
cd llava-vqa-tta

python tta_inference_configurable.py \
    --image cat.jpeg \
    --question "What animal is shown?" \
    --options "Dog" "Cat" "Bird" "Fish" \
    --mode ensemble --passes 24

From Python / a notebook:

from tta_inference_configurable import TTAPredictor

predictor = TTAPredictor(mode="ensemble", passes=24, checkpoint_root=".")
result = predictor.predict(
    "cat.jpeg",
    "What animal is shown?",
    ["Dog", "Cat", "Bird", "Fish"],
)
print(result["answer"], result["answer_text"], result["confidence"])

Options

• --mode ensemble — combine all 3 models (default, best accuracy).
• --mode single --model exp4 — use one model only (exp4, exp3, or exp5b).
• --passes 4 — fast (balanced Latin-square shuffles).
• --passes 24 — thorough (all 24 permutations of the 4 options).
• --checkpoint-root PATH — where the experiment_* folders live (default: .).

Requirements for input

• One image (any format Pillow can open).
• One question.
• Exactly 4 answer options.

Hardware

Runs on GPU (CUDA) if available, otherwise CPU. The ensemble loads three models, so a GPU is recommended for speed; a single model is lighter.

License

Released under the MIT License. Note the base models carry their own licenses (LongCLIP: MIT; Qwen3-0.6B: Apache-2.0) — review those before downstream use.

Citation / Acknowledgements

Implements the LLaVA multimodal architecture with LoRA fine-tuning and test-time augmentation for multiple-choice visual question answering. Built on LongCLIP and Qwen3-0.6B.

Full methodology and results are documented in the accompanying technical report.