llava-v1.6-vicuna-7b-dflash-vlm-clean-overfit

A DFlash speculative-decoding draft model for liuhaotian/llava-v1.6-vicuna-7b, trained on a small (50-row) image+question dataset as a pipeline proof-of-concept, not a production drafter. Part of the VL-FLASH project's Phase 5 (LLaVA-1.6 port).

What this is

• Architecture: DFlashDraftModel, 5 layers, hidden_size=4096, block_size=16 (config: configs/llava-v1.6-vicuna-7b-dflash.json in the training fork).
• Trained on 50 MMStar rows (real images, target-greedy regenerated answers), early-stopped at step 275/375 once pooled training accuracy held sustained above 90%.
• Warm-start: none — trained from scratch (unlike the sibling text-pretrain checkpoint in this same account, which this checkpoint predates).

Measured result (cold-start replay gate, corrected per-row target)

Per-row target = min(answer_token_len - 1, block_size - 1) — LLaVA's real VQA answers are often a single MCQ letter, so accept_len is capped by each row's own answer length, not by block_size.

• Short answers (< 16 tokens, n=26): 26/26 perfect. Every single short answer (including every single-MCQ-letter row) replays at exactly its maximum possible accept_len.
• Long answers (>= 16 tokens, n=24): 8/24 perfect. The remaining 16 were root-caused to a data-generation bug (--max-tokens 60 truncating some training answers mid-generation) — fixed in the pipeline after this checkpoint was trained, not re-trained against the fix. See the repo's NOTES.md "Export + cold-start replay gate" section for the full breakdown.

Reproduce / go further

Full runbook (this checkpoint's exact training run, plus how to scale up to a bigger dataset and a bigger GPU): scripts/gates/llava_text_only/RUNBOOK.md in the training fork.