llava-v1.6-vicuna-7b-dflash-text-pretrain-verify

A DFlash speculative-decoding draft model for liuhaotian/llava-v1.6-vicuna-7b, trained on a small (11-row) text-only dataset — a pipeline verification checkpoint, not a real pretrained model. Part of the VL-FLASH project's Phase 5 (LLaVA-1.6 port), Stage 1 of a planned text-pretrain → multimodal-finetune pipeline.

What this actually is (read before using)

• 6 optimizer steps total, on 11 general-knowledge Q&A rows sourced from databricks/databricks-dolly-15k. This exists to prove the render→capture→loss→backward→optimizer-step path works end-to-end for text-only rows (a real code path added this session to vlm_llava_input_adapter.py — the image field is now optional), not to demonstrate real language-modeling quality. Loss went 11.1 → 8.4 over those 6 steps — real, non-degenerate learning signal, but nowhere near converged.
• Architecture: DFlashDraftModel, 5 layers, hidden_size=4096, block_size=16 — same config as the sibling llava-v1.6-vicuna-7b-dflash-vlm-clean-overfit checkpoint in this account.

Intended use

As a warm-start weights source (model.draft_checkpoint_path) for a real Stage-1 pretraining run at proper scale, per the runbook linked below — not for direct serving or benchmarking as-is.

Reproduce / go further

Full runbook (Stage 1 build/regenerate/train/export, then Stage 2 warm-started from a checkpoint like this one, plus GPU-sizing guidance for scaling up): scripts/gates/llava_text_only/RUNBOOK.md in the training fork.