[QUESTION] Llava VQA performance

[benoriol]

Hello,

What is the expected VQAv2 accuracy of the instruction-finetuned model? Should it reach ~80 as reported by LLaVa1.5? After training for a few thousand steps I get under 70. Not sure if it is just a matter of training for longer or there is any other issue with the training recipe.
Also it seems the LR is 1e-6 vs the 2e-5 reported in LlaVa hyperparameters. Is this intentional? What scores did you get with this recipe?

Thanks,
Benet

Probably for @jon-barker or @trintamaki ?

[Advait251206]

Short answer: no, ~80 VQAv2 is not something you should expect from a short run or from an unverified Megatron-LLaVA recipe without careful hyperparameter alignment. Getting ~65–75 early on is quite common depending on the exact setup.

---

1. Should you expect ~80 VQAv2?

The ~80 VQAv2 number you’re referring to (LLaVA 1.5 / related reports) typically depends on very specific conditions:

• full instruction tuning dataset (not a subset)
• long enough training (often tens of thousands of steps, not a few thousand)
• correct image encoder + projection alignment
• carefully tuned LR schedule
• sometimes higher effective batch size than people assume
• exact eval preprocessing match with VQAv2

So:

❌ “A few thousand steps” is almost always too early
✅ ~80 is a fully converged number under the reference recipe

---

2. Getting <70 after a few thousand steps — is that bad?

Not necessarily.

In most LLaVA-style training runs:

• early phase: ~50–65 VQAv2
• mid training: ~65–75
• convergence: ~75–82 (depending on recipe)

So your result is consistent with:

“training is working but not converged yet”

---

3. Learning rate mismatch (1e-6 vs 2e-5)

This is actually very important.

LLaVA original:

• often uses ~2e-5 for projection / MLP / sometimes LoRA
• sometimes lower LR for full finetuning depending on setup

Megatron / large-scale multimodal setups:

• frequently use much lower LR (1e-6 to 1e-5)

Why?

Because Megatron recipes usually assume:

• larger batch sizes
• more stable distributed training
• risk of destabilizing cross-modal alignment if LR is too high

So yes:

This is likely intentional for stability in large-scale training, not a direct attempt to replicate LLaVA paper hyperparameters.

---

4. Why your results may lag LLaVA 1.5

Common reasons:

(1) Different dataset mixture

LLaVA 1.5 performance depends heavily on:

• CC3M/CC12M-style instruction data
• VQA mixing ratios
• conversation templates

Even small differences matter a lot.

---

(2) Training duration

“few thousand steps” is usually:

• early alignment phase
• not enough for reasoning improvements

---

(3) LR too low (possible)

If you are:

• training only a small projection head

→ 1e-6 may be slightly conservative
→ slows convergence noticeably

---

(4) Missing visual instruction diversity

If dataset lacks:

• multi-turn QA
• reasoning-heavy VQA
• OCR / spatial tasks

→ VQAv2 will plateau lower (~65–70)

---

(5) Eval mismatch

Very common issue:

• prompt format differences
• answer normalization
• VQAv2 evaluation script mismatch

can easily cause 2–5 point differences.

---

5. What I would check first

If I were debugging your run:

Step 1: confirm training is still improving

• is VQAv2 still trending upward after more steps?

Step 2: verify LR schedule

• warmup steps?
• cosine decay?
• peak LR too low?

Step 3: dataset ratio

• instruction vs pure VQA ratio

Step 4: image encoder frozen or not

• frozen ViT vs partially unfrozen matters a lot

---

6. Bottom line

• ❌ <70 after a few thousand steps is not a red flag
• ❌ 80 is not expected early
• ⚠️ LR difference is likely intentional, but could affect convergence speed
• ✅ You likely just need more training + verify dataset + LR schedule

---

If you want, I can help you sanity-check:

• your exact training config
• LR schedule
• dataset mixture
• expected convergence curve

Those usually explain 90% of LLaVA performance gaps.