Skip to content

Add Ministral3 Eagle3 reproduction experiment#50

Draft
andylolu2 wants to merge 38 commits into
deepseek-ai:mainfrom
andylolu2:andy/ministral-3b-repro
Draft

Add Ministral3 Eagle3 reproduction experiment#50
andylolu2 wants to merge 38 commits into
deepseek-ai:mainfrom
andylolu2:andy/ministral-3b-repro

Conversation

@andylolu2

@andylolu2 andylolu2 commented Jul 4, 2026

Copy link
Copy Markdown

Why

This experiment checks whether the DeepSpec Eagle3 recipe can reproduce paper-level accepted-length results when adapted to mistralai/Ministral-3-3B-Instruct-2512. The final report is intentionally included even though the result is negative: the implementation works, but the 10k-subset run does not match the Qwen3-4B Eagle3 reference.

Summary

  • Add Ministral3 Eagle3 model/trainer/evaluator support and 10k Open-PerfectBlend reproduction scripts.
  • Add save-only checkpoints, incremental eval JSON writing, split eval task launching, and plotting utilities.
  • Add a final markdown report with training curves and accepted-length comparison plots.

Reviewer guide

  • Real logic changes: deepspec/modeling/eagle3/ministral3, deepspec/trainer/eagle3_trainer.py, deepspec/eval/base_evaluator.py, eval.py, and checkpoint handling in deepspec/trainer/base_trainer.py.
  • Experiment/config changes: config/eagle3/eagle3_ministral3_3b.py and experiments/ministral3_3b_eagle3/*.sbatch.
  • Report artifacts: experiments/ministral3_3b_eagle3/report.md and report_assets/ contain the final result; the report is the best entry point for reviewing the outcome.

Test plan

  • Ran Python syntax checks for eval.py and shell syntax checks for the split eval sbatch.
  • Ran smoke data/cache/train jobs for Ministral3 support before the full run.
  • Completed the 3000-step DeepSpec Eagle3 training run and final 9-task eval. Final eval JSON: /mnt/vast/runs/andy/deepspec_ministral3_3b_eagle3/eval_10k_step3000_metrics.json.
  • Regenerated report plots from TensorBoard and the completed eval JSON.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant