feat: add inference_speed benchmark with throughput score#148
Open
lwalew wants to merge 2 commits into
Open
Conversation
|
Adds a new `inference_speed` benchmark (leaving `scaling` untouched) that measures MD throughput and how it scales with system size, and turns it into a score. - Reuses the `scaling` dataset via a new `dataset_name` hook on the base Benchmark, so no duplicate data is shipped. - Produces a Hill-function speed score 1 / (1 + (t / t0)^k) on the per-atom step time, averaged over systems; faster models score higher. The score is wall-clock based and only comparable across models run on the same GPU. It contributes to the overall model score. - New GUI page: throughput (ns/day) with log-log axes, power-law fit lines, per-episode variance error bars, and a per-model summary table. - Records per-episode times and the MD timestep on the result. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
lwalew
force-pushed
the
feat/scaling-benchmark-revamp
branch
from
9420481 to
4ec5c8d
Compare
lwalew
changed the title
inference_speed now reports two complementary speeds per system: - Model throughput (atoms/s): the raw forward pass, engine-independent. For mlip models this times the pure network forward on a pre-built graph (mirroring mlip-jax scripts/time_inference.py); for external ASE calculators it forces a recompute on the pre-built atoms (includes their neighbour-list build). Uses warm-up, block-until-ready/numpy-read sync, and slowest-20% outlier trimming. - MD throughput (ns/day): the existing end-to-end short MD run. The speed score now uses the per-atom model forward time (engine-independent) rather than the MD step time. The GUI gains a metric selector across both families and a graph-cutoff column in the summary table. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Adds a new
inference_speedbenchmark. The existingscalingbenchmark is left completely untouched (verified: no diff vsdevelop).What it does
Measures MD throughput and how it scales with system size, and turns it into a score.
scalingdataset via a newdataset_namehook on the baseBenchmark— no duplicate data shipped.1 / (1 + (t/t₀)ᵏ)on the per-atom step time (size-normalised), averaged over systems → faster models score higher. Contributes to the overall model score (scalingstill does not).Caveats
t₀(SCORE_PER_ATOM_STEP_TIME_MIDPOINTininference_speed.py) is a documented placeholder calibrated for H100 — needs tuning against a real run so scores spread sensibly.Tests added for the new benchmark +
compute_speed_score; full suite (132) + ruff + mypy green.🤖 Generated with Claude Code