Skip to content

Add basic benchmarking workflow, create benchmark/ folder with bench script#299

Draft
starsfordummies wants to merge 1 commit into
masterfrom
feature/live_benchmarking
Draft

Add basic benchmarking workflow, create benchmark/ folder with bench script#299
starsfordummies wants to merge 1 commit into
masterfrom
feature/live_benchmarking

Conversation

@starsfordummies

@starsfordummies starsfordummies commented Jan 20, 2025

Copy link
Copy Markdown
Contributor

Ideally this should create some benchmark data in dev/bench/data.js in the gh-pages branch, as well as an accompanying index.html - I have an updated index.html as well but I don't know how to make it such that the pr puts it in the gh-pages folder...

the html file to put in gh-pages should be https://github.com/starsfordummies/testing_livebench/blob/gh-pages/dev/bench/index3.html

More info on the thing in https://github.com/marketplace/actions/continuous-benchmark

@mofeing

mofeing commented Jan 20, 2025

Copy link
Copy Markdown
Collaborator

yeah, don't worry. we can later add it to the gh-pages (although i think we need an automated way).

let's start by adding some benchmarks. @starsfordummies @jofrevalles @Todorbsc what do you think about benchmarking the following?

  • contraction of 2 Tensors (matmul) for Float64 & ComplexF64. at least 3 different sizes (small, mid & large).
  • contraction of 2 Tensors (more complex pattern) with the same variations.
  • MPS overlap for different number of sites (L=10,=30,=60) and maximum bond dimensions
    • for L=10, use 2,32
    • for L=30, use 2,128,1024
    • for L=60, use 128,1024
  • MPS canonization from NonCanonical to Canonical, from NonCanonical to MixedCanonical and from MixedCanonical to MixedCanonical in another site
    • yet to decide the benchmark space

we can benchmark more stuff in the future, but let's start with the following which are key to us right now

@starsfordummies i'm changing this to a draft PR because it's not yet ready to merge

@mofeing mofeing marked this pull request as draft January 20, 2025 15:22
@jofrevalles

Copy link
Copy Markdown
Member

Hey @mofeing @starsfordummies @Todorbsc, I will design some benchmarks for this and let you know. Is this okay?

@mofeing

mofeing commented Jan 22, 2025

Copy link
Copy Markdown
Collaborator

@jofrevalles that sounds great! i would say that feel free to push to this PR so we can see the result

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants