Skip to content

Testing deploying with compiler-generated profiling info for various configs and benchmark_ALE - #258

Draft
manodeep wants to merge 17 commits into
mainfrom
use-compiler-profiling
Draft

manodeep wants to merge 17 commits into
mainfrom
use-compiler-profiling

Conversation

@manodeep

@manodeep manodeep commented Jul 28, 2026

Copy link
Copy Markdown
Collaborator

benchmark_ALE test case had a fairly big performance impact from using a 3-stage compilation process - i) compile with -fprofile-generate & run benchmark ii) combine generated profraw binary files into a single profile file (using llvm-profdata merge) and iii) recompile with -fprofile-use=/path/to/single/profile/file

Ideally, each config is compiled with it's own profile info, but it might be a reasonable trade-off to compile with a single profiling info that is generated from raw profiling data from all the configs and test cases.

This PR is to come up with some performance numbers to help us make that choice.

Performance summary table (25k IAF config)

From a single 30-day run of the experiments, so the numbers have some (unknown) error-bars on them. The profiling info in the first 6 build-cd deployments (i.e., pr258-1 through pr258-6, inclusive) are all from benchmark_ALE case. pr258-11 and pr258-14 using profiling info from a 1-year run with OM3 25k IAF.

yrs/Wday WTime (max ocn/wtime) SUs CPUs yrs/core/Wday Directory
7.5 190.9 (81.3%) 17207 2704 2.79e-03 25km_jra_iaf_Jul28_1yr_pr258-12_profile_generate (MOM6 with 47-perf-rho-remap-diags, 1-year profile generation run)
7.5 15.9 (69.6%) 1429 2704 2.76e-03 25km_jra_iaf_Jul28_30days_pr258-3 ( MOM6 with 47-perf-rho-remap-diags, no profiling info)
7.4 15.9 (69.2%) 1433 2704 2.75e-03 25km_jra_iaf_Jul28_30days_pr258-1 (MOM6 with 47-perf-rho-remap-diags, and 47-perf-rho-remap-diags profiling info)
7.4 15.9 (69.1%) 1437 2704 2.74e-03 25km_jra_iaf_Jul28_30days_pr258-14-perf-remap-rho-profile-use (MOM6 with 47-perf-rho-remap-diags, OM3 25k IAF profiling info for 1 year)
7.3 16.2 (68.7%) 1456 2704 2.71e-03 25km_jra_iaf_Jul28_30days_pr258-4 (MOM6 with 47-perf-rho-remap-diags, dev/gfdl profiling info)
7.2 16.4 (69.0%) 1476 2704 2.67e-03 25km_jra_iaf_Jul28_30days_pr258-11-baseline-profile-use (deployed MOM6, OM3 25k IAF profiling info for 1 year)
7.2 16.5 (70.4%) 1483 2704 2.66e-03 25km_jra_iaf_Jul28_30days_pr258-5 (deployed MOM6, dev/gfdl profiling info)
7.1 201.5 (85.2%) 18161 2704 2.64e-03 25km_jra_iaf_Jul28_1yr_pr258-10_profile_generate (deployed MOM6, 1-year profile generation run)
6.8 17.4 (71.4%) 1570 2704 2.51e-03 25km_jra_iaf_Jul28_30days (baseline with official deployment)
6.5 18.1 (63.4%) 1634 2704 2.41e-03 25km_jra_iaf_Jul28_30days_pr258-6 (deployed MOM6, 47-perf-rho-remap-diags profiling info)

Performance summary table (25k RYF config)

From a set of 3 runs, with repeat=True for a 30-day experiment. The profiling info in the first 6 build-cd deployments (i.e., pr258-1 through pr258-6, inclusive) are all from benchmark_ALE case. pr258-11 and pr258-14 using profiling info from a 1-year run with OM3 25k IAF.

yrs/Wday WTime (max ocn/wtime) SUs CPUs yrs/core/Wday Directory
7.9 15.0 (70.1%) 1348 2704 2.93e-03 25km_jra_ryf_Aug4_30days_pr258-14
7.9 15.0 (70.0%) 1353 2704 2.91e-03 25km_jra_ryf_Aug4_30days_pr258-1
7.8 15.2 (69.5%) 1372 2704 2.87e-03 25km_jra_ryf_Aug4_30days_pr258-4
7.8 15.2 (69.5%) 1374 2704 2.87e-03 25km_jra_ryf_Aug4_30days_pr258-3
7.7 15.3 (70.9%) 1379 2704 2.86e-03 25km_jra_ryf_Aug4_30days_pr258-11
7.5 15.7 (70.6%) 1418 2704 2.78e-03 25km_jra_ryf_Aug4_30days_pr258-5
7.1 16.6 (72.3%) 1494 2704 2.64e-03 25km_jra_ryf_Aug4_30days

Performance summary table (100k IAF config)

From a set of 3 runs, with repeat=True for a 1-year experiment. The profiling info in the first 6 build-cd deployments (i.e., pr258-1 through pr258-6, inclusive) are all from benchmark_ALE case. pr258-11 and pr258-14 using profiling info from a 1-year run with OM3 25k IAF. Since the base config runs on the normal (i.e., cascadelake queue) - this uncovered a hidden variable of the hardware as well -- both pr258-11 and pr258-14 crashed with illegal instruction error.

yrs/Wday WTime (max ocn/wtime) SUs CPUs yrs/core/Wday Directory
27.3 52.7 (91.1%) 422 240 1.14e-01 100km_jra_iaf_Aug5_1year_cascadelake_pr258-3
27.0 53.4 (90.9%) 427 240 1.12e-01 100km_jra_iaf_Aug5_1year_cascadelake_pr258-1
26.8 53.8 (91.5%) 431 240 1.11e-01 100km_jra_iaf_Aug5_1year_cascadelake_pr258-5
25.5 56.5 (92.6%) 452 240 1.06e-01 100km_jra_iaf_Aug5_1year_cascadelake

🚀 The latest prerelease access-om3/pr258-18 at b1cced7 is here: #258 (comment) 🚀


🚀 The latest prerelease access-om3/pr258-20 at 377d53b is here: #258 (comment) 🚀

@manodeep
manodeep requested a review from a team as a code owner July 28, 2026 06:13
@manodeep
manodeep marked this pull request as draft July 28, 2026 06:13
@manodeep
manodeep removed the request for review from a team July 28, 2026 06:13
@github-actions

Copy link
Copy Markdown

The model version in the spack.yaml has not been updated.
You can update it manually.
Alternatively, comment the following to have it updated and committed automatically:

  • !bump major for feature releases
  • !bump minor for bugfixes

@manodeep
manodeep temporarily deployed to Gadi Prerelease July 28, 2026 06:15 — with GitHub Actions Inactive
@github-actions

Copy link
Copy Markdown

🚀 Attempted to deploy access-om3 Prerelease pr258-1 with commit f6d982e

🖥️ Gadi Deployment ❌

@github-actions

Copy link
Copy Markdown

🚀 Attempted to deploy access-om3 Prerelease pr258-1 with commit f6d982e

🖥️ Gadi Deployment ✔️

Usage Instructions

access-om3, defined in spack.yaml, will be deployed to Gadi as:

  • 2026.05.002 as a Release (when merged).
  • pr258-1 as a Prerelease (during this PR).

This Prerelease is accessible on Gadi using:

module use /g/data/vk83/prerelease/modules
module load access-om3/pr258-1

When using the above modules, the binaries shall be on your $PATH.

For advanced users, this Prerelease is also accessible on Gadi via /g/data/vk83/prerelease/apps/spack/1.1/spack in the access-om3-pr258-1 environment.
Due to inode-saving measures, one will have to manually untar the environment metadata before environment activation with tar -xf .spack-env .spack-env.tar. It will require one to have write privileges.

Configuration Information

This Prerelease is deployed using:

  • access-nri/spack on branch 1.1
  • ACCESS-NRI/upstream-spack-packages version access/v1.1
  • access-nri/access-spack-packages version 2026.06.000
  • access-nri/spack-config version 2026.07.001

If the above was not what was expected, commit changes to config/versions.json in this PR, or propose changes to build-cds config/settings.json.

@github-actions

Copy link
Copy Markdown

🚀 Attempted to deploy access-om3 Prerelease pr258-2 with commit 924dbd7

🖥️ Gadi Deployment ❌

@manodeep

Copy link
Copy Markdown
Collaborator Author

!redeploy

@github-actions

Copy link
Copy Markdown

🚀 Attempted to deploy access-om3 Prerelease pr258-3 with commit 924dbd7

🖥️ Gadi Deployment ✔️

Usage Instructions

access-om3, defined in spack.yaml, will be deployed to Gadi as:

  • 2026.05.002 as a Release (when merged).
  • pr258-3 as a Prerelease (during this PR).

This Prerelease is accessible on Gadi using:

module use /g/data/vk83/prerelease/modules
module load access-om3/pr258-3

When using the above modules, the binaries shall be on your $PATH.

For advanced users, this Prerelease is also accessible on Gadi via /g/data/vk83/prerelease/apps/spack/1.1/spack in the access-om3-pr258-3 environment.
Due to inode-saving measures, one will have to manually untar the environment metadata before environment activation with tar -xf .spack-env .spack-env.tar. It will require one to have write privileges.

Configuration Information

This Prerelease is deployed using:

  • access-nri/spack on branch 1.1
  • ACCESS-NRI/upstream-spack-packages version access/v1.1
  • access-nri/access-spack-packages version 2026.06.000
  • access-nri/spack-config version 2026.07.001

If the above was not what was expected, commit changes to config/versions.json in this PR, or propose changes to build-cds config/settings.json.

@manodeep
manodeep temporarily deployed to Gadi Prerelease July 28, 2026 11:39 — with GitHub Actions Inactive
@github-actions

Copy link
Copy Markdown

🚀 Attempted to deploy access-om3 Prerelease pr258-4 with commit 8f6d263

🖥️ Gadi Deployment ✔️

Usage Instructions

access-om3, defined in spack.yaml, will be deployed to Gadi as:

  • 2026.05.002 as a Release (when merged).
  • pr258-4 as a Prerelease (during this PR).

This Prerelease is accessible on Gadi using:

module use /g/data/vk83/prerelease/modules
module load access-om3/pr258-4

When using the above modules, the binaries shall be on your $PATH.

For advanced users, this Prerelease is also accessible on Gadi via /g/data/vk83/prerelease/apps/spack/1.1/spack in the access-om3-pr258-4 environment.
Due to inode-saving measures, one will have to manually untar the environment metadata before environment activation with tar -xf .spack-env .spack-env.tar. It will require one to have write privileges.

Configuration Information

This Prerelease is deployed using:

  • access-nri/spack on branch 1.1
  • ACCESS-NRI/upstream-spack-packages version access/v1.1
  • access-nri/access-spack-packages version 2026.06.000
  • access-nri/spack-config version 2026.07.001

If the above was not what was expected, commit changes to config/versions.json in this PR, or propose changes to build-cds config/settings.json.

@manodeep
manodeep temporarily deployed to Gadi Prerelease July 28, 2026 12:02 — with GitHub Actions Inactive
@github-actions

Copy link
Copy Markdown

🚀 Attempted to deploy access-om3 Prerelease pr258-5 with commit fd9a8be

🖥️ Gadi Deployment ✔️

Usage Instructions

access-om3, defined in spack.yaml, will be deployed to Gadi as:

  • 2026.05.002 as a Release (when merged).
  • pr258-5 as a Prerelease (during this PR).

This Prerelease is accessible on Gadi using:

module use /g/data/vk83/prerelease/modules
module load access-om3/pr258-5

When using the above modules, the binaries shall be on your $PATH.

For advanced users, this Prerelease is also accessible on Gadi via /g/data/vk83/prerelease/apps/spack/1.1/spack in the access-om3-pr258-5 environment.
Due to inode-saving measures, one will have to manually untar the environment metadata before environment activation with tar -xf .spack-env .spack-env.tar. It will require one to have write privileges.

Configuration Information

This Prerelease is deployed using:

  • access-nri/spack on branch 1.1
  • ACCESS-NRI/upstream-spack-packages version access/v1.1
  • access-nri/access-spack-packages version 2026.06.000
  • access-nri/spack-config version 2026.07.001

If the above was not what was expected, commit changes to config/versions.json in this PR, or propose changes to build-cds config/settings.json.

@manodeep
manodeep temporarily deployed to Gadi Prerelease July 28, 2026 12:21 — with GitHub Actions Inactive
@github-actions

Copy link
Copy Markdown

🚀 Attempted to deploy access-om3 Prerelease pr258-6 with commit f8158b7

🖥️ Gadi Deployment ✔️

Usage Instructions

access-om3, defined in spack.yaml, will be deployed to Gadi as:

  • 2026.05.002 as a Release (when merged).
  • pr258-6 as a Prerelease (during this PR).

This Prerelease is accessible on Gadi using:

module use /g/data/vk83/prerelease/modules
module load access-om3/pr258-6

When using the above modules, the binaries shall be on your $PATH.

For advanced users, this Prerelease is also accessible on Gadi via /g/data/vk83/prerelease/apps/spack/1.1/spack in the access-om3-pr258-6 environment.
Due to inode-saving measures, one will have to manually untar the environment metadata before environment activation with tar -xf .spack-env .spack-env.tar. It will require one to have write privileges.

Configuration Information

This Prerelease is deployed using:

  • access-nri/spack on branch 1.1
  • ACCESS-NRI/upstream-spack-packages version access/v1.1
  • access-nri/access-spack-packages version 2026.06.000
  • access-nri/spack-config version 2026.07.001

If the above was not what was expected, commit changes to config/versions.json in this PR, or propose changes to build-cds config/settings.json.

@github-actions

Copy link
Copy Markdown

🚀 Attempted to deploy access-om3 Prerelease pr258-7 with commit 6ea8a56

🖥️ Gadi Deployment ❌

@manodeep

Copy link
Copy Markdown
Collaborator Author

!redeploy

@github-actions

Copy link
Copy Markdown

🚀 Attempted to deploy access-om3 Prerelease pr258-8 with commit 6ea8a56

🖥️ Gadi Deployment ❌

@manodeep

Copy link
Copy Markdown
Collaborator Author

!redeploy

@github-actions

Copy link
Copy Markdown

🚀 Attempted to deploy access-om3 Prerelease pr258-9 with commit 6ea8a56

🖥️ Gadi Deployment ❌

@manodeep
manodeep temporarily deployed to Gadi Prerelease July 29, 2026 22:12 — with GitHub Actions Inactive
@github-actions

Copy link
Copy Markdown

🚀 Attempted to deploy access-om3 Prerelease pr258-14 with commit e45d0f9

🖥️ Gadi Deployment ✔️

Usage Instructions

access-om3, defined in spack.yaml, will be deployed to Gadi as:

  • 2026.05.002 as a Release (when merged).
  • pr258-14 as a Prerelease (during this PR).

This Prerelease is accessible on Gadi using:

module use /g/data/vk83/prerelease/modules
module load access-om3/pr258-14

When using the above modules, the binaries shall be on your $PATH.

For advanced users, this Prerelease is also accessible on Gadi via /g/data/vk83/prerelease/apps/spack/1.1/spack in the access-om3-pr258-14 environment.
Due to inode-saving measures, one will have to manually untar the environment metadata before environment activation with tar -xf .spack-env .spack-env.tar. It will require one to have write privileges.

Configuration Information

This Prerelease is deployed using:

  • access-nri/spack on branch 1.1
  • ACCESS-NRI/upstream-spack-packages version access/v1.1
  • access-nri/access-spack-packages version 2026.06.000
  • access-nri/spack-config version 2026.07.001

If the above was not what was expected, commit changes to config/versions.json in this PR, or propose changes to build-cds config/settings.json.

@manodeep
manodeep temporarily deployed to Gadi Prerelease July 30, 2026 01:13 — with GitHub Actions Inactive
@github-actions

Copy link
Copy Markdown

🚀 Attempted to deploy access-om3 Prerelease pr258-15 with commit 2c323c8

🖥️ Gadi Deployment ✔️

Usage Instructions

access-om3, defined in spack.yaml, will be deployed to Gadi as:

  • 2026.05.002 as a Release (when merged).
  • pr258-15 as a Prerelease (during this PR).

This Prerelease is accessible on Gadi using:

module use /g/data/vk83/prerelease/modules
module load access-om3/pr258-15

When using the above modules, the binaries shall be on your $PATH.

For advanced users, this Prerelease is also accessible on Gadi via /g/data/vk83/prerelease/apps/spack/1.1/spack in the access-om3-pr258-15 environment.
Due to inode-saving measures, one will have to manually untar the environment metadata before environment activation with tar -xf .spack-env .spack-env.tar. It will require one to have write privileges.

Configuration Information

This Prerelease is deployed using:

  • access-nri/spack on branch 1.1
  • ACCESS-NRI/upstream-spack-packages version access/v1.1
  • access-nri/access-spack-packages version 2026.06.000
  • access-nri/spack-config version 2026.07.001

If the above was not what was expected, commit changes to config/versions.json in this PR, or propose changes to build-cds config/settings.json.

@manodeep
manodeep temporarily deployed to Gadi Prerelease July 30, 2026 09:52 — with GitHub Actions Inactive
@github-actions

Copy link
Copy Markdown

🚀 Attempted to deploy access-om3 Prerelease pr258-16 with commit 7bc29a9

🖥️ Gadi Deployment ✔️

Usage Instructions

access-om3, defined in spack.yaml, will be deployed to Gadi as:

  • 2026.05.002 as a Release (when merged).
  • pr258-16 as a Prerelease (during this PR).

This Prerelease is accessible on Gadi using:

module use /g/data/vk83/prerelease/modules
module load access-om3/pr258-16

When using the above modules, the binaries shall be on your $PATH.

For advanced users, this Prerelease is also accessible on Gadi via /g/data/vk83/prerelease/apps/spack/1.1/spack in the access-om3-pr258-16 environment.
Due to inode-saving measures, one will have to manually untar the environment metadata before environment activation with tar -xf .spack-env .spack-env.tar. It will require one to have write privileges.

Configuration Information

This Prerelease is deployed using:

  • access-nri/spack on branch 1.1
  • ACCESS-NRI/upstream-spack-packages version access/v1.1
  • access-nri/access-spack-packages version 2026.06.000
  • access-nri/spack-config version 2026.07.001

If the above was not what was expected, commit changes to config/versions.json in this PR, or propose changes to build-cds config/settings.json.

@github-actions

Copy link
Copy Markdown

🚀 Attempted to deploy access-om3 Prerelease pr258-17 with commit b1cced7

🖥️ Gadi Deployment ❌

@manodeep

Copy link
Copy Markdown
Collaborator Author

!redeploy

@github-actions

Copy link
Copy Markdown

🚀 Attempted to deploy access-om3 Prerelease pr258-18 with commit b1cced7

🖥️ Gadi Deployment ✔️

Usage Instructions

access-om3, defined in spack.yaml, will be deployed to Gadi as:

  • 2026.05.002 as a Release (when merged).
  • pr258-18 as a Prerelease (during this PR).

This Prerelease is accessible on Gadi using:

module use /g/data/vk83/prerelease/modules
module load access-om3/pr258-18

When using the above modules, the binaries shall be on your $PATH.

For advanced users, this Prerelease is also accessible on Gadi via /g/data/vk83/prerelease/apps/spack/1.1/spack in the access-om3-pr258-18 environment.
Due to inode-saving measures, one will have to manually untar the environment metadata before environment activation with tar -xf .spack-env .spack-env.tar. It will require one to have write privileges.

Configuration Information

This Prerelease is deployed using:

  • access-nri/spack on branch 1.1
  • ACCESS-NRI/upstream-spack-packages version access/v1.1
  • access-nri/access-spack-packages version 2026.06.000
  • access-nri/spack-config version 2026.07.001

If the above was not what was expected, commit changes to config/versions.json in this PR, or propose changes to build-cds config/settings.json.

@manodeep

manodeep commented Aug 3, 2026

Copy link
Copy Markdown
Collaborator Author

The compiler flag for the memory layout transformations did not make any significant performance impact

yrs/Wday WTime (max ocn/wtime) SUs CPUs yrs/core/Wday Directory
7.4 15.9 (68.8%) 1434 2704 2.75e-03 25km_jra_iaf_Jul28_30days_pr258-16-qopt-mem-trans3
7.4 15.9 (69.1%) 1437 2704 2.74e-03 25km_jra_iaf_Jul28_30days_pr258-14-perf-remap-rho-profile-use
7.4 16.0 (69.7%) 1444 2704 2.73e-03 25km_jra_iaf_Jul28_30days_pr258-15-qopt-mem-trans4
7.4 16.1 (68.0%) 1447 2704 2.72e-03 25km_jra_iaf_Jul28_30days_pr258-18-qopt-mem-trans2

@manodeep
manodeep temporarily deployed to Gadi Prerelease August 10, 2026 00:27 — with GitHub Actions Inactive
@github-actions

Copy link
Copy Markdown

🚀 Attempted to deploy access-om3 Prerelease pr258-19 with commit ac16be0

🖥️ Gadi Deployment ✔️

Usage Instructions

access-om3, defined in spack.yaml, will be deployed to Gadi as:

  • 2026.05.002 as a Release (when merged).
  • pr258-19 as a Prerelease (during this PR).

This Prerelease is accessible on Gadi using:

module use /g/data/vk83/prerelease/modules
module load access-om3/pr258-19

When using the above modules, the binaries shall be on your $PATH.

For advanced users, this Prerelease is also accessible on Gadi via /g/data/vk83/prerelease/apps/spack/1.1/spack in the access-om3-pr258-19 environment.
Due to inode-saving measures, one will have to manually untar the environment metadata before environment activation with tar -xf .spack-env .spack-env.tar. It will require one to have write privileges.

Configuration Information

This Prerelease is deployed using:

  • access-nri/spack on branch 1.1
  • ACCESS-NRI/upstream-spack-packages version access/v1.1
  • access-nri/access-spack-packages version 2026.06.000
  • access-nri/spack-config version 2026.07.001

If the above was not what was expected, commit changes to config/versions.json in this PR, or propose changes to build-cds config/settings.json.

@manodeep
manodeep deployed to Gadi Prerelease September 14, 2026 10:44 — with GitHub Actions Active
@github-actions

Copy link
Copy Markdown

🚀 Attempted to deploy access-om3 Prerelease pr258-20 with commit 377d53b

🖥️ Gadi Deployment ✔️

Usage Instructions

access-om3, defined in spack.yaml, will be deployed to Gadi as:

  • 2026.05.003 as a Release (when merged).
  • pr258-20 as a Prerelease (during this PR).

This Prerelease is accessible on Gadi using:

module use /g/data/vk83/prerelease/modules
module load access-om3/pr258-20

When using the above modules, the binaries shall be on your $PATH.

For advanced users, this Prerelease is also accessible on Gadi via /g/data/vk83/prerelease/apps/spack/1.1/spack in the access-om3-pr258-20 environment.
Due to inode-saving measures, one will have to manually untar the environment metadata before environment activation with tar -xf .spack-env .spack-env.tar. It will require one to have write privileges.

Configuration Information

This Prerelease is deployed using:

  • access-nri/spack on branch 1.1
  • ACCESS-NRI/upstream-spack-packages version access/v1.1
  • access-nri/access-spack-packages version 2026.08.005
  • access-nri/spack-config version 2026.08.001

If the above was not what was expected, commit changes to the manifest in this PR, or propose changes to build-cds config/settings.json.

@manodeep

Copy link
Copy Markdown
Collaborator Author

!update-configs

@github-actions

Copy link
Copy Markdown

🔧 Opening/Updating Model Configuration PRs in ACCESS-NRI/access-om3-configs

❌ One or more errors occurred with the workflow

Configurations Requested

Configurations requested from profile default:

  • dev-MCW_100km_jra_iaf

  • dev-MC_100km_jra_ryf

  • dev-MC_100km_jra_ryf+wombatlite

  • dev-MC_25km_jra_ryf

Pull Requests Opened/Updated

The following PRs were opened/updated:

More details can be found in the workflow run: https://github.com/ACCESS-NRI/ACCESS-OM3/actions/runs/34836506818

@manodeep

Copy link
Copy Markdown
Collaborator Author

Ughhh I keep forgetting that the MCW branch name needs to be changed for the auto-bot-config-update

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants