Base spack.includes for build-ci

[CodeGat]

Closes #484

Background

We are looking to remove common configuration from individual CR manifests via layered configuration via a spack.include section.

The configuration in this PR is the top level of included configuration, with common configurations for compilers, concretizers, packages and repos. Any of the config in this repository is easily overridable at the CR level.

Note

build-ci@v4-specific CI updates for ASP will be done in #485, this PR is specifically for CRs using build-ci@v4 that consume these base includes, not for ASP to work with build-ci@v4

Structure

Currently, we are looking at having a base level of configuration, which contains information on compilers, packages and repos.

We will also have a pr level of configuration (which inherits base), that is special, specifically for the case where we have an ASP PR that calls CRs, which then include ASP configuration - we want it to use the ASP PRs version of access-spack-packages, not the default api-v2.

As a common entrypoint for both CR and ASP PR, we have a ci level of config, which makes the decision on inheriting base or pr (which then inherits base).

The PR

• Add .github/build-ci/includes/base for config common to all compilers
• Add .github/build-ci/includes/pr for config to be used when access-spack-packages is creating PRs
• Add .github/build-ci/includes/ci as a common entrypoint

Testing

Testing includes in ACCESS-NRI/MOM6#74

[CodeGat]

Before merging, update target to x86_64_v3. Also consider putting it under the base scope rather than the compiler scope

EDIT: Not required, as it is set in https://github.com/ACCESS-NRI/spack-config/blob/main/v1.1/packages.yaml#L12-L13

[CodeGat]

Okay, this is ready for review. I've dropped the CI commit from this PR as it will be done in #485

[harshula]

Hi @CodeGat , We use include in https://github.com/ACCESS-NRI/spack-config/tree/main/v1.1/include . Perhaps for consistency, it would better to replace includes with include?

[CodeGat]

Drafting this to test out ACCESS-NRI/MOM6#74 (comment)

[CodeGat]

I've got this working again, see ACCESS-NRI/MOM6#74 (comment).

However, we need to double-check that we can fold all the compiler info in a single directory so we can be sure this works. Whether that be by putting compiler directives in the spec line, I dunno. But this is looking good! Might need to rename the ASP_ci folder too. It's essentially config used for PRs from this repo.

[CodeGat]

Looks like we can fold compiler info into a single directory by using propagating toolchains in the spec line, eg. %%access_oneapi, so that bit is solved. Example manifest and successful run with concretization.

Now for some naming - for the specific configuration that is used when we are running an access-spack-packages PR - the directory is called .github/build-ci/include/ASP-ci currently, but that seems a bit clunky. Since it is to do with ASP PRs, maybe it should just be .github/build-ci/include/pr, instead. What do people think?

[aidanheerdegen]

What do people think?

I like pr

[CodeGat]

Okay this is finally ready for review!

[harshula]

Hi @CodeGat , Can you please explain the choice of sub-directory names under .github/build-ci/include/ and what they mean?

[CodeGat]

• .github/build-ci/include/ci is the common entrypoint for all build-ci builds - it chooses between including pr (and base) or just base depending on whether it is a PR from ASP or a CR respectively
• .github/build-ci/include/pr is for config to be used when the PR is from this repository. It also includes base.
• .github/build-ci/include/base is a common base for config common to all builds

[harshula]

Hi @CodeGat , Thanks!

• Can .github/build-ci/include/ci be changed to reflect it is the common entrypoint?
• Can .github/build-ci/include/pr be changed to reflect that it contains config for local CI?

For example, Have a look at spack-config (https://github.com/ACCESS-NRI/spack-config/tree/main/v1.1). Could something like this work?:

• .github/build-ci/include/base is .github/build-ci/config/base
• .github/build-ci/include/ci is .github/build-ci/config/include
• .github/build-ci/include/pr is .github/build-ci/config/local

After this discussion, I think I should add a base directory to spack-config!

[CodeGat]

I think that looks good, too. But, should this .github/build-ci/config convention for includes carry over to CRs? Currently they are of the form .github/build-ci/include, see https://github.com/ACCESS-NRI/MOM6/pull/74/changes#diff-f03f8a56c63fecd0bf7d473a48384c6808b0c5b960acc88a217d10db83f0f81dR4

[harshula]

Hi @CodeGat , Yes, that's good idea! Let's make all the .github/build-ci/ directories as similar as possible.

[CodeGat]

Done in 56f3c9d, testing in https://github.com/ACCESS-NRI/MOM6/actions/runs/34789463977

[aidanheerdegen]

I'm approving again, but it seems the same as last time I approved, which is fine, just checking I am not going insane.

[CodeGat]

I've validated that the new v3 image works with these changes, see https://github.com/ACCESS-NRI/MOM6/actions/runs/34799693502?pr=74

Thanks everyone!