Dependabot grouping

[Javagedes]

Current Situation

Currently, our dependabot configuration creates individual pull requests for individual dependencies for (1) rust dependencies, (2) python dependencies, and (3) github-actions (ref: depndabot.yml). We limited dependabot to run only once a week to reduce churn in our various repositories, but this had the side effect of creating multiple pull requests at the same time, leading to a backup of workflows and potentially causes us to hit API limits, resulting in PR CI failures.

I would like to discuss using dependabot::groups to group together various dependency types into a single pull-request to reduce churn.

Dependency Type	pattern	exclude-pattern	Frequency	SemVer	Description
cargo	patina*		Daily	*	Update any patina dependencies as soon as possible. This is mainly for patina-dxe-core-qemu to update it's patina dependencies as a single PR.
cargo		patina*	Weekly	patch, minor	Update all other dependencies at patch and minor level changes once a week as a batch PR.
cargo		patina*	Weekly	major	Update all other dependencies at major level changes once a week as separate PRs.
github-action	*		Weekly	major	Update all github-actions at major level changes once a week
python	*		Weekly	patch, minor	Update all python dependencies at patch and minor level changes once a week as a batch PR
python	*		Weekly	major	Update all python dependencies at major level changes once a week as separate PRs.

Below is a suggested, but untested dependabot file:

version: 2

updates:
  # ----------------------------
  # Cargo - patina* (daily, all semver, single PR)
  # ----------------------------
  - package-ecosystem: "cargo"
    directory: "/"
    schedule:
      interval: "daily"
    allow:
      - dependency-name: "patina*"
    groups:
      patina-dependencies:
        patterns:
          - "patina*"

  # ----------------------------
  # Cargo - non-patina patch/minor (weekly, batched)
  # ----------------------------
  - package-ecosystem: "cargo"
    directory: "/"
    schedule:
      interval: "weekly"
    ignore:
      - dependency-name: "patina*"
    groups:
      cargo-minor-patch:
        update-types:
          - "minor"
          - "patch"

  # ----------------------------
  # Cargo - non-patina major (weekly, separate PRs)
  # ----------------------------
  - package-ecosystem: "cargo"
    directory: "/"
    schedule:
      interval: "weekly"
    ignore:
      - dependency-name: "patina*"
    # Filter to major updates only
    # Using ignore to exclude patch/minor
    ignore:
      - dependency-name: "*"
        update-types:
          - "version-update:semver-patch"
          - "version-update:semver-minor"
      - dependency-name: "patina*"

  # ----------------------------
  # GitHub Actions - major only (weekly)
  # ----------------------------
  - package-ecosystem: "github-actions"
    directory: "/"
    schedule:
      interval: "weekly"
    ignore:
      - dependency-name: "*"
        update-types:
          - "version-update:semver-patch"
          - "version-update:semver-minor"

  # ----------------------------
  # Python - patch/minor (weekly, batched)
  # ----------------------------
  - package-ecosystem: "pip"
    directory: "/"
    schedule:
      interval: "weekly"
    groups:
      python-minor-patch:
        update-types:
          - "minor"
          - "patch"

  # ----------------------------
  # Python - major (weekly, separate PRs)
  # ----------------------------
  - package-ecosystem: "pip"
    directory: "/"
    schedule:
      interval: "weekly"
    ignore:
      - dependency-name: "*"
        update-types:
          - "version-update:semver-patch"
          - "version-update:semver-minor"

[cfernald]

Overall, I agree fully with the strategy, especially the cargo approach of patina daily, the rest weekly with minor versions batched and major split.

Some crates that we use have taken the stance that they are not version 1.0.0 yet and will have breaking changes in minor version updates. I wonder if we need to break those out into their own group and split their PRs.

I think for python and workflows we should make it monthly instead. There isnt much cost in weekly but also neither of those have any impact on the published code so I would prioritize stability over quick updates.

[makubacki]

I was also going to reply that I think Python and GitHub actions should be monthly.

Some crates that we use have taken the stance that they are not version 1.0.0 yet and will have breaking changes in minor version updates. I wonder if we need to break those out into their own group and split their PRs.

Do you think this is going to be significant enough it can't be handled on a per-PR basis?

[cfernald]

Probably not, we can always do this later if it becomes a problem.

[apop5]

From testing, it appears that dependabot only supports one package environment statement. So there can only be one package-ecosystem: "pip"

If two statements exist, it will only use one of them.

[Javagedes]

After doing some reading, it looks like the main limitation is you cannot have different package environments point at the same directory. This means we should be able to work around some of the limitations. I think the main limiting factor is we cannot have a "daily", and "weekly"/"monthly" dependabot configuration pointing at the same directory.

[cfernald]

Should we consider a special workflow (not dependabot) that updates Patina crates specifically then?

[Javagedes]

Yes we should be able to do something like that. I can look at that as a standalone thing before implementing the dependabot grouping.

[Javagedes]

From some light research I see two ways to approach it:

1. A nightly run in patina-dxe-core-qemu that checks if a new release exists on crates.io, and if it does, update and create a PR
2. A coupling of patina and patina-dxe-core-qemu where the patina release workflow triggers a workflow in patina-dxe-core-qemu to create a PR.

I'm not a huge fan of (2) as it creates a weird coupling of the two repositories, but it would lead to immediate PR creation if that is something we want.

[makubacki]

(1) sounds reasonable to me.