fix(ci): push production tags after signing#2231
Open
renner0e wants to merge 8 commits into
Open
Conversation
Up to standards ✅🟢 Issues
|
renner0e
commented
May 17, 2026
If the signing step fails, causes for this can be ghcr jank or a sigstore outage and not testing our changes properly beforehand. Then we have already pushed images to the registry with production tags that users can update to. The clients won't actually update to these images as we enforce verification on the client side to pull images from ghcr.io/ublue-os/ only with a valid signature attached. Otherwise this leads to an "update failed" on the users POV, which is very confusing. This has been a longstanding issue across the org [1] [2] [3]. To workaround this, we push a tag to the registry first that we don't care about and clients don't check against for updates. I chose the job id because it always changes. This also happens to get rid of the action we use for retries to workaround ghcr jank, we were forced to do this in the past because the old podman version we used at the time didn't have those features. Using podman to push signatures and images at the same time is not something that is viable at the moment unfortunately but is definitely the approach we should be using once it works well. This effort can be seen at [4]. Although we are likely hitting [5] here where all layers get invalidated because of this bug. We are effectively pushing full images/no shared layers every time we do podman push. Which causes the overall push step in CI to take upwards of 20 minutes. fixes: ublue-os#2135 [1]: ublue-os/bazzite#4902 [2]: ublue-os/main#643 [3]: ublue-os/bazzite#3044 [4]: containers/podman#27796 [5]: ublue-os#2199 Signed-off-by: renner <renner0@posteo.de>
It seems like cosign tries to verify all .sig files on the registry for that digest, one of which is generated by cosign sign with a static key and one is generated by actions/attest using an OIDC. This exactly broke, which is what we have been using for the longest time: ``` cosign verify --key cosign.pub ghcr.io/ublue-os/aurora:latest Error: no matching attestations: expected key signature, not certificate error during command execution: no matching attestations: expected key signature, not certificate ``` And this currently works to verify the image, although it is of course not the same thing as singing with the ublue cosign key. This will not work going forward after this commit: ``` cosign verify \ --certificate-oidc-issuer https://token.actions.githubusercontent.com \ --certificate-identity-regexp="github.com/ublue-os/aurora/.github/workflows/*" \ ghcr.io/ublue-os/aurora:latest ``` Signed-off-by: renner <renner0@posteo.de>
This is just a minor cosmetic thing and recudes the amount of tags we have because it's static and not changing unlike the run id. Signed-off-by: renner <renner0@posteo.de>
Signed-off-by: renner <renner0@posteo.de>
inffy
previously approved these changes
May 17, 2026
github-merge-queue
Bot
removed this pull request from the merge queue due to failed status checks
Member
Author
|
yeah this was implemented to need |
Or else we are getting this error with new cosign 3.X.X ``` Error: must provide --new-bundle-format or --bundle where applicable with --signing-config or --use-signing-config error during command ``` Signed-off-by: renner <renner0@posteo.de>
if this fails at the beginning because of jank then it's not so bad
``` Error: error getting signing config from TUF: error setting TUF options: error setting TUF mirror: error reading remote.json: open /home/runner/.sigstore/root/remote.json: permission denied ```
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
If the signing step fails, causes for this can be ghcr jank or a sigstore outage and not testing our changes properly beforehand. Then we have already pushed images to the registry with production tags that users can update to.
The clients won't actually update to these images as we enforce verification on the client side to pull images from ghcr.io/ublue-os/ only with a valid signature attached. Otherwise this leads to an "update failed" on the users POV, which is very confusing. This has been a longstanding issue across the org 1 2 3.
To workaround this, we push a tag to the registry first that we don't care about and clients don't check against for updates. I chose the job id because it always changes.
This also happens to get rid of the action we use for retries to workaround ghcr jank, we were forced to do this in the past because the old podman version we used at the time didn't have those features.
Using podman to push signatures and images at the same time is not something that is viable at the moment unfortunately but is definitely the approach we should be using once it works well.
This effort can be seen at 4. Although we are likely hitting 5 here where all layers get invalidated because of this bug. We are effectively pushing full images/no shared layers every time we do podman push. Which causes the overall push step in CI to take upwards of 20 minutes.
fixes: #2135