Create buildkite pipeline to periodically clean up logstash artifact snyk projects

[alexcams]

• enhancement

Release notes

[rn:skip]

What does this PR do?

Creates a new Buildkite workflow to delete stale Logstash artifact snyk projects and th target they belong to.
The pipeline fetches logstash-artifacts Snyk projects and deletes any Snyk artifact-scan projects whose version is no longer tracked, based on https://github.com/logstash-plugins/.ci/blob/1.x/logstash-versions.yml file.

Why is it important/What is the impact to the user?

With the addition of the pipeline to scan logstash artifacts comes the need to clean up "stale" versions. This buildkite workflow resolves this need.

Tests

• Dry-run. Test scenario with hardcoded logstash-versions to track, where 8.19.15 is deleted. As a result, we can see the execution would've delete the correct ones in generic run scenario.

Related issues

• Closes #7055

[github-actions]

🤖 GitHub comments

Just comment with:

• run docs-build : Re-trigger the docs validation. (use unformatted text in the comment!)
• run exhaustive tests : Run the exhaustive tests Buildkite pipeline.

[mergify]

This pull request does not have a backport label. Could you fix it @alexcams? 🙏
To fixup this pull request, you need to add the backport labels for the needed
branches, such as:

• backport-8./d is the label to automatically backport to the 8./d branch. /d is the digit.
• If no backport is necessary, please add the backport-skip label

[donoghuc]

I think we can drastically simplify this.

Logstash artifact scanning uses https://github.com/logstash-plugins/.ci/blob/1.x/logstash-versions.yml to determine which artifacts to scan. Currently, when that file is updated as part of the release process, older versions effectively stop getting updates in Snyk. These abandoned projects are what we need to clean up (which we refer to as "stale"). The sole criterion for whether a project is "stale" and should be cleaned up is whether its version is currently present in logstash-versions.yml. There should be no time-based element in determining staleness.

Therefore, the main logic should be to query Snyk for Logstash artifact projects, compare those project versions to the versions currently in logstash-versions.yml, and delete any Snyk project whose version is not present in the current file.

[alexcams]

@donoghuc Thanks for the detailed explanation! very helpful bc I'm still not really familiarized with Snyk env and was doubting about conditon to consider a project stale. Now it's clear.

I agree that it is way better to follow a version based cleanup, being deterministic.

I updated the workflow according to this behavior and also added a dry-run buildkite run at PR description.

The PR is ready to review, but we'll have a new release next week, it'd be nice to test this on real scenario before merge.

[elasticmachine]

💛 Build succeeded, but was flaky

• Buildkite Build
• Commit: b7c9af5

Failed CI Steps

• 🥼 x-pack unit tests

History

• 💛 Build #4896 was flaky 80b6880
• 💛 Build #4879 was flaky 62b981b

[donoghuc]

This looks much cleaner! Thanks for refactoring. I apologize for not making that more clear when i wrote the original issue. IMO this is a very low risk thing to be running. If it fails to delete something, no problem that is the state we are in already. If it accidentally deletes something we care about, no problem it will be re-generated daily. So i am cool with getting this merged and monitoring it for the upcoming release cycle. If there are bugs or iterations we can file a follow on issue.