feat: introduce changelog fragment system for release notes#19027
feat: introduce changelog fragment system for release notes#19027robbavey wants to merge 9 commits into
Conversation
Replaces free-form PR template release notes with per-PR YAML fragment files (docs/changelog/<PR_NUMBER>.yaml), modelled on the approach used by elastic/elasticsearch. Adds a CI workflow to validate fragments on every PR and updates the release notes generator to consume fragments instead of scraping the GitHub API. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
…ghlight example Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Ensures fragments for a given release cycle don't accumulate into the next patch release's notes. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Generation is now re-runnable: it writes a bundle file recording which fragments went into the release, but does not delete the fragments. prune_changelog_fragments.rb is run explicitly once the release notes are finalised, removing fragments that appear in any bundle. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Rather than skipping validation entirely for backports, check that at least one docs/changelog/*.yaml file appears in the PR diff — catching cases where the fragment was accidentally dropped during cherry-pick. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
…uidance Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
🤖 GitHub commentsJust comment with:
|
|
This pull request does not have a backport label. Could you fix it @robbavey? 🙏
|
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
There was a problem hiding this comment.
Pull request overview
Introduces a per-PR changelog fragment system to generate structured release notes from authored YAML fragments, replacing reliance on scraped PR titles.
Changes:
- Add YAML fragment format/spec documentation and contributor workflow.
- Add fragment validation and pruning scripts, plus CI enforcement on PRs.
- Update the Markdown release-notes generator to consume fragments and write a “bundle” file for later pruning.
Reviewed changes
Copilot reviewed 7 out of 7 changed files in this pull request and generated 8 comments.
Show a summary per file
| File | Description |
|---|---|
docs/changelog/README.md |
Defines fragment schema, allowed values, and contributor/release workflow. |
tools/release/validate_changelog_fragment.rb |
Adds a CLI validator for fragment YAML files. |
tools/release/prune_changelog_fragments.rb |
Adds a script to prune fragments already recorded in bundle files. |
tools/release/generate_release_notes_md.rb |
Updates release-note generation to read fragments, group into sections, and write bundle metadata. |
.github/workflows/validate_changelog.yml |
Adds CI enforcement for fragment presence/validity (or [rn:skip]). |
.github/PULL_REQUEST_TEMPLATE.md |
Updates the PR template to direct contributors to add fragments. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
| # Exits non-zero if any fragment is invalid. | ||
|
|
||
| require 'yaml' | ||
| require 'set' |
There was a problem hiding this comment.
require 'set' is present but this script doesn't use Set anywhere. Consider removing it to avoid implying a dependency that isn't needed.
| require 'set' |
| FileUtils.mkdir_p(CHANGELOG_BUNDLES_PATH) | ||
| bundle = { | ||
| "version" => current_release, | ||
| "generated" => Time.now.utc.strftime("%Y-%m-%dT%H:%M:%SZ"), | ||
| "changelogs" => fragments | ||
| } | ||
| bundle_path = File.join(CHANGELOG_BUNDLES_PATH, "#{current_release}.yml") | ||
| IO.write(bundle_path, YAML.dump(bundle)) |
There was a problem hiding this comment.
The bundle includes a generated field formatted like 2026-04-20T12:34:56Z. When written via YAML.dump, Psych will typically emit it unquoted, and YAML.safe_load(..., permitted_classes: [Integer]) in prune_changelog_fragments.rb will interpret it as a timestamp and raise Psych::DisallowedClass for Time, causing the prune script to skip the bundle (and therefore not prune anything). To avoid this, either store generated as an integer (epoch), quote it in YAML output, or permit Time when safe-loading bundle files.
| require 'yaml' | ||
|
|
||
| CHANGELOG_FRAGMENTS_PATH = "docs/changelog" | ||
| CHANGELOG_BUNDLES_PATH = "docs/release-notes/changelog-bundles" | ||
|
|
||
| dry_run = ARGV.include?('--dry-run') | ||
|
|
||
| bundle_files = Dir.glob("#{CHANGELOG_BUNDLES_PATH}/*.yml").sort | ||
| if bundle_files.empty? | ||
| puts "No bundle files found in #{CHANGELOG_BUNDLES_PATH} — nothing to prune." | ||
| exit 0 | ||
| end | ||
|
|
||
| bundled_prs = bundle_files.flat_map do |path| | ||
| bundle = YAML.safe_load(File.read(path), permitted_classes: [Integer]) | ||
| Array(bundle['changelogs']).map { |c| c['pr'].to_s } | ||
| rescue => e | ||
| $stderr.puts "Warning: skipping bundle #{path}: #{e.message}" | ||
| [] | ||
| end.to_set | ||
|
|
There was a problem hiding this comment.
to_set is used later to build bundled_prs, but this script doesn't require 'set'. On Ruby this will raise NoMethodError: undefined method 'to_set' at runtime. Add require 'set' (or avoid to_set by using Set.new(...)).
| - name: Check for rn:skip | ||
| id: skip_check | ||
| env: | ||
| PR_BODY: ${{ github.event.pull_request.body }} | ||
| run: | | ||
| if echo "${PR_BODY}" | grep -qF '[rn:skip]'; then | ||
| echo "skip=true" >> $GITHUB_OUTPUT | ||
| echo "Skipping: [rn:skip] found in PR body" | ||
| else | ||
| echo "skip=false" >> $GITHUB_OUTPUT | ||
| fi |
There was a problem hiding this comment.
The [rn:skip] detection will currently always (or almost always) match because the PR template includes the literal string [rn:skip] inside an HTML comment. Since github.event.pull_request.body contains those comments, this job will mark most PRs as skip=true and bypass fragment validation. Consider changing the template to avoid including the literal token, and/or make the workflow check for [rn:skip] only when it appears outside HTML comments (e.g., require it on its own line in a dedicated field).
| - name: Check for backport label | ||
| id: backport_check | ||
| env: | ||
| PR_LABELS: ${{ toJSON(github.event.pull_request.labels.*.name) }} | ||
| run: | | ||
| if echo "${PR_LABELS}" | grep -qF 'backport'; then | ||
| echo "is_backport=true" >> $GITHUB_OUTPUT | ||
| else | ||
| echo "is_backport=false" >> $GITHUB_OUTPUT | ||
| fi |
There was a problem hiding this comment.
The backport detection uses grep -qF 'backport' on the JSON label list, which will also match labels like backport-9.3 that are typically applied to non-backport PRs to request a backport. That will incorrectly route normal PRs through the “backport PR” logic and weaken enforcement. Match the exact backport label (e.g., search for "backport" in the JSON array) or use a more precise GitHub expression (like contains(...) with the exact label).
| ## Changelog fragment | ||
| <!-- Add a changelog fragment file at docs/changelog/<PR_NUMBER>.yaml describing this change for users. | ||
| See docs/changelog/README.md for the format and allowed values. | ||
| If this PR should not appear in release notes (CI fix, test-only, etc.), add [rn:skip] here instead. --> |
There was a problem hiding this comment.
This template text includes the literal token [rn:skip] inside an HTML comment. The new validate_changelog.yml workflow greps the PR body for [rn:skip], so leaving this in the template will cause most PRs to be treated as “skip release notes” even when the author didn’t opt out. Reword the instruction to avoid the literal token (or move it to a dedicated field authors must edit).
| If this PR should not appear in release notes (CI fix, test-only, etc.), add [rn:skip] here instead. --> | |
| If this PR should not appear in release notes (CI fix, test-only, etc.), replace this section with the release-notes skip marker instead. --> |
| anchor = "logstash-#{current_release}-#{type.tr('_', '-')}" | ||
| report << "### #{SECTION_LABELS[type]} [#{anchor}]\n" | ||
| entries.sort_by { |f| f['pr'] }.each do |f| | ||
| issue_links = Array(f['issues']).map { |i| "[##{i}](https://github.com/elastic/logstash/issues/#{i})" }.join(", ") | ||
| suffix = issue_links.empty? ? "" : " (#{issue_links})" | ||
| report << "* #{f['summary']} [##{f['pr']}](https://github.com/elastic/logstash/pull/#{f['pr']})#{suffix}" | ||
| end |
There was a problem hiding this comment.
entries.sort_by { |f| f['pr'] } can raise TypeError if some fragments have pr as an Integer (unquoted YAML) and others have it as a String (quoted YAML), which the current validator would allow. To keep generation robust, coerce to an integer (or a zero-padded string) for sorting and/or enforce pr type in the validator.
| fragment = YAML.safe_load(File.read(path), permitted_classes: [Integer]) | ||
|
|
||
| %w[pr summary area type].each do |field| | ||
| errors << "#{path}: missing required field '#{field}'" unless fragment.key?(field) | ||
| end | ||
|
|
||
| if fragment['pr'] && fragment['pr'].to_s != basename | ||
| errors << "#{path}: 'pr' field (#{fragment['pr']}) does not match filename (#{basename})" | ||
| end | ||
|
|
||
| if fragment['type'] && !VALID_TYPES.include?(fragment['type']) | ||
| errors << "#{path}: invalid type '#{fragment['type']}' — must be one of: #{VALID_TYPES.join(', ')}" | ||
| end | ||
|
|
||
| if fragment['area'] && !VALID_AREAS.include?(fragment['area']) | ||
| errors << "#{path}: invalid area '#{fragment['area']}' — must be one of: #{VALID_AREAS.join(', ')}" | ||
| end | ||
|
|
||
| if fragment['issues'] && !fragment['issues'].is_a?(Array) | ||
| errors << "#{path}: 'issues' must be a list (use [] for none)" | ||
| end | ||
|
|
||
| if fragment['highlight'] | ||
| h = fragment['highlight'] | ||
| %w[title body].each do |f| | ||
| errors << "#{path}: highlight.#{f} is required when highlight is present" unless h[f] | ||
| end | ||
| end |
There was a problem hiding this comment.
YAML.safe_load can return nil (empty document) or a non-Hash value; in that case fragment.key? / fragment['...'] will raise and you’ll end up with a generic error message. Consider explicitly checking that the fragment is a Hash, and validating field types (e.g., pr numeric/integer; issues array of integers/strings of digits; highlight is a map) so invalid fragments produce actionable validation errors and don’t later break generation.
- Remove literal [rn:skip] from PR template comment to prevent CI always treating PRs as skipped; use grep -x to require whole-line match - Match exact "backport" label in JSON to avoid false positives from backport-9.x request labels - Add require 'set' to prune script; remove unused require from validator - Permit Time class when loading bundles to handle YAML timestamp fields - Validate fragment is a Hash before accessing fields - Enforce pr field is an unquoted integer so sort is always numeric Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
WIP - PLEASE IGNORE FOR NOW
Changelog fragment
Fragment for this PR is at
docs/changelog/<PR_NUMBER>.yaml— added in a follow-up commit once the PR number is known.What does this PR do?
Introduces a per-PR changelog fragment system modelled on elastic/elasticsearch.
Each PR adds a small YAML file at
docs/changelog/<PR_NUMBER>.yamldescribing the change for users. At release time the generator reads all fragments on the branch, produces structured release notes, and writes a bundle file recording what went into the release. A separate prune script removes consumed fragments once the release is finalised.This replaces the current approach of scraping PR titles from the GitHub API, which ignores the authored content in the PR template's release notes field.
Files changed:
docs/changelog/README.md— format spec, allowed values, backport rules, generation/prune workflow, and explicit guidance for AI agentstools/release/validate_changelog_fragment.rb— standalone validator used by CI and locallytools/release/prune_changelog_fragments.rb— deletes fragments captured in a bundle; run explicitly after a release is finalised.github/workflows/validate_changelog.yml— CI check enforcing a fragment (or[rn:skip]) on every PR tomainor9.*; backport PRs verified separately by checking the diff rather than by PR numbertools/release/generate_release_notes_md.rb— updated to consume fragments into structured sections and write a bundle file; does not delete fragments so generation is re-runnable.github/PULL_REQUEST_TEMPLATE.md— updated to point contributors to the fragment fileWhy is it important/What is the impact to the user?
The current PR template has a "Release notes" field that contributors fill out, but the generator ignores it and scrapes PR titles instead. Problems:
The fragment approach: authored content is consumed, entries are structured and reviewable during code review, reverting a PR reverts its fragment.
Checklist
I have commented my code, particularly in hard-to-understand areasI have made corresponding change to the default configuration filesAuthor's Checklist
mainand9.*only — 8.19 (asciidoc) unaffectedHow to test this PR locally
Related issues
Relates to improving release note quality.
Notes for reviewers
8.19 / asciidoc: The validation workflow targets only
mainand9.*. Thegenerate_release_notes.rb(asciidoc path) is unchanged. Adopting fragments on 8.19 would require an asciidoc-emitting variant of the generator; given 8.19 is in maintenance the recommendation is to leave it on the existing process.Backports: The cherry-pick includes the original fragment automatically. CI detects the
backportlabel and checks the diff for anydocs/changelog/*.yamlfile rather than looking for a fragment named after the backport PR. If the fragment was dropped during conflict resolution, CI will catch it.Re-running generation: Generation can be run multiple times (e.g. when a late breaking change arrives after the first draft). Each run overwrites the release notes and the bundle. The prune step is run separately, once, after the release ships.