feat: introduce changelog fragment system for release notes

.github/PULL_REQUEST_TEMPLATE.md

@@ -6,8 +6,10 @@ Please label this PR with the release version and one of the following labels, d
 - doc
 -->
 
-## Release notes
-<!-- Add content to appear in  [Release Notes](https://www.elastic.co/guide/en/logstash/current/releasenotes.html), or add [rn:skip] to leave this PR out of release notes -->
+## Changelog fragment
+<!-- Add a changelog fragment file at docs/changelog/<PR_NUMBER>.yaml describing this change for users.
+     See docs/changelog/README.md for the format and allowed values.
+     If this PR should not appear in release notes (CI fix, test-only, etc.), add [rn:skip] here instead. -->
 
 
 ## What does this PR do?

.github/workflows/validate_changelog.yml

@@ -0,0 +1,79 @@
+name: Validate changelog fragment
+
+on:
+  pull_request:
+    types: [opened, synchronize, reopened, edited]
+    branches:
+      - main
+      - 9.*
+
+jobs:
+  validate:
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v4
+        with:
+          fetch-depth: 0
+
+      - name: Set up Ruby
+        uses: ruby/setup-ruby@v1
+        with:
+          ruby-version: '3.3'
+
+      - name: Check for rn:skip
+        id: skip_check
+        env:
+          PR_BODY: ${{ github.event.pull_request.body }}
+        run: |
+          if echo "${PR_BODY}" | grep -qF '[rn:skip]'; then
+            echo "skip=true" >> $GITHUB_OUTPUT
+            echo "Skipping: [rn:skip] found in PR body"
+          else
+            echo "skip=false" >> $GITHUB_OUTPUT
+          fi
+
+      - name: Check for backport label
+        id: backport_check
+        env:
+          PR_LABELS: ${{ toJSON(github.event.pull_request.labels.*.name) }}
+        run: |
+          if echo "${PR_LABELS}" | grep -qF 'backport'; then
+            echo "is_backport=true" >> $GITHUB_OUTPUT
+          else
+            echo "is_backport=false" >> $GITHUB_OUTPUT
+          fi
+
+      - name: Validate fragment exists and is valid (normal PR)
+        if: steps.skip_check.outputs.skip == 'false' && steps.backport_check.outputs.is_backport == 'false'
+        run: |
+          PR_NUMBER="${{ github.event.pull_request.number }}"
+          FRAGMENT="docs/changelog/${PR_NUMBER}.yaml"
+
+          if [[ ! -f "${FRAGMENT}" ]]; then
+            echo "ERROR: No changelog fragment found at ${FRAGMENT}."
+            echo ""
+            echo "Add a fragment file (see docs/changelog/README.md) or add [rn:skip] to the"
+            echo "PR description to skip release notes for this PR."
+            exit 1
+          fi
+
+          ruby tools/release/validate_changelog_fragment.rb "${FRAGMENT}"
+
+      - name: Validate fragment present in diff (backport PR)
+        if: steps.skip_check.outputs.skip == 'false' && steps.backport_check.outputs.is_backport == 'true'
+        run: |
+          BASE="${{ github.event.pull_request.base.sha }}"
+          HEAD="${{ github.event.pull_request.head.sha }}"
+
+          FRAGMENT=$(git diff --name-only "${BASE}".."${HEAD}" -- 'docs/changelog/*.yaml' | head -1)
+
+          if [[ -z "${FRAGMENT}" ]]; then
+            echo "ERROR: Backport PR contains no changelog fragment in docs/changelog/."
+            echo ""
+            echo "The cherry-pick should have included the fragment from the original PR."
+            echo "Check that the fragment file was not dropped during conflict resolution."
+            exit 1
+          fi
+
+          echo "Found fragment in diff: ${FRAGMENT}"
+          ruby tools/release/validate_changelog_fragment.rb "${FRAGMENT}"

docs/changelog/19027.yaml

@@ -0,0 +1,5 @@
+pr: 19027
+summary: "Introduce per-PR changelog fragment system for structured release notes"
+area: docs
+type: feature
+issues: []

docs/changelog/README.md

@@ -0,0 +1,151 @@
+# Changelog Fragments
+
+Each pull request that affects the Logstash release notes should include a YAML fragment file in this directory named `<PR_NUMBER>.yaml`.
+
+This applies to human contributors and AI agents (Claude, Copilot, and others) equally. If you are an AI agent opening or assisting with a PR, you must follow all rules in this document.
+
+## Format
+
+```yaml
+pr: 12345
+summary: "Brief, user-facing description of the change"
+area: core
+type: enhancement
+issues:
+  - 12344
+```
+
+### Required fields
+
+| Field | Description |
+|-------|-------------|
+| `pr` | Pull request number (must match the filename) |
+| `summary` | One-line, user-facing description. Write for the person upgrading, not the reviewer. |
+| `area` | Component area — see allowed values below |
+| `type` | Type of change — see allowed values below |
+
+### Optional fields
+
+| Field | Description |
+|-------|-------------|
+| `issues` | List of linked issue numbers (use `[]` if none) |
+| `highlight` | For notable features — adds an extended description to release highlights |
+
+### Allowed values for `area`
+
+- `core` — general Logstash core changes
+- `performance` — throughput, memory, or CPU improvements
+- `pq` — persistent queue
+- `dlq` — dead letter queue
+- `docs` — documentation-only changes
+- `monitoring` — x-pack monitoring, metrics, health API
+- `central management` — Kibana-based central pipeline management
+- `pipeline->pipeline` — pipeline-to-pipeline communication
+
+### Allowed values for `type`
+
+- `bug` — bug fix
+- `enhancement` — improvement to an existing feature
+- `feature` — new feature
+- `breaking_change` — removes or incompatibly changes existing behaviour
+- `deprecation` — marks something for future removal
+- `dependency` — dependency update (include CVE number in summary if security-related)
+- `doc` — documentation only
+
+## Skipping release notes
+
+If your PR should not appear in release notes (e.g. CI fixes, test-only changes), add `[rn:skip]` to the PR description instead of creating a fragment file. This applies to backport PRs too — see below.
+
+## Highlight format
+
+For significant features, add a `highlight` block:
+
+```yaml
+pr: 18377
+summary: "Add wait_for_status and timeout parameters to the Logstash root API endpoint"
+area: monitoring
+type: feature
+issues: []
+highlight:
+  title: "Wait for status on the Logstash API"
+  notable: true
+  body: |-
+    The Logstash root endpoint `/` now accepts `wait_for_status` and `timeout`
+    query parameters. When set, the call blocks until Logstash reaches (or
+    exceeds) the requested status, or the timeout expires. This makes it
+    straightforward to script startup readiness checks without polling.
+```
+
+## Backports
+
+When a PR is backported to a maintenance branch (e.g. `9.3`), the automated
+backport bot cherry-picks the original commits, which includes the fragment
+file. **Do not create a new fragment for the backport PR.**
+
+The original fragment (named after the original PR number) will appear in the
+release notes of every version it lands in. For example, if PR #500 is merged
+to `main` and backported to `9.3`:
+
+- `9.4.0` release notes will include the entry from `500.yaml` (via `main`)
+- `9.3.x` release notes will include the same entry (via the cherry-pick)
+
+### CI behaviour for backport PRs
+
+CI does not require a fragment named after the backport PR number. Instead it
+verifies that **at least one `docs/changelog/*.yaml` file appears in the PR
+diff** — confirming the fragment was not accidentally dropped during
+cherry-pick conflict resolution.
+
+If a fragment is missing from a backport PR diff, check whether it was dropped
+during conflict resolution and re-add it manually.
+
+If the backport genuinely needs no release notes entry (e.g. a CI-only fix),
+add `[rn:skip]` to the backport PR description.
+
+### Branch scope
+
+Fragment validation only runs on PRs targeting `main` and `9.*` branches.
+PRs targeting `8.19` (which uses a different asciidoc-based release notes
+process) are not affected.
+
+## Release notes generation
+
+Release notes are generated by `tools/release/generate_release_notes_md.rb`.
+Running the generator is safe to repeat: it reads all `*.yaml` files currently
+in this directory, writes the release notes, and records the consumed fragments
+in a bundle file at `docs/release-notes/changelog-bundles/<version>.yml`.
+**It does not delete fragment files.** This means the generator can be re-run
+as many times as needed — for example, when a late-breaking change arrives
+after the first draft.
+
+## Pruning fragments after release
+
+Once the release notes for a version are finalised and the release has shipped,
+run the prune script to remove fragment files that have been captured in a bundle:
+
+```bash
+# Preview what would be deleted
+ruby tools/release/prune_changelog_fragments.rb --dry-run
+
+# Delete and stage
+ruby tools/release/prune_changelog_fragments.rb
+git add docs/changelog/
+git commit -m "Prune changelog fragments for <version>"
+```
+
+The prune step reads all bundle files and deletes any fragment whose PR number
+appears in a bundle. It is safe to run repeatedly.
+
+## Validation
+
+To validate fragment files locally before pushing:
+
+```bash
+# Validate a specific fragment
+ruby tools/release/validate_changelog_fragment.rb docs/changelog/12345.yaml
+
+# Validate all fragments
+ruby tools/release/validate_changelog_fragment.rb --all
+```
+
+CI runs this check automatically on every PR.

tools/release/generate_release_notes_md.rb

@@ -26,8 +26,23 @@
 require 'yaml'
 require 'json'
 require 'net/http'
+require 'fileutils'
 
 RELEASE_NOTES_PATH = "docs/release-notes/index.md"
+CHANGELOG_FRAGMENTS_PATH = "docs/changelog"
+CHANGELOG_BUNDLES_PATH = "docs/release-notes/changelog-bundles"
+
+SECTION_ORDER = %w[feature enhancement bug breaking_change deprecation dependency doc].freeze
+SECTION_LABELS = {
+  "feature"        => "New features",
+  "enhancement"    => "Enhancements",
+  "bug"            => "Bug fixes",
+  "breaking_change" => "Breaking changes",
+  "deprecation"    => "Deprecations",
+  "dependency"     => "Dependency updates",
+  "doc"            => "Documentation",
+}.freeze
+
 release_branch = ARGV[0]
 previous_release_tag = ARGV[1]
 user = ARGV[2]
@@ -45,28 +60,48 @@
 release_notes_entry_index = coming_tag_index || release_notes.find_index {|line| line.match(/^## .*\[logstash-.*-release-notes\]$/) }
 
 unless coming_tag_index
-  report << "## #{current_release} [logstash-#{current_release}-release-notes]\n\n"
-  report << "### Features and enhancements [logstash-#{current_release}-features-enhancements]\n"
+  report << "## #{current_release} [logstash-#{current_release}-release-notes]\n"
 end
 
-plugin_changes = {}
+# Load changelog fragments and group by type
+fragments = Dir.glob("#{CHANGELOG_FRAGMENTS_PATH}/*.yaml").sort.map do |path|
+  YAML.safe_load(File.read(path), permitted_classes: [Integer])
+rescue => e
+  $stderr.puts "Warning: skipping #{path}: #{e.message}"
+  nil
+end.compact
+
+highlights = fragments.select { |f| f['highlight']&.fetch('notable', false) }
+unless highlights.empty?
+  report << "### Highlights [logstash-#{current_release}-highlights]\n"
+  highlights.each do |f|
+    h = f['highlight']
+    report << "#### #{h['title']}\n"
+    report << h['body'].to_s
+    report << ""
+  end
+end
 
-report <<  "---------- GENERATED CONTENT STARTS HERE ------------"
-report <<  "=== Logstash Pull Requests with label v#{current_release}\n"
+by_type = fragments.group_by { |f| f['type'] }
+
+SECTION_ORDER.each do |type|
+  entries = by_type[type]
+  next unless entries&.any?
 
-uri = URI.parse("https://api.github.com/search/issues?q=repo:elastic/logstash+is:pr+is:closed+label:v#{current_release}&sort=created&order=asc")
-pull_requests = JSON.parse(Net::HTTP.get(uri))
-pull_requests['items'].each do |prs|
-  report << "* #{prs['title']} #{prs['html_url']}[##{prs['number']}]"
+  anchor = "logstash-#{current_release}-#{type.tr('_', '-')}"
+  report << "### #{SECTION_LABELS[type]} [#{anchor}]\n"
+  entries.sort_by { |f| f['pr'] }.each do |f|
+    issue_links = Array(f['issues']).map { |i| "[##{i}](https://github.com/elastic/logstash/issues/#{i})" }.join(", ")
+    suffix = issue_links.empty? ? "" : " (#{issue_links})"
+    report << "* #{f['summary']} [##{f['pr']}](https://github.com/elastic/logstash/pull/#{f['pr']})#{suffix}"
+  end
+  report << ""
 end
-report << ""
-
-report <<  "=== Logstash Commits between #{release_branch} and #{previous_release_tag}\n"
-report <<  "Computed with \"git log --pretty=format:'%h -%d %s (%cr) <%an>' --abbrev-commit --date=relative v#{previous_release_tag}..#{release_branch}\""
-report <<  ""
-logstash_prs = `git log --pretty=format:'%h -%d %s (%cr) <%an>' --abbrev-commit --date=relative v#{previous_release_tag}..#{release_branch}`
-report << logstash_prs
-report << "\n=== Logstash Plugin Release Changelogs ==="
+
+plugin_changes = {}
+
+report <<  "---------- GENERATED CONTENT STARTS HERE ------------"
+report <<  "=== Logstash Plugin Release Changelogs ==="
 report << "Computed from \"git diff v#{previous_release_tag}..#{release_branch} *.release\""
 result = `git diff v#{previous_release_tag}..#{release_branch} *.release`.split("\n")
 
@@ -115,6 +150,18 @@
 
 IO.write(RELEASE_NOTES_PATH, release_notes.join("\n"))
 
+# Write a bundle file recording which fragments went into this release.
+# This is the input for prune_changelog_fragments.rb, which is run separately
+# once the release notes are finalised — keeping generation re-runnable.
+FileUtils.mkdir_p(CHANGELOG_BUNDLES_PATH)
+bundle = {
+  "version"    => current_release,
+  "generated"  => Time.now.utc.strftime("%Y-%m-%dT%H:%M:%SZ"),
+  "changelogs" => fragments
+}
+bundle_path = File.join(CHANGELOG_BUNDLES_PATH, "#{current_release}.yml")
+IO.write(bundle_path, YAML.dump(bundle))
+
 if token.nil?
   puts "No token provided, skipping commit and push"
   exit
@@ -123,7 +170,8 @@
 puts "Creating commit.."
 branch_name = "update_release_notes_#{Time.now.to_i}"
 `git checkout -b #{branch_name}`
-`git commit #{RELEASE_NOTES_PATH} -m "Update release notes for #{current_release}"`
+`git add #{RELEASE_NOTES_PATH} #{bundle_path}`
+`git commit -m "Update release notes for #{current_release}"`
 
 puts "Pushing commit.."
 `git remote set-url origin https://x-access-token:#{token}@github.com/elastic/logstash.git`