chore(ci): add code-block testing workflow#6810
Merged
jstirnaman merged 18 commits intomasterfrom Apr 22, 2026
Merged
Conversation
Add GitHub Actions workflow to run code-block tests on PRs. Features: - Automatic detection of changed content files - Matrix strategy to run product tests in parallel - 30-minute timeout for test execution - Smart test selection based on changed files - Mock credentials for CI environment - Detailed test summaries and artifact uploads Also adds `test:codeblocks:parallel` script to package.json for running tests in parallel locally.
Implement parallel test execution by language and test result caching to significantly improve test performance. Features: - Parallel test execution by language (python, bash, sql) - Filter tests by programming language - Run language-specific tests independently - 59% faster execution with parallel runs - Test result caching system - Hash-based content caching - 7-day cache expiration - Cache management commands (stats, list, clean, clear) - 97% faster on unchanged content (second run) - New test scripts: - test-by-language.sh: Filter and test specific languages - cached-test.sh: Cache test results by content hash - manage-test-cache.sh: Manage cache entries - New package.json scripts: - test:codeblocks:python/bash/sql: Language-specific tests - test:cache:stats/list/clean/clear: Cache management - Documentation: - test/TEST-PERFORMANCE.md: Comprehensive performance guide - DOCS-TESTING.md: Added performance optimization section Performance improvements: - Sequential: ~45 minutes - Parallel: ~18 minutes (59% faster) - Cached (2nd run): ~5 seconds (97% faster) Analysis: - 766 testable code blocks (sh: 582, bash: 90, python: 10, sql: 46, shell: 38) - Language aliases supported (bash→sh/shell, python→py, sql→influxql) - Smart cache invalidation on content changes
Contributor
There was a problem hiding this comment.
Pull request overview
Adds CI automation and local tooling to speed up pytest-based code-block testing by selecting affected products, running suites in parallel, and introducing language-focused and cached test helpers.
Changes:
- Add a GitHub Actions workflow to run code-block tests on PRs with product selection + matrix execution.
- Add shell scripts for language-filtered runs and local caching/management of codeblock test results.
- Extend docs and
package.jsonwith new testing commands and performance guidance.
Reviewed changes
Copilot reviewed 7 out of 8 changed files in this pull request and generated 10 comments.
Show a summary per file
| File | Description |
|---|---|
.github/workflows/test.yml |
New PR workflow to detect changed content and run code-block tests via a product matrix. |
package.json |
Adds parallel/local language-specific codeblock test commands + cache management commands. |
test/scripts/test-by-language.sh |
New helper intended to run tests against markdown files containing a target code-fence language. |
test/scripts/cached-test.sh |
New helper to skip rerunning tests when target content hash hasn’t changed. |
test/scripts/manage-test-cache.sh |
New helper to inspect/clean/clear cached test results. |
DOCS-TESTING.md |
Adds a “Performance Optimization” section describing the new workflows/scripts. |
test/TEST-PERFORMANCE.md |
New extended documentation on codeblock testing performance strategies. |
.gitignore |
Ignores .test-cache directory created by caching helpers. |
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
Contributor
|
@jstirnaman I've opened a new pull request, #6811, to work on those changes. Once the pull request is ready, I'll request review from you. |
jstirnaman
commented
Feb 11, 2026
Contributor
Author
jstirnaman
left a comment
jstirnaman
left a comment
There was a problem hiding this comment.
@copilot Replace default products with Core and Telegraf
7 tasks
Contributor
|
@jstirnaman I've opened a new pull request, #6812, to work on those changes. Once the pull request is ready, I'll request review from you. |
* feat(ci): add per-product codeblock testing with default group - Add support for all products: core, enterprise, v2, v1, telegraf, cloud, cloud-dedicated, cloud-serverless, clustered, explorer - Define default test group (core + telegraf) when no product specified - Exclude cloud products from automatic CI (manual dispatch only) - Add placeholder scripts for products without pytest services - Normalize product name handling (core, influxdb3_core, influxdb3-core) - Log informative messages when excluded products' content changes * chore(ci): make codeblock tests manual-only and informational - Remove pull_request trigger, keep only workflow_dispatch - Change all exit codes to 0 so workflow never blocks PRs - Use warnings instead of errors for failed tests - Simplify job from detect-changes to parse-inputs * feat(ci): make codeblock tests informational on PRs, manual-only execution - PRs now trigger detection-only mode that suggests which products to test - Actual test execution only runs via manual workflow_dispatch - Add detect-test-products.js script using existing content-utils library - Properly resolve shared content to affected product pages - Non-blocking: PRs won't fail due to codeblock test issues * Update .github/workflows/test.yml Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> * fix(ci): add guard for empty products array after parsing Co-authored-by: jstirnaman <212227+jstirnaman@users.noreply.github.com> * fix(ci): remove redundant output before exit 1 Co-authored-by: jstirnaman <212227+jstirnaman@users.noreply.github.com> --------- Co-authored-by: Jason Stirnaman <jstirnaman@influxdata.com> Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com> Co-authored-by: jstirnaman <212227+jstirnaman@users.noreply.github.com>
4 tasks
The workflow triggers on changes to test/**, Dockerfile.pytest, and compose.yaml, but detect-changes only grepped content/**/*.md. Harness-only PRs would trigger the workflow and then report 'No content changes'. Detect harness paths separately. When only harness files change, suggest the default group (core + telegraf). When both content and harness files change, merge the default group into the detected product list so that harness changes are covered. Supersedes #6811, which carried the same idea against an earlier revision of this branch.
Contributor
Vale Style Check Results
Warnings (21)
Showing first 20 of 21 warnings. ✅ Check passed |
When matrix.product == influxdb3_core, generate an offline admin token, write it to test/.influxdb3/core/.token (the compose secret path) and to content/influxdb3/core/.env.test, then start the influxdb3-core compose service with --admin-token-file. The pytest container shares the compose network and reaches the server by service name (influxdb3-core:8181). This is the first product in the workflow that actually runs against a live instance instead of mock credentials. Enterprise can follow the same pattern once we settle on license handling. Also capture server logs as an artifact and tear down the stack on job exit so matrix jobs don't leak state.
The influxdb:3-core image runs as a non-root user. Host-created bind-mount dirs are owned by the runner user and not writable by the container, so the server crashes on startup with: failed to persist catalog checkpoint file ... PermissionDenied Make test/.influxdb3 world-writable before `compose up`.
The influxdb:3-core container runs as a non-root user. When a 0600
host file is bind-mounted through docker compose secrets to
/run/secrets/, the container's user can't read it:
ERROR Failed to initialize admin token from file:
Failed to read admin token file: Permission denied
Loosen to 0644. Docs recommend 0600 for production — this is CI with
a throwaway token, not a secrets-management pattern.
The previous probe used `curl -fsS /ping` without a token. In Core 3.9 the /ping endpoint requires auth by default (can be disabled with --disable-authz=ping), so the server returned 401 and curl treated it as a failure. The server was actually running and responsive — we were just asking it the wrong way. Include the admin token in the readiness request. As a bonus, this verifies the token was loaded correctly before pytest gets its turn.
The workflow comments and the test step's `continue-on-error: true` both signal that code-block failures are informational, not gating. But a later step explicitly `exit 1`s when test-status == failed, which undoes the non-blocking intent and turns the job red. Replace the hard fail with a `::warning::` annotation so the signal still surfaces in the PR UI and job summary but doesn't fail the matrix job. Summary and artifact steps run regardless.
Mirrors the influxdb3-core-pytest service but targets content/influxdb3/enterprise/ and uses an enterprise-specific tmp volume. Enables yarn test:codeblocks:influxdb3_enterprise to run against the influxdb3-enterprise server on the compose network. Also declares named volumes influxdb3-core-tmp and influxdb3-enterprise-tmp that were previously referenced but not declared at the top level.
Mirrors the Core startup pattern with three Enterprise-specific pieces: - Requires INFLUXDB3_ENTERPRISE_LICENSE_EMAIL repo secret. Writes it to $HOME/influxdata-docs/.influxdb3/enterprise/.env where the compose env_file directive reads it. - Preconfigured offline admin token at admin-token.json (same schema as Core), mounted via the influxdb3-enterprise-admin-token secret. - Starts the service with `--profile shared` (Enterprise lives in that profile, not the default). Readiness probe uses the admin token against /ping on host port 8181. Loop extends to 120s since Enterprise activation can take longer than Core's first-boot catalog creation. Also replaces the test-case skip for influxdb3_enterprise with the real yarn script, and adds parallel log-capture / teardown steps so Enterprise runs don't leak state between matrix jobs.
test:codeblocks:default ran Core && Telegraf serially, meaning Core failure skipped Telegraf entirely and the total runtime was sum of both. Background both, wait on each PID, OR the exit codes so the overall command fails if either leg fails.
… suggestions Enterprise: Trial activation requires out-of-band email verification which can't complete in headless CI. The previous workflow happily started the server and then timed out waiting for a license that would never arrive. Gate startup on a new INFLUXDB3_ENTERPRISE_LICENSE_BLOB secret (base64-encoded trial_or_home_license file from a manually activated instance). When the secret is missing, skip Enterprise pytest cleanly with a ::notice:: annotation explaining how to enable. When present, decode into /var/lib/influxdb3/data/cluster0/trial_or_home_license so the server skips activation. The Enterprise startup step now carries an id so the downstream test case can check the skip-pytest output. Suggest-tests: The PR-mode summary listed v1, explorer, and any other content paths as actionable suggestions even though their yarn scripts are no-op echoes. Partition the detected products into 'runnable' (have a compose pytest service) and 'not yet runnable' buckets and label them separately so users don't burn time running no-ops.
Add a Code-block-testing-in-CI subsection that explains: - How pull-request vs workflow_dispatch triggers differ - Which products run against live servers vs mock credentials - Where the workflow lives and how to trigger it manually Add an Enterprise provisioning recipe that walks through: - Registering a verification email - Activating Enterprise locally to produce a license file - Uploading the base64-encoded license as INFLUXDB3_ENTERPRISE_LICENSE_BLOB - Re-triggering CI to confirm Enterprise starts Callout flags that trial licenses expire so the blob needs periodic refresh. Mirrors the inline ::notice:: the workflow surfaces when the secret is missing.
The harness-only and workflow_dispatch fallback paths suggested/ran only Core + Telegraf. Now that Enterprise is a working product in CI (skipped cleanly without INFLUXDB3_ENTERPRISE_LICENSE_BLOB, run live with it), it belongs in the default group — otherwise PRs that touch harness files never surface Enterprise in Suggest Tests, and manual dispatches with no products specified skip it.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Add GitHub Actions workflow to run code-block tests on PRs.
Features:
Also adds
test:codeblocks:parallelscript to package.jsonfor running tests in parallel locally.