Improves pipeline testing and code documentation by alexiswl · Pull Request #255 · OrcaBus/platform-cdk-constructs

alexiswl · 2026-08-06T10:49:46Z

Adds a new Python package orcabus-pipeline-test-utils to provide shared test utilities for OrcaBus Pipeline Orchestrator services. This package:

Includes pytest fixtures for AWS mocks, Lambda context, and event building.
Offers ASL structural validation, placeholder resolution, and cross-reference checking.
Provides a Docker-based Step Functions Local harness for integration testing.
Implements post-deployment smoke tests for Lambda, Step Functions, and SSM parameters.

Enhances code clarity by adding comprehensive module, class, and function docstrings across numerous Python Lambda layers and submodules in fastapi_tools, icav2_tools, mart_tools, and orcabus_api_tools.

Introduces new steering documentation files that outline the platform's product overview, project structure, and tech stack, aiding project understanding.

Updates the FASTQ API tools with distinct job models (FastqJob, FastqSetJob) and adds new helper functions for querying and creating FASTQ set jobs.

New Feature

Adds orcabus-pipeline-test-utils package for shared testing utilities.
Introduces new job models and querying capabilities for the Fastq API.

Bug Fixes

Addresses known bugs or regressions.

Breaking Changes

Introduces backward-incompatible changes.

Documentation

Adds comprehensive docstrings to Python modules and functions.
Includes new .kiro/steering documentation files.

Tests

Adds new tests to cover the introduced functionality (e.g., property-based tests for ASL validation and assertions).

Configuration Changes

Updates project configuration (e.g., pyproject.toml, .secrets.baseline).

Dependency Updates

Adds new runtime or development dependencies.

Copilot

Pull request overview

Adds a new shared Python package (orcabus-pipeline-test-utils) for pipeline/orchestrator testing (fixtures, ASL validation, Step Functions Local harness, smoke checks), while also improving clarity of existing Python Lambda-layer modules via docstrings and expanding FASTQ API job modeling/helpers. It also introduces new steering docs under .kiro/steering to document product/structure/tech stack.

Changes:

Introduces orcabus-pipeline-test-utils with pytest fixtures, ASL placeholder/reference validation utilities, smoke checks, and a Docker-based Step Functions Local test runner.
Adds/updates OrcaBus API tools (notably FASTQ) to support distinct job models and new job-query helpers.
Adds extensive docstrings across multiple Python Lambda layers and introduces .kiro/steering documentation.

Reviewed changes

Copilot reviewed 123 out of 124 changed files in this pull request and generated 5 comments.

Show a summary per file

File	Description
packages/orcabus-pipeline-test-utils/tests/init.py	Adds test package marker/docstring.
packages/orcabus-pipeline-test-utils/tests/test_aws_mocks.py	Tests for moto-based AWS mock fixtures.
packages/orcabus-pipeline-test-utils/tests/test_event_builder.py	Tests for event builder fixture behavior.
packages/orcabus-pipeline-test-utils/tests/test_lambda_arn_checker.py	Unit tests for Lambda ARN placeholder reference checking.
packages/orcabus-pipeline-test-utils/tests/test_lambda_context.py	Tests for mock Lambda context fixture/dataclass.
packages/orcabus-pipeline-test-utils/tests/test_placeholder_resolver.py	Unit tests for placeholder resolution + placeholder map loading.
packages/orcabus-pipeline-test-utils/tests/test_property_execution_assertion.py	Property tests for execution status assertion behavior.
packages/orcabus-pipeline-test-utils/tests/test_property_lambda_arn_refs.py	Property tests for Lambda ARN placeholder cross-references.
packages/orcabus-pipeline-test-utils/tests/test_property_map_cardinality.py	Property tests for Map-state output cardinality assertion.
packages/orcabus-pipeline-test-utils/tests/test_property_smoke_validation.py	Property tests for smoke-check pass/fail classification rules.
packages/orcabus-pipeline-test-utils/tests/test_smoke_lambda.py	Unit tests for Lambda DryRun smoke check.
packages/orcabus-pipeline-test-utils/tests/test_smoke_sfn.py	Unit tests for Step Functions state machine smoke check.
packages/orcabus-pipeline-test-utils/tests/test_smoke_ssm.py	Unit tests for SSM parameter existence smoke check.
packages/orcabus-pipeline-test-utils/src/orcabus_pipeline_test_utils/init.py	Declares package + version metadata.
packages/orcabus-pipeline-test-utils/src/orcabus_pipeline_test_utils/conftest_plugin.py	Registers pytest plugin to expose shared fixtures automatically.
packages/orcabus-pipeline-test-utils/src/orcabus_pipeline_test_utils/fixtures/init.py	Documents/shared exports for fixture module.
packages/orcabus-pipeline-test-utils/src/orcabus_pipeline_test_utils/fixtures/aws_mocks.py	Implements session-scoped moto clients + dummy credentials fixture.
packages/orcabus-pipeline-test-utils/src/orcabus_pipeline_test_utils/fixtures/event_builder.py	Implements simple event builder fixture.
packages/orcabus-pipeline-test-utils/src/orcabus_pipeline_test_utils/fixtures/lambda_context.py	Implements mock Lambda context + factory fixture.
packages/orcabus-pipeline-test-utils/src/orcabus_pipeline_test_utils/asl_validation/init.py	Public exports for ASL validation utilities.
packages/orcabus-pipeline-test-utils/src/orcabus_pipeline_test_utils/asl_validation/placeholder_resolver.py	Resolves `${__...__}` placeholders with generated or mapped values.
packages/orcabus-pipeline-test-utils/src/orcabus_pipeline_test_utils/asl_validation/reference_checker.py	Validates ASL state references + Lambda ARN placeholders vs config.
packages/orcabus-pipeline-test-utils/src/orcabus_pipeline_test_utils/smoke/init.py	Defines `SmokeTestResult` and smoke module exports.
packages/orcabus-pipeline-test-utils/src/orcabus_pipeline_test_utils/smoke/lambda_check.py	Adds Lambda DryRun smoke check.
packages/orcabus-pipeline-test-utils/src/orcabus_pipeline_test_utils/smoke/sfn_check.py	Adds Step Functions state machine smoke check.
packages/orcabus-pipeline-test-utils/src/orcabus_pipeline_test_utils/smoke/ssm_check.py	Adds SSM parameter existence/readability smoke check.
packages/orcabus-pipeline-test-utils/src/orcabus_pipeline_test_utils/sfn_local/init.py	Documents Step Functions Local harness modules.
packages/orcabus-pipeline-test-utils/src/orcabus_pipeline_test_utils/sfn_local/main.py	Adds module entry point for SFN-local runner.
packages/orcabus-pipeline-test-utils/src/orcabus_pipeline_test_utils/sfn_local/client.py	Wraps boto3 Step Functions client for local endpoint + helpers.
packages/orcabus-pipeline-test-utils/src/orcabus_pipeline_test_utils/sfn_local/container.py	Manages Step Functions Local Docker container lifecycle.
packages/orcabus-pipeline-test-utils/src/orcabus_pipeline_test_utils/sfn_local/mock_config.py	Fluent builder for MockConfigFile.json structure.
packages/orcabus-pipeline-test-utils/src/orcabus_pipeline_test_utils/sfn_local/runner.py	CLI runner orchestrating container start/pytest/cleanup with timeout.
packages/orcabus-pipeline-test-utils/pyproject.toml	Adds packaging metadata, deps, and pytest plugin entrypoint.
packages/lambda/layers/orcabus_api_tools/src/orcabus_api_tools/workflow/init.py	Improves docstrings and switches to relative imports.
packages/lambda/layers/orcabus_api_tools/src/orcabus_api_tools/workflow/create_helpers.py	Adds docstrings (and updates module header).
packages/lambda/layers/orcabus_api_tools/src/orcabus_api_tools/workflow/errors.py	Adds module/class docstrings for workflow errors.
packages/lambda/layers/orcabus_api_tools/src/orcabus_api_tools/workflow/globals.py	Adds module docstring/header.
packages/lambda/layers/orcabus_api_tools/src/orcabus_api_tools/workflow/models.py	Adds docstrings to workflow TypedDict models.
packages/lambda/layers/orcabus_api_tools/src/orcabus_api_tools/utils/init.py	Adds module docstring for utilities package.
packages/lambda/layers/orcabus_api_tools/src/orcabus_api_tools/utils/aws_helpers.py	Adds docstrings for AWS helper functions.
packages/lambda/layers/orcabus_api_tools/src/orcabus_api_tools/utils/miscell.py	Adds docstring for misc utilities module.
packages/lambda/layers/orcabus_api_tools/src/orcabus_api_tools/utils/requests_helpers.py	Adds docstrings and improves `HTTPError` re-raise detail preservation.
packages/lambda/layers/orcabus_api_tools/src/orcabus_api_tools/sequence/init.py	Adds docstrings for sequence service helpers.
packages/lambda/layers/orcabus_api_tools/src/orcabus_api_tools/sequence/globals.py	Adds docstring for sequence globals.
packages/lambda/layers/orcabus_api_tools/src/orcabus_api_tools/sequence/models.py	Adds TypedDict docstrings for sequence models.
packages/lambda/layers/orcabus_api_tools/src/orcabus_api_tools/metadata/init.py	Adds module docstring and reorganizes imports.
packages/lambda/layers/orcabus_api_tools/src/orcabus_api_tools/metadata/errors.py	Adds module/class docstrings for metadata errors.
packages/lambda/layers/orcabus_api_tools/src/orcabus_api_tools/metadata/globals.py	Adds docstring for metadata globals.
packages/lambda/layers/orcabus_api_tools/src/orcabus_api_tools/metadata/library_helpers.py	Adds module docstring and cleans import formatting.
packages/lambda/layers/orcabus_api_tools/src/orcabus_api_tools/metadata/models.py	Adds docstrings to metadata TypedDict models.
packages/lambda/layers/orcabus_api_tools/src/orcabus_api_tools/mart/init.py	Adds mart module docstring.
packages/lambda/layers/orcabus_api_tools/src/orcabus_api_tools/mart/aws_helpers.py	Adds docstrings for Athena/S3 helper functions.
packages/lambda/layers/orcabus_api_tools/src/orcabus_api_tools/mart/dataframe_helpers.py	Adds docstring for dtype mapping helper.
packages/lambda/layers/orcabus_api_tools/src/orcabus_api_tools/mart/globals.py	Adds mart globals docstring.
packages/lambda/layers/orcabus_api_tools/src/orcabus_api_tools/mart/models.py	Adds docstrings for Athena/pandas dtype enums.
packages/lambda/layers/orcabus_api_tools/src/orcabus_api_tools/icav2_wes/init.py	Adds docstrings for ICAv2 WES toolkit wrappers.
packages/lambda/layers/orcabus_api_tools/src/orcabus_api_tools/icav2_wes/globals.py	Adds docstring for ICAv2 WES globals.
packages/lambda/layers/orcabus_api_tools/src/orcabus_api_tools/icav2_wes/models.py	Adds docstrings to ICAv2 WES TypedDict models.
packages/lambda/layers/orcabus_api_tools/src/orcabus_api_tools/icav2_wes/query_helpers.py	Adds docstring to ICAv2 WES query helpers.
packages/lambda/layers/orcabus_api_tools/src/orcabus_api_tools/icav2_wes/update_helpers.py	Adds docstring to ICAv2 WES update helpers.
packages/lambda/layers/orcabus_api_tools/src/orcabus_api_tools/filemanager/init.py	Adds docstrings for File Manager toolkit wrappers.
packages/lambda/layers/orcabus_api_tools/src/orcabus_api_tools/filemanager/errors.py	Adds module/class docstrings for File Manager errors.
packages/lambda/layers/orcabus_api_tools/src/orcabus_api_tools/filemanager/file_helpers.py	Adds docstrings to File Manager helper functions.
packages/lambda/layers/orcabus_api_tools/src/orcabus_api_tools/filemanager/globals.py	Adds docstring for File Manager globals.
packages/lambda/layers/orcabus_api_tools/src/orcabus_api_tools/filemanager/models.py	Adds module docstring + TypedDict docstrings.
packages/lambda/layers/orcabus_api_tools/src/orcabus_api_tools/fastq/init.py	Adds toolkit docstrings + exports new helper.
packages/lambda/layers/orcabus_api_tools/src/orcabus_api_tools/fastq/create_helpers.py	Replaces outdated doc with create-helper docstring.
packages/lambda/layers/orcabus_api_tools/src/orcabus_api_tools/fastq/globals.py	Adds docstring for FASTQ globals.
packages/lambda/layers/orcabus_api_tools/src/orcabus_api_tools/fastq/job_helpers.py	Updates job helper docs and returns typed job models.
packages/lambda/layers/orcabus_api_tools/src/orcabus_api_tools/fastq/models.py	Introduces `FastqJob`/`FastqSetJob` and job-type split; adds docstrings.
packages/lambda/layers/orcabus_api_tools/src/orcabus_api_tools/fastq/query_helpers.py	Adds `get_fastq_set_jobs` and returns typed job lists.
packages/lambda/layers/orcabus_api_tools/src/orcabus_api_tools/fastq/workflow_helpers.py	Adds docstrings for workflow conversion helpers.
packages/lambda/layers/orcabus_api_tools/src/orcabus_api_tools/fastq_unarchiving/init.py	Adds docstrings for unarchiving toolkit wrappers.
packages/lambda/layers/orcabus_api_tools/src/orcabus_api_tools/fastq_unarchiving/create_helpers.py	Adds docstring for create job helper.
packages/lambda/layers/orcabus_api_tools/src/orcabus_api_tools/fastq_unarchiving/globals.py	Adds docstring for unarchiving globals.
packages/lambda/layers/orcabus_api_tools/src/orcabus_api_tools/fastq_unarchiving/models.py	Adds docstrings for unarchiving models.
packages/lambda/layers/orcabus_api_tools/src/orcabus_api_tools/fastq_unarchiving/query_helpers.py	Adds docstrings for unarchiving query helpers.
packages/lambda/layers/orcabus_api_tools/src/orcabus_api_tools/fastq_unarchiving/update_helpers.py	Adds docstrings for unarchiving update helpers.
packages/lambda/layers/orcabus_api_tools/src/orcabus_api_tools/fastq_decompression/init.py	Adds docstrings for decompression toolkit wrappers.
packages/lambda/layers/orcabus_api_tools/src/orcabus_api_tools/fastq_decompression/create_helpers.py	Adds docstring for decompression create job helper.
packages/lambda/layers/orcabus_api_tools/src/orcabus_api_tools/fastq_decompression/globals.py	Adds docstring for decompression globals.
packages/lambda/layers/orcabus_api_tools/src/orcabus_api_tools/fastq_decompression/models.py	Adds docstrings for decompression models.
packages/lambda/layers/orcabus_api_tools/src/orcabus_api_tools/fastq_decompression/query_helpers.py	Adds docstrings for decompression query helpers.
packages/lambda/layers/orcabus_api_tools/src/orcabus_api_tools/fastq_decompression/update_helpers.py	Adds docstrings for decompression update helpers.
packages/lambda/layers/orcabus_api_tools/src/orcabus_api_tools/deploy_status/init.py	Adds docstring for deploy status endpoint wrapper.
packages/lambda/layers/orcabus_api_tools/src/orcabus_api_tools/deploy_status/query_helpers.py	Adds docstring for deploy status query helper.
packages/lambda/layers/orcabus_api_tools/src/orcabus_api_tools/data_sharing/init.py	Adds toolkit docstrings + request wrapper functions.
packages/lambda/layers/orcabus_api_tools/src/orcabus_api_tools/data_sharing/globals.py	Adds docstring for data sharing globals.
packages/lambda/layers/orcabus_api_tools/src/orcabus_api_tools/data_sharing/models.py	Adds docstrings for data sharing models.
packages/lambda/layers/orcabus_api_tools/src/orcabus_api_tools/data_sharing/query_helpers.py	Adds docstrings for data sharing query helpers.
packages/lambda/layers/orcabus_api_tools/src/orcabus_api_tools/data_sharing/update_helpers.py	Adds docstrings for data sharing update helpers.
packages/lambda/layers/mart_tools/src/mart_tools/init.py	Adds docstring for mart_tools layer package.
packages/lambda/layers/mart_tools/src/mart_tools/mart/init.py	Adds mart module docstring.
packages/lambda/layers/mart_tools/src/mart_tools/mart/aws_helpers.py	Adds docstrings for Athena/S3 helper functions.
packages/lambda/layers/mart_tools/src/mart_tools/mart/dataframe_helpers.py	Adds docstring for dtype mapping helper.
packages/lambda/layers/mart_tools/src/mart_tools/mart/globals.py	Adds docstring for mart globals.
packages/lambda/layers/mart_tools/src/mart_tools/mart/models.py	Adds docstrings for Athena/pandas dtype enums.
packages/lambda/layers/icav2_tools/src/icav2_tools/init.py	Adds docstrings for ICAv2 tools helpers.
packages/lambda/layers/icav2_tools/src/icav2_tools/aws_helpers.py	Adds docstrings for ICAv2 AWS helper functions.
packages/lambda/layers/icav2_tools/src/icav2_tools/globals.py	Adds docstring for ICAv2 tools globals.
packages/lambda/layers/fastapi_tools/src/fastapi_tools/init.py	Adds FastAPI tools module docstring.
packages/lambda/layers/fastapi_tools/src/fastapi_tools/globals.py	Adds docstring for FastAPI tools globals.
packages/lambda/layers/fastapi_tools/src/fastapi_tools/pagination_helpers.py	Adds docstrings for pagination helpers/models.
.kiro/steering/product.md	Adds product overview steering documentation.
.kiro/steering/structure.md	Adds repository structure steering documentation.
.kiro/steering/tech.md	Adds tech stack/build/test steering documentation.

Suppressed comments (1)

packages/orcabus-pipeline-test-utils/tests/test_property_smoke_validation.py:27

SmokeTestResult is imported but never used in this module; remove it to avoid unused-import lint failures.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Shared Python test utilities for OrcaBus pipeline services including: - Pytest plugin with moto-based AWS mocks, Lambda context, and event builder fixtures - ASL validation (structural checks, placeholder resolution, reference checking) - Step Functions Local Docker-based integration test harness - Post-deployment smoke tests (Lambda DryRun, SFN describe, SSM parameter checks)

Copilot

Pull request overview

Copilot reviewed 38 out of 39 changed files in this pull request and generated no new comments.

Suppressed comments (5)

packages/orcabus-pipeline-test-utils/README.md:108

The placeholder resolver section suggests it handles CDK ${Token[...]} placeholders and the example passes the returned string directly into validate_asl_definition, but resolve_placeholders only replaces ${__...__} tokens and returns a string, while validate_asl_definition expects a parsed dict. This example will fail as written and is likely to confuse consumers.

Use `placeholder_resolver` to substitute CDK token placeholders (e.g. `${Token[...]}}`)
in ASL definitions before validation:

```python
from orcabus_pipeline_test_utils.asl_validation.placeholder_resolver import resolve_placeholders

packages/orcabus-pipeline-test-utils/README.md:123

The README example calls check_lambda_arn_references(asl_definition, known_lambda_arns=[...]), but the actual function signature expects a CDK lambda config map (e.g. {"lambdas": {"name": {"placeholder": "${__...__}", ...}}}) as the second argument. As written, this snippet won’t run and misdocuments the API.

from orcabus_pipeline_test_utils.asl_validation.reference_checker import check_lambda_arn_references

errors = check_lambda_arn_references(asl_definition, known_lambda_arns=["arn:aws:lambda:..."])
assert errors == []

packages/orcabus-pipeline-test-utils/src/orcabus_pipeline_test_utils/smoke/ssm_check.py:65

check_ssm_parameters_exist only handles ClientError. Any other exception (e.g. network/endpoint issues, credential provider errors that aren't surfaced as ClientError) will currently bubble out and crash the smoke test run instead of returning a failed SmokeTestResult, unlike the Lambda/SFN checks which handle generic exceptions.

        except ClientError as exc:
            error_code = exc.response["Error"]["Code"]
            error_message = exc.response["Error"].get("Message", str(exc))
            error_type = "auth" if error_code in _AUTH_ERROR_CODES else "config"

packages/orcabus-pipeline-test-utils/README.md:25

README references an sfn_local module, but there is no orcabus_pipeline_test_utils/sfn_local package in this PR. Either add the missing implementation or update the module list/docs so consumers don’t try importing a module that doesn’t exist.

This issue also appears in the following locations of the same file:

line 104
line 120

| `sfn_local` | Docker-based Step Functions Local integration tests | Pre-deployment (integration tests) |

packages/orcabus-pipeline-test-utils/src/orcabus_pipeline_test_utils/sfn_teststate/assertions.py:312

PathAssertion.states_visited currently returns the next_state values from each result (excluding the last), which are the transitions rather than the actual states executed. This doesn’t match the docstring and can’t reliably report visited states unless the state name is captured alongside each TestStateResult (e.g., add a state_name field or have test_path return the visited state list).

        """Return the list of state names visited (from nextState of prior + first state)."""
        # We can't get state names directly from TestStateResult since
        # the API doesn't echo the state name. We track via the caller.
        # This is a convenience for the terminal status.
        return [r.next_state for r in self._results[:-1]] if self._results else []

alexiswl requested a lite review from Copilot August 6, 2026 10:49

alexiswl self-assigned this Aug 6, 2026

alexiswl added the enhancement New feature or request label Aug 6, 2026

Copilot started reviewing on behalf of alexiswl August 6, 2026 10:50 View session

Copilot AI reviewed Aug 6, 2026

View reviewed changes

alexiswl force-pushed the test/add-pipeline-tests branch 2 times, most recently from 673055d to 74b19d6 Compare August 6, 2026 23:53

alexiswl force-pushed the test/add-pipeline-tests branch from 74b19d6 to 7064abd Compare August 6, 2026 23:54

alexiswl added 8 commits August 7, 2026 16:45

feat: Introduce common assertion utilities

d7bb18a

feat: Remove Step Functions Local test harness

a72a91b

feat: Introduce Step Functions TestState API test harness

787f56e

docs: Update package description for TestState API

fba8746

Feat: Allow injecting boto3 client into TestStateClient

a9ea5bf

Feat: Enhance test_state API with variables and fix mock serialization

e8eb575

Chore: Add boto3-stubs for improved type checking

aedbba7

Address copilot suggestions

729812f

alexiswl requested a lite review from Copilot August 10, 2026 08:16

Copilot started reviewing on behalf of alexiswl August 10, 2026 08:16 View session

Copilot AI reviewed Aug 10, 2026

View reviewed changes

alexiswl added 3 commits August 10, 2026 18:46

feat: Introduce Step Functions TestState API for integration testing

f43e36f

refactor: Update package descriptions and build config for TestState API

b7d9141

fix(smoke): Add generic exception handling for SSM parameter checks

b30b14a

Navigation Menu

Uh oh!

Improves pipeline testing and code documentation - #255

Improves pipeline testing and code documentation#255
alexiswl wants to merge 12 commits into
mainfrom
test/add-pipeline-tests

alexiswl commented Aug 6, 2026

Uh oh!

Copilot AI left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Uh oh!

Conversation

alexiswl commented Aug 6, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants