AWS dynamic dispatcher with transient secrets - #9316
Open
allisonkarlitskaya wants to merge 11 commits into
Open
Conversation
![github-advanced-security[bot]](https://avatars.githubusercontent.com/in/57789?s=60&v=4){kind=small}
| print(f" - bots ref: {args.bots_ref}") | ||
| print(f" - bots sha: {sha}") | ||
| print(f" - bots url: {bots_url}") | ||
| print(f" - secrets: {list(secrets)}") |
| duration: timedelta = timedelta(hours=1), | ||
| ) -> Iterator[tuple[str, JsonValue]]: | ||
|
|
||
| logger.debug("providing secrets %r with duration %r", secrets, duration) |
| match output: | ||
| case 'env': | ||
| print(f'export AWS_ACCESS_KEY_ID={shlex.quote(key.access)}') | ||
| print(f'export AWS_SECRET_ACCESS_KEY={shlex.quote(key.secret)}') |
Comment on lines
+51
to
+56
| print(json.dumps({ | ||
| 'Version': 1, | ||
| 'AccessKeyId': key.access, | ||
| 'SecretAccessKey': key.secret, | ||
| 'SessionToken': key.token, | ||
| })) |
There's no real decent upstream stubs for pika but they're kinda small so we can do our own.
This file defines various constants pertaining to our AWS account. Users of it will be introduced in the following commits. In general things in this directory will need to run on our future dispatcher, which will be running Debian trixie, which only has Python 3.13. Add a ruff override.
In 7c6e4af ("sts: increase ticket duration to 12 hours") we increased the session duration to 12 hours for all roles, but that information is actually tied to the specific role and most of them don't support longer sessions. Move that to to the SAMLTarget dataclass and take the chance to use the various constants from `lib/aws/accounts.py`.
Move secret expansion out of JobContext.__aenter__() and into a new
prepare_secrets() method that is called from run_container() with the
job's requested secrets and a tmpdir for materialized inline values.
This means container.secrets entries in the config are only expanded
when a job actually requests them — secrets that no job asks for sit
inert, even if their %{name} references can't be resolved. This
unblocks putting secret wiring in the default config without it
exploding for unconfigured secrets.
The two error cases (no wiring for a requested secret, wiring
references an undefined value) raise LookupError with a descriptive
message, caught in run_container() and surfaced as a job Failure.
Assisted-by: Claude Opus 4.6 <noreply@anthropic.com>
Our log bucket and image download stores are now entirely derivable from our AWS account information, so let's do that. We still keep this file separate: these used to come from Linode and may well some day be replaced (or supplemented) by non-AWS things again.
This will help us serialize minted S3 keys when spinning up instances in the new dispatcher.
Add an option to attach the journal output from the current systemd invocation to the log generated by job-runner. This can be helpful for showing how job-runner itself was invoked, assuming it's done via a one-off unit.
Modify `saml-login` to use the constants from `lib/aws/accounts.py` instead of selecting the first target in `gssapi_saml_sts.py` (which is now defined in terms of the former, anyway). Add the ability to authenticate as arbitrary roles by passing the role name as a positional on the command line. For compatibility, the default is still to authenticate for downloading CI images. Add a new --output= CLI argument which allows generating a POSIX shell block (`--output=env`) which can be sourced to set `AWS_*` environment variables. Even better is a `--output=awscreds` which can be used as a credential helper in `~/.aws/credentials`, like so: ``` [admin] credential_process = /path/to/saml-login --output=awscreds 727920394381-admin ```
This is the start of a centralized definition of the shape of our CI infrastructure inside of AWS. It replaces and expands on the existing `ansible/aws/setup-s3-buckets.yml` in the `cockpituous` repository by adding the IAM roles for SSO-authenticated `image-download` and the roles that will be used by the new task dispatcher. It also provisions an ASG (autoscaling group) which ensures that a single machine is always running to function as the dispatcher. Currently no actual dispatcher is running in that machine, but the roles are configured and we can use it for testing. The contents of this directory use the `boto3` Python library (the official AWS API for Python) and assume that you're authenticated against a suitable IAM Role. Further automation around this point will come soon. The main entry point here is an `infractl` command which has three subcommands: - `sync`: ensures all of our deployed resources match the configuration defined in `infra_definitions.py` and reports on any resources which aren't managed in that file - `dispatcher`: allows querying the status of the dispatcher, temporarily taking it offline, restarting it, and has an `ssh` command to ssh to it - `runners`: lists current CI runs in progress The "meat" of the infrastructure definition is inside of `lib/aws/infra_definitions.py` which is written in an imperative style but can effectively be treated as declarative data. It's backstopped by a series of functions in `ensure_resource.py` which are about ensuring that the resource in question has the desired shape. I considered a very large number of alternative approaches before, during, and after developing this but all of the existing solutions have a number of problems: - depending on "community" modules for with incomplete support of the resources we're deploying (Ansible) - needing to keep a browser open with documentation in order to write yaml dialects (Ansible, CloudFormation) with poor editor support or custom DSLs (Terraform/OpenTofu). - requirement for extra "state" storage volumes and/or databases for locking (Terraform/OpenTofu) - "state" that is used as the basis for reconciliation but can itself differ from reality when "emergency fixes" are made in the AWS web console (Terraform/OpenTofu/CF) - no tracking of resources outside of what is mentioned in the plan or the state file (all) Our own custom-rolled solution has its own drawbacks, mainly that we have to maintain the helpers in `ensure_resource.py`, and trust that they produce correct results. It also doesn't have support for dry-runs or diffing. On the other hand, it's written in Python (understood by everyone on the team) and the type annotations and editor support are extremely good, even down to detecting typos in instance types and region names. The closest contenders were probably AWS CDK and the unified AWS Cloud Control API. AWS CDK is similar to what we're doing and has a nice `cdk` CLI tool for deployment but is a massive unpackaged (on Fedora) dependency which is a somewhat lossy Python binding wrapped around a Node.js core with a bridge between them. Cloud control is interesting but somewhat unwieldy due to the massive number of sub-resource types and the need to understand JSON schemas and produce JSON merge patches to interact with it properly. It also doesn't give a huge advantage over the "traditional" APIs we're already using.
This script has been useful for me when creating test runs using a local AMQP server. In general, you can do something like this: podman run --rm -it -p 5672:5672 docker.io/library/rabbitmq python3 -m test.generate_test_jobs --amqp > slugs python3 -m lib.aws.dispatcher --param amqp-server=localhost:5672 and then inspect the generated logs for failures. `slugs` will contain a list of the relevant slugs on the logs server.
The dashboard html and JS have been written against and checked with Biome and TypeScript. Let's add configs for those (and stubs for Lit) and wire them up to test/run.
allisonkarlitskaya
force-pushed
the
ec3-thingy
branch
from
August 4, 2026 18:09
581e265 to
98a3754
Compare
| ) -> None: | ||
| ssm = boto3.client("ssm", region_name=CI_RUNNER_REGION) | ||
|
|
||
| print(f" - {name} ({param_type}, {len(value)} bytes)") |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This is a massive change. It moves us to being able to dispatch jobs on AWS, but that undersells it substantially.
The three main benefits here (and the resultant complexity) are:
The
lib/aws/README.mdfile is probably a good place to start reading.