Create auto-refreshing boto3 sessions for long-running calls to AWS services

[adivekar-utexas]

Below is a detailed design specification document for the auto‑refreshing AWS STS session solution that leverages boto3, botocore’s RefreshableCredentials.

Detailed Design Specification: Auto‑Refreshing AWS STS Session

1. Overview

This design describes an auto‑refreshing AWS session utility that automatically renews temporary credentials obtained by assuming an IAM role via AWS STS. It integrates with boto3 and botocore to provide a persistent session that always uses up‑to‑date credentials. Pydantic v2 is used to validate and transform the credentials metadata, ensuring correctness and consistency.

2. Objectives

• Seamless Credential Refresh: Automatically refresh temporary credentials before they expire.
• Pydantic Validation: Validate and standardize credential metadata using Pydantic v2.
• Abstraction Layer: Provide a high-level session interface that allows creation of AWS service clients without manual credential management.
• Flexibility: Allow configuration of role ARN, session name, desired duration, and region.
• Error Handling & Fallback: Gracefully handle errors (e.g., missing IAM permissions) and fallback to default behavior when needed.

3. Scope

This solution is focused on:

• Creating an auto‑refreshing boto3 session for an assumed IAM role.
• Supporting the automatic refresh of temporary credentials through a refresh callback.
• Incorporating Pydantic v2 models to ensure proper formatting and validation of credentials metadata.

4. Functional Requirements

1. Assume Role Functionality:

   • Must call sts:assume_role to retrieve temporary credentials.
   • Accept a role ARN, session name, and duration as inputs.
   • Optionally use provided base credentials; otherwise, use the default boto3 credential chain.

2. Credential Refresh:

   • Automatically refresh credentials before expiration.
   • Use botocore’s RefreshableCredentials with a refresh callback method.
   • The refresh callback should re‐assume the role with the specified duration.

3. Metadata Validation:

   • Validate the temporary credentials (access key, secret key, session token, expiry) using a Pydantic v2 model.
   • Convert the expiry time (if provided as a datetime) to an ISO‑formatted string.

4. Session Creation:

   • Create and maintain a boto3 session (built on a botocore session) that always uses the latest credentials.
   • Provide a method for creating service clients (e.g., S3, Bedrock, etc.) using the auto‑refreshing session.

5. Non‑Functional Requirements

• Performance: The credential refresh process should introduce minimal overhead.
• Reliability: The solution should handle transient errors from AWS STS gracefully.
• Security: Only use temporary credentials; do not store or log sensitive information.
• Extensibility: The design should allow for future enhancements, such as chaining multiple role assumptions.

6. System Architecture and Components

6.1 AutoRefreshingSession Class

• Responsibilities:

  • Encapsulate the creation and management of a botocore session with auto‑refreshable credentials.
  • Provide a high‑level API to create AWS service clients.

• Key Attributes:

  • role_arn: Target IAM role ARN to assume.
  • session_name: Unique name for the STS session.
  • duration_seconds: Requested duration for temporary credentials.
  • region_name: AWS region for client creation.
  • botocore_session: Underlying botocore session updated with RefreshableCredentials.
  • boto3_session: A boto3 session built on the botocore session.

• Key Methods:

  • refresh_credentials():
    • Calls STS assume_role with DurationSeconds equal to duration_seconds.
    • Retrieves and validates credentials metadata.
    • Updates the botocore session with new RefreshableCredentials.
  • client(service_name, **kwargs):
    • Creates a boto3 client using the auto‑refreshing session.

• Configuration Options:

  • Ability to set the role ARN, session name, duration, and region.
  • Can be configured to always set the max duration for the assumed role.

6.2 CredentialsMetadata Pydantic Model

• Responsibilities:

  • Validate the temporary credentials metadata.
  • Ensure that the expiry_time field is an ISO‑formatted string.

• Fields:

  • access_key: AWS access key ID.
  • secret_key: AWS secret access key.
  • token: Session token.
  • expiry_time: Expiration time (ISO‑formatted).

• Validators:

  • A field validator to convert a datetime to an ISO‑formatted string, ensuring uniform formatting.

7. Interaction Flow

1. Initialization:

   • The user instantiates an AutoRefreshingSession with parameters (role ARN, session name, duration, region).
   • A botocore session is created.

2. Initial Credential Fetch:

   • refresh_credentials() is called.
   • STS assume_role is invoked with the provided parameters.
   • Response metadata is validated using the CredentialsMetadata model.
   • RefreshableCredentials are created and set in the botocore session.

3. Client Creation:

   • The user calls the client() method to create a boto3 client.
   • The client uses the botocore session which automatically refreshes credentials as needed.

4. Auto-Refresh Mechanism:

   • As the temporary credentials approach expiration, botocore automatically triggers the refresh callback (refresh_credentials), updating the session with fresh credentials.

8. Error Handling Strategy

• STS Assume Role Errors:
  • If the assume_role call fails, an exception will be raised. The refresh method should propagate the error for higher-level handling.
• Pydantic Validation:
  • If the credentials metadata does not pass validation, the error is raised, ensuring only valid data is used.
• Fallback Behavior:
  • Optionally, if certain permissions are missing (e.g., for calling iam.get_role), the code logs a warning and falls back to not setting the DurationSeconds parameter (thereby using default expiration).

9. Logging and Monitoring

• Logging:
  • Informational logging at key stages (credential assumption, refresh, and expiration) using Python’s built‑in logging module.
• Monitoring:
  • Monitor logs for repeated refresh failures, which may indicate permission issues or configuration errors.

10. Security Considerations

• Sensitive Data Handling:
  • Do not log sensitive credentials. Only log non‑sensitive metadata (e.g., expiration time).
• Least Privilege:
  • The assumed role must have the minimum permissions required to perform the assume_role call.
• Credential Lifecycle:
  • Temporary credentials are automatically refreshed, reducing the risk of long‑term credential exposure.