fix(aws): omit static key/secret/token for RefreshableCredentials in …#4004
Open
tderk wants to merge 2 commits into
Open
fix(aws): omit static key/secret/token for RefreshableCredentials in …#4004tderk wants to merge 2 commits into
tderk wants to merge 2 commits into
Conversation
…to_s3fs_credentials When the underlying boto3 default chain returns RefreshableCredentials (ECS task role, EKS IRSA, EC2 instance profile, assumed role, SSO), extracting access_key/secret_key/session_token into plain strings and passing them as static s3fs kwargs freezes the credentials at pipeline init time. After the provider rotates the temporary token mid-process, dlt keeps sending the stale snapshot and S3 PutObject dies with ExpiredToken on long-running writes. Detect the RefreshableCredentials case and omit key/secret/token so s3fs falls back to its own aiobotocore default chain, which honors the same provider and refreshes transparently before expiry. Static credentials (env vars, explicit IAM user keys) are passed through unchanged. Closes dlt-hub#4003
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
…to_s3fs_credentials
When the underlying boto3 default chain returns RefreshableCredentials (ECS task role, EKS IRSA, EC2 instance profile, assumed role, SSO), extracting access_key/secret_key/session_token into plain strings and passing them as static s3fs kwargs freezes the credentials at pipeline init time. After the provider rotates the temporary token mid-process, dlt keeps sending the stale snapshot and S3 PutObject dies with ExpiredToken on long-running writes.
Detect the RefreshableCredentials case and omit key/secret/token so s3fs falls back to its own aiobotocore default chain, which honors the same provider and refreshes transparently before expiry.
Static credentials (env vars, explicit IAM user keys) are passed through unchanged.
Closes #4003
Description
Related Issues
Additional Context