Retry HTTP 429 responses as throttling errors

[thundergolfer]

S3-compatible backends like Cloudflare R2 return HTTP 429 for bandwidth throttling instead of S3's native 503 SlowDown response. The CRT retry logic is keyed on AWS_ERROR_S3_SLOW_DOWN (which maps to 503), so try_parse_throttled never matches for 429. The 429 falls through to _ => None in try_parse_generic_error, becomes a terminal ResponseError, and the FUSE mount dies with EIO.

This adds a 429 arm to the try_parse_generic_error match that returns S3RequestError::Throttled(...), enabling the existing exponential backoff retry strategy (up to 10 attempts, 500ms base with full jitter) for these responses. The 429 arm parses the XML response body to extract the actual error code and message, so error metadata accurately reflects the real response (HTTP 429 / ServiceUnavailable) rather than hardcoding 503 / SlowDown.

To support this, S3RequestError::Throttled was changed from a unit variant to Throttled(ClientErrorMetadata). The existing CRT-error path (AWS_ERROR_S3_SLOW_DOWN, i.e. S3's 503 SlowDown) continues to populate the metadata with the same synthetic 503/SlowDown values as before.

Observed in production when a Modal CloudBucketMount backed by R2 failed immediately on the first GetObject with: "You have exceeded your available bandwidth limit." (HTTP 429, <Code>ServiceUnavailable</Code>).

Does this change impact existing behavior?

Yes — HTTP 429 responses that were previously treated as terminal errors are now retried. No other response codes are affected. The Throttled variant is now a tuple variant carrying ClientErrorMetadata, which is a breaking change to the enum shape but does not affect runtime behavior for the existing 503 path (same metadata values are produced).

Does this change need a changelog entry? Does it require a version change?

Not required — this is a Modal fork-only change for R2 compatibility, not an upstream contribution.

Human review checklist

• The 429 arm parses the XML body best-effort (falls back to None for error code/message if parsing fails). Confirm this is acceptable vs. providing a generic fallback message.
• The Throttled variant is now Throttled(ClientErrorMetadata) — any downstream code pattern-matching on Throttled will need updating. Only s3_crt_client.rs matches on it within this repo.
• This was validated with a unit test using a synthetic R2 response, not against a live R2 endpoint.

---

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license and I agree to the terms of the Developer Certificate of Origin (DCO).

Link to Devin session: https://modal.devinenterprise.com/sessions/1742473af31243ae8ce6830f659271eb
Requested by: @thundergolfer

[devin-ai-integration]

🤖 Devin AI Engineer

I'll be helping with this pull request! Here's what you should know:

✅ I will automatically:

• Address comments on this PR. Add '(aside)' to your comment to have me ignore it.
• Look at CI failures and help fix them

Note: I can only respond to comments from users who have write access to this repository.

⚙️ Control Options:

• Disable automatic comment and CI monitoring