Skip to content

fix(lambda): rate-limit retry warn log to avoid log storms#6318

Merged
trinity-1686a merged 6 commits intomainfrom
fmassot/rate-limit-lambda-retry-warn
May 5, 2026
Merged

fix(lambda): rate-limit retry warn log to avoid log storms#6318
trinity-1686a merged 6 commits intomainfrom
fmassot/rate-limit-lambda-retry-warn

Conversation

@fmassot
Copy link
Copy Markdown
Collaborator

@fmassot fmassot commented Apr 19, 2026

Summary

  • Lambda invocation retries emit a WARN log on every attempt. Under heavy rate-limiting this floods customer logs, 700k+ logs was observed.
  • Switch to quickwit_common::rate_limited_warn!(limit_per_min = 10, ...) so at most 10 warnings are emitted per minute regardless of retry volume.

Test plan

  • cargo check -p quickwit-lambda-client passes
  • Manual: trigger Lambda rate-limiting and confirm log volume is bounded

🤖 Generated with Claude Code

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
@fmassot fmassot requested a review from trinity-1686a April 19, 2026 17:22
fmassot and others added 3 commits April 21, 2026 17:22
When the rate limiter opens a new window after suppressing messages,
it now emits a preceding log at the same level:
"suppressed N similar log messages in the last minute"

Closes #6322.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
… enum

The three possible outcomes (log normally, log after suppression, suppress)
are now distinct enum variants, removing the implicit invariant that the
skipped count is only meaningful when the bool is true.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
@fmassot fmassot mentioned this pull request Apr 26, 2026
11 tasks
@fmassot fmassot requested a review from trinity-1686a May 4, 2026 19:01
@trinity-1686a trinity-1686a enabled auto-merge (squash) May 5, 2026 07:50
@trinity-1686a trinity-1686a merged commit 374839d into main May 5, 2026
9 checks passed
@trinity-1686a trinity-1686a deleted the fmassot/rate-limit-lambda-retry-warn branch May 5, 2026 08:47
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants