Skip to content

Add celery config to adjust retry settings#98

Merged
audiodude merged 1 commit intomainfrom
celery-redis
May 1, 2025
Merged

Add celery config to adjust retry settings#98
audiodude merged 1 commit intomainfrom
celery-redis

Conversation

@audiodude
Copy link
Copy Markdown
Owner

I've gotten multiple user reports now that previews have "hung" and not completed. This is definitely an issue with the Celery worker.

Looking in the worker logs, I see:

redis.exceptions.ConnectionError: Error while reading from fly-rainfall-redis.upstash.io:6379 : (104, 'Connection reset by peer')

It seems like it's losing the connection to the Upstash Redis. In this PR, I've attempted to stop using a worker pool (so connections don't get stale), use infinite retries, and retry on startup.

Hopefully this will fix the issue.

@codecov-commenter
Copy link
Copy Markdown

codecov-commenter commented May 1, 2025

⚠️ Please install the 'codecov app svg image' to ensure uploads and comments are reliably processed by Codecov.

Codecov Report

Attention: Patch coverage is 25.00000% with 6 lines in your changes missing coverage. Please review.

Project coverage is 96.59%. Comparing base (90fade6) to head (6f5da35).

Files with missing lines Patch % Lines
rainfall/celeryconfig.py 0.00% 6 Missing ⚠️

❗ Your organization needs to install the Codecov GitHub app to enable full functionality.

Additional details and impacted files
@@            Coverage Diff             @@
##             main      #98      +/-   ##
==========================================
- Coverage   96.78%   96.59%   -0.20%     
==========================================
  Files          37       38       +1     
  Lines        2958     2965       +7     
==========================================
+ Hits         2863     2864       +1     
- Misses         95      101       +6     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@audiodude audiodude merged commit f8ed2b0 into main May 1, 2025
3 checks passed
@audiodude audiodude deleted the celery-redis branch May 3, 2025 17:15
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants