Task rate limits

[mbertrand]

What are the relevant tickets?

Closes https://github.com/mitodl/hq/issues/11394

Description (What does it do?)

Adds a rate_limited_task decorator that makes use of redis lock keys to prevent a task from running more than once every n seconds (3600 for get_mit_edx_data, 900 for get_xpro_data, get_oll_data, and get_mitxonline_data).

How can this be tested?

• docker compose up

• Run ./manage.py backpopulate_mitxonline_data

• Try running it again afterward. You should see this in the logs:

  Skipping ratelimit:learning_resources.tasks.get_mitxonline_data: rate-limited (900s)

  And the command should finish quickly:

  Population of MITX Online data finished, took 0.041112 seconds
  Task skipped (rate-limited). See celery logs for details.

• Wait 15 minutes and try again. This time it should process normally:

  Population of MITX Online data finished, took 77.92632 seconds
  Populated 268 resources. See celery logs for details.
  

Additional Context

[github-actions]

OpenAPI Changes

No changes detected

View full changelog

Unexpected changes? Ensure your branch is up-to-date with main (consider rebasing).

[Copilot]

Pull request overview

Adds a reusable task rate-limiting decorator (backed by the configured Redis cache) and applies it to several ETL Celery tasks to prevent repeated invocations within a fixed time window.

Changes:

• Introduces rate_limited_task(wait_time, key=None) in main/decorators.py using an atomic cache.add on the redis cache.
• Adds unit tests covering rate-limiting behavior and key selection.
• Applies the decorator to selected ETL tasks in learning_resources/tasks.py (3600s for MIT edX; 900s for MITxOnline/xPro/OLL).

Reviewed changes

Copilot reviewed 3 out of 3 changed files in this pull request and generated 4 comments.

File	Description
main/decorators.py	Adds the Redis-cache-backed rate limit decorator used to skip frequent task invocations.
main/decorators_test.py	Adds tests for rate-limiting behavior and cache key selection.
learning_resources/tasks.py	Wraps several ETL tasks with the new rate-limiting decorator.