Skip to content

optimise use of inv queries; tighten hpa watch predicate - #190

Merged
lukebond merged 2 commits into
mainfrom
luke/hpa-predicate-and-optimise-query
Aug 14, 2026
Merged

optimise use of inv queries; tighten hpa watch predicate#190
lukebond merged 2 commits into
mainfrom
luke/hpa-predicate-and-optimise-query

Conversation

@lukebond

Copy link
Copy Markdown
Contributor

we have observed in production when the HPA feature is used that excessive invocation status query load is placed on the restate server. on investigation, two issues were turned up, that this PR addresses:

  1. the owned HPA resources are watched without a generation predicate, meaning they will trigger a reconcile whenever they change, even a status change that we don't care about. HPA's by default reconcile every 15s and we were seeing these trigger a reconcile of the RestateDeployment CR every time
  2. the query to fetch invocation statuses, which ultimately scans the sys_invocation_status table, was being executed unnecessarily on each one of these reconciles. i've optimised this to only be called when necessary

together these changes mean that the expensive query runs far less often. in the common case it's currently called every 15s and that has been removed. it still re-reconciles after 30s on error, but that should be exceptional.

i'll explain the optimisation, because it might not be obvious from the diff. the expensive /query was being called from within list_deployments() and passed to plan_registration and cleanup_old_replicasets(). the former usage does not require the full query unless there is a version to register or promote, or a foreign takeover; in the simple case of a single version we can get the info from a cheaper source: GET /services. in the case of cleanup_old_replicasets, the invocation counts are needed, so the costly query needs to run. my optimisation is to avoid calling list_deployments in the simple case.

note that there will still be scenarios where the query is being called a lot, and if we find that it's an issue we can consider some of the other options that have been explored, such as rate limiting and caching. the fix in this PR cuts out the call entirely when it is not necessary, so is the right place to start. given the query is currently firing every 15s, i expect this PR to make a big difference.

@tillrohrmann tillrohrmann left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for implementing this optimization @lukebond. LGTM. +1 for merging :-)

@lukebond
lukebond merged commit bb0f555 into main Aug 14, 2026
8 checks passed
@github-actions github-actions Bot locked and limited conversation to collaborators Aug 14, 2026
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants