feat(agent): publish pod deletion cost from in-flight turns - #3144
feat(agent): publish pod deletion cost from in-flight turns#3144daryllimyt wants to merge 1 commit into
Conversation
Kubernetes ReplicaSet scale-down ranks pods by readiness and age, not load, so KEDA can delete an agent-executor pod with ten in-flight 30-minute turns while an idle pod survives. Publish the in-flight run_agent_activity count to the pod's controller.kubernetes.io/pod-deletion-cost annotation so scale-down prefers the emptiest pod. Best-effort by design: PATCHes the pod via the in-cluster API with the service-account token, updates only when the count changes, and after three consecutive failures (e.g. missing RBAC) disables itself for the process lifetime with a single warning.
daryllimyt
added
enhancement
|
✅ No security or compliance issues detected. Reviewed everything up to e61e453. Security Overview
Detected Code Changes
|
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: e61e453e3e
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| if not self._enabled or self._count == count: | ||
| self._publishing = False | ||
| return |
There was a problem hiding this comment.
When the PATCH for a stable count fails transiently, published remains false but _count == count still stops the publishing loop. With the default single concurrent activity, a failed 0→1 update is therefore never retried until that long-running turn ends, leaving the pod advertised as idle and preferred for scale-down for the entire turn. Retry unsuccessful publications with bounded backoff instead of treating an unchanged count as successfully settled.
Useful? React with 👍 / 👎.
| # Lazy singleton - no lifespan required. | ||
| _pod_deletion_cost_publisher: PodDeletionCostPublisher | None = None |
There was a problem hiding this comment.
When this container process restarts inside an existing pod after publishing a positive cost, Kubernetes retains that pod annotation, but the lazy singleton starts its local count at zero without publishing it. If the restarted worker remains idle, the stale positive cost persists indefinitely and can cause ReplicaSet scale-down to delete a genuinely busy lower-cost pod instead; initialize the publisher during worker startup and publish zero before accepting activities.
Useful? React with 👍 / 👎.
1 participant
Why
Kubernetes ReplicaSet scale-down picks pods by its default ranking (unready first, then youngest), not by load. With KEDA scaling the agent-executor on queue depth, a scale-down can delete the pod running ten in-flight 30-minute agent turns while an idle pod survives — and pod termination is irreversible, so those turns spend up to 31 minutes draining for nothing.
What
The worker now publishes its in-flight
run_agent_activitycount to its own pod'scontroller.kubernetes.io/pod-deletion-costannotation, so the ReplicaSet always prefers deleting the emptiest pod.tracecat/agent/executor/deletion_cost.py: best-effort publisher. DirecthttpxPATCH (application/merge-patch+json) against the in-cluster API — nokubernetesdependency. Service-account token re-read per request (projected tokens rotate), CA-verified TLS, 5s timeout.TRACECAT__AGENT_EXECUTOR_POD_DELETION_COST_ENABLED(default true),KUBERNETES_SERVICE_HOSTset, SA token file present, andTRACECAT__K8S_POD_NAME/TRACECAT__K8S_POD_NAMESPACEprovided via the downward API. Anywhere else (compose, dev, Fargate) every call is a no-op.run_agent_activityincrements on entry and decrements infinally. Probe activities are not counted (millisecond noise).Chart counterpart (downward-API env + namespaced Role/RoleBinding with
get/patchon pods): TracecatHQ/k8s#77. Without it this code self-disables harmlessly.Testing
test_agent_activities.py.ruff,basedpyright --warnings, and the touched test files all clean (64 passed).Summary by cubic
Publish the agent-executor pod’s deletion cost based on its in-flight turns so scale-down prefers the emptiest pod and avoids draining long-running work. This reduces wasted time when KEDA scales down pods.
New Features
run_agent_activitycount to the pod’scontroller.kubernetes.io/pod-deletion-costviahttpxPATCH (CA-verified TLS, 5s timeout). Updates only when the count changes and coalesces concurrent updates.run_agent_activitynow increments on start and decrements in finally; probe activities are ignored.Migration
TRACECAT__K8S_POD_NAMEandTRACECAT__K8S_POD_NAMESPACEvia the downward API, ensureKUBERNETES_SERVICE_HOSTand the service-account token are present, and grant the SAget/patchonpods. Feature flagTRACECAT__AGENT_EXECUTOR_POD_DELETION_COST_ENABLEDdefaults to true; otherwise this is a no-op.Written for commit e61e453. Summary will update on new commits.