Optimize queued_jobs_by_label query

[huydhn]

Summary

/api/clickhouse/queued_jobs_by_label is the second HUD endpoint to hit ClickHouse's 32 GiB total-memory limit (MEMORY_LIMIT_EXCEEDED ... While executing ReplacingSorted). Same root cause as #8088: FINAL on workflow_job over a multi-day window, repeated across two CTEs.

This applies the same optimization pattern as #8088 to queued_jobs_by_label:

• Drop FINAL on workflow_job; replace with a manual ReplacingMergeTree dedup via LIMIT 1 BY id ORDER BY _inserted_at DESC, scoped to the candidate id set.
• Pre-filter workflow_run (status, repo, run_id in candidates) before FINAL so the JOIN's right side is tiny.
• Consolidate the per-CTE FINAL JOINs into a single latest_jobs + latest_workflows pair so the EC2 and ARC paths share one scan instead of two.

Measurements (production ClickHouse, current state)

Scenario	wall	mem	read_rows	read_bytes
OLD	3.3-3.7 s	2.72 GB	7.09 M	4.79 GB
NEW	3.5-3.8 s	0.61-1.00 GB	6.84 M	4.11 GB

≈3-4× peak-memory reduction per query; result rows identical (numeric queue_s drift between runs is just CURRENT_TIMESTAMP() advancing, not a correctness change).

Test plan

• OLD vs NEW return the same (count, machine_type) tuples on prod.
• Peak memory drops from ~2.7 GB to ~1 GB on prod.
• Monitor hud.pytorch.org/api/clickhouse/queued_jobs_by_label after rollout — error rate should drop from non-zero to ~0%.

[vercel]

The latest updates on your projects. Learn more about Vercel for GitHub.

Project	Deployment	Actions	Updated (UTC)
torchci	Ready	Preview	May 16, 2026 6:22am