[OPIK-6941] [BE] [FE] feat: workspace-level span metric aggregation in dashboards#7365
[OPIK-6941] [BE] [FE] feat: workspace-level span metric aggregation in dashboards#7365alexkuzmik wants to merge 5 commits into
Conversation
…n dashboards Add a workspace metrics endpoint (POST /v1/private/workspaces/metrics/spans) that aggregates span metrics (count, token usage, cost) across a set of projects — empty set means all projects in the workspace — with optional provider/model breakdown and configurable interval. The DAO builds the whole series in SQL (groupArray) like the existing workspace cost query; the service stays thin and returns WorkspaceMetricResponse. A dedicated WorkspaceSpanMetricRequest keeps the shared WorkspaceMetricRequest contract untouched. Frontend: the dashboard metric widget's Project field becomes a multi-select (reusing ProjectsSelectBox). One project keeps the existing per-project behavior and full metric set; two or more narrows the metric list to the span trio and routes to the workspace endpoint. The form is otherwise identical to main. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Add multi-project support to the single-metric (stat card) widget, mirroring the time-series widget: the Project field is a multi-select, and selecting two or more projects narrows the metric list to the span totals (count, token usage, cost) served by the workspace endpoint with interval=TOTAL, rendered as a single value. Extract the shared workspace-metric logic into dedicated modules to avoid duplicating flow-specific data across widgets: - lib/dashboard/workspaceMetrics.ts: supported metric names, editor/stat options, isWorkspaceMetric, and project-selection resolution (single vs workspace). - api/projects/useMetricData.ts: picks the per-project vs workspace metrics source, so MetricChartContainer no longer branches inline. The stats-endpoint metric catalog (metrics.ts) and useWorkspaceMetric are kept free of cross-flow constants. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
github-actions
Bot
added
java
⏱️ pre-commit per-hook timing
⏭️ 37 skipped (no matching files changed)
|
|
🔄 Test environment deployment process has started Phase 1: Deploying base version You can monitor the progress here. |
![baz-reviewer[bot]](https://avatars.githubusercontent.com/in/933528?s=60&v=4){kind=small}
| @Override | ||
| public Mono<List<WorkspaceMetricResponse.Result>> getSpanCount(@NonNull WorkspaceSpanMetricRequest request) { | ||
| return template.nonTransaction(connection -> getSpanMetric(request, connection, | ||
| request.hasBreakdown() ? GET_SPAN_COUNT_WITH_BREAKDOWN : GET_SPAN_COUNT, "workspaceSpanCount") | ||
| .flatMapMany(this::rowToDataPoint) | ||
| .collectList()); | ||
| } | ||
|
|
||
| @Override | ||
| public Mono<List<WorkspaceMetricResponse.Result>> getSpanTokenUsage(@NonNull WorkspaceSpanMetricRequest request) { | ||
| return template.nonTransaction(connection -> getSpanMetric(request, connection, | ||
| request.hasBreakdown() ? GET_SPAN_TOKEN_USAGE_WITH_BREAKDOWN : GET_SPAN_TOKEN_USAGE, |
There was a problem hiding this comment.
Duplicate span query builder logic
WorkspaceMetricsDAO.getSpanMetric duplicates the ClickHouse template/binding logic already in ProjectMetricsDAO.getMetric, so changes to bucket, breakdown, or template handling will drift between the two span-metric queries — should we extract the shared builder and parameterize the query differences?
Want Baz to fix this for you? Activate Fixer
There was a problem hiding this comment.
Skipping — the duplication is intentional. Workspace span metrics are the per-project queries with project_id IN (...) in place of project_id = ...; keeping them as exact copies was a deliberate choice for this feature so the two paths stay behaviorally identical (enforced by the parity tests). Extracting a shared cross-DAO builder would couple the workspace and per-project DAOs and is a larger refactor out of scope here — noted as a possible follow-up.
🤖 Reply posted via /address-github-pr-comments
There was a problem hiding this comment.
Thanks, that makes sense — keeping these as intentional copies to preserve DAO parity and avoid broader coupling is a helpful constraint to know. I’ll save this to memory once the PR is merged.
| private void validate(WorkspaceSpanMetricRequest request) { | ||
| if (request.metricType() == null) { | ||
| throw new BadRequestException("'metric_type' must be provided"); | ||
| } | ||
| if (request.interval() == null) { | ||
| throw new BadRequestException("'interval' must be provided"); | ||
| } | ||
| if (!WorkspaceMetricsDAO.SUPPORTED_SPAN_METRICS.contains(request.metricType())) { | ||
| throw new BadRequestException("Unsupported metric type '%s'. Supported: %s" | ||
| .formatted(request.metricType(), WorkspaceMetricsDAO.SUPPORTED_SPAN_METRICS)); | ||
| } | ||
| if (request.hasBreakdown()) { |
There was a problem hiding this comment.
Token-usage breakdown validation untested
SpanMetricsTest doesn't hit the MetricType.SPAN_TOKEN_USAGE + missing subMetric branch in validate, so a future refactor could drop the 400 guard and still pass tests — should we add a dedicated or parameterized case for a WorkspaceSpanMetricRequest with subMetric null or blank and assert the 400 response/message?
Want Baz to fix this for you? Activate Fixer
Other fix methods
Prompt for AI Agents
Before applying, verify this suggestion against the current code. In
apps/opik-backend/src/main/java/com/comet/opik/domain/WorkspaceMetricsService.java
around lines 93-114, the `validate(WorkspaceSpanMetricRequest request)` method adds a
guard: when `metricType` is `MetricType.SPAN_TOKEN_USAGE` and
`request.breakdown().subMetric` is blank, it throws a `BadRequestException` with the
“'sub_metric' is required…” message. Add a regression test in the existing
`SpanMetricsTest` that posts a `WorkspaceSpanMetricRequest` with
`metricType=SPAN_TOKEN_USAGE` and a breakdown where `subMetric` is `null` and/or blank,
and assert the API responds with 400 plus the exact error message (or a stable substring
match). Prefer a parameterized test covering both `null` and `""`/whitespace so this
branch stays covered across refactors.
There was a problem hiding this comment.
Fixed in 694e4fc — added tokenUsageBreakdown_withoutSubMetric_returnsBadRequest to SpanMetricsTest, parameterized over null / empty / blank sub_metric, asserting the endpoint returns 400 for a SPAN_TOKEN_USAGE breakdown with a missing sub_metric. Added the equivalent durationBreakdown_withoutSubMetric_returnsBadRequest guard alongside the new duration metric.
🤖 Reply posted via /address-github-pr-comments
There was a problem hiding this comment.
Commit 694e4fc addressed this comment by adding a parameterized regression test for SPAN_TOKEN_USAGE requests with subMetric null, empty, and blank whitespace. The new tokenUsageBreakdown_withoutSubMetric_returnsBadRequest case asserts the API returns 400, so the missing-subMetric validation branch is now covered.
|
✅ Test environment is now available! To configure additional Environment variables for your environment, run [Deploy Opik AdHoc Environment workflow] (https://github.com/comet-ml/comet-deployment/actions/workflows/deploy_opik_adhoc_env.yaml) Access Information
The deployment has completed successfully and the version has been verified. |
… metrics Extend workspace-level span aggregation to match the per-project endpoint more closely: - Span filters: WorkspaceSpanMetricRequest carries a `filters` list, bound in getSpanMetric via FilterQueryBuilder (span + span-feedback-score strategies), reusing the filter slots already present in the copied SPAN_FILTERED_PREFIX. The time-series and single-metric widgets forward span filters to the workspace endpoint in multi-project mode. - Duration percentiles: SPAN_DURATION joins the supported workspace span metrics, computing quantiles(0.5, 0.9, 0.99) DB-side and fanning them into p50/p90/p99 series (single percentile per group when a provider/model breakdown is set). It is exposed in the time-series widget only, since a percentile distribution has no single-value stat-card representation. Tests: duration parity vs the per-project endpoint, a filtered parity case proving the filter narrows results, and validation that a token-usage or duration breakdown without sub_metric returns 400. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
| <ProjectWidgetFiltersSection | ||
| control={form.control} | ||
| fieldName={isTraceSource ? "traceFilters" : "spanFilters"} | ||
| fieldName={useSpanFilters ? "spanFilters" : "traceFilters"} | ||
| projectId={projectId} | ||
| filterType={isTraceSource ? "trace" : "span"} | ||
| filterType={useSpanFilters ? "span" : "trace"} |
There was a problem hiding this comment.
Span filter errors never clear
In multi-project + trace-source mode, ProjectWidgetFiltersSection uses spanFilters, but the effect still keys currentFilters and filterKey off isTraceSource, so edits don’t change the watched length and clearErrors never runs for spanFilters — formState.errors.spanFilters can stay visible after the field is fixed. Should we key the watcher and clearErrors off useSpanFilters too, including the v2 editor copy?
Want Baz to fix this for you? Activate Fixer
Other fix methods
Prompt for AI Agents
Before applying, verify this suggestion against the current code. In
apps/opik-frontend/src/v1/pages-shared/dashboards/widgets/ProjectStatsCardWidget/ProjectStatsCardEditor.tsx
around lines 348-352 (where `ProjectWidgetFiltersSection` chooses
`fieldName`/`filterType` using `useSpanFilters`), update the related `useEffect`/watcher
logic that derives `currentFilters` and `filterKey` to also use `useSpanFilters` instead
of only `isTraceSource`. Specifically, ensure the watched array length and the
`clearErrors`/cleanup path are keyed to the same field (`spanFilters` vs `traceFilters`)
that the UI renders, so editing span filters reliably removes stale
`formState.errors.spanFilters` after correction. Refactor by introducing a single source
of truth (e.g., compute `activeFieldName` and `activeFilterKey` from `useSpanFilters`)
and use it consistently in both the watcher and error clearing.
| Optional.ofNullable(request.filters()) | ||
| .ifPresent(filters -> { | ||
| FilterQueryBuilder.bind(statement, filters, FilterStrategy.SPAN); | ||
| FilterQueryBuilder.bind(statement, filters, FilterStrategy.SPAN_FEEDBACK_SCORES); | ||
| FilterQueryBuilder.bind(statement, filters, FilterStrategy.SPAN_FEEDBACK_SCORES_IS_EMPTY); |
There was a problem hiding this comment.
Filter strategies duplicated
The filter-strategy list is duplicated in the SQL template and the ClickHouse binding, so a new strategy can be added on one side but not the other, leaving placeholders missing or the payload ignored — should we drive both from a single list/loop so each strategy is added once?
Want Baz to fix this for you? Activate Fixer
Duration across projects is percentile-only (p50/p90/p99); grouping a percentile by provider/model isn't needed. Remove the duration breakdown query and quantile sub_metric handling; the service now rejects a duration request with a breakdown (400), and the widgets hide the group-by control for duration in multi-project mode and never send a breakdown for it. Group by and span filters remain for the other workspace span metrics. Also fixes the filtered-parity test to type the filter list explicitly so it compiles under ECJ. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Python SDK E2E Tests Results (Python 3.14)285 tests 282 ✅ 5m 7s ⏱️ Results for commit 38d5fba. ♻️ This comment has been updated with latest results. |
alexkuzmik
added
test-environment
|
🔄 Test environment deployment process has started Phase 1: Deploying base version You can monitor the progress here. |
|
✅ Test environment is now available! To configure additional Environment variables for your environment, run [Deploy Opik AdHoc Environment workflow] (https://github.com/comet-ml/comet-deployment/actions/workflows/deploy_opik_adhoc_env.yaml) Access Information
The deployment has completed successfully and the version has been verified. |
alexkuzmik
added
test-environment
|
🔄 Test environment deployment process has started Phase 1: Deploying base version You can monitor the progress here. |
|
✅ Test environment is now available! To configure additional Environment variables for your environment, run [Deploy Opik AdHoc Environment workflow] (https://github.com/comet-ml/comet-deployment/actions/workflows/deploy_opik_adhoc_env.yaml) Access Information
The deployment has completed successfully and the version has been verified. |
… only Narrow the workspace span feature to a single metric — span token usage — with provider/model grouping and span filters. Drop span count, span cost, and span duration from workspace aggregation (they remain single-project only): - Backend: SUPPORTED_SPAN_METRICS is just SPAN_TOKEN_USAGE; removed the count/cost/ duration queries and DAO methods, and the service now returns 400 for any other metric type. Grouping and filter wiring are unchanged. - Frontend: the time-series and stat-card metric lists expose only token usage in multi-project mode; removed the now-dead duration-in-workspace guards and the unused SPAN_COST metric-name entry. Tests: SpanMetricsTest now covers token-usage parity, provider breakdown, filtered parity, all-vs-subset, workspace isolation, and rejects the unsupported metrics. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
| // Copied verbatim from ProjectMetricsDAO.SPAN_FILTERED_PREFIX. The only change is the project predicate: | ||
| // the per-project `AND project_id = :project_id` becomes an optional IN-list so the same query can span a set | ||
| // of projects (or, when project_ids is absent, every project in the workspace). workspace_id stays mandatory. | ||
| private static final String SPAN_FILTERED_PREFIX = """ | ||
| WITH feedback_scores_deduped AS ( |
There was a problem hiding this comment.
Span filter SQL duplicated
The SPAN_FILTERED_PREFIX CTE here is copied from ProjectMetricsDAO.SPAN_FILTERED_PREFIX with only the project predicate changed, so any span-filtering fix has to be made twice and the workspaces/projects queries can drift — should we extract the shared SQL and pass project_id = :project_id vs project_id IN :project_ids as a parameter?
Want Baz to fix this for you? Activate Fixer
2 participants
Details
Dashboards could previously chart metrics for a single project only. This adds workspace-level aggregation so a metric widget can span multiple projects, addressing the need to watch total span usage (tokens, cost) across a whole workspace — e.g. to track provider rate limits broken down by model/provider.
POST /v1/private/workspaces/metrics/spansaggregates span metrics (count, token usage, cost) across a set of projects, with optional provider/model/type breakdown. Series are built DB-side withgroupArrayfor parity with the existing per-project queries; the service stays thin.WorkspaceSpanMetricRequestDTO so the sharedWorkspaceMetricRequest(cost/feedback summaries) is left untouched.lib/dashboard/workspaceMetrics.tsandapi/projects/useMetricData.tsso flow-specific constants are not duplicated across widgets or leaked into shared modules.Change checklist
Issues
AI-WATERMARK
AI-WATERMARK: yes
Testing
mvn testforWorkspacesResourceTest— the newSpanMetricsTestnested class covers per-project parity, provider/model breakdown, all-projects vs subset, cross-project isolation, and request validation (10 tests, green).mvn spotless:check— green.npm run lintandnpm run typecheck— green.Documentation
N/A — no user-facing documentation changes in this PR.