Skip to content

perf: precompute org page contributors leaderboard #4501

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
epipav merged 2 commits into from
Aug 25, 2026
+106 −35
Merged

perf: precompute org page contributors leaderboard #4501

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
0686845
perf: precompute org page contributors leaderboard
epipav Aug 24, 2026
56d7404
Merge branch 'main' into perf/org-page-contributors-precompute
epipav Aug 25, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
16 changes: 16 additions & 0 deletions services/libs/tinybird/datasources/org_page_contributors_copy_ds.datasource
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,16 @@
DESCRIPTION >
Precomputed all-time contribution counts per (organizationId, memberId) for the org
page contributors leaderboard. Rebuilt nightly by org_page_contributors_copy_pipe.
One row per (organizationId, memberId) pair (~2M rows total). Used by
org_page_contributors.pipe for cheap request-time lookups instead of scanning the
full activityRelations cleaned bucket union (~744M rows) per request.

SCHEMA >
`organizationId` String,
`memberId` String,
`contributionCount` UInt64,
`computedAt` DateTime

ENGINE ReplacingMergeTree
ENGINE_SORTING_KEY organizationId, memberId
ENGINE_VER computedAt
108 changes: 73 additions & 35 deletions services/libs/tinybird/pipes/org_page_contributors.pipe
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,6 +1,10 @@
DESCRIPTION >
Top contributors for a given organization leaderboard.
Returns members sorted by contribution count within the specified date range.
When no startDate/endDate is given (the dominant traffic shape), reads precomputed
per-(organizationId, memberId) counts from org_page_contributors_copy_ds (keyed
lookup) instead of scanning the full activityRelations cleaned bucket union
(~744M rows) per request. Date-filtered requests fall back to the full scan.

TAGS "Organization page"

Expand All @@ -14,49 +18,83 @@ SQL >
NODE org_page_contributors_activity_aggregates
SQL >
%
{% set use_precomputed = 1 %}
{% if defined(startDate) %} {% set use_precomputed = 0 %} {% end %}
{% if defined(endDate) %} {% set use_precomputed = 0 %} {% end %}
{% if Boolean(count, false) %}
SELECT count(distinct memberId)
FROM activityRelations_deduplicated_cleaned_bucket_union
WHERE
organizationId = (SELECT id FROM org_slug_lookup)
{% if defined(startDate) %}
AND timestamp
>= {{ DateTime(startDate, description="Filter activity timestamp after") }}
{% end %}
{% if defined(endDate) %}
AND timestamp < {{ DateTime(endDate, description="Filter activity timestamp before") }}
{% end %}
{% if use_precomputed == 1 %}
SELECT count()
FROM org_page_contributors_copy_ds
WHERE organizationId = (SELECT id FROM org_slug_lookup)
Comment on lines +25 to +28
{% else %}
SELECT count(distinct memberId)
FROM activityRelations_deduplicated_cleaned_bucket_union
WHERE
organizationId = (SELECT id FROM org_slug_lookup)
{% if defined(startDate) %}
AND timestamp
>= {{ DateTime(startDate, description="Filter activity timestamp after") }}
{% end %}
{% if defined(endDate) %}
AND timestamp
< {{ DateTime(endDate, description="Filter activity timestamp before") }}
{% end %}
{% end %}
{% else %}
SELECT
memberId,
count() as "contributionCount",
ROUND(COUNT(*) * 100.0 / SUM(COUNT(*)) OVER (), 2) as "contributionPercentage"
FROM activityRelations_deduplicated_cleaned_bucket_union
WHERE
organizationId = (SELECT id FROM org_slug_lookup)
{% if defined(startDate) %}
AND timestamp
>= {{ DateTime(startDate, description="Filter activity timestamp after") }}
{% end %}
{% if defined(endDate) %}
AND timestamp < {{ DateTime(endDate, description="Filter activity timestamp before") }}
{% end %}
GROUP BY memberId
ORDER BY contributionCount DESC, memberId DESC
LIMIT {{ Int32(limit, 10) }}
OFFSET {{ Int32(offset, 0) }}
{% if use_precomputed == 1 %}
SELECT
memberId,
contributionCount,
ROUND(
contributionCount * 100.0 / SUM(contributionCount) OVER (), 2
) as "contributionPercentage"
FROM org_page_contributors_copy_ds
WHERE organizationId = (SELECT id FROM org_slug_lookup)
ORDER BY contributionCount DESC, memberId DESC
LIMIT {{ Int32(limit, 10) }}
OFFSET {{ Int32(offset, 0) }}
{% else %}
SELECT
memberId,
count() as "contributionCount",
ROUND(COUNT(*) * 100.0 / SUM(COUNT(*)) OVER (), 2) as "contributionPercentage"
FROM activityRelations_deduplicated_cleaned_bucket_union
WHERE
organizationId = (SELECT id FROM org_slug_lookup)
{% if defined(startDate) %}
AND timestamp
>= {{ DateTime(startDate, description="Filter activity timestamp after") }}
{% end %}
{% if defined(endDate) %}
AND timestamp
< {{ DateTime(endDate, description="Filter activity timestamp before") }}
{% end %}
GROUP BY memberId
ORDER BY contributionCount DESC, memberId DESC
LIMIT {{ Int32(limit, 10) }}
OFFSET {{ Int32(offset, 0) }}
{% end %}
{% end %}

NODE org_page_contributors_leaderboard
SQL >
%
{% set use_precomputed = 1 %}
{% if defined(startDate) %} {% set use_precomputed = 0 %} {% end %}
{% if defined(endDate) %} {% set use_precomputed = 0 %} {% end %}
{% if Boolean(count, false) %}
SELECT count(distinct memberId) as count
FROM activityRelations_deduplicated_cleaned_bucket_union
WHERE
organizationId = (SELECT id FROM org_slug_lookup)
{% if defined(startDate) %} AND timestamp >= {{ DateTime(startDate) }} {% end %}
{% if defined(endDate) %} AND timestamp < {{ DateTime(endDate) }} {% end %}
{% if use_precomputed == 1 %}
SELECT count() as count
FROM org_page_contributors_copy_ds
WHERE organizationId = (SELECT id FROM org_slug_lookup)
{% else %}
SELECT count(distinct memberId) as count
FROM activityRelations_deduplicated_cleaned_bucket_union
WHERE
organizationId = (SELECT id FROM org_slug_lookup)
{% if defined(startDate) %} AND timestamp >= {{ DateTime(startDate) }} {% end %}
{% if defined(endDate) %} AND timestamp < {{ DateTime(endDate) }} {% end %}
{% end %}
{% else %}
SELECT
m.id,
Expand Down
17 changes: 17 additions & 0 deletions services/libs/tinybird/pipes/org_page_contributors_copy_pipe.pipe
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,17 @@
DESCRIPTION >
Nightly copy pipe that precomputes all-time contribution counts per
(organizationId, memberId) for the org page contributors leaderboard.

TAGS "Organization page"

NODE org_page_contributors_copy_pipe_data
SQL >
SELECT organizationId, memberId, count() AS contributionCount, now() AS computedAt
FROM activityRelations_deduplicated_cleaned_bucket_union
WHERE organizationId != ''
GROUP BY organizationId, memberId

@cursor cursor Bot Aug 25, 2026

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Contributors include stars and forks

High Severity

The new COPY aggregates every activity row per (organizationId, memberId) with no activityTypes / code-contribution filter. Stargazers, forkers, and similar engagement actors become “contributors,” which can inflate leaderboard counts by an order of magnitude. Sibling org-page COPY pipes (org_page_kpis, org_page_contributors_timeseries, org_page_projects) already restrict to code contributions.

Fix in Cursor Fix in Web

Triggered by learned rule: Tinybird pipes counting contributors must filter by activityTypes

Reviewed by Cursor Bugbot for commit 56d7404. Configure here.

@cursor cursor Bot Aug 25, 2026

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Empty memberId counted as contributor

Medium Severity

The COPY filters organizationId != '' but not memberId != '', so unresolved/empty member IDs become one row per org. The new precomputed count() paths then count that row as a real contributor, while the leaderboard INNER JOIN to members_sorted drops it—so count mode and the listed leaderboard disagree by one.

Additional Locations (2)
Fix in Cursor Fix in Web

Triggered by learned rule: Tinybird pipes with count(DISTINCT memberId/organizationId) must filter empty string values

Reviewed by Cursor Bugbot for commit 56d7404. Configure here.


TYPE COPY
TARGET_DATASOURCE org_page_contributors_copy_ds
COPY_MODE replace
COPY_SCHEDULE 50 3 * * *
Comment thread
epipav marked this conversation as resolved.
Loading