feat: Add GCP Managed Kafka metrics to existing KAFKACLUSTER and KAFKATOPIC entities (NR-565785)#2981
Open
sparsi17 wants to merge 4 commits into
Open
feat: Add GCP Managed Kafka metrics to existing KAFKACLUSTER and KAFKATOPIC entities (NR-565785)#2981sparsi17 wants to merge 4 commits into
sparsi17 wants to merge 4 commits into
Conversation
…tities (NR-565785) Add gcp: query blocks to golden_metrics.yml and dashboard.json for KAFKACLUSTER and KAFKATOPIC to support GCP Managed Kafka metrics. KAFKACLUSTER golden metrics added: - gcpCpuUsage: rate(sum(gcp.managedkafka.cpu.core_usage_time), 1 minute) - gcpOfflinePartitions: average(gcp.managedkafka.offline_partitions) - gcpRequestLatency: average(gcp.managedkafka.request_latencies) WHERE percentile='99' KAFKATOPIC golden metrics added: - gcpMessagesIn: sum(gcp.managedkafka.message_in_count) - gcpBytesIn: sum(gcp.managedkafka.byte_in_count) - gcpTopicErrors: sum(gcp.managedkafka.topic_error_count) Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>
… GCP-only entries (NR-565785) Add gcp: query blocks to existing metrics where GCP Managed Kafka has a direct equivalent per GCP Cloud Monitoring docs: KAFKACLUSTER: - offlinePartitions: added gcp.managedkafka.offline_partitions - Removed duplicate gcpOfflinePartitions standalone entry KAFKATOPIC: - bytesInPerSecond: added gcp.managedkafka.byte_in_count - bytesOutPerSecond: added gcp.managedkafka.byte_out_count - messagesInPerSecond: added gcp.managedkafka.message_in_count - Removed duplicate gcpBytesIn/gcpMessagesIn standalone entries - Also updated titles to remove OTel-only prefix No GCP equivalent found (kept as-is): - brokerCount, TopicCount, partitionCount (no GCP managed kafka metric) - consumerGroupLag, underReplicatedPartitions, partitionsWithNonPreferredLeader Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>
…AFKACLUSTER/KAFKATOPIC (NR-565785) golden_metrics: - KAFKACLUSTER.partitionCount: add gcp block (gcp.managedkafka.partitions = 138, confirmed live) - KAFKACLUSTER: offlinePartitions already has gcp block summary_metrics (KAFKACLUSTER): - Add providerAccountName + gcpProjectId tags - Add gcpCpuUsage + gcpRequestLatency references - Clean up existing titles summary_metrics (KAFKATOPIC): - Add providerAccountName + gcpProjectId tags - Add messagesInPerSecond + gcpTopicErrors references - Clean up titles (remove OTel/NR source prefixes) Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>
…ks (NR-565785) Golden metrics validator requires same aggregation function across all query blocks: - partitionCount: sum() → latest() (matches newRelic: and opentelemetry: blocks) - offlinePartitions: average() → max() (matches newRelic: and opentelemetry: blocks) Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>
entityBot
requested changes
Jun 26, 2026
entityBot
left a comment
entityBot
left a comment
Collaborator
There was a problem hiding this comment.
beep boop bop.
I have found some errors processing these changes:
- No valid, approved ARB ticket found in the PR description. This PR has to be reviewed by the API Review Board, please include a link to the ticket of that review in the PR description. Check how to start the process here or reach
#api-review-boardchannel in case of doubts
| @@ -1,13 +1,13 @@ | |||
| { | |||
| "name": "GCP Cloud Storage GCS Bucket", | |||
| "name": "GCP Managed Kafka Cluster", | |||
Contributor
There was a problem hiding this comment.
Where is this dashboard.json is referenced? generally dashboard.json should reference in definition.yml. And always we should first make changes in stg.yml files and get them merged and test properly on staging. .yml files are for production. If stg files doesn't exist then .yml will be for all.
| { | ||
| "accountId": 0, | ||
| "query": "SELECT sum(`gcp.kubernetes.pod.network.received_bytes_count`) AS 'Bytes Received' FROM Metric WHERE `entity.guid` = '{{entity.id}}' TIMESERIES AUTO" | ||
| "query": "FROM Metric SELECT sum(`gcp.managedkafka.message_in_count`) AS 'Messages In' WHERE `entity.guid` = '{{entity.guid}}' TIMESERIES AUTO" |
Contributor
There was a problem hiding this comment.
We don't need to keep entity.guid filter in entity platform dashboard files.
Contributor
|
@sparsi17 Can you please add test cases for the gcp data. Please refer existing tests folder in each entity type of kafka. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adds GCP Managed Kafka as a new
gcp:data source to the existingKAFKACLUSTERandKAFKATOPICentity types.KAFKACLUSTER: partitionCount + offlinePartitions get
gcp:blocks; new gcpCpuUsage + gcpRequestLatency golden metrics; updated summary_metrics + dashboard.jsonKAFKATOPIC: bytesInPerSecond + bytesOutPerSecond + messagesInPerSecond get
gcp:blocks; new gcpTopicErrors; updated summary_metrics + dashboard.jsonTest plan
collector.name = 'gcp-cloud-monitoring'🤖 Generated with Claude Code
ARB: https://new-relic.atlassian.net/browse/NR-567584