feat: Add GCP V2 Managed Kafka entity definitions for 6 entity types (NR-565785) by sparsi17 · Pull Request #2935 · newrelic/entity-definitions

sparsi17 · 2026-06-11T10:51:09Z

GCP V2 Managed Kafka entity support for 4 entity types (NR-565785).

New entities:

GCPMANAGEDKAFKATOPICPARTITION: consumer lag, partition size, log offsets
GCPMANAGEDKAFKACONNECTOR: task states, record errors, active records
GCPMANAGEDKAFKACONNECTCLUSTER: worker CPU, memory, request rate
GCPMANAGEDKAFKATOPICPARTITIONCONSUMERGROUP: offset lag (only available metric)

All entities include: definition.yml, golden_metrics.yml, summary_metrics.yml, dashboard.json, UI definition.

🤖 Generated with Claude Code

ARB: https://new-relic.atlassian.net/browse/NR-567584

…(NR-565785)

entityBot

beep boop bop.

I have found some errors processing these changes:

Error VALIDATING domainType: 'INFRA-GCPMANAGEDKAFKACLUSTER' on Golden metrics for key 'SELECT rate(sum(gcp.managedkafka.cpu.core_usage_time), 1 minute) / average(gcp.managedkafka.cpu.limit) * 100 FROM Metric WHERE entity.guid IN ('guid') FACET entity.name TIMESERIES' - Exception: Error validating the query.
Error VALIDATING domainType: 'INFRA-GCPMANAGEDKAFKACLUSTER' on Golden metrics for key 'SELECT rate(sum(gcp.managedkafka.cpu.core_usage_time), 1 minute) / average(gcp.managedkafka.cpu.limit) * 100 FROM Metric WHERE entity.guid IN ('guid') FACET entity.name TIMESERIES' - analytics.nrql.parser.DiracParserException: Error at line 1 position 97, unexpected 'rate(sum(gcp.managedkafka.cpu.core_usage_time), 1 minute) / average(gcp.managedkafka.cpu.limit'
Error VALIDATING domainType: 'INFRA-GCPMANAGEDKAFKACLUSTER' on Golden metrics for key 'rate(sum(gcp.managedkafka.cpu.core_usage_time), 1 minute) / average(gcp.managedkafka.cpu.limit) * 100' - Exception: The NRQL validator was not able to parse the selection.
No valid, approved ARB ticket found in the PR description. This PR has to be reviewed by the API Review Board, please include a link to the ticket of that review in the PR description. Check how to start the process here or reach #api-review-board channel in case of doubts

entityBot · 2026-06-14T12:06:31Z

⚠️ Review Required ⚠️

👋 This PR has seen no activity in over 3 days and is still awaiting approval from: @newrelic/BEYOND

Please review or provide an update.

Let's validate those changes.

entityBot

beep boop bop.

I have found some errors processing these changes:

Error VALIDATING domainType: 'INFRA-GCPMANAGEDKAFKACLUSTER' on Golden metrics for key 'SELECT rate(sum(gcp.managedkafka.cpu.core_usage_time), 1 minute) / average(gcp.managedkafka.cpu.limit) * 100 FROM Metric WHERE entity.guid IN ('guid') FACET entity.name TIMESERIES' - Exception: Error validating the query.
Error VALIDATING domainType: 'INFRA-GCPMANAGEDKAFKACLUSTER' on Golden metrics for key 'SELECT rate(sum(gcp.managedkafka.cpu.core_usage_time), 1 minute) / average(gcp.managedkafka.cpu.limit) * 100 FROM Metric WHERE entity.guid IN ('guid') FACET entity.name TIMESERIES' - analytics.nrql.parser.DiracParserException: Error at line 1 position 97, unexpected 'rate(sum(gcp.managedkafka.cpu.core_usage_time), 1 minute) / average(gcp.managedkafka.cpu.limit'
Error VALIDATING domainType: 'INFRA-GCPMANAGEDKAFKACLUSTER' on Golden metrics for key 'rate(sum(gcp.managedkafka.cpu.core_usage_time), 1 minute) / average(gcp.managedkafka.cpu.limit) * 100' - Exception: The NRQL validator was not able to parse the selection.
No valid, approved ARB ticket found in the PR description. This PR has to be reviewed by the API Review Board, please include a link to the ticket of that review in the PR description. Check how to start the process here or reach #api-review-board channel in case of doubts

…CPU golden metric (NR-565785)

…ic (NR-565785)

…afka entities (NR-565785)

…sub-cluster entities (NR-565785) - Topic, TopicPartition, ConsumerGroup: add gcp.cluster (present in resource labels) - Connector: add gcp.connect_cluster (present in resource labels) - Cluster, ConnectCluster: keep gcp.projectId + gcp.region only (top-level, no parent)

entityBot · 2026-06-20T08:10:35Z

⚠️ Review Required ⚠️

👋 This PR has seen no activity in over 3 days and is still awaiting approval from: @newrelic/BEYOND

Please review or provide an update.

entityBot · 2026-06-23T11:22:30Z

⚠️ Review Required ⚠️

👋 This PR has seen no activity in over 3 days and is still awaiting approval from: @newrelic/BEYOND

Please review or provide an update.

…latencies (NR-565785) GCP stores managedkafka.request_latencies with numeric percentile labels ('50', '75', '95', '99', '999') — not 'p99'. - dashboard.json: WHERE metric.percentile = 'p99' → '99' - golden_metrics.yml: add where: "metric.percentile = '99'" to requestLatencies Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>

sparsi17 · 2026-06-24T17:48:39Z

Phase 3a Validation Report — 6 Managed Kafka Entities

Account: 10876283 | Collector: gcp-cloud-monitoring | Date: 2026-06-24

Summary Table

Entity	golden_metrics	dashboard	Overall
GCPMANAGEDKAFKACLUSTER	3/3	6/6	✅
GCPMANAGEDKAFKACONNECTCLUSTER	3/3	4/4	✅
GCPMANAGEDKAFKACONNECTOR	3/3	5/5	⚠️ No active connectors
GCPMANAGEDKAFKATOPIC	3/3	5/5	✅
GCPMANAGEDKAFKATOPICPARTITION	3/3	4/4	✅
GCPMANAGEDKAFKATOPICPARTITIONCONSUMERGROUP	1/1	2/2	⚠️ No consumer group activity

Results by Entity

GCPMANAGEDKAFKACLUSTER

Metric	Status	Result
cpuCoreUsageTime	✅	1.595 vCPU/s
offlinePartitions	⚠️	~0 (healthy cluster)
requestLatencies (WHERE metric.percentile = '99')	✅	335.63 ms
cluster_byte_in_count	✅	367,905 bytes
cluster_byte_out_count	✅	353,326 bytes
cluster_message_in_count	✅	3,654 messages
memory.usage	✅	1.03 GB

GCPMANAGEDKAFKACONNECTCLUSTER

Metric	Status	Result
cpuWorkerCoreUsageTime	✅	1.699 vCPU/s
memoryWorkerUsage	✅	1.994 GB
connectorRequestRate	✅	0.349 req/s

GCPMANAGEDKAFKACONNECTOR

Metric	Status	Result
connectorTasks	⚠️	0 (no active tasks in test account)
connectorTotalErrorRecordCount	⚠️	0 (no errors — healthy)
connectorSinkActiveRecords	⚠️	null (no sink connectors configured)

GCPMANAGEDKAFKATOPIC

Metric	Status	Result
message_in_count	✅	3,846 messages
byte_in_count	✅	399,662 bytes
byte_out_count	✅	417,323 bytes
topic_error_count	⚠️	0 (no errors)
topic_request_count	✅	250,009,866 requests

GCPMANAGEDKAFKATOPICPARTITION

Metric	Status	Result
consumer_lag	⚠️	0
byte_size	✅	6.17 MB
last_offset	✅	169,622
first_offset	✅	92,226
log_segments	✅	1.08

GCPMANAGEDKAFKATOPICPARTITIONCONSUMERGROUP

Metric	Status	Result
offsetLag	⚠️	null (no active consumer groups with committed offsets)

Notes

All queries execute without errors
Bug fixed in this PR (commit 3bf4d368): metric.percentile = 'p99' → '99' — the 'p' prefix is not stored in GCP data
Connector/ConsumerGroup entities show ⚠️ due to no active connectors/consumer groups in test account, not a schema issue
Definitions are ready for merge

… (NR-565785) Moving to separate PRs. Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>

…tities (NR-565785) Add gcp: query blocks to golden_metrics.yml and dashboard.json for KAFKACLUSTER and KAFKATOPIC to support GCP Managed Kafka metrics. KAFKACLUSTER golden metrics added: - gcpCpuUsage: rate(sum(gcp.managedkafka.cpu.core_usage_time), 1 minute) - gcpOfflinePartitions: average(gcp.managedkafka.offline_partitions) - gcpRequestLatency: average(gcp.managedkafka.request_latencies) WHERE percentile='99' KAFKATOPIC golden metrics added: - gcpMessagesIn: sum(gcp.managedkafka.message_in_count) - gcpBytesIn: sum(gcp.managedkafka.byte_in_count) - gcpTopicErrors: sum(gcp.managedkafka.topic_error_count) Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>

… GCP-only entries (NR-565785) Add gcp: query blocks to existing metrics where GCP Managed Kafka has a direct equivalent per GCP Cloud Monitoring docs: KAFKACLUSTER: - offlinePartitions: added gcp.managedkafka.offline_partitions - Removed duplicate gcpOfflinePartitions standalone entry KAFKATOPIC: - bytesInPerSecond: added gcp.managedkafka.byte_in_count - bytesOutPerSecond: added gcp.managedkafka.byte_out_count - messagesInPerSecond: added gcp.managedkafka.message_in_count - Removed duplicate gcpBytesIn/gcpMessagesIn standalone entries - Also updated titles to remove OTel-only prefix No GCP equivalent found (kept as-is): - brokerCount, TopicCount, partitionCount (no GCP managed kafka metric) - consumerGroupLag, underReplicatedPartitions, partitionsWithNonPreferredLeader Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>

Let's validate those changes.

entityBot

beep boop bop.

I have found some errors processing these changes:

Error VALIDATING domainType: 'INFRA-KAFKACLUSTER' on Golden metrics for key 'SELECT average(gcp.managedkafka.offline_partitions) FROM Metric WHERE entity.guid IN ('guid') FACET entity.name TIMESERIES' - Exception: The select clause uses a different value function than the rest.

Let's validate those changes.

entityBot

beep boop bop.

I have found some errors processing these changes:

Error VALIDATING domainType: 'INFRA-KAFKACLUSTER' on Golden metrics for key 'SELECT average(gcp.managedkafka.offline_partitions) FROM Metric WHERE entity.guid IN ('guid') FACET entity.name TIMESERIES' - Exception: The select clause uses a different value function than the rest.

…AFKACLUSTER/KAFKATOPIC (NR-565785) golden_metrics: - KAFKACLUSTER.partitionCount: add gcp block (gcp.managedkafka.partitions = 138, confirmed live) - KAFKACLUSTER: offlinePartitions already has gcp block summary_metrics (KAFKACLUSTER): - Add providerAccountName + gcpProjectId tags - Add gcpCpuUsage + gcpRequestLatency references - Clean up existing titles summary_metrics (KAFKATOPIC): - Add providerAccountName + gcpProjectId tags - Add messagesInPerSecond + gcpTopicErrors references - Clean up titles (remove OTel/NR source prefixes) Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>

Let's validate those changes.

entityBot

beep boop bop.

I have found some errors processing these changes:

Error VALIDATING domainType: 'INFRA-KAFKACLUSTER' on Golden metrics for key 'SELECT sum(gcp.managedkafka.partitions) FROM Metric WHERE entity.guid IN ('guid') FACET entity.name TIMESERIES' - Exception: The select clause uses a different value function than the rest.
Error VALIDATING domainType: 'INFRA-KAFKACLUSTER' on Golden metrics for key 'SELECT average(gcp.managedkafka.offline_partitions) FROM Metric WHERE entity.guid IN ('guid') FACET entity.name TIMESERIES' - Exception: The select clause uses a different value function than the rest.

…ks (NR-565785) Golden metrics validator requires same aggregation function across all query blocks: - partitionCount: sum() → latest() (matches newRelic: and opentelemetry: blocks) - offlinePartitions: average() → max() (matches newRelic: and opentelemetry: blocks) Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>

…-kafka-existing-entities (NR-565785) Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>

feat: Add GCP V2 Managed Kafka entity definitions for 6 entity types …

85b9c64

…(NR-565785)

entityBot added the ARB:pending label Jun 11, 2026

entityBot previously requested changes Jun 11, 2026

View reviewed changes

entityBot requested a review from a team June 11, 2026 10:51

entityBot previously requested changes Jun 16, 2026

View reviewed changes

sparsi17 added 2 commits June 16, 2026 16:06

fix: Remove invalid division aggregation from GCPMANAGEDKAFKACLUSTER …

145ddb3

…CPU golden metric (NR-565785)

fix: Restore original title and unit for cpuCoreUsageTime golden metr…

a56966d

…ic (NR-565785)

sparsi17 force-pushed the feat/gcp-managedkafka-entities branch from abe67d9 to a56966d Compare June 16, 2026 10:41

entityBot added ARB:approved and removed ARB:pending labels Jun 16, 2026

entityBot previously approved these changes Jun 16, 2026

View reviewed changes

fix: Use standard gcp.projectId + gcp.region goldenTags for Managed K…

991b2f7

…afka entities (NR-565785)

sparsi17 dismissed entityBot’s stale review via 991b2f7 June 16, 2026 13:04

entityBot previously approved these changes Jun 16, 2026

View reviewed changes

sparsi17 dismissed entityBot’s stale review via 619ffca June 16, 2026 13:09

entityBot previously approved these changes Jun 16, 2026

View reviewed changes

fix: Use camelCase gcp.connectCluster goldenTag (NR-565785)

8aaeed4

sparsi17 dismissed entityBot’s stale review via 8aaeed4 June 16, 2026 13:12

entityBot previously approved these changes Jun 16, 2026

View reviewed changes

sparsi17 dismissed entityBot’s stale review via 3bf4d36 June 24, 2026 17:44

entityBot previously approved these changes Jun 24, 2026

View reviewed changes

Merge branch 'main' into feat/gcp-managedkafka-entities

f2ff9b1

entityBot previously approved these changes Jun 25, 2026

View reviewed changes

chore: Remove GCPMANAGEDKAFKACLUSTER and GCPMANAGEDKAFKATOPIC from PR…

f248b30

… (NR-565785) Moving to separate PRs. Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>

sparsi17 dismissed entityBot’s stale review via f248b30 June 25, 2026 12:54

entityBot previously approved these changes Jun 25, 2026

View reviewed changes

sparsi17 dismissed entityBot’s stale review via 676c6c6 June 26, 2026 12:18

entityBot previously approved these changes Jun 26, 2026

View reviewed changes

entityBot requested a review from a team June 26, 2026 12:18

sparsi17 dismissed entityBot’s stale review via 29c9cd9 June 26, 2026 12:26

entityBot previously requested changes Jun 26, 2026

View reviewed changes

Merge branch 'main' into feat/gcp-managedkafka-entities

30fc752

entityBot previously requested changes Jun 26, 2026

View reviewed changes

entityBot requested changes Jun 26, 2026

View reviewed changes

entityBot previously approved these changes Jun 26, 2026

View reviewed changes

chore: Remove KAFKACLUSTER and KAFKATOPIC changes — moved to feat/gcp…

be7c40f

…-kafka-existing-entities (NR-565785) Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>

sparsi17 dismissed entityBot’s stale review via be7c40f June 26, 2026 13:08

entityBot approved these changes Jun 26, 2026

View reviewed changes

entityBot removed the request for review from a team June 26, 2026 13:08

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

feat: Add GCP V2 Managed Kafka entity definitions for 6 entity types (NR-565785)#2935

feat: Add GCP V2 Managed Kafka entity definitions for 6 entity types (NR-565785)#2935
sparsi17 wants to merge 15 commits into
newrelic:mainfrom
sparsi17:feat/gcp-managedkafka-entities

sparsi17 commented Jun 11, 2026 •

edited

Loading

Uh oh!

entityBot left a comment

Uh oh!

entityBot commented Jun 14, 2026

Uh oh!

entityBot left a comment

Uh oh!

entityBot commented Jun 20, 2026

Uh oh!

entityBot commented Jun 23, 2026

Uh oh!

sparsi17 commented Jun 24, 2026

Uh oh!

entityBot left a comment

Uh oh!

entityBot left a comment

Uh oh!

entityBot left a comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Uh oh!

Conversation

sparsi17 commented Jun 11, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

entityBot left a comment

Choose a reason for hiding this comment

Uh oh!

entityBot commented Jun 14, 2026

Uh oh!

entityBot left a comment

Choose a reason for hiding this comment

Uh oh!

entityBot commented Jun 20, 2026

Uh oh!

entityBot commented Jun 23, 2026

Uh oh!

sparsi17 commented Jun 24, 2026

Phase 3a Validation Report — 6 Managed Kafka Entities

Summary Table

Results by Entity

GCPMANAGEDKAFKACLUSTER

GCPMANAGEDKAFKACONNECTCLUSTER

GCPMANAGEDKAFKACONNECTOR

GCPMANAGEDKAFKATOPIC

GCPMANAGEDKAFKATOPICPARTITION

GCPMANAGEDKAFKATOPICPARTITIONCONSUMERGROUP

Notes

Uh oh!

entityBot left a comment

Choose a reason for hiding this comment

Uh oh!

entityBot left a comment

Choose a reason for hiding this comment

Uh oh!

entityBot left a comment

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

sparsi17 commented Jun 11, 2026 •

edited

Loading