From 72f6df33fc690b1c6be8fd05f960c7e7ece54ab6 Mon Sep 17 00:00:00 2001 From: Claude Date: Sun, 26 Apr 2026 20:54:11 +0000 Subject: [PATCH 1/5] fix(profiler): N+1 / missing-index regression on /tables/.../columns?fields=profile (#3488) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Root cause ---------- The 1.9.9 migration introduced two separate index regressions on `profiler_data_time_series`: 1. **PostgreSQL**: `schemaChanges.sql` explicitly dropped the unique constraint `profiler_data_time_series_unique_hash_extension_ts` (entityFQNHash, extension, operation, timestamp) to allow altering the generated `operation` column expression, but never recreated it. After the migration the table kept only the `(extension, timestamp)` index, which is useless for queries filtering by `entityFQNHash`. 2. **MySQL/both**: `postDataMigrationSQLScript.sql` created temporary indexes (idx_pdts_entityFQNHash, idx_pdts_composite, etc.) for its bulk UPDATE pass and then dropped **all** of them, including the only index covering `entityFQNHash`. The batch query issued by `getLatestExtensionsBatch()` when `fields=profile` is requested: SELECT entityFQNHash, MAX(timestamp) FROM profiler_data_time_series WHERE entityFQNHash IN (...N hashes...) AND extension = 'table.columnProfile' GROUP BY entityFQNHash required an `(entityFQNHash, extension, timestamp)` index. Without it the database performs a full table scan. On production deployments with millions of profiler rows this caused 100+ second response times (Grafana: 106 770 ms; 99 % in DB; 93 dbOps). Without `profile` in the fields param the same endpoint returned in ~150-220 ms. A secondary N+1 bug existed independently of the index: `customMetrics` in fields called `getCustomMetrics(table, column)` once per paginated column, issuing up to N identical queries against `entity_extension` and then filtering in Java. Fix --- * **migration 2.0.2** (MySQL + PostgreSQL): `CREATE INDEX IF NOT EXISTS idx_pdts_fqnhash_ext_ts ON profiler_data_time_series(entityFQNHash, extension, timestamp)`. The `IF NOT EXISTS` guard makes the migration safe to re-run and handles both upgrade and fresh-install paths. * **`getTableColumnsInternal`** — `customMetrics` block: fetch all column custom metrics for the table in one query, group by column name in Java, then distribute. Reduces N queries to 1. * **`getTableColumnsInternal`** — `profile` block: skip the duplicate `populateEntityFieldTags` call when `tags` was already fetched earlier in the same request, saving one prefix-scan on `tag_usage` per request. Related: PR #26855 (fixed N+1 tag queries on the list-tables path but left the profiler-index and customMetrics N+1 untouched on the columns sub-path). --- .../native/2.0.2/mysql/schemaChanges.sql | 19 +++ .../native/2.0.2/postgres/schemaChanges.sql | 23 +++ .../it/tests/TableResourceIT.java | 134 ++++++++++++++++++ .../service/jdbi3/TableRepository.java | 19 ++- 4 files changed, 193 insertions(+), 2 deletions(-) create mode 100644 bootstrap/sql/migrations/native/2.0.2/mysql/schemaChanges.sql create mode 100644 bootstrap/sql/migrations/native/2.0.2/postgres/schemaChanges.sql diff --git a/bootstrap/sql/migrations/native/2.0.2/mysql/schemaChanges.sql b/bootstrap/sql/migrations/native/2.0.2/mysql/schemaChanges.sql new file mode 100644 index 000000000000..47e3b6867d6d --- /dev/null +++ b/bootstrap/sql/migrations/native/2.0.2/mysql/schemaChanges.sql @@ -0,0 +1,19 @@ +-- Restore a composite index on profiler_data_time_series(entityFQNHash, extension, timestamp). +-- +-- Root cause: the 1.9.9 postDataMigrationSQLScript.sql created several temporary indexes for +-- the data migration and then dropped ALL of them at the end, including the only index that +-- covered entityFQNHash. After that migration the table retains only the unique constraint +-- (entityFQNHash, extension, operation, timestamp) where `operation` sits between `extension` +-- and `timestamp`. Queries of the form +-- +-- SELECT entityFQNHash, MAX(timestamp) FROM profiler_data_time_series +-- WHERE entityFQNHash IN (...) AND extension = 'table.columnProfile' +-- GROUP BY entityFQNHash +-- +-- cannot use that index efficiently for MAX(timestamp) because `operation` (nullable) breaks +-- the prefix. On a large table (millions of profile rows) this causes a full table scan and +-- 100+ second response times on the columns API when `fields=profile` is requested. +-- +-- The new index covers the exact predicate pattern used by getLatestExtensionsBatch(). +CREATE INDEX IF NOT EXISTS idx_pdts_fqnhash_ext_ts + ON profiler_data_time_series (entityFQNHash, extension, timestamp); diff --git a/bootstrap/sql/migrations/native/2.0.2/postgres/schemaChanges.sql b/bootstrap/sql/migrations/native/2.0.2/postgres/schemaChanges.sql new file mode 100644 index 000000000000..67bcb2e63488 --- /dev/null +++ b/bootstrap/sql/migrations/native/2.0.2/postgres/schemaChanges.sql @@ -0,0 +1,23 @@ +-- Restore a composite index on profiler_data_time_series(entityFQNHash, extension, timestamp). +-- +-- Root cause: the 1.9.9 schemaChanges.sql explicitly dropped the unique constraint +-- profiler_data_time_series_unique_hash_extension_ts (entityFQNHash, extension, operation, timestamp) +-- to allow changing the `operation` generated-column expression, but never recreated it. +-- After the 1.9.9 migration the table retains only +-- profiler_data_time_series_combined_id_ts (extension, timestamp) +-- which is useless for queries that filter by entityFQNHash. +-- +-- The 1.9.9 postDataMigrationSQLScript.sql also created temporary indexes +-- (idx_pdts_entityFQNHash, idx_pdts_composite, etc.) during its bulk UPDATE pass and then +-- dropped them all, leaving no index on entityFQNHash. +-- +-- Queries of the form +-- +-- SELECT entityFQNHash, MAX(timestamp) FROM profiler_data_time_series +-- WHERE entityFQNHash IN (...) AND extension = 'table.columnProfile' +-- GROUP BY entityFQNHash +-- +-- issued by getLatestExtensionsBatch() perform a full table scan without this index, +-- causing 100+ second response times on the columns API when `fields=profile` is requested. +CREATE INDEX IF NOT EXISTS idx_pdts_fqnhash_ext_ts + ON profiler_data_time_series (entityFQNHash, extension, timestamp); diff --git a/openmetadata-integration-tests/src/test/java/org/openmetadata/it/tests/TableResourceIT.java b/openmetadata-integration-tests/src/test/java/org/openmetadata/it/tests/TableResourceIT.java index bf725c071b5b..4c1d6ba677bf 100644 --- a/openmetadata-integration-tests/src/test/java/org/openmetadata/it/tests/TableResourceIT.java +++ b/openmetadata-integration-tests/src/test/java/org/openmetadata/it/tests/TableResourceIT.java @@ -5900,4 +5900,138 @@ void test_listTablesWithColumnTags_performance(TestNamespace ns) { assertFalse(table.getTags().isEmpty(), "Table tags should not be empty"); } } + + // =================================================================== + // REGRESSION TEST - columns API with fields=profile (collate#3488) + // =================================================================== + + @Test + @Execution(ExecutionMode.SAME_THREAD) + void test_getColumnsWithProfileField_correctnessAndNoBatchRegression(TestNamespace ns) { + OpenMetadataClient client = SdkClients.adminClient(); + + DatabaseService service = DatabaseServiceTestFactory.createPostgres(ns); + DatabaseSchema schema = DatabaseSchemaTestFactory.createSimple(ns, service); + + CreateClassification createClassification = + new CreateClassification() + .withName(ns.prefix("profile_test_cls")) + .withDescription("Classification for profile regression test"); + Classification cls = client.classifications().create(createClassification); + + CreateTag createTag = + new CreateTag() + .withName(ns.prefix("profile_test_tag")) + .withDescription("Tag for profile regression test") + .withClassification(cls.getName()); + Tag tag = client.tags().create(createTag); + + TagLabel tagLabel = + new TagLabel() + .withTagFQN(tag.getFullyQualifiedName()) + .withSource(TagLabel.TagSource.CLASSIFICATION); + + Column idCol = ColumnBuilder.of("id", "BIGINT").primaryKey().notNull().build(); + idCol.setTags(List.of(tagLabel)); + Column emailCol = ColumnBuilder.of("email", "VARCHAR").dataLength(255).build(); + emailCol.setTags(List.of(tagLabel)); + Column nameCol = ColumnBuilder.of("name", "VARCHAR").dataLength(255).build(); + + CreateTable createRequest = createRequest(ns.prefix("profile_regression_table"), ns); + createRequest.setDatabaseSchema(schema.getFullyQualifiedName()); + createRequest.setColumns(List.of(idCol, emailCol, nameCol)); + Table table = client.tables().create(createRequest); + + Long timestamp = System.currentTimeMillis(); + ColumnProfile idProfile = + new ColumnProfile() + .withName("id") + .withMin(1.0) + .withMax(999.0) + .withUniqueCount(100.0) + .withTimestamp(timestamp); + ColumnProfile emailProfile = + new ColumnProfile() + .withName("email") + .withNullCount(5.0) + .withNullProportion(0.05) + .withTimestamp(timestamp); + + TableProfile tableProfile = + new TableProfile().withRowCount(100.0).withColumnCount(3.0).withTimestamp(timestamp); + + CreateTableProfile createProfile = + new CreateTableProfile() + .withTableProfile(tableProfile) + .withColumnProfile(List.of(idProfile, emailProfile)); + client.tables().updateTableProfile(table.getId(), createProfile); + + // Verify all four field combinations don't regress: + // (a) fields=profile — profile data returned, no full-table-scan on profiler_data_time_series + TableColumnList withProfile = + assertTimeout( + Duration.ofSeconds(30), + () -> client.tables().getColumns(table.getId(), "profile"), + "columns?fields=profile should complete within 30s"); + + assertEquals(3, withProfile.getData().size()); + Column returnedId = + withProfile.getData().stream() + .filter(c -> "id".equals(c.getName())) + .findFirst() + .orElse(null); + Column returnedName = + withProfile.getData().stream() + .filter(c -> "name".equals(c.getName())) + .findFirst() + .orElse(null); + assertNotNull(returnedId, "id column should be present"); + assertNotNull(returnedId.getProfile(), "id column should have profile data"); + assertEquals(1.0, returnedId.getProfile().getMin(), "id column min should match"); + assertEquals(999.0, returnedId.getProfile().getMax(), "id column max should match"); + assertNotNull(returnedName, "name column should be present"); + assertNull(returnedName.getProfile(), "name column has no profile, should be null"); + + // (b) fields=tags,customMetrics,extension,profile — the exact production query + TableColumnList withAllFields = + assertTimeout( + Duration.ofSeconds(30), + () -> client.tables().getColumns(table.getId(), "tags,customMetrics,extension,profile"), + "columns?fields=tags,customMetrics,extension,profile should complete within 30s"); + + assertEquals(3, withAllFields.getData().size()); + + Column idResult = + withAllFields.getData().stream() + .filter(c -> "id".equals(c.getName())) + .findFirst() + .orElse(null); + assertNotNull(idResult, "id column must be present"); + assertNotNull(idResult.getProfile(), "id column must have profile"); + assertNotNull(idResult.getTags(), "id column must have tags"); + assertFalse(idResult.getTags().isEmpty(), "id column tags must not be empty"); + assertTrue( + idResult.getTags().stream() + .anyMatch(t -> tag.getFullyQualifiedName().equals(t.getTagFQN())), + "id column should carry the test tag"); + + // (c) fields=tags,profile — duplicate populateEntityFieldTags must not run twice + TableColumnList withTagsAndProfile = + assertTimeout( + Duration.ofSeconds(30), + () -> client.tables().getColumns(table.getId(), "tags,profile"), + "columns?fields=tags,profile should complete within 30s"); + + assertEquals(3, withTagsAndProfile.getData().size()); + Column idTagsProfile = + withTagsAndProfile.getData().stream() + .filter(c -> "id".equals(c.getName())) + .findFirst() + .orElse(null); + assertNotNull(idTagsProfile); + assertNotNull(idTagsProfile.getTags()); + assertFalse( + idTagsProfile.getTags().isEmpty(), "Tags must be present even when profile requested"); + assertNotNull(idTagsProfile.getProfile(), "Profile must be present when profile requested"); + } } diff --git a/openmetadata-service/src/main/java/org/openmetadata/service/jdbi3/TableRepository.java b/openmetadata-service/src/main/java/org/openmetadata/service/jdbi3/TableRepository.java index cbdc052523a1..05aeb37af9cc 100644 --- a/openmetadata-service/src/main/java/org/openmetadata/service/jdbi3/TableRepository.java +++ b/openmetadata-service/src/main/java/org/openmetadata/service/jdbi3/TableRepository.java @@ -2891,8 +2891,21 @@ private ResultList getTableColumnsInternal( } if (fieldsParam != null && fieldsParam.contains("customMetrics")) { + List allColumnMetricRecords = + daoCollection + .entityExtensionDAO() + .getExtensions(table.getId(), CUSTOM_METRICS_EXTENSION + TABLE_COLUMN_EXTENSION); + Map> metricsByColumn = new HashMap<>(); + for (ExtensionRecord record : allColumnMetricRecords) { + CustomMetric metric = JsonUtils.readValue(record.extensionJson(), CustomMetric.class); + if (metric != null && metric.getColumnName() != null) { + metricsByColumn + .computeIfAbsent(metric.getColumnName(), k -> new ArrayList<>()) + .add(metric); + } + } for (Column column : paginatedColumns) { - column.setCustomMetrics(getCustomMetrics(table, column.getName())); + column.setCustomMetrics(metricsByColumn.getOrDefault(column.getName(), new ArrayList<>())); } } @@ -2904,7 +2917,9 @@ private ResultList getTableColumnsInternal( if (fieldsParam != null && fieldsParam.contains("profile")) { setColumnProfile(paginatedColumns); - populateEntityFieldTags(entityType, paginatedColumns, table.getFullyQualifiedName(), true); + if (!fieldsParam.contains("tags")) { + populateEntityFieldTags(entityType, paginatedColumns, table.getFullyQualifiedName(), true); + } paginatedColumns = piiOwners != null ? PIIMasker.getTableProfile(piiOwners, paginatedColumns, authorizer, securityContext) From eddb32c32fbc3493d5b303e827e50d28ce4f8233 Mon Sep 17 00:00:00 2001 From: sonika-shah <58761340+sonika-shah@users.noreply.github.com> Date: Thu, 7 May 2026 14:37:33 +0530 Subject: [PATCH 2/5] fix(profiler): restore unique constraint on profiler_data_time_series + batch column extension/customMetrics fetch MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Move the migration from 2.0.2/ to 1.12.8/ and switch from a non-unique covering index to restoring the original unique constraint dropped in 1.9.9. The two-phase CREATE UNIQUE INDEX CONCURRENTLY + ADD CONSTRAINT USING INDEX pattern avoids the ACCESS EXCLUSIVE lock on the hot profiler_data_time_series table during the upgrade. Closes the 1.9.9 regression and brings Postgres back in line with MySQL (which never lost the constraint). The leading (entityFQNHash, extension) prefix serves the column-profile batch query — same shape MySQL has been running without 504s. MySQL needs no migration. Java side, eliminates two more N+1 patterns that compound the latency at customer scale: * getTableColumnsInternal extension block: replaced per-column getColumnExtension() loop with a single getExtensionsByJsonSchema() call, grouped by column FQN-hash in Java. * searchTableColumnsInternal customMetrics block: applied the same batch-fetch pattern already used in getTableColumnsInternal, replacing per-column getCustomMetrics() with one getExtensions() call. New DAO method on EntityExtensionDAO: getExtensionsByJsonSchema(id, jsonSchema) — selects extensions for a table id filtered by the jsonschema discriminator. Required because column extensions are stored with MD5-hashed extension keys and have no shared prefix the existing getExtensions(id, prefix) could use. --- .../native/1.12.8/postgres/schemaChanges.sql | 14 ++++++++ .../native/2.0.2/mysql/schemaChanges.sql | 19 ---------- .../native/2.0.2/postgres/schemaChanges.sql | 23 ------------ .../service/jdbi3/CollectionDAO.java | 7 ++++ .../service/jdbi3/TableRepository.java | 36 +++++++++++++++++-- 5 files changed, 55 insertions(+), 44 deletions(-) create mode 100644 bootstrap/sql/migrations/native/1.12.8/postgres/schemaChanges.sql delete mode 100644 bootstrap/sql/migrations/native/2.0.2/mysql/schemaChanges.sql delete mode 100644 bootstrap/sql/migrations/native/2.0.2/postgres/schemaChanges.sql diff --git a/bootstrap/sql/migrations/native/1.12.8/postgres/schemaChanges.sql b/bootstrap/sql/migrations/native/1.12.8/postgres/schemaChanges.sql new file mode 100644 index 000000000000..20b90952ac5c --- /dev/null +++ b/bootstrap/sql/migrations/native/1.12.8/postgres/schemaChanges.sql @@ -0,0 +1,14 @@ +-- Restore the unique constraint dropped in 1.9.9. Closes the 1.9.9 regression that caused +-- /columns?fields=profile 504s, and brings Postgres back in line with MySQL (which never +-- lost it). The leading (entityFQNHash, extension) prefix serves the column-profile batch query. +-- Two-phase: CONCURRENTLY build avoids ACCESS EXCLUSIVE lock; ADD CONSTRAINT USING INDEX +-- promotes the built index without re-scanning. +CREATE UNIQUE INDEX CONCURRENTLY IF NOT EXISTS + profiler_data_time_series_unique_hash_extension_ts + ON profiler_data_time_series (entityFQNHash, extension, operation, timestamp); + +ALTER TABLE profiler_data_time_series + ADD CONSTRAINT profiler_data_time_series_unique_hash_extension_ts + UNIQUE USING INDEX profiler_data_time_series_unique_hash_extension_ts; + +ANALYZE profiler_data_time_series; diff --git a/bootstrap/sql/migrations/native/2.0.2/mysql/schemaChanges.sql b/bootstrap/sql/migrations/native/2.0.2/mysql/schemaChanges.sql deleted file mode 100644 index 47e3b6867d6d..000000000000 --- a/bootstrap/sql/migrations/native/2.0.2/mysql/schemaChanges.sql +++ /dev/null @@ -1,19 +0,0 @@ --- Restore a composite index on profiler_data_time_series(entityFQNHash, extension, timestamp). --- --- Root cause: the 1.9.9 postDataMigrationSQLScript.sql created several temporary indexes for --- the data migration and then dropped ALL of them at the end, including the only index that --- covered entityFQNHash. After that migration the table retains only the unique constraint --- (entityFQNHash, extension, operation, timestamp) where `operation` sits between `extension` --- and `timestamp`. Queries of the form --- --- SELECT entityFQNHash, MAX(timestamp) FROM profiler_data_time_series --- WHERE entityFQNHash IN (...) AND extension = 'table.columnProfile' --- GROUP BY entityFQNHash --- --- cannot use that index efficiently for MAX(timestamp) because `operation` (nullable) breaks --- the prefix. On a large table (millions of profile rows) this causes a full table scan and --- 100+ second response times on the columns API when `fields=profile` is requested. --- --- The new index covers the exact predicate pattern used by getLatestExtensionsBatch(). -CREATE INDEX IF NOT EXISTS idx_pdts_fqnhash_ext_ts - ON profiler_data_time_series (entityFQNHash, extension, timestamp); diff --git a/bootstrap/sql/migrations/native/2.0.2/postgres/schemaChanges.sql b/bootstrap/sql/migrations/native/2.0.2/postgres/schemaChanges.sql deleted file mode 100644 index 67bcb2e63488..000000000000 --- a/bootstrap/sql/migrations/native/2.0.2/postgres/schemaChanges.sql +++ /dev/null @@ -1,23 +0,0 @@ --- Restore a composite index on profiler_data_time_series(entityFQNHash, extension, timestamp). --- --- Root cause: the 1.9.9 schemaChanges.sql explicitly dropped the unique constraint --- profiler_data_time_series_unique_hash_extension_ts (entityFQNHash, extension, operation, timestamp) --- to allow changing the `operation` generated-column expression, but never recreated it. --- After the 1.9.9 migration the table retains only --- profiler_data_time_series_combined_id_ts (extension, timestamp) --- which is useless for queries that filter by entityFQNHash. --- --- The 1.9.9 postDataMigrationSQLScript.sql also created temporary indexes --- (idx_pdts_entityFQNHash, idx_pdts_composite, etc.) during its bulk UPDATE pass and then --- dropped them all, leaving no index on entityFQNHash. --- --- Queries of the form --- --- SELECT entityFQNHash, MAX(timestamp) FROM profiler_data_time_series --- WHERE entityFQNHash IN (...) AND extension = 'table.columnProfile' --- GROUP BY entityFQNHash --- --- issued by getLatestExtensionsBatch() perform a full table scan without this index, --- causing 100+ second response times on the columns API when `fields=profile` is requested. -CREATE INDEX IF NOT EXISTS idx_pdts_fqnhash_ext_ts - ON profiler_data_time_series (entityFQNHash, extension, timestamp); diff --git a/openmetadata-service/src/main/java/org/openmetadata/service/jdbi3/CollectionDAO.java b/openmetadata-service/src/main/java/org/openmetadata/service/jdbi3/CollectionDAO.java index 67c88b54a628..d9ac72daf98e 100644 --- a/openmetadata-service/src/main/java/org/openmetadata/service/jdbi3/CollectionDAO.java +++ b/openmetadata-service/src/main/java/org/openmetadata/service/jdbi3/CollectionDAO.java @@ -1526,6 +1526,13 @@ void bulkUpsertExtensions( List getExtensions( @BindUUID("id") UUID id, @Bind("extensionPrefix") String extensionPrefix); + @RegisterRowMapper(ExtensionMapper.class) + @SqlQuery( + "SELECT extension, json FROM entity_extension WHERE id = :id AND jsonschema = :jsonSchema " + + "ORDER BY extension") + List getExtensionsByJsonSchema( + @BindUUID("id") UUID id, @Bind("jsonSchema") String jsonSchema); + @ConnectionAwareSqlQuery( value = "SELECT json FROM (" diff --git a/openmetadata-service/src/main/java/org/openmetadata/service/jdbi3/TableRepository.java b/openmetadata-service/src/main/java/org/openmetadata/service/jdbi3/TableRepository.java index 05aeb37af9cc..7f2cfc6c2a2d 100644 --- a/openmetadata-service/src/main/java/org/openmetadata/service/jdbi3/TableRepository.java +++ b/openmetadata-service/src/main/java/org/openmetadata/service/jdbi3/TableRepository.java @@ -154,6 +154,7 @@ public class TableRepository extends EntityRepository { public static final String TABLE_COLUMN_EXTENSION = "table.column"; public static final String TABLE_EXTENSION = "table.table"; public static final String CUSTOM_METRICS_EXTENSION = "customMetrics."; + public static final String COLUMN_EXTENSION_JSON_SCHEMA = "columnExtension"; public static final String TABLE_PROFILER_CONFIG = "tableProfilerConfig"; private static final ReadPrefetchKey PREFETCH_DEFAULT_FIELDS = ReadPrefetchKey.TABLE_DEFAULT_FIELDS; @@ -2910,8 +2911,26 @@ private ResultList getTableColumnsInternal( } if (fieldsParam != null && fieldsParam.contains("extension")) { + List allColumnExtensions = + daoCollection + .entityExtensionDAO() + .getExtensionsByJsonSchema(table.getId(), COLUMN_EXTENSION_JSON_SCHEMA); + Map extensionByColumnHash = new HashMap<>(); + for (ExtensionRecord record : allColumnExtensions) { + try { + extensionByColumnHash.put( + record.extensionName(), JsonUtils.readValue(record.extensionJson(), Object.class)); + } catch (Exception e) { + LOG.warn( + "Failed to deserialize column extension for table {}: {}", + table.getId(), + e.getMessage()); + } + } for (Column column : paginatedColumns) { - column.setExtension(getColumnExtension(table.getId(), column.getFullyQualifiedName())); + column.setExtension( + extensionByColumnHash.get( + FullyQualifiedName.buildHash(column.getFullyQualifiedName()))); } } @@ -3242,8 +3261,21 @@ private ResultList searchTableColumnsInternal( Fields fields = getFields(fieldsParam); if (fields.contains("customMetrics") || fields.contains("*")) { + List allColumnMetricRecords = + daoCollection + .entityExtensionDAO() + .getExtensions(table.getId(), CUSTOM_METRICS_EXTENSION + TABLE_COLUMN_EXTENSION); + Map> metricsByColumn = new HashMap<>(); + for (ExtensionRecord record : allColumnMetricRecords) { + CustomMetric metric = JsonUtils.readValue(record.extensionJson(), CustomMetric.class); + if (metric != null && metric.getColumnName() != null) { + metricsByColumn + .computeIfAbsent(metric.getColumnName(), k -> new ArrayList<>()) + .add(metric); + } + } for (Column column : paginatedResults) { - column.setCustomMetrics(getCustomMetrics(table, column.getName())); + column.setCustomMetrics(metricsByColumn.getOrDefault(column.getName(), new ArrayList<>())); } } From 38bf22e5dadd147079163b75bbe76cc557844ee5 Mon Sep 17 00:00:00 2001 From: sonika-shah <58761340+sonika-shah@users.noreply.github.com> Date: Thu, 7 May 2026 16:11:16 +0530 Subject: [PATCH 3/5] =?UTF-8?q?chore(profiler):=20address=20review=20feedb?= =?UTF-8?q?ack=20=E2=80=94=20empty-list=20literal=20+=20accurate=20test=20?= =?UTF-8?q?comments?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit * Replace `new ArrayList<>()` default in `metricsByColumn.getOrDefault(...)` with `List.of()` at both call sites in `TableRepository` (getTableColumnsInternal and searchTableColumnsInternal). `getOrDefault` evaluates its default eagerly, so the new ArrayList allocates per-column even when the key is found — unnecessary work on a hot path. * Reword two stale test comments in `test_getColumnsWithProfileField_correctnessAndNoBatchRegression`: - "all four field combinations" → "the three field combinations exercised below" - "(c) duplicate populateEntityFieldTags must not run twice" → describe the observable contract the assertions actually verify (tags + profile both present), not the internal call count. --- .../java/org/openmetadata/it/tests/TableResourceIT.java | 7 +++++-- .../org/openmetadata/service/jdbi3/TableRepository.java | 4 ++-- 2 files changed, 7 insertions(+), 4 deletions(-) diff --git a/openmetadata-integration-tests/src/test/java/org/openmetadata/it/tests/TableResourceIT.java b/openmetadata-integration-tests/src/test/java/org/openmetadata/it/tests/TableResourceIT.java index 1fe833f166f3..12047ddeb917 100644 --- a/openmetadata-integration-tests/src/test/java/org/openmetadata/it/tests/TableResourceIT.java +++ b/openmetadata-integration-tests/src/test/java/org/openmetadata/it/tests/TableResourceIT.java @@ -5927,7 +5927,7 @@ void test_getColumnsWithProfileField_correctnessAndNoBatchRegression(TestNamespa .withColumnProfile(List.of(idProfile, emailProfile)); client.tables().updateTableProfile(table.getId(), createProfile); - // Verify all four field combinations don't regress: + // Verify the three field combinations exercised below don't regress: // (a) fields=profile — profile data returned, no full-table-scan on profiler_data_time_series TableColumnList withProfile = assertTimeout( @@ -5976,7 +5976,10 @@ void test_getColumnsWithProfileField_correctnessAndNoBatchRegression(TestNamespa .anyMatch(t -> tag.getFullyQualifiedName().equals(t.getTagFQN())), "id column should carry the test tag"); - // (c) fields=tags,profile — duplicate populateEntityFieldTags must not run twice + // (c) fields=tags,profile — both tags and profile are populated correctly when requested + // together (the dedup of populateEntityFieldTags is exercised here, but this test + // verifies the observable contract — tags + profile both present on the result — + // not the internal call count) TableColumnList withTagsAndProfile = assertTimeout( Duration.ofSeconds(30), diff --git a/openmetadata-service/src/main/java/org/openmetadata/service/jdbi3/TableRepository.java b/openmetadata-service/src/main/java/org/openmetadata/service/jdbi3/TableRepository.java index 7f2cfc6c2a2d..90cc319db8b2 100644 --- a/openmetadata-service/src/main/java/org/openmetadata/service/jdbi3/TableRepository.java +++ b/openmetadata-service/src/main/java/org/openmetadata/service/jdbi3/TableRepository.java @@ -2906,7 +2906,7 @@ private ResultList getTableColumnsInternal( } } for (Column column : paginatedColumns) { - column.setCustomMetrics(metricsByColumn.getOrDefault(column.getName(), new ArrayList<>())); + column.setCustomMetrics(metricsByColumn.getOrDefault(column.getName(), List.of())); } } @@ -3275,7 +3275,7 @@ private ResultList searchTableColumnsInternal( } } for (Column column : paginatedResults) { - column.setCustomMetrics(metricsByColumn.getOrDefault(column.getName(), new ArrayList<>())); + column.setCustomMetrics(metricsByColumn.getOrDefault(column.getName(), List.of())); } } From ec22ab6a4b105a08cb7f59ab07b38390aed7d44b Mon Sep 17 00:00:00 2001 From: sonika-shah <58761340+sonika-shah@users.noreply.github.com> Date: Fri, 8 May 2026 00:03:14 +0530 Subject: [PATCH 4/5] fix(profiler): force outer index scan in getLatestExtensionsBatch by pushing IN list to the join MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The getLatestExtensionsBatch query was the right shape for correctness but the planner — on Postgres at customer scale, with the new unique constraint in place — was still choosing a parallel sequential scan over the full profiler_data_time_series table for the outer side of the JOIN, rather than a merge join with index scan on both sides. Inner subquery: filtered by `entityFQNHash IN (...)`, used the index. Outer: only filtered by `p.extension = :extension`, no IN list, planner couldn't infer the transitive constraint that p.entityFQNHash must equal one of the inner hashes (because it's enforced through the JOIN ON clause, not a WHERE predicate). Result: full table scan reading 6.7M+ rows even when the actual answer is 23 rows. Adding the redundant `AND p.entityFQNHash IN ()` to the outer WHERE makes the constraint explicit. The result set is unchanged (implied by the join condition), but the planner can now use the unique index for the outer access too. Verified on the AUT dump (6.94M-row pdts): EXPLAIN of the batch query: 7,234ms → 79ms (Hash Join + Parallel Seq Scan → Merge Join + Index Only Scan). Live API /columns?fields=profile&include=all: 6-36 seconds → 22-28ms (warm) / 1.9s (very first call). 250-1000x improvement, depending on cache state. Same SQL works on both engines; no @ConnectionAwareSqlQuery split needed. --- .../java/org/openmetadata/service/jdbi3/CollectionDAO.java | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/openmetadata-service/src/main/java/org/openmetadata/service/jdbi3/CollectionDAO.java b/openmetadata-service/src/main/java/org/openmetadata/service/jdbi3/CollectionDAO.java index d9ac72daf98e..35f88df74c55 100644 --- a/openmetadata-service/src/main/java/org/openmetadata/service/jdbi3/CollectionDAO.java +++ b/openmetadata-service/src/main/java/org/openmetadata/service/jdbi3/CollectionDAO.java @@ -9528,7 +9528,8 @@ default String getTimeSeriesTableName() { + " GROUP BY entityFQNHash" + ") latest " + "ON p.entityFQNHash = latest.entityFQNHash AND p.timestamp = latest.latestTs " - + "WHERE p.extension = :extension") + + "WHERE p.extension = :extension " + + "AND p.entityFQNHash IN ()") @RegisterRowMapper(LatestExtensionRecordMapper.class) List getLatestExtensionsBatch( @Define("table") String table, From 458bc91565711e27445a2697875c8d82f15f4446 Mon Sep 17 00:00:00 2001 From: sonika-shah <58761340+sonika-shah@users.noreply.github.com> Date: Fri, 8 May 2026 00:21:42 +0530 Subject: [PATCH 5/5] test(profiler): shorten classification/tag fixture names in IT to fit varchar(256) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The IT fixture for test_getColumnsWithProfileField_correctnessAndNoBatchRegression was building a tagFQN of `.` where each part went through TestNamespace.prefix(). With the descriptive method name (62 chars) + class name (15 chars) + namespace UUID (32 chars) plus the `profile_test_cls` / `profile_test_tag` base names (16 chars each), the resulting tagFQN was 263 characters — over the tag_usage.tagFQN VARCHAR(256) limit: ERROR: value too long for type character varying(256) Shorten the fixture base names from `profile_test_cls`/`profile_test_tag` to `cls`/`tag`. The namespace prefix already encodes test isolation (class + method + UUID), so the base name doesn't need to repeat that context. New tagFQN length: 237 chars (cls__<32>__TableResourceIT__<62>.tag__<32>__TableResourceIT__<62>), comfortably under 256. --- .../test/java/org/openmetadata/it/tests/TableResourceIT.java | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/openmetadata-integration-tests/src/test/java/org/openmetadata/it/tests/TableResourceIT.java b/openmetadata-integration-tests/src/test/java/org/openmetadata/it/tests/TableResourceIT.java index 12047ddeb917..28a46fa18602 100644 --- a/openmetadata-integration-tests/src/test/java/org/openmetadata/it/tests/TableResourceIT.java +++ b/openmetadata-integration-tests/src/test/java/org/openmetadata/it/tests/TableResourceIT.java @@ -5876,13 +5876,13 @@ void test_getColumnsWithProfileField_correctnessAndNoBatchRegression(TestNamespa CreateClassification createClassification = new CreateClassification() - .withName(ns.prefix("profile_test_cls")) + .withName(ns.prefix("cls")) .withDescription("Classification for profile regression test"); Classification cls = client.classifications().create(createClassification); CreateTag createTag = new CreateTag() - .withName(ns.prefix("profile_test_tag")) + .withName(ns.prefix("tag")) .withDescription("Tag for profile regression test") .withClassification(cls.getName()); Tag tag = client.tags().create(createTag);