Skip to content

fix(clickhouse): extend MV TO-clause regex to cover POPULATE, UUID, and three-part names (#26265)#27827

Closed
Vaishnavi-Cailum wants to merge 4 commits into
open-metadata:mainfrom
Vaishnavi-Cailum:vdixit/26265-clickhouse-mv-lineage-improvements
Closed

fix(clickhouse): extend MV TO-clause regex to cover POPULATE, UUID, and three-part names (#26265)#27827
Vaishnavi-Cailum wants to merge 4 commits into
open-metadata:mainfrom
Vaishnavi-Cailum:vdixit/26265-clickhouse-mv-lineage-improvements

Conversation

@Vaishnavi-Cailum
Copy link
Copy Markdown

Summary

Fixes gaps in the MATERIALIZED_VIEW_TO_PATTERN regex introduced by #27553 for issue #26265.

Four additional ClickHouse DDL forms were not matched:

  • POPULATE keyword before TO — valid ClickHouse DDL supported since v22.x
  • UUID clause — emitted by system.tables DDL output (e.g. CREATE MATERIALIZED VIEW db.mv UUID '...' TO db.target)
  • Three-part qualified names — catalog.schema.mv TO catalog2.schema2.target
  • Bare FROM without ASCREATE MATERIALIZED VIEW mv TO target (...) FROM source (no AS SELECT)

Changes:

  • _QUALIFIED_IDENT extended to match up to three dot-separated parts
  • Added (?:\s+UUID\s+\S+)? optional clause after MV name
  • Added (?:\s+POPULATE)? optional clause before TO
  • Added FROM\b as an accepted token after the target table (alongside ( and AS)
  • 4 new regression tests in TestGetMvToTargetTable

Builds on top of the approach in #27553 (same query_lineage_producer override, same get_mv_to_target_table helper) — no changes to that logic, only the regex coverage is extended.

Test plan

  • All existing TestGetMvToTargetTable tests still pass
  • 4 new tests pass: test_populate_keyword_before_to, test_on_cluster_and_populate_before_to, test_uuid_clause, test_three_part_qualified_names
  • Run: cd ingestion && python -m pytest tests/unit/topology/database/test_clickhouse_utils.py -v

🤖 Generated with Claude Code

abhay-codes07 and others added 3 commits April 20, 2026 16:45
… clause (open-metadata#26265)

Signed-off-by: Abhay Singh <abhaysingh0293@gmail.com>
Signed-off-by: Abhay Singh <abhaysingh0293@gmail.com>
…nd three-part names

Fixes gaps in MATERIALIZED_VIEW_TO_PATTERN missed by PR open-metadata#27553:
- POPULATE keyword before TO (valid ClickHouse DDL form)
- UUID clause emitted by system.tables DDL output
- Three-part qualified names (catalog.schema.table)
- FROM keyword after target (bare FROM without AS keyword)

Adds four regression tests for each new case.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
@Vaishnavi-Cailum Vaishnavi-Cailum requested a review from a team as a code owner April 29, 2026 15:57
@github-actions
Copy link
Copy Markdown
Contributor

Hi there 👋 Thanks for your contribution!

The OpenMetadata team will review the PR shortly! Once it has been labeled as safe to test, the CI workflows
will start executing and we'll be able to make sure everything is working as expected.

Let us know if you need any help!

Comment thread ingestion/src/metadata/ingestion/source/database/clickhouse/utils.py Outdated
… annotation

Tuple from typing was shadowed by Tuple = create_sqlalchemy_type("Tuple")
on line 81. Using the built-in tuple[str, str] (Python 3.10+) removes the
ambiguity entirely and is safe under from __future__ import annotations.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
@github-actions
Copy link
Copy Markdown
Contributor

Hi there 👋 Thanks for your contribution!

The OpenMetadata team will review the PR shortly! Once it has been labeled as safe to test, the CI workflows
will start executing and we'll be able to make sure everything is working as expected.

Let us know if you need any help!

@gitar-bot
Copy link
Copy Markdown

gitar-bot Bot commented Apr 29, 2026

Code Review ✅ Approved 1 resolved / 1 findings

Extends the ClickHouse MV TO-clause regex to support POPULATE, UUID, and three-part table names. Shadowed typing.Tuple import has been resolved.

✅ 1 resolved
Quality: typing.Tuple shadowed by SQLAlchemy Tuple at module level

📄 ingestion/src/metadata/ingestion/source/database/clickhouse/utils.py:18 📄 ingestion/src/metadata/ingestion/source/database/clickhouse/utils.py:63 📄 ingestion/src/metadata/ingestion/source/database/clickhouse/utils.py:81
from typing import Optional, Tuple (line 18) is shadowed by Tuple = create_sqlalchemy_type("Tuple") (line 81). The function annotation Optional[Tuple[str, str]] on line 63 works today only because Python evaluates it before the reassignment executes. However, this is fragile: if the function is moved below line 81, or if from __future__ import annotations is added (which defers evaluation), the annotation will resolve to the SQLAlchemy type and break at runtime when inspected (e.g., by type checkers, Pydantic, or get_type_hints()).

Options

Display: compact → Showing less information.

Comment with these commands to change:

Compact
gitar display:verbose         

Was this helpful? React with 👍 / 👎 | Gitar

@Vaishnavi-Cailum
Copy link
Copy Markdown
Author

Closing — reopened as #27832 from correct account.

@Vaishnavi-Cailum Vaishnavi-Cailum deleted the vdixit/26265-clickhouse-mv-lineage-improvements branch April 29, 2026 17:26
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants