feat(query-runner): storage config + websocket notifications#27542
feat(query-runner): storage config + websocket notifications#27542Khairajani wants to merge 22 commits intomainfrom
Conversation
Adds schema + infrastructure for the Collate Query Runner to stream results
to object storage and notify the UI via websocket when a query completes.
- queryRunnerRequest: runtime-injected storageConfig {bucketName, prefix}
and resultPath. Credentials never enter the payload — the worker fetches
them via a server callback so secret: refs resolve in-worker via
CustomSecretStr.
- queryRunnerResponse: resultPath for S3-backed results (mutually exclusive
with inline results).
- WebSocketManager: queryRunnerChannel constant.
- WebsocketNotificationHandler + QueryRunnerMessage: COMPLETED/FAILED
notifications for the UI hook.
- SOCKET_EVENTS.QUERY_RUNNER_CHANNEL so the UI hook can subscribe.
Paired with open-metadata/openmetadata-collate query_runner_socket_flow.
|
Hi there 👋 Thanks for your contribution! The OpenMetadata team will review the PR shortly! Once it has been labeled as Let us know if you need any help! |
Khairajani
added
Ready for testing
|
The Java checkstyle failed. Please run You can install the pre-commit hooks with |
✅ TypeScript Types Auto-UpdatedThe generated TypeScript types have been automatically updated based on JSON schema changes in this PR. |
Jest test CoverageUI tests summary
|
🔴 Playwright Results — 2 failure(s), 12 flaky✅ 4014 passed · ❌ 2 failed · 🟡 12 flaky · ⏭️ 86 skipped
Genuine Failures (failed on all attempts)❌
|
|
The Java checkstyle failed. Please run You can install the pre-commit hooks with |
0.3.x's connector calls tds.skipall, but python-tds 1.x moved that to tds_base.skipall — every server-side cursor fetch raises AttributeError on TABNAME / COLINFO tokens. 1.0+ uses the correct module path.
Code Review 👍 Approved with suggestions 0 resolved / 1 findingsImplements storage configuration and WebSocket notification infrastructure for query runner results while bumping sqlalchemy-pytds. Remove the duplicate route definition for 'KNOWLEDGE_CENTER_PAGE' and 'KNOWLEDGE_CENTER' in constants.ts. 💡 Quality: Duplicate route: KNOWLEDGE_CENTER_PAGE and KNOWLEDGE_CENTER📄 openmetadata-ui/src/main/resources/ui/src/constants/constants.ts:156-157 Line 156 ( 🤖 Prompt for agentsOptionsDisplay: compact → Showing less information. Comment with these commands to change:
Was this helpful? React with 👍 / 👎 | Gitar |
|
|
1 participant
Summary
Adds the OSS-side pieces that let the Collate Query Runner stream results to object storage and notify the UI via websocket when a query completes. All Collate-specific logic lives in
open-metadata/openmetadata-collate#query_runner_socket_flow; this PR only lands the schema/infrastructure pieces that must live in OSS.queryRunnerRequest.json: runtime-injectedstorageConfig(bucket + prefix, no credentials) andresultPath. Worker fetches credentials via a separate callback sosecret:refs resolve in-worker viaCustomSecretStr.queryRunnerResponse.json:resultPathfor S3-backed results (mutually exclusive with inlineresults).WebSocketManager: newqueryRunnerChannelconstant.WebsocketNotificationHandler+ newQueryRunnerMessage: COMPLETED / FAILED notifications for the UI hook.SOCKET_EVENTS.QUERY_RUNNER_CHANNEL: TS constant so the UI hook can subscribe.Test plan
query_runner_socket_flowbuilds against this branch end-to-end (test-connection + real BigQuery query verified locally — S3 streaming, WebSocket → UI fetch).🤖 Generated with Claude Code
Summary by Gitar
sqlalchemy-pytdsto~=1.0iningestion/setup.pyto fix cursor fetch issues withpython-tds1.x.KNOWLEDGE_CENTER_CLASSIFICATION,CREATE_PAGE_HASH, and delay values inconstants.ts.KNOWLEDGE_CENTERand associated sub-pages inconstants.ts.This will update automatically on new commits.