ci: use UC REST API for schema create/drop in Reyden nightly#505
Open
eric-wang-1990 wants to merge 19 commits into
Open
ci: use UC REST API for schema create/drop in Reyden nightly#505eric-wang-1990 wants to merge 19 commits into
eric-wang-1990 wants to merge 19 commits into
Conversation
eric-wang-1990
requested review from
gopalldb,
lidavidm and
msrathore-db
as code owners
Reyden warehouse rejects CREATE/DROP SCHEMA via the SQL statement API
("Unsupported CREATE type: SCHEMA"). Switch to the Unity Catalog REST
API (/api/2.1/unity-catalog/schemas) for both provision and cleanup.
eric-wang-1990
force-pushed
the
ci/reyden-rest-nightly
branch
from
cbac8d3 to
e3903a3
Compare
Reyden does not support DDL. Remove the per-run schema create/seed/drop steps and point the nightly at the pre-existing main.adbc_testing schema instead.
Reyden does not support any DDL (CREATE/DROP schema/table). Split the two concerns: SETUP_HTTP_PATH (TEST_PECO_WAREHOUSE_HTTP_PATH) handles schema provision, seeding, and cleanup; DATABRICKS_HTTP_PATH (TEST_PECO_REYDEN_HTTP_PATH) is used only in the test connection config.
Add workflow_call to e2e-tests.yml with an optional execution_http_path secret. Setup/teardown always use TEST_PECO_WAREHOUSE_HTTP_PATH (supports DDL); test execution uses execution_http_path when provided, falling back to the same regular warehouse. reyden-rest-nightly.yml is now a 10-line wrapper that calls e2e-tests.yml with protocol=rest and TEST_PECO_REYDEN_HTTP_PATH as the execution path.
Replace explicit secret passing (which dropped DATABRICKS_HOST and credentials) with secrets: inherit. Control Reyden routing via a boolean input use_reyden_for_execution instead of a passed secret.
Add e2e_only boolean input to e2e-tests.yml workflow_call. When true,
passes --filter FullyQualifiedName!~Tests.Unit to dotnet test, skipping
the bulk of unit tests. The test script now forwards extra args to
dotnet test via ${@:2}.
Set e2e_only: true in reyden-rest-nightly.yml.
Reyden supports SELECT and INSERT but not CREATE TABLE, DROP TABLE, UPDATE, DELETE, SHOW COLUMNS, KEY type (SHOW PRIMARY KEYS/CROSS REFERENCES), or bare SET statements. Add isReadOnly and mutableTable config flags to gate tests appropriately: - Seed step provisions adbc_testing_mutable via the regular warehouse so INSERT tests have a pre-existing table to write into on Reyden - isReadOnly=true skips DDL-based and unsupported-metadata tests - INSERT tests route to the pre-provisioned table when isReadOnly; unique IDs + WHERE clause isolate rows across parallel runs - UPDATE and DELETE tests skip entirely under isReadOnly - TestServerSidePropertyOnSeaPath skips (bare SET not supported) Closes #505
xUnit1024 errors on CanInteractUsingSetOptions and CanGetColumns because xUnit does not allow same-named methods in an inheritance chain. The new keyword is intentional here — we need to intercept the inherited tests to add Skip.If guards for Reyden. Suppress the analyzer error with pragma around both methods.
Adds an append-only history dashboard for the Reyden REST nightly E2E run, served from the existing gh-pages branch at https://adbc-drivers.github.io/databricks/e2e-nightly/. - parse-trx-to-json.py: TRX -> per-run JSON with totals, pass rate, and a coarse failure "signature" per failure (404 warehouse, read-only write rejected, assertion mismatch, ...) so 100+ failures group into a few causes. - update-e2e-dashboard.py: merges each run into e2e-nightly/data/runs.json (keyed by run+protocol, capped history) and writes full per-run detail. - e2e-dashboard/index.html: static page with summary cards, a Chart.js pass/fail/skip trend + pass-rate line, latest-run failure analysis, and a clickable run-history table that lazy-loads each run's failures. - e2e-tests.yml: emit + upload a TRX; new opt-in publish_dashboard input and a publish-dashboard job that runs always() so failed runs are still recorded. - reyden-rest-nightly.yml: opt in (publish_dashboard) and grant contents:write. Co-authored-by: Isaac
## What The raw pass-rate on the SEA/Reyden nightly (~61% on the latest run) is dominated by **expected** failures — unsupported DDL/types, no Thrift endpoint on a SEA-only warehouse, no CloudFetch — which buries the genuine driver bugs. This adds a coarse **root-cause category** on top of the existing per-failure signature so the dashboard separates expected noise from the real backlog. On the latest run (127 failures) the split is **51 expected Reyden gaps / 76 real issues**. ## How it classifies Classification follows the failing step, which the message already encodes: - A test with a `CREATE TABLE/SCHEMA` step Reyden can't run fails **at that step** with an `Unsupported …` message → **Reyden capability gap (expected)**. - A value/cast mismatch means setup succeeded and the `INSERT→SELECT→DELETE` round-trip returned wrong data → **Real issue** (e.g. a SEA-path serialization difference). - Missing warehouse / read-only / auth / timeout / transport → **Environment / infra**. ## Changes - **`parse-trx-to-json.py`**: refined `signature_for()` (Thrift-on-SEA, CloudFetch, unsupported-feature buckets; `PARSE_SYNTAX_ERROR` ordered before the broad assertion bucket), added `category_for()` + per-failure `category` + a `by_category` rollup. - **`update-e2e-dashboard.py`**: propagate `by_category` into the `runs.json` summary row. - **`index.html`**: "By root-cause category" rollup with a color-coded legend, and failure detail grouped by category → signature. Degrades gracefully for older runs without `by_category`. Validated against the latest run's data and syntax-checked (`py_compile` + `node --check`). This pull request and its description were written by Isaac.
) ## What Adds the per-error granularity requested for the E2E nightly dashboard. - **Splits** the single `Reyden unsupported feature` signature into the specific errors so each gap shows its own count: `Unsupported statement: SHOW COLUMNS` (13), `Unsupported CREATE type: SCHEMA` (4), `Unsupported feature: CREATE OR REPLACE TABLE` (2), `Unsupported type: INTERVAL` (2), plus a generic fallback. All map to the **Reyden capability gap** category. - **Dashboard**: replaces the flat signature/class tables with a **"By error (root cause)"** table — `Category | Error | Count | What it is | Tests` — where "What it is" is a plain-language description (`SIGNATURE_DESCRIPTIONS`) and "Tests" lists the affected `Class.Method`s (from the run detail file). ## Changes - `.github/scripts/parse-trx-to-json.py` — `signature_for` splits the unsupported bucket; `_SIGNATURE_CATEGORY` maps the new signatures to the gap category (old combined signature kept for back-compat). - `.github/e2e-dashboard/index.html` — `SIGNATURE_DESCRIPTIONS` + `sigDetailTable()`; `renderAnalysis` now fetches the run detail and renders the per-error table. ## Verification Generated the dashboard locally from the latest run's data (re-classified) and rendered it in a browser — the split, descriptions, and test lists display correctly. Scripts `py_compile` clean and the dashboard JS passes `node --check`. This pull request and its description were written by Isaac.
Drops the csharp/test/E2E/* edits introduced by the Reyden read-only adaptation (commit 167e356) and restores them to origin/main. Per PR #505 review the E2E test edits should not ship as part of the Reyden nightly infrastructure PR — they belong in a separate change that proposes a read-only-warehouse-aware test model independently. Files restored from origin/main: csharp/test/E2E/DatabricksTestConfiguration.cs csharp/test/E2E/ServerSidePropertyE2ETest.cs csharp/test/E2E/StatementExecution/StatementExecutionDriverE2ETests.cs csharp/test/E2E/StatementTests.cs Non-E2E commits (workflow plumbing, classifier, dashboard, etc.) are unaffected. Co-authored-by: Isaac
The "make e2e-tests callable + Reyden nightly as thin wrapper" refactor (ce48179) leaked Reyden-specific routing (use_reyden_for_execution, EXECUTION_HTTP_PATH, CREATE TABLE adbc_testing_mutable, isReadOnly / mutableTable connection.json fields, the publish-dashboard job, …) into the generic e2e-tests workflow. With the C# read-only test adaptation reverted in 37243c5 those knobs no longer have a consumer, leaving e2e-tests.yml carrying dead Reyden code on every PR run. This commit moves all Reyden-specific scaffolding out of e2e-tests.yml and back into reyden-rest-nightly.yml as a self-contained workflow: - e2e-tests.yml restored byte-for-byte to origin/main (no workflow_call, no Reyden routing, no dashboard, no TRX upload, no e2e_only filter). - reyden-rest-nightly.yml rebuilt from the pre-refactor 6a9501f version with the additions that lived briefly in e2e-tests.yml: - CREATE TABLE adbc_testing_mutable in the seed step (Reyden supports INSERT but not CREATE TABLE). - isReadOnly / mutableTable fields in the generated connection.json. - --filter FullyQualifiedName!~Tests.Unit on the test invocation (nightly only runs E2E). - TRX logger + upload-artifact step. - publish-dashboard job that parses TRX and pushes to gh-pages (RUN_READ_ONLY is hard-coded 'true' since this workflow always exercises the read-only Reyden warehouse). - permissions: contents: write at workflow scope for the gh-pages push. Helper scripts (.github/scripts/parse-trx-to-json.py, .github/scripts/update-e2e-dashboard.py) and .github/e2e-dashboard/ index.html are unchanged; the consolidated publish-dashboard job calls them in place. Trade-off accepted: ~180 lines of duplication between the two workflow files. Justified because Reyden's needs (split setup/execution warehouses, pre-created mutable table, read-only-tagged dashboard) don't fit a generic e2e-tests parameterization cleanly, and the called-workflow extensibility model GitHub Actions provides is too limited for the kind of hooks Reyden would need. Co-authored-by: Isaac
The "By error (root cause)" table capped the Tests column at 8 names and showed a static "+N more". Turn that into a clickable toggle that expands the full deduped test list and collapses back to "show fewer". Co-authored-by: Isaac
Three failure modes are confirmed Reyden backend limitations, not driver bugs, but the categorizer was filing two of them under "Real issue / to investigate" via the generic value-mismatch bucket: - rows_affected = -1 on INSERT/UPDATE (known Reyden bug) — surfaced as "Expected: 1 / Actual: -1" - no hive_metastore catalog — surfaced as Expected "hive_metastore" / Actual "main" Add two specific signatures ahead of the generic assertion bucket and map them to CAT_REYDEN_GAP. (SHOW COLUMNS and PK/FK metadata were already classified as gaps.) On the latest run this moves 71 failures out of "Real issue" (80 -> 9), leaving only genuine driver-side items (syntax error, col_0 alias, AfterInsert 2/0, cast mismatch). Co-authored-by: Isaac
The "Real issue / to investigate" bucket now contains only the confirmed Reyden SEA backend bugs filed under Epic SC-222102, each given its own signature: - SHOW SCHEMAS/TABLES IN ALL CATALOGS rejected (SC-233357) - bare SET rejected (SC-233354) - result schema omits column aliases / col_0 (SC-233356) - ANSI strict-cast on heterogeneous ARRAY/MAP (SC-233355) Generic value-mismatch and DML/DDL-rejected are remapped to the expected Reyden capability-gap bucket (e.g. the non-reproducible write-not-persisted case), since on this Reyden-only nightly everything outside the four tracked bugs is a known limitation. Latest run: Real 9 -> 8 (4 tracked defects), gaps 107 -> 108. Co-authored-by: Isaac
The known Reyden rows_affected=-1 backend bug now has a tracking ticket; surface it in the signature description. Stays in the expected-gap bucket. Co-authored-by: Isaac
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Provision per-run schemawith[USER_ERROR] Unsupported CREATE type: SCHEMA/api/2.1/unity-catalog/schemas) for both schema creation and deletionCREATE TABLE/INSERTDML which Reyden does supportTest plan
reyden-rest-nightlyviaworkflow_dispatchafter merge to verify schema is created and tests runFixes first-run failure from #504.
This PR was created with GitHub MCP.