Skip to content

feat(transformations): add external data sources client API#2715

Draft
Khyat-Cognite wants to merge 2 commits into
feat/onelake-edc-data-modelfrom
feat/onelake-edc-api
Draft

feat(transformations): add external data sources client API#2715
Khyat-Cognite wants to merge 2 commits into
feat/onelake-edc-data-modelfrom
feat/onelake-edc-api

Conversation

@Khyat-Cognite

@Khyat-Cognite Khyat-Cognite commented Jul 9, 2026

Copy link
Copy Markdown

CDF-28251

Summary

Slice 2 of 3 — splits PR #2713.

Adds the client API for managing Fabric OneLake external data sources under client.transformations.external_data_sources. Depends on PR #2714 (data classes).

What's in this PR

Production code

  • cognite/client/_api/transformations/external_data.py — async list, upsert, delete, verify_usability
  • cognite/client/_sync_api/transformations/external_data.py — generated sync wrapper
  • cognite/client/_api/transformations/__init__.py + sync __init__.py — wire external_data_sources sub-API
  • cognite/client/_cognite_client.py — Sphinx doc-build stub
  • cognite/client/testing.pyCogniteClientMock / AsyncCogniteClientMock autospec wiring

Tests (co-located)

  • tests/tests_unit/test_api/test_transformation_external_data.py — HTTP mock tests for all four endpoints
  • tests/tests_integration/test_api/test_transformations/test_external_data.py — Fabric-gated lifecycle (upsert → list → verify_usability → delete); skips without FABRIC_* env vars

SDK surface

client.transformations.external_data_sources.list()
client.transformations.external_data_sources.upsert(source)
client.transformations.external_data_sources.delete("my-source")
client.transformations.external_data_sources.verify_usability("my-source")

Upsert always sends "format": "one_lake".

What came before

PR Status Scope
1 — Data model #2714 Data classes, IAM capability, serialization unit tests

What comes next

PR Branch Scope
3 — Docs feat/onelake-edc-docs User guide, quickstart, API reference page

Merge #2714 first, then this PR (rebase onto master after #2714 lands, or merge stacked).

Test plan

  • pytest tests/tests_unit/test_api/test_transformation_external_data.py
  • verify-sync-codegen CI green
  • Integration tests skip without Fabric credentials

Parent

Replaces the API portion of #2713. Close #2713 after all three slices are merged to master.

@codecov

codecov Bot commented Jul 9, 2026

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 82.81250% with 33 lines in your changes missing coverage. Please review.
✅ Project coverage is 93.70%. Comparing base (f7888df) to head (d0ccc58).
⚠️ Report is 1 commits behind head on feat/onelake-edc-data-model.

Files with missing lines Patch % Lines
...est_api/test_transformations/test_external_data.py 37.73% 33 Missing ⚠️
Additional details and impacted files
@@                      Coverage Diff                      @@
##           feat/onelake-edc-data-model    #2715    +/-   ##
=============================================================
  Coverage                        93.69%   93.70%            
=============================================================
  Files                              506      510     +4     
  Lines                            51373    51573   +200     
=============================================================
+ Hits                             48133    48325   +192     
- Misses                            3240     3248     +8     
Files with missing lines Coverage Δ
cognite/client/_api/transformations/__init__.py 75.70% <100.00%> (+0.46%) ⬆️
...gnite/client/_api/transformations/external_data.py 100.00% <100.00%> (ø)
cognite/client/_cognite_client.py 95.83% <100.00%> (+0.01%) ⬆️
...gnite/client/_sync_api/transformations/__init__.py 94.64% <100.00%> (+0.19%) ⬆️
.../client/_sync_api/transformations/external_data.py 100.00% <100.00%> (ø)
cognite/client/testing.py 100.00% <100.00%> (ø)
...unit/test_api/test_transformation_external_data.py 100.00% <100.00%> (ø)
...est_api/test_transformations/test_external_data.py 37.73% <37.73%> (ø)

... and 9 files with indirect coverage changes

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@Khyat-Cognite
Khyat-Cognite force-pushed the feat/onelake-edc-data-model branch from 98e71c4 to ad1e459 Compare July 9, 2026 08:53
@Khyat-Cognite
Khyat-Cognite force-pushed the feat/onelake-edc-api branch from 746eaa8 to 8a93be4 Compare July 9, 2026 08:53
@Khyat-Cognite
Khyat-Cognite requested a review from a team July 9, 2026 09:00
@Khyat-Cognite
Khyat-Cognite force-pushed the feat/onelake-edc-api branch 2 times, most recently from 180cf64 to caaafc9 Compare July 9, 2026 10:36
@Khyat-Cognite

Copy link
Copy Markdown
Author

/gemini review

@gemini-code-assist gemini-code-assist Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request introduces the TransformationExternalDataAPI (and its synchronous counterpart) to manage Fabric OneLake external data sources for transformations, supporting listing, upserting, deleting, and verifying usability. The review feedback suggests improving type safety by adding @overload signatures to the upsert method and ensuring integration tests are not silently skipped in CI environments by checking for existing environment variables before looking for a .env file.

Comment thread cognite/client/_api/transformations/external_data.py
Comment thread cognite/client/_api/transformations/external_data.py
@Khyat-Cognite
Khyat-Cognite marked this pull request as ready for review July 14, 2026 06:51
@Khyat-Cognite
Khyat-Cognite requested review from a team as code owners July 14, 2026 06:51

@gemini-code-assist gemini-code-assist Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request introduces the TransformationExternalDataAPI (and its sync counterpart) to manage Fabric OneLake external data sources for transformations, including list, upsert, delete, and usability verification endpoints. The feedback suggests adding support for the ignore_unknown_ids parameter in the delete method to maintain consistency with the rest of the SDK, updating the unit tests accordingly, and refactoring the integration test teardown to avoid masking potential setup failures in the finally block.

Comment thread cognite/client/_api/transformations/external_data.py Outdated
Comment thread tests/tests_unit/test_api/test_transformation_external_data.py
Comment thread tests/tests_integration/test_api/test_transformations/test_external_data.py Outdated
Comment thread cognite/client/_api/transformations/external_data.py
Comment thread cognite/client/_api/transformations/external_data.py
@Khyat-Cognite Khyat-Cognite self-assigned this Jul 15, 2026
@Khyat-Cognite
Khyat-Cognite requested a review from a team July 15, 2026 05:20
@Khyat-Cognite
Khyat-Cognite force-pushed the feat/onelake-edc-api branch 4 times, most recently from fc22e73 to cbe7e16 Compare July 16, 2026 09:41
Wire async and sync TransformationExternalDataAPI (list, upsert, delete,
verify_usability), client mocks, and unit plus Fabric-gated integration tests.
Uses OneLakeExternalDataSourceWrite for registration and documents
as_write(client_secret=...) for credential re-upsert from listed sources.
@Khyat-Cognite
Khyat-Cognite force-pushed the feat/onelake-edc-api branch from cbe7e16 to d564e77 Compare July 16, 2026 10:12
Match hosted extractors pattern after making the read base an ABC.
@Khyat-Cognite
Khyat-Cognite marked this pull request as draft July 16, 2026 11:45
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants