feat(transformations): add external data sources client API#2715
feat(transformations): add external data sources client API#2715Khyat-Cognite wants to merge 2 commits into
Conversation
Codecov Report❌ Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## feat/onelake-edc-data-model #2715 +/- ##
=============================================================
Coverage 93.69% 93.70%
=============================================================
Files 506 510 +4
Lines 51373 51573 +200
=============================================================
+ Hits 48133 48325 +192
- Misses 3240 3248 +8
🚀 New features to boost your workflow:
|
98e71c4 to
ad1e459
Compare
746eaa8 to
8a93be4
Compare
180cf64 to
caaafc9
Compare
|
/gemini review |
There was a problem hiding this comment.
Code Review
This pull request introduces the TransformationExternalDataAPI (and its synchronous counterpart) to manage Fabric OneLake external data sources for transformations, supporting listing, upserting, deleting, and verifying usability. The review feedback suggests improving type safety by adding @overload signatures to the upsert method and ensuring integration tests are not silently skipped in CI environments by checking for existing environment variables before looking for a .env file.
There was a problem hiding this comment.
Code Review
This pull request introduces the TransformationExternalDataAPI (and its sync counterpart) to manage Fabric OneLake external data sources for transformations, including list, upsert, delete, and usability verification endpoints. The feedback suggests adding support for the ignore_unknown_ids parameter in the delete method to maintain consistency with the rest of the SDK, updating the unit tests accordingly, and refactoring the integration test teardown to avoid masking potential setup failures in the finally block.
fc22e73 to
cbe7e16
Compare
Wire async and sync TransformationExternalDataAPI (list, upsert, delete, verify_usability), client mocks, and unit plus Fabric-gated integration tests. Uses OneLakeExternalDataSourceWrite for registration and documents as_write(client_secret=...) for credential re-upsert from listed sources.
cbe7e16 to
d564e77
Compare
Match hosted extractors pattern after making the read base an ABC.
CDF-28251
Summary
Slice 2 of 3 — splits PR #2713.
Adds the client API for managing Fabric OneLake external data sources under
client.transformations.external_data_sources. Depends on PR #2714 (data classes).What's in this PR
Production code
cognite/client/_api/transformations/external_data.py— asynclist,upsert,delete,verify_usabilitycognite/client/_sync_api/transformations/external_data.py— generated sync wrappercognite/client/_api/transformations/__init__.py+ sync__init__.py— wireexternal_data_sourcessub-APIcognite/client/_cognite_client.py— Sphinx doc-build stubcognite/client/testing.py—CogniteClientMock/AsyncCogniteClientMockautospec wiringTests (co-located)
tests/tests_unit/test_api/test_transformation_external_data.py— HTTP mock tests for all four endpointstests/tests_integration/test_api/test_transformations/test_external_data.py— Fabric-gated lifecycle (upsert → list → verify_usability → delete); skips withoutFABRIC_*env varsSDK surface
Upsert always sends
"format": "one_lake".What came before
What comes next
feat/onelake-edc-docsMerge #2714 first, then this PR (rebase onto
masterafter #2714 lands, or merge stacked).Test plan
pytest tests/tests_unit/test_api/test_transformation_external_data.pyverify-sync-codegenCI greenParent
Replaces the API portion of #2713. Close #2713 after all three slices are merged to
master.