feature(connectors/bright_data_unlocker): add Bright Data Web Unlocke…#40
Open
adomhamza wants to merge 3 commits into
Open
feature(connectors/bright_data_unlocker): add Bright Data Web Unlocke…#40adomhamza wants to merge 3 commits into
adomhamza wants to merge 3 commits into
Conversation
Contributor
There was a problem hiding this comment.
Pull request overview
This PR adds a new Connector SDK example (bright_data_unlocker) that fetches content via Bright Data’s Web Unlocker API, flattens/normalizes responses, and upserts results into an unlocker_results table.
Changes:
- Introduces a new
connector.pywithschema()/update(), URL parsing, and a local debug entrypoint. - Adds helper modules for Unlocker API requests + retries, response flattening/upsert utilities, and configuration validation.
- Adds example
configuration.jsonand an example-specificREADME.md.
Reviewed changes
Copilot reviewed 7 out of 7 changed files in this pull request and generated 14 comments.
Show a summary per file
| File | Description |
|---|---|
| bright_data_unlocker/connector.py | New connector entrypoint with schema, update flow, URL parsing, and debug block. |
| bright_data_unlocker/helpers/unlocker.py | Web Unlocker API client with retry/backoff and response normalization. |
| bright_data_unlocker/helpers/data_processing.py | Flattening and PK-safe result normalization + upsert routine. |
| bright_data_unlocker/helpers/validation.py | Configuration validation helper. |
| bright_data_unlocker/helpers/init.py | Helper exports for simplified imports. |
| bright_data_unlocker/configuration.json | Example configuration file committed with placeholders/defaults. |
| bright_data_unlocker/README.md | Usage and setup documentation for the example connector. |
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
Contributor
|
Hi @adomhamza , Could you please rebase your PR so that the latest checks can run and pass? Thank you! |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
…r connector
Description of Change
unlocker_resultstable with primary key(requested_url, result_index).schema(), scalarop.upsert()calls, retry logic with exponential backoff for transient HTTP errors, configuration validation, and state checkpointing.Files added
connector.py—schema(),update(), URL parsing, debug entry pointhelpers/unlocker.py— Web Unlocker API requests and retry logichelpers/data_processing.py— flattening, field discovery, PK-safe upsertshelpers/validation.py— configuration validationconfiguration.json— placeholder values (no secrets)README.md— setup, configuration, and usage documentationTesting
Checklist
Some tips and links to help validate your PR:
fivetran debugcommand.