Skip to content

feature(connectors/bright_data_unlocker): add Bright Data Web Unlocke…#40

Open
adomhamza wants to merge 3 commits into
fivetran:mainfrom
adomhamza:feat/bright_data_unlocker
Open

feature(connectors/bright_data_unlocker): add Bright Data Web Unlocke…#40
adomhamza wants to merge 3 commits into
fivetran:mainfrom
adomhamza:feat/bright_data_unlocker

Conversation

@adomhamza

@adomhamza adomhamza commented Jul 2, 2026

Copy link
Copy Markdown

…r connector

Description of Change

  • Adds a new Connector SDK example that syncs web page content from Bright Data's Web Unlocker API to Fivetran.
  • Supports single or multiple URLs (comma-separated, newline-separated, or JSON array string), flattens nested JSON responses, and upserts results to an unlocker_results table with primary key (requested_url, result_index).
  • Follows Connector SDK best practices: explicit PK column types in schema(), scalar op.upsert() calls, retry logic with exponential backoff for transient HTTP errors, configuration validation, and state checkpointing.

Files added

  • connector.pyschema(), update(), URL parsing, debug entry point
  • helpers/unlocker.py — Web Unlocker API requests and retry logic
  • helpers/data_processing.py — flattening, field discovery, PK-safe upserts
  • helpers/validation.py — configuration validation
  • configuration.json — placeholder values (no secrets)
  • README.md — setup, configuration, and usage documentation

Testing

image1 image3 image2

Checklist

Some tips and links to help validate your PR:

  • Tested the connector with fivetran debug command.
  • Added/Updated example-specific README.md file, see the README template for the required structure and guidelines.
  • Followed Python Coding Standards, refer here

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR adds a new Connector SDK example (bright_data_unlocker) that fetches content via Bright Data’s Web Unlocker API, flattens/normalizes responses, and upserts results into an unlocker_results table.

Changes:

  • Introduces a new connector.py with schema() / update(), URL parsing, and a local debug entrypoint.
  • Adds helper modules for Unlocker API requests + retries, response flattening/upsert utilities, and configuration validation.
  • Adds example configuration.json and an example-specific README.md.

Reviewed changes

Copilot reviewed 7 out of 7 changed files in this pull request and generated 14 comments.

Show a summary per file
File Description
bright_data_unlocker/connector.py New connector entrypoint with schema, update flow, URL parsing, and debug block.
bright_data_unlocker/helpers/unlocker.py Web Unlocker API client with retry/backoff and response normalization.
bright_data_unlocker/helpers/data_processing.py Flattening and PK-safe result normalization + upsert routine.
bright_data_unlocker/helpers/validation.py Configuration validation helper.
bright_data_unlocker/helpers/init.py Helper exports for simplified imports.
bright_data_unlocker/configuration.json Example configuration file committed with placeholders/defaults.
bright_data_unlocker/README.md Usage and setup documentation for the example connector.

Comment thread bright_data_unlocker/configuration.json
Comment thread bright_data_unlocker/connector.py
Comment thread bright_data_unlocker/connector.py
Comment thread bright_data_unlocker/connector.py
Comment thread bright_data_unlocker/connector.py
Comment thread bright_data_unlocker/README.md
Comment thread bright_data_unlocker/README.md
Comment thread bright_data_unlocker/README.md
Comment thread bright_data_unlocker/README.md Outdated
Comment thread bright_data_unlocker/README.md
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
@fivetran-JenasVimal

Copy link
Copy Markdown
Contributor

Hi @adomhamza ,

Could you please rebase your PR so that the latest checks can run and pass?

Thank you!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants