feature(connectors/bright_data_scrape): add Bright Data Web Scraper C…#41
Open
adomhamza wants to merge 6 commits into
Open
feature(connectors/bright_data_scrape): add Bright Data Web Scraper C…#41adomhamza wants to merge 6 commits into
adomhamza wants to merge 6 commits into
Conversation
Author
Contributor
There was a problem hiding this comment.
Pull request overview
This PR adds a new Connector SDK example (bright_data_scrape) that syncs web scraping results from Bright Data’s Web Scraper API into a Fivetran destination, including helper modules for scraping/polling, validation, and result flattening.
Changes:
- Introduces a new Bright Data scraper connector (
connector.py) with schema, URL parsing/validation, state checkpointing, and upsert logic. - Adds helper modules for API job triggering + snapshot polling (
helpers/scrape.py), result flattening/field discovery (helpers/data_processing.py), and configuration validation (helpers/validation.py). - Adds end-user documentation (
README.md) and a placeholderconfiguration.json.
Reviewed changes
Copilot reviewed 7 out of 7 changed files in this pull request and generated 18 comments.
Show a summary per file
| File | Description |
|---|---|
| bright_data_scrape/connector.py | Core Connector SDK implementation (schema/update, URL parsing, sync/upsert, state). |
| bright_data_scrape/helpers/scrape.py | Bright Data API trigger + snapshot polling + response parsing utilities. |
| bright_data_scrape/helpers/data_processing.py | Flattening and row-shaping utilities for scraped results. |
| bright_data_scrape/helpers/validation.py | Configuration validation helper. |
| bright_data_scrape/helpers/init.py | Helper re-exports for simplified imports. |
| bright_data_scrape/configuration.json | Placeholder configuration values (no secrets). |
| bright_data_scrape/README.md | Example documentation (setup/configuration/usage). |
Comment on lines
+32
to
+39
| def schema(configuration: dict): | ||
| """ | ||
| Define the schema function which lets you configure the schema your connector delivers. | ||
| See the technical reference documentation for more details on the schema function: | ||
| https://fivetran.com/docs/connectors/connector-sdk/technical-reference#schema | ||
| Args: | ||
| configuration: a dictionary that holds the configuration settings for the connector. | ||
| """ |
Comment on lines
+55
to
+66
| def update(configuration: dict, state: dict): | ||
| """ | ||
| Define the update function which lets you configure how your connector fetches data. | ||
| See the technical reference documentation for more details on the update function: | ||
| https://fivetran.com/docs/connectors/connector-sdk/technical-reference#update | ||
| Args: | ||
| configuration: a dictionary that holds the configuration settings for the connector. | ||
| state: a dictionary that holds the state of the connector. | ||
| """ | ||
| # Validate the configuration to ensure it contains all required values | ||
| validate_configuration(configuration=configuration) | ||
|
|
Comment on lines
+195
to
+199
| # Update state with sync information | ||
| state["last_scrape_urls"] = valid_urls | ||
| state["last_scrape_count"] = len(processed_results) | ||
|
|
||
| op.checkpoint(state) |
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
Contributor
|
Hi @adomhamza , Could you please rebase your PR so that the latest checks can run and pass Thank you! |
Author
Done. |
Comment on lines
+15
to
+17
| ## Getting started | ||
|
|
||
| Refer to the [Connector SDK Setup Guide](https://fivetran.com/docs/connectors/connector-sdk/setup-guide) to get started. |
Contributor
There was a problem hiding this comment.
Add this instead
Getting started
Refer to the Connector SDK Setup Guide to get started.
To initialize a new Connector SDK project using this connector as a starting point, run:
fivetran init --template bright_data_scrape
fivetran init initializes a new Connector SDK project by setting up the project structure, configuration files, and a connector you can run immediately with fivetran debug. For more information on fivetran init, refer to the Connector SDK init documentation.
Include the following note only if the connector requires a configuration.json file to run.
Note: Ensure you have updated the
configuration.jsonfile with the necessary parameters before runningfivetran debug. See the Configuration file section for details on the required configuration parameters.
Comment on lines
+20
to
+23
| # Import required classes from fivetran_connector_sdk | ||
| from fivetran_connector_sdk import Connector | ||
| from fivetran_connector_sdk import Logging as log | ||
| from fivetran_connector_sdk import Operations as op |
Contributor
There was a problem hiding this comment.
# For reading configuration from a JSON file
import json
# Import required classes from fivetran_connector_sdk
from fivetran_connector_sdk import Connector
# For enabling Logs in your connector code
from fivetran_connector_sdk import Logging as log
# For supporting Data operations like upsert(), update(), delete() and checkpoint()
from fivetran_connector_sdk import Operations as op
Please add these comments
Co-authored-by: Jenas Anton Vimal <jenas.vimal@fivetran.com>
Co-authored-by: Jenas Anton Vimal <jenas.vimal@fivetran.com>
Co-authored-by: Jenas Anton Vimal <jenas.vimal@fivetran.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
…onnector.
Description of Change
/datasets/v3/trigger, polls snapshot completion, flattens nested JSON responses, and upserts results to ascrape_resultstable with primary key(url, result_index).schema(), scalarop.upsert()calls, retry logic with exponential backoff, configuration validation, URL validation, state checkpointing, and key-presence checks during snapshot/JSONL data extraction. No extra third-party dependencies beyond the pre-installed Fivetran environment.Features
log.warning()before syncFiles added
connector.py—schema(),update(), URL parsing/validation, result processing, upsert logic, debug entry pointhelpers/scrape.py— job triggering, snapshot polling, retry logic, response parsinghelpers/data_processing.py— flattening and field discovery utilitieshelpers/validation.py— configuration validationconfiguration.json— placeholder values (no secrets)README.md— setup, configuration, and usage documentationTesting
Checklist
Some tips and links to help validate your PR:
fivetran debugcommand.