feature(connectors/bright_data_unlocker): add Bright Data Web Unlocke…

bright_data_unlocker/README.md

@@ -0,0 +1,92 @@
+# Bright Data Web Unlocker Connector Example
+
+## Connector overview
+
+This connector syncs web page content from Bright Data's Web Unlocker API to your Fivetran destination. It fetches unlocked page data for one or more URLs, flattens nested JSON responses, and upserts results to an `unlocker_results` table.
+
+## Requirements
+
+- [Supported Python versions](https://github.com/fivetran/connector_sdk/blob/main/README.md#requirements)
+- Operating system:
+  - Windows: 10 or later (64-bit only)
+  - macOS: 13 (Ventura) or later (Apple Silicon [arm64] or Intel [x86_64])
+  - Linux: Distributions such as Ubuntu 20.04 or later, Debian 10 or later, or Amazon Linux 2 or later (arm64 or x86_64)
+
+## Getting started
+
+Refer to the [Connector SDK Setup Guide](https://fivetran.com/docs/connector-sdk/setup-guide) to get started.
+
+Note: Ensure that the `configuration.json` file is not checked into version control to protect sensitive information.
+
+## Features
+
+- Fetches unlocked web content via Bright Data Web Unlocker API
+- Supports multiple URL input formats (single URL, comma-separated, newline-separated, JSON array string)
+- Flattens nested JSON structures for analysis
+- Dynamically discovers fields from API responses
+- Includes retry logic with exponential backoff for transient API errors
+- Checkpoints state after each sync
+
+## Configuration file
+
+```json
+{
+  "api_token": "<YOUR_BRIGHT_DATA_API_TOKEN>",
+  "unlocker_url": "<YOUR_UNLOCKER_URL>",
+  "country": "us",
+  "data_format": "markdown",
+  "zone": "web_unlocker1",
+  "format_param": "json"
+}
+```
+
+Configuration parameters:
+
+- `api_token` (required): Your Bright Data API token
+- `unlocker_url` (required): URL or URLs to fetch via the Web Unlocker
+- `zone` (optional): Bright Data unlocker zone identifier. Defaults to `web_unlocker1`
+- `country` (optional): ISO 3166-1 alpha-2 country code. Defaults to `us`
+- `method` (optional): HTTP method. Defaults to `GET`
+- `format_param` (optional): Response format (`json` or `html`). Defaults to `json`
+- `data_format` (optional): Content format such as `markdown` or `html`
+
+## Requirements file
+
+This connector does not require any additional Python packages beyond what is pre-installed in the Fivetran environment.
+
+Note: The `fivetran_connector_sdk:latest` and `requests:latest` packages are pre-installed in the Fivetran environment. To avoid dependency conflicts, do not declare them in your `requirements.txt`.
+
+## Authentication
+
+The Bright Data API uses Bearer token authentication. Obtain your API token from the Bright Data dashboard at https://brightdata.com/cp/setting/users.
+
+## Data handling
+
+1. Configuration validation — refer to `validate_configuration()` in `helpers/validation.py`
+2. URL parsing — refer to `parse_unlocker_urls()` in `connector.py`
+3. API requests — refer to `perform_web_unlocker()` in `helpers/unlocker.py`
+4. Result processing — refer to `process_unlocker_result()` in `helpers/data_processing.py`
+5. Data upsertion — refer to `process_and_upsert_results()`
+6. State checkpointing — refer to `op.checkpoint()` in `update()`
+
+## Error handling
+
+- Retry logic for transient HTTP errors (408, 429, 500, 502, 503, 504) with exponential backoff — refer to `_execute_unlocker_request()` in `helpers/unlocker.py`
+- Non-retryable failures raise `RuntimeError` with API error details
+- Primary key validation issues are logged once per sync via `log.warning()`
+
+## Tables created
+
+| Table Name          | Primary Key                    | Description                                              |
+|---------------------|--------------------------------|----------------------------------------------------------|
+| `UNLOCKER_RESULTS`  | `requested_url`, `result_index`  | Flattened web unlocker results for each requested URL.   |
+
+## Additional files
+
+- `helpers/validation.py` — Configuration parameter validation
+- `helpers/unlocker.py` — Bright Data Web Unlocker API interaction and retry logic
+- `helpers/data_processing.py` — Data flattening, field discovery, and upsert utilities
+
+## Additional considerations
+
+The examples provided are intended to help you effectively use Fivetran's Connector SDK. While we've tested the code, Fivetran cannot be held responsible for any unexpected or negative consequences that may arise from using these examples. For inquiries, please reach out to our Support team.

bright_data_unlocker/configuration.json

@@ -0,0 +1,8 @@
+{
+  "api_token": "<YOUR_BRIGHT_DATA_API_TOKEN>",
+  "unlocker_url": "<YOUR_UNLOCKER_URL>",
+  "country": "us",
+  "data_format": "markdown",
+  "zone": "web_unlocker1",
+  "format_param": "json"
+}

bright_data_unlocker/connector.py

@@ -0,0 +1,176 @@
+"""This connector syncs web unlocker results from Bright Data's Web Unlocker API to Fivetran.
+See the Technical Reference documentation
+(https://fivetran.com/docs/connectors/connector-sdk/technical-reference#update)
+and the Best Practices documentation
+(https://fivetran.com/docs/connectors/connector-sdk/best-practices) for details
+"""
+
+# For reading configuration from a JSON file
+import json
+
+# Helper functions for data processing, validation, and API interaction
+from helpers import (
+    collect_all_fields,
+    perform_web_unlocker,
+    process_and_upsert_results,
+    process_unlocker_result,
+    validate_configuration,
+)
+
+# For supporting Connector operations like Update() and Schema()
+from fivetran_connector_sdk import Connector
+
+# For enabling Logs in your connector code
+from fivetran_connector_sdk import Logging as log
+
+# For supporting Data operations like Upsert(), Update(), Delete() and checkpoint()
+from fivetran_connector_sdk import Operations as op
+
+__UNLOCKER_TABLE = "unlocker_results"
+
+
+def schema(configuration: dict):
+    """
+    Define the schema function which lets you configure the schema your connector delivers.
+    See the technical reference documentation for more details on the schema function:
+    https://fivetran.com/docs/connectors/connector-sdk/technical-reference#schema
+    Args:
+        configuration: a dictionary that holds the configuration settings for the connector.
+    """
+    return [
+        {
+            "table": __UNLOCKER_TABLE,
+            "primary_key": [
+                "requested_url",
+                "result_index",
+            ],
+            "columns": {
+                "requested_url": "STRING",
+                "result_index": "INT",
+            },
+        }
+    ]
+
+
+def update(configuration: dict, state: dict):
+    """
+    Define the update function which lets you configure how your connector fetches data.
+    See the technical reference documentation for more details on the update function:
+    https://fivetran.com/docs/connectors/connector-sdk/technical-reference#update
+    Args:
+        configuration: a dictionary that holds the configuration settings for the connector.
+        state: a dictionary that holds the state of the connector.
+    """
+    log.warning("Example: Connectors : Bright Data Web Unlocker")
+
+    validate_configuration(configuration=configuration)
+
+    new_state = dict(state) if state else {}
+
+    unlocker_url_input = configuration.get("unlocker_url", "")
+    urls = parse_unlocker_urls(unlocker_url_input)
+
+    if urls:
+        sync_unlocker_urls(configuration=configuration, urls=urls, state=new_state)
+
+    # Save the progress by checkpointing the state. This is important for ensuring that the sync
+    # process can resume from the correct position in case of next sync or interruptions.
+    # Learn more about how and where to checkpoint by reading our best practices documentation
+    # (https://fivetran.com/docs/connectors/connector-sdk/best-practices#largedatasetrecommendation).
+    op.checkpoint(state=new_state)
+
+
+def sync_unlocker_urls(configuration: dict, urls: list, state: dict):
+    """
+    Fetch unlocker results for the requested URLs and upsert them to Fivetran.
+    Args:
+        configuration: Configuration dictionary containing unlocker parameters.
+        urls: List of URLs to unlock/fetch.
+        state: Current connector state.
+    """
+    api_token = configuration.get("api_token")
+    country = configuration.get("country")
+    data_format = configuration.get("data_format")
+    format_param = configuration.get("format_param")
+    method = configuration.get("method") or "GET"
+    unlocker_zone = configuration.get("zone")
+
+    payload = urls if len(urls) > 1 else urls[0]
+    unlocker_results = perform_web_unlocker(
+        api_token=api_token,
+        url=payload,
+        zone=unlocker_zone,
+        country=country,
+        method=method,
+        format_param=format_param,
+        data_format=data_format,
+    )
+
+    if not isinstance(unlocker_results, list):
+        unlocker_results = [unlocker_results]
+
+    processed_results = []
+    for index, result in enumerate(unlocker_results):
+        requested_url = result.get("requested_url") if isinstance(result, dict) else None
+        if not requested_url:
+            requested_url = urls[index % len(urls)]
+        processed_results.append(process_unlocker_result(result, requested_url, index))
+
+    if not processed_results:
+        log.warning("No unlocker results returned from API")
+        return
+
+    log.info(f"Upserting {len(processed_results)} unlocker results to Fivetran")
+
+    all_fields = collect_all_fields(processed_results)
+    process_and_upsert_results(processed_results, all_fields, __UNLOCKER_TABLE)
+
+    state["last_unlocker_urls"] = urls
+    state["last_unlocker_count"] = len(processed_results)
+
+
+def parse_unlocker_urls(unlocker_url_input) -> list:
+    """
+    Normalize the unlocker_url configuration value into a list of URLs.
+    Args:
+        unlocker_url_input: The unlocker_url configuration value (various formats supported).
+    Returns:
+        list: List of normalized URL strings.
+    """
+    if not unlocker_url_input:
+        return []
+
+    if isinstance(unlocker_url_input, list):
+        return [
+            item.strip() for item in unlocker_url_input if isinstance(item, str) and item.strip()
+        ]
+
+    if isinstance(unlocker_url_input, str):
+        try:
+            parsed = json.loads(unlocker_url_input)
+            if isinstance(parsed, list):
+                return [item.strip() for item in parsed if isinstance(item, str) and item.strip()]
+            if isinstance(parsed, str) and parsed.strip():
+                return [parsed.strip()]
+        except (json.JSONDecodeError, TypeError):
+            pass
+
+        if "," in unlocker_url_input:
+            return [item.strip() for item in unlocker_url_input.split(",") if item.strip()]
+
+        if "\n" in unlocker_url_input:
+            return [item.strip() for item in unlocker_url_input.split("\n") if item.strip()]
+
+        return [unlocker_url_input.strip()] if unlocker_url_input.strip() else []
+
+    return []
+
+
+connector = Connector(update=update, schema=schema)
+
+
+if __name__ == "__main__":
+    with open("configuration.json", "r") as f:
+        configuration = json.load(f)
+
+    connector.debug(configuration=configuration)

bright_data_unlocker/helpers/__init__.py

@@ -0,0 +1,22 @@
+"""Helper module exports for easy importing."""
+
+# For exporting the helper functions
+from .data_processing import (
+    collect_all_fields,
+    process_and_upsert_results,
+    process_unlocker_result,
+)
+
+# For performing the web unlocker
+from .unlocker import perform_web_unlocker
+
+# For validating the configuration
+from .validation import validate_configuration
+
+__all__ = [
+    "collect_all_fields",
+    "perform_web_unlocker",
+    "process_and_upsert_results",
+    "process_unlocker_result",
+    "validate_configuration",
+]