diff --git a/examples/README.md b/examples/README.md index 760efd5f9..a2914e7e6 100644 --- a/examples/README.md +++ b/examples/README.md @@ -17,6 +17,8 @@ These are graded examples designed to help you get started with the Connector SD - [configuration](https://github.com/fivetran/connector_sdk/tree/main/examples/quickstart_examples/configuration) - This example shows how to use secrets. +- [configuration_form](https://github.com/fivetran/connector_sdk/tree/main/examples/quickstart_examples/configuration_form) - This example shows how to define a connector setup form using `ConfigurationForm`, `form_field`, and `Test`. It demonstrates all available field types (plain text, password, dropdown, toggle, and descriptive dropdown) and how to register a connection test. + - [multiple_code_files](https://github.com/fivetran/connector_sdk/tree/main/examples/quickstart_examples/multiple_code_files_with_sub_directory_structure) - This example shows how you can write a complex connector comprising multiple `.py` files. - [using_pd_dataframes](https://github.com/fivetran/connector_sdk/tree/main/examples/quickstart_examples/using_pd_dataframes) - This example shows the use of Pandas DataFrames to manipulate data prior to sending to Fivetran. diff --git a/examples/quickstart_examples/configuration_form/README.md b/examples/quickstart_examples/configuration_form/README.md new file mode 100644 index 000000000..ce513e8e0 --- /dev/null +++ b/examples/quickstart_examples/configuration_form/README.md @@ -0,0 +1,76 @@ +# Configuration form example + +## Connector overview + +This example demonstrates how to define a connector setup form using `ConfigurationForm`, `form_field`, and `Test` from the Fivetran Connector SDK. It covers plain text, password, dropdown, and toggle fields, including dropdown options with labels and descriptions. It also shows how to register a connection test that Fivetran runs when the user clicks **Test Connection** during setup. + +The API fields (`api_base_url`, `api_key`) in the configuration form illustrate how a real connector would collect connection details. The `configuration.json` file provided contains sample values for running the connector locally. + +Refer to `def configuration_form()` and `def connection_test()` in `connector.py` for the main setup form and test implementation. + +## Example UI + +### Configuration form + +![Configuration form UI](./images/configuration_form.png) + +### Setup tests + +![Setup tests UI](./images/setup_tests.png) + +## Requirements + +- [Supported Python versions](https://github.com/fivetran/fivetran_connector_sdk/blob/main/README.md#requirements) +- Operating system: + - Windows: 10 or later (64-bit only) + - macOS: 13 (Ventura) or later (Apple Silicon [arm64] or Intel [x86_64]) + - Linux: Distributions such as Ubuntu 20.04 or later, Debian 10 or later, or Amazon Linux 2 or later (arm64 or x86_64) + +## Getting started + +Refer to the [Connector SDK Setup Guide](https://fivetran.com/docs/connector-sdk/setup-guide) to get started. + +For available CLI commands, refer to the [Connector SDK Commands](https://fivetran.com/docs/connector-sdk/connector-development-and-configuration/connector-sdk-commands) reference. + +1. Run the `fivetran configuration` command to interactively provide values for each form field. This generates (or overrides) a `configuration.json` file in the project directory. + + ```bash + fivetran configuration + ``` + +2. Optionally, run setup tests to validate your inputs without running a full sync. + + ```bash + fivetran configuration --test + ``` + +3. Run the connector locally using the generated `configuration.json`. + + ```bash + fivetran debug --configuration configuration.json + ``` + +## Features + +- Demonstrates form field types: `TextField` (plain text and password variants), `DropdownField`, and `ToggleField` +- Registers a connection test function that Fivetran calls by its `__name__` during connector setup +- Shows how to read and use configuration form values inside `update()` +- Uses the `data_range` dropdown with option labels and descriptions to control how much sample data the sync reads +- Optionally logs extraction volume when the metrics toggle is enabled +- Supports the `fivetran configuration` command, which interactively prompts for each form field value and generates (or overrides) `configuration.json` — the resulting file can then be used with `fivetran debug --configuration configuration.json` to run the connector locally or with `fivetran deploy` to deploy it. Setup tests registered via `add_test()` can be run independently using `fivetran configuration --test`, which is useful for validating credentials, field values, or connection health without running a full sync + +## Requirements file + +This connector has no third-party dependencies and does not include a `requirements.txt` file. + +> Note: The `fivetran_connector_sdk:latest` and `requests:latest` packages are pre-installed in the Fivetran environment. To avoid dependency conflicts, do not declare them in your `requirements.txt`. + +## Data handling + +The example uses the JSONPlaceholder API to illustrate the sync pattern. Configuration values from the form are read in `update()` to control sync behavior. `data_range` limits how many records are fetched for the demo (`last_7_days`, `last_30_days`, or `all_time`), `batch_size` shows how a page limit would be applied, and `enable_metrics` controls whether extraction stats are logged. + +Refer to `def update()` for details. + +## Additional considerations + +The examples provided are intended to help you effectively use Fivetran's Connector SDK. While we've tested the code, Fivetran cannot be held responsible for any unexpected or negative consequences that may arise from using these examples. For inquiries, please reach out to our Support team. diff --git a/examples/quickstart_examples/configuration_form/connector.py b/examples/quickstart_examples/configuration_form/connector.py new file mode 100644 index 000000000..7d152f295 --- /dev/null +++ b/examples/quickstart_examples/configuration_form/connector.py @@ -0,0 +1,271 @@ +# This is a simple example for how to use ConfigurationForm, form_field, and Test +# with the fivetran_connector_sdk module to build a connector with a setup form. +# It shows all available field types (TextField, DropdownField, ToggleField) +# and how to register a setup test that Fivetran runs when the user clicks "Test Connection". +# See the Technical Reference documentation (https://fivetran.com/docs/connectors/connector-sdk/technical-reference#update) +# and the Best Practices documentation (https://fivetran.com/docs/connectors/connector-sdk/best-practices) for details. + +# Import required classes from fivetran_connector_sdk +# For supporting Connector operations like Update() and Schema() +from fivetran_connector_sdk import Connector + +# For enabling Logs in your connector code +from fivetran_connector_sdk import Logging as log + +# For supporting Data operations like Upsert(), Update(), Delete() and checkpoint() +from fivetran_connector_sdk import Operations as op + +# For building the setup form shown to users when configuring the connector in Fivetran +from fivetran_connector_sdk import ConfigurationForm + +# For defining individual form fields (text inputs, dropdowns, toggles, etc.) +from fivetran_connector_sdk import form_field + +# For returning success/failure responses from setup test functions +from fivetran_connector_sdk import Test + +# For reading configuration from the local configuration.json file during debug runs +import json +import os +import re + +# For making HTTP requests to the example API in the connection test and sync logic +import requests + +DATA_RANGE_LIMITS = { + "last_7_days": 7, + "last_30_days": 30, + "all_time": None, +} + + +def configuration_form(): + """ + Define the setup form shown to users when configuring this connector in Fivetran. + Add fields using add_field() and register connection tests using add_test(). + Fivetran calls this method when rendering the connector setup UI. + Returns: + ConfigurationForm: the completed form with fields and tests. + """ + config_form = ConfigurationForm() + + log.info("Example: QuickStart Examples - Configuration Form start") + + # Plain text field — visible input, suitable for non-sensitive values like URLs + config_form.add_field( + form_field.TextField( + name="api_base_url", + label="API Base URL", + description="The base URL for your REST API endpoint.", + required=True, + field_type=form_field.TextField.plain_text, + placeholder="https://api.example.com/v1", + ) + ) + + # Password field — masked input, suitable for secrets like API keys + config_form.add_field( + form_field.TextField( + name="api_key", + label="API Key", + description="Your API authentication key. This value is stored securely.", + required=True, + field_type=form_field.TextField.password, + placeholder="your_api_key_here", + ) + ) + + # Dropdown field — lets the user select one option from a fixed list + config_form.add_field( + form_field.DropdownField( + name="batch_size", + label="Batch Size", + description="Number of records to fetch per API request.", + fields=[ + form_field.DropdownFieldParam(value=10), + form_field.DropdownFieldParam(value=100), + form_field.DropdownFieldParam(value=500), + ], + required=True, + ) + ) + + # Toggle field — boolean on/off switch + config_form.add_field( + form_field.ToggleField( + name="enable_metrics", + label="Enable Metrics", + description="Log extraction volume metrics (record count and bytes) during each sync.", + ) + ) + + # Dropdown — lets the user select one option from a fixed list but each option + # includes an explanation (label and description) + config_form.add_field( + form_field.DropdownField( + name="data_range", + label="Data Range", + fields=[ + form_field.DropdownFieldParam( + value="last_7_days", + label="Last 7 Days", + description="Fetches records created or updated in the last 7 days.", + ), + form_field.DropdownFieldParam( + value="last_30_days", + label="Last 30 Days", + description="Fetches records created or updated in the last 30 days.", + ), + form_field.DropdownFieldParam( + value="all_time", + label="All Time", + description="Fetches all available historical records.", + ), + ], + ) + ) + + # Register a setup test — Fivetran calls connection_test() when the user clicks "Test Connection" + config_form.add_test(label="Test connection", func=connection_test) + + return config_form + + +def connection_test(configuration: dict): + """ + Validate that the API credentials are correct and the endpoint is reachable. + Fivetran calls this function by its __name__ during connector setup. + Args: + configuration: a dictionary that holds the configuration settings for the connector. + Returns: + Test response indicating success or failure. + """ + test = Test() + api_base_url = configuration.get("api_base_url", "").rstrip("/") + api_key = configuration.get("api_key", "") + + if not api_base_url: + return test.failure("api_base_url is required.") + if not re.match(r"^https?://", api_base_url): + return test.failure("api_base_url must start with http:// or https://.") + if not api_key: + return test.failure("api_key is required.") + + try: + response = requests.get( + api_base_url, + headers={"Authorization": f"Bearer {api_key}"}, + timeout=10, + ) + if response.status_code == 200: + log.info("Connection test passed: API returned 200") + return test.success() + else: + log.warning(f"Connection test failed: API returned status {response.status_code}") + return test.failure(f"API returned status code: {response.status_code}") + except requests.exceptions.Timeout: + return test.failure("Connection timed out. Verify your api_base_url is reachable.") + except requests.exceptions.ConnectionError: + return test.failure("Could not connect to the API. Check your api_base_url.") + except requests.exceptions.RequestException as e: + return test.failure(f"Request failed: {e}") + + +def schema(configuration: dict): + """ + Define the schema function which lets you configure the schema your connector delivers. + See the technical reference documentation for more details on the schema function: + https://fivetran.com/docs/connector-sdk/technical-reference/connector-sdk-code/connector-sdk-methods#schema + Args: + configuration: a dictionary that holds the configuration settings for the connector. + """ + return [ + { + "table": "post", + "primary_key": ["id"], + } + ] + + +def update(configuration: dict, state: dict): + """ + Define the update function, which is a required function, and is called by Fivetran during each sync. + See the technical reference documentation for more details on the update function + https://fivetran.com/docs/connectors/connector-sdk/technical-reference#update + Args: + configuration: A dictionary containing connection details + state: A dictionary containing state information from previous runs + The state dictionary is empty for the first sync or for any full re-sync + """ + log.warning("Example: QuickStart Examples - Configuration Form") + + api_base_url = configuration.get("api_base_url", "").rstrip("/") + api_key = configuration.get("api_key", "") + batch_size = int(configuration.get("batch_size", 100)) + is_metrics_enabled = str(configuration.get("enable_metrics", "false")).lower() == "true" + data_range = configuration.get("data_range", "all_time") + max_records = DATA_RANGE_LIMITS.get(data_range) + cursor = 0 + + total_records = 0 + + while True: + remaining_records = None if max_records is None else max_records - total_records + if remaining_records is not None and remaining_records <= 0: + break + + current_batch_size = ( + batch_size if remaining_records is None else min(batch_size, remaining_records) + ) + + response = requests.get( + f"{api_base_url}/posts", + headers={"Authorization": f"Bearer {api_key}"}, + params={"_start": cursor, "_limit": current_batch_size}, + timeout=30, + ) + response.raise_for_status() + posts = response.json() + + if not posts: + break + + for post in posts: + # The 'upsert' operation is used to insert or update data in the destination table. + # The first argument is the name of the destination table. + # The second argument is a dictionary containing the record to be upserted. + op.upsert(table="post", data=post) + + total_records += len(posts) + cursor += len(posts) + + # Save the progress by checkpointing the state. This is important for ensuring that the sync process can resume + # from the correct position in case of next sync or interruptions. + # You should checkpoint even if you are not using incremental sync, as it tells Fivetran it is safe to write to destination. + # For large datasets, checkpoint regularly (e.g., every N records) not only at the end. + # Learn more about how and where to checkpoint by reading our best practices documentation + # (https://fivetran.com/docs/connector-sdk/best-practices#optimizingperformancewhenhandlinglargedatasets). + op.checkpoint({"cursor": cursor}) + + if len(posts) < current_batch_size: + break + + if is_metrics_enabled: + log.info(f"Sync complete for data_range={data_range}: {total_records} records extracted") + + +# This creates the connector object that will use the update, schema, and configuration_form +# functions defined in this connector.py file. +connector = Connector(update=update, schema=schema, configuration_form=configuration_form) + +# Check if the script is being run as the main module. +# This is Python's standard entry method allowing your script to be run directly from the command line or IDE 'run' button. +# This is useful for debugging while you write your code. Note this method is not called by Fivetran when executing your connector in production. +# Please test using the Fivetran debug command prior to finalizing and deploying your connector. +if __name__ == "__main__": + # Open the configuration.json file and load its contents into a dictionary. + config_path = os.path.join(os.path.dirname(os.path.abspath(__file__)), "configuration.json") + with open(config_path, "r") as f: + configuration = json.load(f) + # Adding this code to your `connector.py` allows you to test your connector by running your file directly from your IDE. + connector.debug(configuration=configuration) diff --git a/examples/quickstart_examples/configuration_form/images/configuration_form.png b/examples/quickstart_examples/configuration_form/images/configuration_form.png new file mode 100644 index 000000000..611d05075 Binary files /dev/null and b/examples/quickstart_examples/configuration_form/images/configuration_form.png differ diff --git a/examples/quickstart_examples/configuration_form/images/setup_tests.png b/examples/quickstart_examples/configuration_form/images/setup_tests.png new file mode 100644 index 000000000..11b57dbd4 Binary files /dev/null and b/examples/quickstart_examples/configuration_form/images/setup_tests.png differ