example(connector_sdk) : re-raised new connector#39
Open
fivetran-JenasVimal wants to merge 5 commits into
Open
example(connector_sdk) : re-raised new connector#39fivetran-JenasVimal wants to merge 5 commits into
fivetran-JenasVimal wants to merge 5 commits into
Conversation
3 tasks
Contributor
There was a problem hiding this comment.
Pull request overview
Adds a new people_ai Connector SDK example (ported from an older repo) that authenticates to the People.ai API via OAuth2 client credentials, paginates through activities endpoints, and upserts results into destination tables.
Changes:
- Added
people_ai/connector.pyimplementing OAuth token acquisition, retry/backoff, pagination, and upserts. - Added
people_ai/README.mddocumenting usage, configuration, and produced tables. - Added
people_ai/configuration.jsontemplate plus a (currently empty)people_ai/requirements.txt.
Reviewed changes
Copilot reviewed 4 out of 4 changed files in this pull request and generated 22 comments.
| File | Description |
|---|---|
| people_ai/connector.py | Implements the People.ai API sync logic (auth, pagination, retries, upserts, checkpointing). |
| people_ai/README.md | Documents connector behavior, setup, configuration parameters, and tables. |
| people_ai/configuration.json | Provides placeholder configuration keys for API credentials. |
| people_ai/requirements.txt | Added file intended for dependencies (currently empty). |
| @@ -0,0 +1 @@ | |||
|
|
|||
Comment on lines
+14
to
+19
| ## Requirements | ||
| - [Supported Python versions](https://github.com/fivetran/fivetran_connector_sdk/blob/main/README.md#requirements) | ||
| - Operating system: | ||
| - Windows: 10 or later (64-bit only) | ||
| - macOS: 13 (Ventura) or later (Apple Silicon [arm64] or Intel [x86_64]) | ||
| - Linux: Ubuntu 20.04 or later, Debian 10 or later, or Amazon Linux 2 or later (arm64 or x86_64) |
Comment on lines
+47
to
+50
| ## Requirements file | ||
| This connector has no external dependencies and does not require a `requirements.txt` file. | ||
|
|
||
| Note: The `fivetran_connector_sdk:latest` and `requests:latest` packages are pre-installed in the Fivetran environment. To avoid dependency conflicts, do not declare them in your `requirements.txt`. |
Comment on lines
+104
to
+108
| ## Additional files | ||
| - `connector.py` – Contains all core logic: `schema`, `update`, `get_page`, `sync_base_activities`, `sync_activity_type`, `get_access_token`. | ||
| - `configuration.json` – Contains API credentials (`api_key`, `api_secret`). | ||
| - `requirements.txt` – Lists any third-party Python libraries required (e.g., `requests`). | ||
|
|
Comment on lines
+62
to
+71
| """ | ||
| Define the schema function which lets you configure the schema | ||
| your connector delivers. | ||
| See the technical reference documentation | ||
| for more details on the schema function: | ||
| https://fivetran.com/docs/connectors/connector-sdk/technical-reference#schema | ||
|
|
||
| Args: | ||
| configuration: a dictionary that holds the configuration settings for the connector. | ||
| """ |
Comment on lines
+21
to
+25
| ## Getting started | ||
| Refer to the [Connector SDK Setup Guide](https://fivetran.com/docs/connectors/connector-sdk/setup-guide) for setup instructions. | ||
|
|
||
| For local testing, this example includes a `__main__` block that reads `configuration.json` and runs `connector.debug(...)`. | ||
|
|
Comment on lines
+91
to
+103
| ## Tables created | ||
| This connector creates two tables, `ACTIVITY` and `PARTICIPANTS`. | ||
|
|
||
| ### `ACTIVITY` | ||
| - Primary key: `uid` | ||
| - Selected columns (not exhaustive): | ||
| `uid`, `sub_type`, `created_at`, `activity_type`, `updated_at` | ||
|
|
||
| ### `PARTICIPANTS` | ||
| - Primary key: `uid`, `email` | ||
| - Selected columns (not exhaustive): | ||
| `uid`, `email`, `status`, `name`, `ingested_at`, `phone_number` | ||
|
|
| raise e # If re-auth fails, raise the exception | ||
| else: | ||
| # If 401 and max reauth retries reached | ||
| log.severe( |
| except requests.exceptions.HTTPError as e: | ||
| # The retry logic is now inside get_page, | ||
| # so an error here means it failed permanently | ||
| log.severe(f"Permanent failure fetching base activities" f" at offset {offset}: {e}") |
| except requests.exceptions.HTTPError as e: | ||
| # The retry logic is now inside get_page, | ||
| # so an error here means it failed permanently | ||
| log.severe(f"Permanent failure fetching {activity_type}" f" at offset {offset}: {e}") |
fivetran-sahilkhirwal
previously approved these changes
Jul 2, 2026
fivetran-sahilkhirwal
left a comment
fivetran-sahilkhirwal
left a comment
Contributor
There was a problem hiding this comment.
changes look good
For tests, please rebase after this is merged: #30
fivetran-JenasVimal
force-pushed
the
re-raise_people-ai
branch
from
0523275 to
8c3fc1b
Compare
| @@ -0,0 +1 @@ | |||
|
|
|||
Comment on lines
+14
to
+19
| ## Requirements | ||
| - [Supported Python versions](https://github.com/fivetran/fivetran_connector_sdk/blob/main/README.md#requirements) | ||
| - Operating system: | ||
| - Windows: 10 or later (64-bit only) | ||
| - macOS: 13 (Ventura) or later (Apple Silicon [arm64] or Intel [x86_64]) | ||
| - Linux: Ubuntu 20.04 or later, Debian 10 or later, or Amazon Linux 2 or later (arm64 or x86_64) |
| - `api_key`: Your People.ai API key (client ID). | ||
| - `api_secret`: Your People.ai API secret (client secret). | ||
|
|
||
| Note: Ensure that `configuration.json` is not committed to version control. Both configuration values are required; the connector will raise an error if either is missing. |
Comment on lines
+47
to
+50
| ## Requirements file | ||
| This connector has no external dependencies and does not require a `requirements.txt` file. | ||
|
|
||
| Note: The `fivetran_connector_sdk:latest` and `requests:latest` packages are pre-installed in the Fivetran environment. To avoid dependency conflicts, do not declare them in your `requirements.txt`. |
Comment on lines
+104
to
+108
| ## Additional files | ||
| - `connector.py` – Contains all core logic: `schema`, `update`, `get_page`, `sync_base_activities`, `sync_activity_type`, `get_access_token`. | ||
| - `configuration.json` – Contains API credentials (`api_key`, `api_secret`). | ||
| - `requirements.txt` – Lists any third-party Python libraries required (e.g., `requests`). | ||
|
|
Comment on lines
+402
to
+413
| """ | ||
| Define the update function which lets you configure | ||
| how your connector fetches data. | ||
| See the technical reference documentation for more details | ||
| on the update function: https://fivetran.com/docs/connectors/connector-sdk/technical-reference#update | ||
|
|
||
| Args: | ||
| configuration: A dictionary containing connection details. | ||
| state: A dictionary containing state information from previous runs. | ||
| The state dictionary is empty for the first sync or for any | ||
| full re-sync. | ||
| """ |
Comment on lines
+458
to
+459
| limit=100000, | ||
| ) |
Comment on lines
+468
to
+477
| # Check if the script is being run as the main module. | ||
| # This is Python's standard entry method allowing your script to be run directly from the command line or IDE 'run' button. | ||
| # This is useful for debugging while you write your code. Note this method is not called by Fivetran when executing your connector in production. | ||
| # Please test using the Fivetran debug command prior to finalizing and deploying your connector. | ||
| if __name__ == "__main__": | ||
| # Open the configuration.json file and load its contents into a dictionary. | ||
| with open("configuration.json", "r") as f: | ||
| configuration = json.load(f) | ||
| # Adding this code to your `connector.py` allows you to test your connector by running your file directly from your IDE. | ||
| connector.debug(configuration=configuration) |
Comment on lines
+236
to
+242
| page = get_page( | ||
| access_token, | ||
| reauth_func, | ||
| activity_type=None, | ||
| limit=limit, | ||
| offset=offset, | ||
| ) |
Comment on lines
+21
to
+25
| ## Getting started | ||
| Refer to the [Connector SDK Setup Guide](https://fivetran.com/docs/connectors/connector-sdk/setup-guide) for setup instructions. | ||
|
|
||
| For local testing, this example includes a `__main__` block that reads `configuration.json` and runs `connector.debug(...)`. | ||
|
|
Comment on lines
+62
to
+71
| """ | ||
| Define the schema function which lets you configure the schema | ||
| your connector delivers. | ||
| See the technical reference documentation | ||
| for more details on the schema function: | ||
| https://fivetran.com/docs/connectors/connector-sdk/technical-reference#schema | ||
|
|
||
| Args: | ||
| configuration: a dictionary that holds the configuration settings for the connector. | ||
| """ |
Comment on lines
+402
to
+413
| """ | ||
| Define the update function which lets you configure | ||
| how your connector fetches data. | ||
| See the technical reference documentation for more details | ||
| on the update function: https://fivetran.com/docs/connectors/connector-sdk/technical-reference#update | ||
|
|
||
| Args: | ||
| configuration: A dictionary containing connection details. | ||
| state: A dictionary containing state information from previous runs. | ||
| The state dictionary is empty for the first sync or for any | ||
| full re-sync. | ||
| """ |
Comment on lines
+39
to
+40
| # The unused imports 'Sequence' and 'Literal' have been removed. | ||
| from typing import Any, Dict, List, Callable, Optional |
Comment on lines
+158
to
+162
| log.severe( | ||
| f"Failed to fetch page after {__MAX_RETRIES + 1} " | ||
| f"due to {status_code} error." | ||
| ) | ||
| raise e |
Comment on lines
+166
to
+167
| log.severe(f"Received unrecoverable HTTP error {status_code}") | ||
| raise e |
Comment on lines
+21
to
+24
| ## Getting started | ||
| Refer to the [Connector SDK Setup Guide](https://fivetran.com/docs/connectors/connector-sdk/setup-guide) for setup instructions. | ||
|
|
||
| For local testing, this example includes a `__main__` block that reads `configuration.json` and runs `connector.debug(...)`. |
Comment on lines
+47
to
+50
| ## Requirements file | ||
| This connector has no external dependencies and does not require a `requirements.txt` file. | ||
|
|
||
| Note: The `fivetran_connector_sdk:latest` and `requests:latest` packages are pre-installed in the Fivetran environment. To avoid dependency conflicts, do not declare them in your `requirements.txt`. |
Comment on lines
+104
to
+108
| ## Additional files | ||
| - `connector.py` – Contains all core logic: `schema`, `update`, `get_page`, `sync_base_activities`, `sync_activity_type`, `get_access_token`. | ||
| - `configuration.json` – Contains API credentials (`api_key`, `api_secret`). | ||
| - `requirements.txt` – Lists any third-party Python libraries required (e.g., `requests`). | ||
|
|
Comment on lines
+52
to
+56
| ## Authentication | ||
| - Type: OAuth2 Client Credentials | ||
| - Token URL: `https://api.people.ai/auth/v1/tokens` | ||
| - Headers: `Content-Type: application/x-www-form-urlencoded` | ||
| - Grant Type: `client_credentials` |
Comment on lines
+94
to
+98
| ### `ACTIVITY` | ||
| - Primary key: `uid` | ||
| - Selected columns (not exhaustive): | ||
| `uid`, `sub_type`, `created_at`, `activity_type`, `updated_at` | ||
|
|
Comment on lines
+61
to
+70
| """ | ||
| Define the schema function which lets you configure the schema | ||
| your connector delivers. | ||
| See the technical reference documentation | ||
| for more details on the schema function: | ||
| https://fivetran.com/docs/connectors/connector-sdk/technical-reference#schema | ||
|
|
||
| Args: | ||
| configuration: a dictionary that holds the configuration settings for the connector. | ||
| """ |
Comment on lines
+401
to
+412
| """ | ||
| Define the update function which lets you configure | ||
| how your connector fetches data. | ||
| See the technical reference documentation for more details | ||
| on the update function: https://fivetran.com/docs/connectors/connector-sdk/technical-reference#update | ||
|
|
||
| Args: | ||
| configuration: A dictionary containing connection details. | ||
| state: A dictionary containing state information from previous runs. | ||
| The state dictionary is empty for the first sync or for any | ||
| full re-sync. | ||
| """ |
Comment on lines
+157
to
+160
| log.severe( | ||
| f"Failed to fetch page after {__MAX_RETRIES + 1} " | ||
| f"due to {status_code} error." | ||
| ) |
Comment on lines
+165
to
+166
| log.severe(f"Received unrecoverable HTTP error {status_code}") | ||
| raise e |
Comment on lines
+242
to
+246
| except requests.exceptions.HTTPError as e: | ||
| # The retry logic is now inside get_page, | ||
| # so an error here means it failed permanently | ||
| log.severe(f"Permanent failure fetching base activities" f" at offset {offset}: {e}") | ||
| break |
Comment on lines
+192
to
+194
| except Exception as e: | ||
| log.severe(f"Failed to refresh token: {e}") | ||
| raise e # If re-auth fails, raise the exception |
Comment on lines
+197
to
+203
| log.severe( | ||
| f"Authentication failed after {__REAUTH_RETRY_COUNT + 1}" | ||
| " re-authentication attempts." | ||
| ) | ||
| raise requests.exceptions.HTTPError( | ||
| "401 Client Error: Unauthorized (Max re-auth retries reached)" | ||
| ) |
Comment on lines
+427
to
+432
| try: | ||
| access_token = get_access_token(api_key, api_secret) | ||
| except requests.exceptions.RequestException as e: | ||
| log.severe(f"FATAL: Initial authentication failed: {e}") | ||
| return # Stop the sync | ||
|
|
Comment on lines
+235
to
+241
| page = get_page( | ||
| access_token, | ||
| reauth_func, | ||
| activity_type=None, | ||
| limit=limit, | ||
| offset=offset, | ||
| ) |
| while True: | ||
| try: | ||
| # Pass the reauth_func | ||
| page = get_page(access_token, reauth_func, activity_type, limit=limit, offset=offset) |
fivetran-JenasVimal
force-pushed
the
re-raise_people-ai
branch
from
3e46920 to
fd7caa1
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description of Change
Re-raising this connector from old repo
Testing
This is actively being used by a customer. Also, tested locally this morning here
Checklist
Some tips and links to help validate your PR:
fivetran debugcommand.