-
Notifications
You must be signed in to change notification settings - Fork 5
example(connector_sdk) : re-raised new connector #39
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Closed
+621
−0
Closed
Changes from 20 commits
Commits
Show all changes
21 commits
Select commit
Hold shift + click to select a range
5935c36
re-raised new connector
fivetran-JenasVimal 3124427
Update People.ai connector listing
fivetran-JenasVimal bc141cb
Handle label permission failures
fivetran-JenasVimal 1227113
Revert "Handle label permission failures"
fivetran-JenasVimal fd7caa1
Format People.ai connector
fivetran-JenasVimal ddb9f27
changed readme.md
fivetran-JenasVimal 8bbed6e
Potential fix for pull request finding
fivetran-JenasVimal e9e8004
Potential fix for pull request finding
fivetran-JenasVimal 1960ffb
Potential fix for pull request finding
fivetran-JenasVimal 65b80bb
Potential fix for pull request finding
fivetran-JenasVimal 64f754f
changed logs
fivetran-JenasVimal 7907eff
Made changes to code
fivetran-JenasVimal f5c4937
Address People.ai README review comments
fivetran-JenasVimal b405c50
Format People.ai connector
fivetran-JenasVimal 1d706bc
Potential fix for pull request finding
fivetran-JenasVimal 49b7892
Potential fix for pull request finding
fivetran-JenasVimal daf0775
Potential fix for pull request finding
fivetran-JenasVimal 5886608
Potential fix for pull request finding
fivetran-JenasVimal 29934a3
Potential fix for pull request finding
fivetran-JenasVimal a489ddd
Apply repository Python formatting
fivetran-JenasVimal b376a3e
Made co-pilot changes
fivetran-JenasVimal File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Some comments aren't visible on the classic Files Changed page.
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,118 @@ | ||
| # People.ai Connector Example | ||
|
|
||
| ## Connector overview | ||
|
|
||
| This connector syncs activity data from the People.ai API into a destination using the Fivetran Connector SDK. It retrieves base activity records from `/v0/public/activities` and participant records from `/v0/public/activities/participants`. | ||
|
|
||
| The connector uses OAuth2 client credentials authentication, refreshes the access token after an unauthorized response, retries transient API failures with exponential backoff, and writes records page by page to avoid loading the full API result set into memory. | ||
|
|
||
| ## Requirements | ||
|
|
||
| - [Supported Python versions](https://github.com/fivetran/community_connectors/blob/main/README.md#requirements) | ||
| - Operating system: | ||
| - Windows: 10 or later (64-bit only) | ||
| - macOS: 13 (Ventura) or later (Apple Silicon [arm64] or Intel [x86_64]) | ||
| - Linux: Distributions such as Ubuntu 20.04 or later, Debian 10 or later, or Amazon Linux 2 or later (arm64 or x86_64) | ||
|
|
||
|
fivetran-JenasVimal marked this conversation as resolved.
|
||
| ## Getting started | ||
|
|
||
| Refer to the [Connector SDK Setup Guide](https://fivetran.com/docs/connectors/connector-sdk/setup-guide) to get started. | ||
|
|
||
| To initialize a new Connector SDK project using this connector as a starting point, run: | ||
|
|
||
| ```shell | ||
| fivetran init --template people_ai | ||
| ``` | ||
|
|
||
| `fivetran init` initializes a new Connector SDK project by setting up the project structure, configuration files, and a connector you can run immediately with `fivetran debug`. For more information on `fivetran init`, refer to the [Connector SDK `init` documentation](https://fivetran.com/docs/connector-sdk/connector-development-and-configuration/connector-sdk-commands#fivetraninit). | ||
|
|
||
| > Note: Ensure you have updated the `configuration.json` file with the necessary parameters before running `fivetran debug`. See the [Configuration file](#configuration-file) section for details on the required configuration parameters. | ||
|
|
||
|
fivetran-JenasVimal marked this conversation as resolved.
fivetran-meetmahna marked this conversation as resolved.
fivetran-JenasVimal marked this conversation as resolved.
|
||
| ## Features | ||
|
|
||
| - Syncs base activity records from `/v0/public/activities`. | ||
| - Syncs participant records from `/v0/public/activities/participants`. | ||
| - Authenticates with OAuth2 client credentials. | ||
| - Refreshes the access token once after a `401 Unauthorized` response. | ||
| - Retries transient server and network failures with exponential backoff. | ||
| - Uses offset-based pagination and writes each page with `op.upsert(...)`. | ||
| - Checkpoints progress after each successfully written page. | ||
|
|
||
| ## Configuration file | ||
|
|
||
| The connector requires the following configuration parameters: | ||
|
|
||
| ```json | ||
| { | ||
| "api_key": "<YOUR_PEOPLE_AI_API_KEY>", | ||
| "api_secret": "<YOUR_PEOPLE_AI_API_SECRET>" | ||
| } | ||
| ``` | ||
|
|
||
| - `api_key` - Your People.ai OAuth client ID or API key. | ||
| - `api_secret` - Your People.ai OAuth client secret. | ||
|
|
||
| > Note: When submitting connector code as a [Community Connector](https://github.com/fivetran/community_connectors/tree/main) in the open-source [Connector SDK repository](https://github.com/fivetran/community_connectors/tree/main), ensure the `configuration.json` file has placeholder values. When adding the connector to your production repository, ensure that the `configuration.json` file is not checked into version control to protect sensitive information. | ||
|
|
||
| ## Authentication | ||
|
|
||
| The connector uses OAuth2 client credentials authentication in `get_access_token()`. | ||
|
|
||
| - Token URL: `https://api.people.ai/auth/v1/tokens` | ||
| - Grant type: `client_credentials` | ||
| - Request content type: `application/x-www-form-urlencoded` | ||
| - Access token usage: `Authorization: Bearer <access_token>` | ||
|
|
||
| The `update()` function performs initial authentication before syncing data. The connector passes a `reauthenticate()` closure into the page-fetching functions so `get_page()` can request a new token and retry the request after a `401 Unauthorized` response. | ||
|
|
||
|
fivetran-JenasVimal marked this conversation as resolved.
|
||
| ## Pagination | ||
|
|
||
| People.ai activity endpoints use offset-based pagination. The connector handles pagination in `sync_base_activities()` and `sync_activity_type()`. | ||
|
|
||
| - The base activities endpoint uses the `limit` and `offset` query parameters with a page size of 1000. | ||
| - The participant activity endpoint uses the `limit` and `offset` query parameters with a page size of 100000. | ||
| - Pagination stops when the API returns an empty page or a page with fewer records than the requested limit. | ||
| - Each page is upserted before the connector advances and checkpoints the offset. | ||
|
|
||
| ## Data handling | ||
|
|
||
| The connector processes records in pages and writes them to destination tables using `op.upsert(...)`. | ||
|
|
||
| - `schema()` defines the `activity` table with primary key `uid`. | ||
| - `schema()` defines the `participants` table with primary key `uid`, `email`. | ||
| - `sync_base_activities()` copies each base activity record and renames a `subject` field to `api_subject` when present. | ||
| - `sync_activity_type()` writes participant records to the `participants` table. | ||
| - The connector stores the most recently processed page offsets in `state["activity_offset"]` and `state["participants_offset"]` after each successful page write. | ||
|
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Where are we using |
||
|
|
||
| This example resumes page-based syncs from the checkpointed offsets stored in state on subsequent runs, so it can continue from the last successfully written page. | ||
|
|
||
|
fivetran-JenasVimal marked this conversation as resolved.
|
||
| ## Error handling | ||
|
|
||
| The connector implements retry and authentication handling in `get_page()`. | ||
|
|
||
| - `401 Unauthorized` responses trigger one access token refresh and request retry. | ||
| - `5xx` server responses are retried up to five times with exponential backoff. | ||
| - Network and timeout errors are retried up to five times with exponential backoff. | ||
| - Other HTTP errors are treated as unrecoverable and raised. | ||
| - Missing `api_key` or `api_secret` values cause early validation failure in `validate_configuration()`. | ||
| - The connector uses `fivetran_connector_sdk.Logging` for status, retry, and error messages. | ||
|
|
||
| ## Tables created | ||
|
|
||
| The connector creates the following tables: | ||
|
|
||
| Columns are inferred from the API response. The primary keys configured in `schema()` are: | ||
|
|
||
| | Table name | Primary key | Description | | ||
| |------------|-------------|-------------| | ||
| | `activity` | `uid` | Base People.ai activity records from `/v0/public/activities`. | | ||
| | `participants` | `uid`, `email` | Participant activity records from `/v0/public/activities/participants`. | | ||
|
|
||
|
fivetran-JenasVimal marked this conversation as resolved.
|
||
| ## Additional files | ||
|
|
||
| - **`connector.py`** - Contains the connector implementation, including schema definition, authentication, pagination, retry handling, and data writes. | ||
| - **`configuration.json`** - Contains placeholder People.ai credential fields used by the connector. | ||
|
|
||
|
fivetran-JenasVimal marked this conversation as resolved.
fivetran-JenasVimal marked this conversation as resolved.
fivetran-JenasVimal marked this conversation as resolved.
fivetran-JenasVimal marked this conversation as resolved.
|
||
| ## Additional considerations | ||
|
|
||
| The examples provided are intended to help you effectively use Fivetran's Connector SDK. While we've tested the code, Fivetran cannot be held responsible for any unexpected or negative consequences that may arise from using these examples. For inquiries, please reach out to our Support team. | ||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,4 @@ | ||
| { | ||
| "api_key": "<YOUR_PEOPLE_AI_API_KEY>", | ||
| "api_secret": "<YOUR_PEOPLE_AI_API_SECRET>" | ||
| } |
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Uh oh!
There was an error while loading. Please reload this page.