Skip to content

feature(connector_sdk) : Add FHIR R4 Healthcare Intelligence connector example#601

Open
fivetran-JenasVimal wants to merge 2 commits into
mainfrom
re-raise/all_things_ai_databricks-fm-fhir-healthcare-intelligence
Open

feature(connector_sdk) : Add FHIR R4 Healthcare Intelligence connector example#601
fivetran-JenasVimal wants to merge 2 commits into
mainfrom
re-raise/all_things_ai_databricks-fm-fhir-healthcare-intelligence

Conversation

@fivetran-JenasVimal

@fivetran-JenasVimal fivetran-JenasVimal commented Jun 29, 2026

Copy link
Copy Markdown
Contributor

Jira ticket

RD-1234449
Re-raising because @fivetran-davidmillman had CLA checks failing

Description of Change

Adds a new connector under all_things_ai/tutorials/databricks-fm-fhir-healthcare-intelligence
Syncs Patient, Condition, Observation, and MedicationRequest resources from any FHIR R4-compliant server (default: HAPI FHIR public test server, no auth required)
Implements the Hybrid (Discovery + Debate) AI pattern using Databricks ai_query(): a Discovery phase for population health risk stratification, and a Debate phase where a Clinical Risk Analyst and Resource Allocation Analyst debate per-patient intervention priorities, producing a consensus with a disagreement flag
Produces 8 destination tables: patients, conditions, observations, medications, population_insights, clinical_assessments, resource_assessments, debate_consensus
Follows the Hybrid pattern established by PRs #570 and #571
Updates main README.md with entry for new connector

Test plan

  • fivetran debug run locally — SYNC SUCCEEDED
  • Deployed to Fivetran and triggered production sync — SUCCEEDED
  • black --line-length 99 passes
  • flake8 --extend-ignore=E203,E501,B008 passes
  • All configuration.json values are angle-bracket placeholders
  • No credentials in submitted files

fivetran debug results

SDK Version: 2.8.1
Sync: SUCCEEDED (00:02:24)
Records: 3 patients, 2 AI-enriched (max_patients=3, max_enrichments=2)

Table Records
patients 3
population_insights 1
clinical_assessments 2
resource_assessments 2
debate_consensus 2
Total upserts 10
Raw fivetran debug terminal output
20-Apr 13:59:11.030 INFO sdk fivetran-connector-sdk 2.8.1 is available. (a newer release exists)
20-Apr 13:59:11.030 INFO sdk run 'pip install --upgrade fivetran-connector-sdk' to update
20-Apr 13:59:11.433 INFO sdk debugging connector at: .../databricks-fm-fhir-healthcare-intelligence
20-Apr 13:59:11.730 INFO sdk starting connector tester
20-Apr 13:59:11.730 INFO debugger connector tester version: 2.26.0410.001
20-Apr 13:59:13.126 INFO debugger previous state: {"last_sync": "2026-04-20T18:49:43.854397+00:00"}
20-Apr 13:59:14.223 INFO sdk calling schema()
20-Apr 13:59:14.233 INFO debugger schema change detected: tester.patients
20-Apr 13:59:14.233 INFO debugger schema change detected: tester.conditions
20-Apr 13:59:14.234 INFO debugger schema change detected: tester.observations
20-Apr 13:59:14.234 INFO debugger schema change detected: tester.medications
20-Apr 13:59:14.234 INFO debugger schema change detected: tester.population_insights
20-Apr 13:59:14.234 INFO debugger schema change detected: tester.clinical_assessments
20-Apr 13:59:14.234 INFO debugger schema change detected: tester.resource_assessments
20-Apr 13:59:14.234 INFO debugger schema change detected: tester.debate_consensus
20-Apr 13:59:14.238 INFO sdk calling update()
20-Apr 13:59:14.238 WARNING Example: all_things_ai/tutorials : databricks-fm-fhir-healthcare-intelligence
20-Apr 13:59:14.238 INFO FHIR base URL: https://hapi.fhir.org/baseR4
20-Apr 13:59:14.238 INFO Max patients: 3, condition filter: none
20-Apr 13:59:14.238 INFO Hybrid analysis ENABLED: model=databricks-claude-sonnet-4-6
20-Apr 13:59:14.239 INFO Phase 1 (MOVE): Fetching clinical data from FHIR
20-Apr 13:59:14.239 INFO Fetching patients (condition filter: none, max: 3)
20-Apr 13:59:14.699 INFO Fetched 3 patient resources
20-Apr 13:59:23.511 INFO Phase 1 complete: 3 patients fetched
20-Apr 13:59:23.512 INFO Phase 2 (DISCOVERY): AI population health analysis
20-Apr 13:59:23.512 INFO Calling ai_query() for population health discovery
20-Apr 13:59:23.578 INFO debugger checkpoint recorded: {"last_sync": "2026-04-20T18:49:43.854397+00:00"}
20-Apr 13:59:57.487 INFO Discovery phase complete
20-Apr 13:59:57.487 INFO Phase 3 (DEBATE): Clinical vs Resource debate per patient
20-Apr 13:59:57.487 INFO Starting debate for 2 patients (3 ai_query() calls each)
20-Apr 13:59:57.513 INFO debugger table created: tester.population_insights
20-Apr 13:59:57.526 INFO debugger checkpoint recorded: {"last_sync": "2026-04-20T18:49:43.854397+00:00"}
20-Apr 14:01:00.335 INFO debugger table created: tester.clinical_assessments
20-Apr 14:01:00.352 INFO debugger table created: tester.resource_assessments
20-Apr 14:01:00.368 INFO debugger table created: tester.debate_consensus
20-Apr 14:01:00.381 INFO debugger checkpoint recorded: {"last_sync": "2026-04-20T18:49:43.854397+00:00"}
20-Apr 14:01:37.568 INFO Debate complete: 2 patients debated, 1 with disagreement flags
20-Apr 14:01:37.568 INFO Sync complete: 3 patients, all clinical phases done
20-Apr 14:01:37.611 INFO debugger checkpoint recorded: {"last_sync": "2026-04-20T18:49:43.854397+00:00"}
20-Apr 14:01:37.611 INFO debugger checkpoint recorded: {"last_sync": "2026-04-20T19:01:37.568371+00:00"}
20-Apr 14:01:37.612 INFO debugger Final checkpoint: committing any remaining operations
20-Apr 14:01:37.612 INFO debugger SYNC SUCCEEDED — total elapsed 00:02:24
Operation       | Counts
----------------+------------
Upserts         | 10
Updates         | 0
Deletes         | 0
Truncates       | 0
Schema changes  | 8
Checkpoints     | 5

Copilot AI review requested due to automatic review settings June 29, 2026 19:54
@fivetran-JenasVimal fivetran-JenasVimal requested review from a team as code owners June 29, 2026 19:54
@github-actions

Copy link
Copy Markdown

🧹 Python Code Quality Check

✅ No issues found in Python Files.

🔍 See how this check works

This comment is auto-updated with every commit.

@fivetran-JenasVimal fivetran-JenasVimal force-pushed the re-raise/all_things_ai_databricks-fm-fhir-healthcare-intelligence branch from acd6fdf to a4038d9 Compare June 29, 2026 19:56

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Adds a new AI tutorial connector example under all_things_ai/tutorials/ that syncs clinical data from a FHIR R4 server and performs Databricks ai_query()-based hybrid (Discovery + Debate) enrichment, plus updates the root README to reference the new tutorial.

Changes:

  • Added the FHIR R4 Healthcare Intelligence tutorial connector (connector code, tutorial README, and configuration template).
  • Added a requirements.txt for the new tutorial.
  • Updated the repo README.md (badges/links and tutorial list) to include the new connector entry and adjust various GitHub URLs.

Reviewed changes

Copilot reviewed 4 out of 4 changed files in this pull request and generated 7 comments.

Show a summary per file
File Description
README.md Updates badges/links, requirements text, and AI tutorial list to include the new connector
all_things_ai/tutorials/databricks-fm-fhir-healthcare-intelligence/connector.py New FHIR R4 + Databricks hybrid (Discovery + Debate) tutorial connector implementation
all_things_ai/tutorials/databricks-fm-fhir-healthcare-intelligence/README.md New tutorial documentation for the connector (setup, configuration, behavior)
all_things_ai/tutorials/databricks-fm-fhir-healthcare-intelligence/configuration.json Configuration template with placeholders for FHIR and Databricks settings
all_things_ai/tutorials/databricks-fm-fhir-healthcare-intelligence/requirements.txt Declares Python dependencies for the tutorial

Comment thread all_things_ai/tutorials/databricks-fm-fhir-healthcare-intelligence/connector.py Outdated
Comment thread all_things_ai/tutorials/databricks-fm-fhir-healthcare-intelligence/connector.py Outdated
@fivetran-JenasVimal fivetran-JenasVimal force-pushed the re-raise/all_things_ai_databricks-fm-fhir-healthcare-intelligence branch from a4038d9 to d48956a Compare June 29, 2026 20:00
@fivetran-rishabhghosh fivetran-rishabhghosh requested review from fivetran-meetmahna and fivetran-sahilkhirwal and removed request for a team June 29, 2026 20:13

@fivetran-sahilkhirwal fivetran-sahilkhirwal left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good to me
please get review from @fivetran-anushkaparashar as well before merging

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

ai-assisted/yes risk/low size/XXL PR size: extra extra large

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants