Configure database variables in dataflow by csafreen · Pull Request #2685 · OHDSI/Data2Evidence

csafreen · 2026-06-10T12:45:40Z

Merge Checklist

Please cross check this list if additions / modifications needs to be done on top of your core changes and tick them off. Reviewer can as well glance through and help the developer if something is missed out.

Automated Tests (Jasmine integration tests, Unit tests, and/or Performance tests)
Updated Manual tests / Demo Config
Documentation (Application guide, Admin guide, Markdown, Readme and/or Wiki)
Verified that local development environment is working with latest changes (integrated with latest develop branch)
following best practices in code review doc

Copilot

Pull request overview

This PR adds first-class “database variables” to the Flow UI so users can define reusable database configurations (name → database code + schema) in the Variables panel, then reference them from DB reader/writer nodes. It also wires the new databases payload through save/export/import/run and into the job plugin / Prefect parameter transformation so DB nodes can resolve variable names to concrete database codes at execution time.

Changes:

Extend flow state/DTOs to persist databases: DatabaseVariable[] alongside existing variables/import libs.
Add a new Databases section in the Variables drawer (with a dedicated DatabaseVariableForm) and update DB reader/writer drawers to select from configured database variables.
Update job plugin parameter transformation to pass databases through and resolve DB variable names to concrete DB codes/schemas for db_reader/db_writer nodes.

Reviewed changes

Copilot reviewed 19 out of 19 changed files in this pull request and generated 6 comments.

Show a summary per file

File	Description
plugins/ui/apps/flow/src/features/flow/types/state.ts	Adds `databases` to the root flow state shape.
plugins/ui/apps/flow/src/features/flow/types/dataflow.state.ts	Adds `databases` to exported/saved flow DTOs.
plugins/ui/apps/flow/src/features/flow/types/common.ts	Introduces `DatabaseVariable` type.
plugins/ui/apps/flow/src/features/flow/reducers.ts	Adds `databases` to initial state and a `replaceDatabases` action.
plugins/ui/apps/flow/src/features/flow/containers/Node/NodeTypes/DbWriterNode/DbWriterDrawer.tsx	Switches DB selection to configured database variables and auto-fills schema on selection.
plugins/ui/apps/flow/src/features/flow/containers/Node/NodeTypes/DbReaderNode/DbReaderDrawer.tsx	Switches DB selection to configured database variables.
plugins/ui/apps/flow/src/features/flow/containers/FlowLayout.tsx	Loads `databases` into Redux from fetched dataflow revisions.
plugins/ui/apps/flow/src/features/flow/containers/Flow/SaveFlow/SaveFlowDialog.tsx	Persists `databases` when saving flows (including template creation path).
plugins/ui/apps/flow/src/features/flow/containers/Flow/RunFlow/RunFlowButton.tsx	Includes `databases` in test-run request payload.
plugins/ui/apps/flow/src/features/flow/containers/Flow/ImportFlow/ImportFlowButton.tsx	Imports `databases` from JSON and stores in Redux.
plugins/ui/apps/flow/src/features/flow/containers/Flow/FlowVariables/FlowVariablesDrawer.tsx	Adds Databases editor UI and applies `replaceDatabases` on Apply.
plugins/ui/apps/flow/src/features/flow/containers/Flow/FlowVariables/FlowVariablesButton.tsx	Updates badge count to include databases + variables.
plugins/ui/apps/flow/src/features/flow/containers/Flow/FlowVariables/DatabaseVariableForm/DatabaseVariableForm.tsx	New form component for editing database variables.
plugins/ui/apps/flow/src/features/flow/containers/Flow/FlowVariables/DatabaseVariableForm/DatabaseVariableForm.scss	Styles for the new database variable form row.
plugins/ui/apps/flow/src/features/flow/containers/Flow/ExportFlow/ExportFlowButton.tsx	Exports `databases` into the JSON export.
plugins/functions/jobplugins/src/utils/DataflowParser.ts	Passes databases through and resolves db node `database` from variable name → db code (+ schema defaulting for writer).
plugins/functions/jobplugins/src/types.ts	Adds `DatabaseVariable` and includes `databases` in flow/Prefect parameter types.
plugins/flows/data_transformation/dataflow_ui_plugin/nodes.py	Injects database variables into Python node execution context via `SimpleNamespace`.
plugins/flows/data_transformation/dataflow_ui_plugin/flow.py	Accepts `databases` in flow signature and passes them into node execution (notably Python nodes).

            case NodeType.CSV | NodeType.DBREADER | NodeType.CONCEPTMAPPING | NodeType.FILE | NodeType.WHITERABBIT:
                result = _node.task(task_run_context)
            case NodeType.PYTHON:
-                result = _node.task(input, shared_variables, importlibs, task_run_context)
+                result = _node.task(input, shared_variables, importlibs, task_run_context, databases or [])


+    def test(self, _input: dict[str, Result],
+             shared_variables: dict[str, str],
+             importlibs: list[str],
             task_run_context):
        return self.task(_input, shared_variables, importlibs, task_run_context)


            onChange={(e: SelectChangeEvent<string>) =>
              onFormDataChange({ database: e.target.value })
            }
-            disabled={isLoadingDatabases}
+            disabled={databases.length === 0}


@@ -154,16 +163,27 @@ export const DbWriterDrawer: FC<DbWriterDrawerProps> = ({
          <Select
            value={formData.database}


  const handleApply = useCallback(() => {
    dispatch(replaceVariables(localVariables));
    dispatch(replaceImportLibs(localImportLibs));
+    dispatch(replaceDatabases(localDatabases));
    dispatch(markStatusAsDraft());
    handleClose();
-  }, [localVariables, localImportLibs, handleClose]);
+  }, [localVariables, localImportLibs, localDatabases, handleClose]);


 export interface IReactFlow {
  nodes: IReactFlowNode[];
  edges: IReactFlowEdge[];
  variables: KeyValue[];
  importLibs: string[];
+  databases: DatabaseVariable[];
 }


brandantck · 2026-06-19T03:43:54Z

+
+    const dbList = await dbCredentialsAPI.getDbList();
+    const exist = dbList.find((db) => db.code === env.FHIR_DATABASE_CODE);
+    if (exist && exist.dialect === "trex") {


We have a dialect that is trex?

FHIR db entry is created with dialect trex cause of the way its being connected in DBReader nodes.

Configure database variables to dataflow

97a4d5c

Copilot AI review requested due to automatic review settings June 10, 2026 12:45

csafreen requested review from KangJingA, SantanM, brandantck, hengxian-jiang, p-hoffmann and suwarnoong as code owners June 10, 2026 12:45

csafreen temporarily deployed to github June 10, 2026 12:45 — with GitHub Actions Inactive

Copilot started reviewing on behalf of csafreen June 10, 2026 12:45 View session

csafreen temporarily deployed to github June 10, 2026 12:46 — with GitHub Actions Inactive

Copilot AI reviewed Jun 10, 2026

View reviewed changes

Update

470999d

csafreen temporarily deployed to github June 11, 2026 07:00 — with GitHub Actions Inactive

csafreen temporarily deployed to github June 11, 2026 07:01 — with GitHub Actions Inactive

Improve error messages

b5a7f57

csafreen temporarily deployed to github June 11, 2026 07:53 — with GitHub Actions Inactive

csafreen temporarily deployed to github June 16, 2026 05:54 — with GitHub Actions Inactive

csafreen temporarily deployed to github June 16, 2026 05:55 — with GitHub Actions Inactive

csafreen mentioned this pull request Jun 16, 2026

Update templates to use database variables data2evidence/templates#36

Merged

6 tasks

Zhimin-arya requested changes Jun 18, 2026

View reviewed changes

Comment thread plugins/fhir_functions/fhir-init/index.ts

Comment thread plugins/flows/_shared_flow_utils/dao/trexdao.py

Fix

45b6695

csafreen temporarily deployed to github June 19, 2026 01:11 — with GitHub Actions Inactive

csafreen requested a review from Zhimin-arya June 19, 2026 01:12

Zhimin-arya approved these changes Jun 19, 2026

View reviewed changes

hengxian-jiang reviewed Jun 19, 2026

View reviewed changes

Comment thread ...rc/features/flow/containers/Flow/FlowVariables/DatabaseVariableForm/DatabaseVariableForm.tsx Outdated

Refactor

95c8b94

csafreen temporarily deployed to github June 19, 2026 04:44 — with GitHub Actions Inactive

csafreen requested a review from hengxian-jiang June 19, 2026 04:44

csafreen temporarily deployed to github June 19, 2026 04:50 — with GitHub Actions Inactive

hengxian-jiang approved these changes Jun 19, 2026

View reviewed changes

brandantck reviewed Jun 19, 2026

View reviewed changes

csafreen requested a review from brandantck June 19, 2026 05:53

brandantck approved these changes Jun 19, 2026

View reviewed changes

csafreen enabled auto-merge June 19, 2026 06:11

Merge branch 'develop' into csafreen/dataflow_database_var

d99a041

csafreen temporarily deployed to github June 19, 2026 06:51 — with GitHub Actions Inactive

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Configure database variables in dataflow#2685

Configure database variables in dataflow#2685
csafreen wants to merge 10 commits into
developfrom
csafreen/dataflow_database_var

csafreen commented Jun 10, 2026

Uh oh!

Copilot AI left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

brandantck Jun 19, 2026

Uh oh!

csafreen Jun 19, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

		@@ -154,16 +163,27 @@ export const DbWriterDrawer: FC<DbWriterDrawerProps> = ({
		<Select
		value={formData.database}

Conversation

csafreen commented Jun 10, 2026

Merge Checklist

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Uh oh!

Uh oh!

Uh oh!

brandantck Jun 19, 2026

Choose a reason for hiding this comment

Uh oh!

csafreen Jun 19, 2026

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants