Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,7 @@

---8<-- "snippets/connectors/integration-connector-intro.md"

The purpose of the integration daemon is to minimise the effort required to integrate a third party technology into the open metadata ecosystem. They handle:
The purpose of the [integration daemon](/concepts/integration-daemon) is to minimise the effort required to integrate a third party technology into the open metadata ecosystem. They handle:

* Management of configuration - including user security information.
* Starting and stopping of your integration logic.
Expand All @@ -16,6 +16,7 @@ The purpose of the integration daemon is to minimise the effort required to inte

This means you can focus on interacting with the third party technology and mapping its metadata to open metadata in your integration connector.

Integration connectors are also useful for tasks that need to run regularly. Egeria uses integration connectors to monitor the health of the open metadata ecosystem and to add its own insights.

## Integration connector interface

Expand Down Expand Up @@ -168,7 +169,7 @@ The connection object is stored in the `connectionProperties` instance variable

### Accessing context

The context is retrieved using the `getContext()` method. This is a synchronized method that can be called from multiple threads, that occurs when the connector is using listeners.
The context is retrieved using the `getIntegrationContext()` method. This is a synchronized method that can be called from multiple threads, that occurs when the connector is using listeners.

### Registering a listener with open metadata

Expand All @@ -190,7 +191,7 @@ Your integration connector registers itself as a listener in the `start()` metho
:
:

myContext = super.getContext();
myContext = super.getIntegrationContext();

if (myContext != null)
{
Expand Down Expand Up @@ -242,7 +243,7 @@ Ideally your integration connector should use an embedded [digital resource conn

final String methodName = "start";

myContext = super.getContext();
myContext = super.getIntegrationContext();

if (myContext != null)
{
Expand Down
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.

Large diffs are not rendered by default.

Original file line number Diff line number Diff line change
@@ -0,0 +1,126 @@
<!-- SPDX-License-Identifier: CC-BY-4.0 -->
<!-- Copyright Contributors to the Egeria project. -->

# Self-registering integration connectors

You may wish to write an integration connector that searches for particular types of metadata elements and registered them as its own catalog targets for ongoing processing.

In this pattern:

* The integration connector inherits from `DynamicIntegrationConnectorBase`.
* The `start()` method is minimal, simply initializing any resources needed to perform the discovery of new metadata elements of interest.
* The discovery logic is implemented in the `refresh()` method. It compares the existing list of catalog targets with the list of interesting metadata elements retrieved form the metadata repositories. Any new ones are added as catalog targets. Any that are no longer of interest are removed.
* The processing logic is implemented in the *Target Processor* class.
* The `disconnect()` method cleans up any resources.

## Integration connector implementation

### Inheritance

The integration connector inherits from [DynamicIntegrationConnectorBase](https://odpi.github.io/egeria/org/odpi/openmetadata/frameworks/integration/connectors/DynamicIntegrationConnectorBase.html). This base class manages the lifecycle of the integration connector and calling of the catalog targets.

```java
public class MySelfRegisteringIntegrationConnector extends DynamicIntegrationConnectorBase
{

}
```

If the connector will also process events then it should also implement the [OpenMetadataEventListener](https://odpi.github.io/egeria/org/odpi/openmetadata/frameworks/openmetadata/events/OpenMetadataEventListener.html) interface and its `processEvent()` method. If this interface is implemented, the `DynamicIntegrationConnectorBase` will register and event listener for you.

### Start method

Typically the `start()` method simply outputs an audit log message to indicate that it is starting up.

### Refresh method

The `refresh()` method has three jobs to perform:

* discover if there are new metadata elements of interest
* register any new metadata elements as catalog targets
* refresh all catalog targets

This code snippet shows how to discover the unique identifiers (GUIDs) of elements that are already catalog targets.
```java
try
{
Set<String> = new HashSet<>();

AssetClient assetClient = integrationContext.getAssetClient();
int startFrom = 0;

List<OpenMetadataRootElement> catalogTargetList = assetClient.getCatalogTargets(integrationContext.getIntegrationConnectorGUID(),
assetClient.getQueryOptions(startFrom, integrationContext.getMaxPageSize()));

while (catalogTargetList != null)
{
for (OpenMetadataRootElement catalogTarget : catalogTargetList)
{
if (catalogTarget != null)
{
knownCatalogTargets.add(catalogTarget.getElementHeader().getGUID());
}
}

startFrom = startFrom + integrationContext.getMaxPageSize();
catalogTargetList = assetClient.getCatalogTargets(integrationContext.getIntegrationConnectorGUID(),
assetClient.getQueryOptions(startFrom, integrationContext.getMaxPageSize()));
}

/*
* Process any new data hubs.
*/
CollectionClient collectionClient = integrationContext.getCollectionClient(OpenMetadataType.DATA_HUB.typeName);
startFrom = 0;

List<OpenMetadataRootElement> dataHubs = collectionClient.findCollections(null,
collectionClient.getSearchOptions(startFrom, integrationContext.getMaxPageSize()));

while (dataHubs != null)
{
for (OpenMetadataRootElement dataHub : dataHubs)
{
if ((dataHub != null) && (! knownCatalogTargets.contains(dataHub.getElementHeader().getGUID())) && (dataHub.getProperties() instanceof DataHubProperties dataHubProperties))
{
/*
* This is a new data hub. Add it as a catalog target.
*/
CatalogTargetProperties catalogTargetProperties = new CatalogTargetProperties();
catalogTargetProperties.setCatalogTargetName(dataHubProperties.getDisplayName() + "(" + dataHub.getElementHeader().getGUID() + ")");
assetClient.addCatalogTarget(integrationContext.getIntegrationConnectorGUID(),
dataHub.getElementHeader().getGUID(),
assetClient.getMakeAnchorOptions(false),
catalogTargetProperties);
}
}

startFrom = startFrom + integrationContext.getMaxPageSize();
dataHubs = collectionClient.findCollections(null,
collectionClient.getSearchOptions(startFrom, integrationContext.getMaxPageSize()));
}
}
catch (Exception error)
{
auditLog.logMessage(methodName,
LiskovAuditCode.UNEXPECTED_EXCEPTION.getMessageDefinition(connectorName,
error.getClass().getName(),
methodName,
error.getMessage()));
}

super.refresh();
}
```
It is up to the implementation to determine how to discover the metadata elements of interest. However using appropriate queries, it should be possible to identify anly elements that are not in






???+ info "Code Example"
See the [LiskovDataHubManagerConnector](https://github.com/odpi/egeria/blob/main/open-metadata-implementation/adapters/open-connectors/nanny-connectors/src/main/java/org/odpi/openmetadata/adapters/connectors/liskov/DataHubManagerConnector.java)



--8<-- "snippets/abbr.md"
2 changes: 1 addition & 1 deletion site/docs/types/5/0505-Schema-Attributes.svg
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
18 changes: 9 additions & 9 deletions site/docs/types/5/area-5-models-and-schema.drawio
Original file line number Diff line number Diff line change
Expand Up @@ -440,7 +440,7 @@
</mxGraphModel>
</diagram>
<diagram name="0505-Schema-Attributes" id="4GrBEy_7Aw3WSbm6_cvN">
<mxGraphModel dx="1234" dy="835" grid="0" gridSize="10" guides="1" tooltips="1" connect="1" arrows="1" fold="1" page="1" pageScale="1" pageWidth="1169" pageHeight="827" math="0" shadow="0">
<mxGraphModel dx="1222" dy="835" grid="0" gridSize="10" guides="1" tooltips="1" connect="1" arrows="1" fold="1" page="1" pageScale="1" pageWidth="1169" pageHeight="827" math="0" shadow="0">
<root>
<mxCell id="dp80E5BfLT-rZpdb8q4U-0" />
<mxCell id="dp80E5BfLT-rZpdb8q4U-1" parent="dp80E5BfLT-rZpdb8q4U-0" />
Expand Down Expand Up @@ -477,17 +477,17 @@
<mxGeometry height="20" width="107" x="759" y="222.5" as="geometry" />
</mxCell>
<mxCell id="dp80E5BfLT-rZpdb8q4U-46" parent="dp80E5BfLT-rZpdb8q4U-1" style="text;html=1;strokeColor=none;fillColor=none;align=center;verticalAlign=middle;whiteSpace=wrap;rounded=0;" value="nestedAttributes" vertex="1">
<mxGeometry height="20" width="107" x="759" y="407.25" as="geometry" />
<mxGeometry height="20" width="107" x="759" y="424" as="geometry" />
</mxCell>
<mxCell id="dp80E5BfLT-rZpdb8q4U-47" parent="dp80E5BfLT-rZpdb8q4U-1" style="text;html=1;strokeColor=none;fillColor=none;align=center;verticalAlign=middle;whiteSpace=wrap;rounded=0;" value="*" vertex="1">
<mxGeometry height="20" width="40" x="757" y="382.5" as="geometry" />
<mxGeometry height="20" width="40" x="757" y="390.5" as="geometry" />
</mxCell>
<mxCell id="dp80E5BfLT-rZpdb8q4U-48" parent="dp80E5BfLT-rZpdb8q4U-1" style="text;html=1;strokeColor=none;fillColor=none;align=center;verticalAlign=middle;whiteSpace=wrap;rounded=0;" value="0..1" vertex="1">
<mxGeometry height="20" width="40" x="759" y="253" as="geometry" />
</mxCell>
<mxCell id="z0DgIlncA7POvNjI8am9-0" edge="1" parent="dp80E5BfLT-rZpdb8q4U-1" style="endArrow=none;dashed=1;html=1;" target="z0DgIlncA7POvNjI8am9-3" value="">
<mxGeometry height="50" relative="1" width="50" as="geometry">
<mxPoint x="377" y="249.5" as="sourcePoint" />
<mxPoint x="367" y="250" as="sourcePoint" />
<mxPoint x="864" y="430" as="targetPoint" />
</mxGeometry>
</mxCell>
Expand All @@ -509,8 +509,8 @@
<mxCell id="dp80E5BfLT-rZpdb8q4U-23" parent="dp80E5BfLT-rZpdb8q4U-1" style="html=1;strokeColor=#996185;align=center;fillColor=#e6d0de;gradientColor=#d5739d;" value="«entity»&lt;br&gt;&lt;b&gt;SchemaAttribute&lt;/b&gt;" vertex="1">
<mxGeometry height="50" width="240" x="521.5" y="225.5" as="geometry" />
</mxCell>
<mxCell id="dp80E5BfLT-rZpdb8q4U-24" parent="dp80E5BfLT-rZpdb8q4U-1" style="rounded=0;whiteSpace=wrap;html=1;strokeColor=#666666;align=left;fontColor=#333333;fillColor=#f5f5f5;" value="&lt;font style=&quot;white-space: normal&quot;&gt;allowsDuplicateValues : boolean&lt;br&gt;orderedValues : boolean&lt;br&gt;&lt;/font&gt;&lt;span&gt;defaultValueOverride : string&lt;br&gt;minimumLength : int&lt;br&gt;length : int&lt;br&gt;precision : int&lt;br&gt;isNullable : boolean&lt;br&gt;&lt;/span&gt;&lt;font style=&quot;white-space: normal&quot;&gt;nativeClass : string&lt;br&gt;aliases : array&amp;lt;string&amp;gt;&lt;br&gt;sortOrder : DataItemSortOrder&lt;br&gt;&lt;/font&gt;" vertex="1">
<mxGeometry height="169.5" width="240" x="522" y="275.5" as="geometry" />
<mxCell id="dp80E5BfLT-rZpdb8q4U-24" parent="dp80E5BfLT-rZpdb8q4U-1" style="rounded=0;whiteSpace=wrap;html=1;strokeColor=#666666;align=left;fontColor=#333333;fillColor=#f5f5f5;" value="&lt;font style=&quot;white-space: normal&quot;&gt;namespacePath : string&lt;/font&gt;&lt;div&gt;&lt;font style=&quot;white-space: normal&quot;&gt;allowsDuplicateValues : boolean&lt;br&gt;orderedValues : boolean&lt;br&gt;&lt;/font&gt;&lt;span&gt;defaultValueOverride : string&lt;br&gt;minimumLength : int&lt;br&gt;length : int&lt;br&gt;precision : int&lt;br&gt;isNullable : boolean&lt;br&gt;&lt;/span&gt;&lt;font style=&quot;white-space: normal&quot;&gt;nativeClass : string&lt;br&gt;aliases : array&amp;lt;string&amp;gt;&lt;br&gt;sortOrder : DataItemSortOrder&lt;br&gt;&lt;/font&gt;&lt;/div&gt;" vertex="1">
<mxGeometry height="183.5" width="240" x="522" y="275.5" as="geometry" />
</mxCell>
<mxCell id="dp80E5BfLT-rZpdb8q4U-27" parent="dp80E5BfLT-rZpdb8q4U-1" style="rounded=0;whiteSpace=wrap;html=1;strokeColor=#666666;align=left;fillColor=#f5f5f5;fontColor=#333333;" value="schemaTypeName : string&lt;div&gt;qualifiedName : string&lt;br&gt;displayName : string&lt;br&gt;description : string&lt;br&gt;versionIdentifier : string&lt;/div&gt;&lt;div&gt;category : string&lt;br&gt;usage : string&lt;br&gt;encodingStandard : string&lt;br&gt;namespacePath : string&lt;br&gt;dataType : string&lt;br&gt;defaultValue : string&lt;br&gt;fixedValue : string&lt;br&gt;additionalProperties : map&amp;lt;string, String&amp;gt;&lt;br&gt;&lt;div style=&quot;text-align: left&quot;&gt;&lt;/div&gt;&lt;/div&gt;" vertex="1">
<mxGeometry height="219" width="241" x="521" y="531" as="geometry" />
Expand All @@ -522,15 +522,15 @@
<mxGeometry relative="1" as="geometry">
<Array as="points">
<mxPoint x="796" y="251" />
<mxPoint x="796" y="403" />
<mxPoint x="796" y="413" />
</Array>
</mxGeometry>
</mxCell>
<mxCell id="dp80E5BfLT-rZpdb8q4U-42" parent="dp80E5BfLT-rZpdb8q4U-1" style="html=1;strokeColor=#996185;align=center;fillColor=#e6d0de;gradientColor=#d5739d;" value="«relationship»&lt;br&gt;&lt;b&gt;NestedSchemaAttribute&lt;/b&gt;" vertex="1">
<mxGeometry height="53" width="216" x="841" y="265" as="geometry" />
<mxGeometry height="53" width="216" x="841" y="273" as="geometry" />
</mxCell>
<mxCell id="dp80E5BfLT-rZpdb8q4U-43" parent="dp80E5BfLT-rZpdb8q4U-1" style="rounded=0;whiteSpace=wrap;html=1;strokeColor=#666666;align=left;fillColor=#f5f5f5;fontColor=#333333;" value="&lt;span style=&quot;forced-color-adjust: none; color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-ligatures: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; orphans: 2; text-indent: 0px; text-transform: none; widows: 2; word-spacing: 0px; -webkit-text-stroke-width: 0px; white-space: normal; background-color: rgb(251, 251, 251); text-decoration-thickness: initial; text-decoration-style: initial; text-decoration-color: initial;&quot;&gt;position : int&lt;/span&gt;&lt;br style=&quot;forced-color-adjust: none; color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-ligatures: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; orphans: 2; text-indent: 0px; text-transform: none; widows: 2; word-spacing: 0px; -webkit-text-stroke-width: 0px; white-space: normal; background-color: rgb(251, 251, 251); text-decoration-thickness: initial; text-decoration-style: initial; text-decoration-color: initial;&quot;&gt;&lt;font style=&quot;forced-color-adjust: none; color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-ligatures: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; orphans: 2; text-indent: 0px; text-transform: none; widows: 2; word-spacing: 0px; -webkit-text-stroke-width: 0px; white-space: normal; background-color: rgb(251, 251, 251); text-decoration-thickness: initial; text-decoration-style: initial; text-decoration-color: initial;&quot;&gt;minCardinality : int&lt;br style=&quot;forced-color-adjust: none;&quot;&gt;maxCardinality : int&lt;/font&gt;&lt;div&gt;&lt;font style=&quot;forced-color-adjust: none; color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-ligatures: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; orphans: 2; text-indent: 0px; text-transform: none; widows: 2; word-spacing: 0px; -webkit-text-stroke-width: 0px; white-space: normal; text-decoration-thickness: initial; text-decoration-style: initial; text-decoration-color: initial;&quot;&gt;&lt;span style=&quot;color: rgb(51, 51, 51);&quot;&gt;coverageCategory : CoverageCategory&lt;/span&gt;&lt;/font&gt;&lt;/div&gt;" vertex="1">
<mxGeometry height="78" width="215" x="841" y="318" as="geometry" />
<mxGeometry height="78" width="215" x="841" y="326" as="geometry" />
</mxCell>
<mxCell id="z0DgIlncA7POvNjI8am9-3" parent="dp80E5BfLT-rZpdb8q4U-1" style="html=1;strokeColor=#996185;align=center;fillColor=#e6d0de;gradientColor=#d5739d;" value="«relationship»&lt;br&gt;&lt;b&gt;AttributeForSchema&lt;/b&gt;" vertex="1">
<mxGeometry height="53" width="219" x="257" y="347.5" as="geometry" />
Expand Down
4 changes: 2 additions & 2 deletions site/docs/types/7/0705-Data-Sharing.md
Original file line number Diff line number Diff line change
Expand Up @@ -10,10 +10,10 @@ Data sharing describes the process of sharing data between different organizatio

## DataHub entity

The *DataHub* entity describes a collection of data assets that are available for sharing, as long as the requester satisfies the requirements laid down by the data owner.
The *DataHub* entity describes a [collection](/types/0/0021-Collections) of data assets that are available for sharing, as long as the requester satisfies the requirements laid down by the data owner.

## DataSharingRequest entity

The *DataSharingRequest* entity describes a request for data sharing. It is used to track the status of the request and gather the details of the request (such as the requested data specification) and the data sharing agreement and related resources.
The *DataSharingRequest* entity describes a request to another party for them to share data. It is a type of [ToDo](/types/1/0135-Actions-For-People) since there is work involved. It is used to track the status of the request and gather the details of the request (such as the requested data specification) and the data sharing agreement and related resources.

--8<-- "snippets/abbr.md"
2 changes: 1 addition & 1 deletion site/docs/types/7/0705-Data-Sharing.svg
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Loading