Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion site/docs/concepts/asset.md
Original file line number Diff line number Diff line change
Expand Up @@ -52,6 +52,6 @@ Egeria's [Asset Maker API](/services/omvs/asset-maker/overview) provide the spec

Egeria's [Open Metadata Repository Services (OMRS)](/services/omrs) provides the ability to store and extract information about assets in a distributed collection of servers called an [open metadata repository cohort](/concepts/cohort-member). The cohort provides both peer-to-peer exchange of metadata via an event bus topic and federated queries between different members of the cohort. Egeria provides a [metadata access store](/concepts/metadata-access-store), a [metadata access point](/concepts/metadata-access-point) and a [repository proxy](/concepts/repository-proxy) server that are all able to join a cohort. The repository proxy supports the integration of third party servers (typically [asset managers](/software-capability/#asset-manager)) into the cohort. The mapping between the third party server's APIs and the open metadata APIs in this case is implemented in a [repository connector](/concepts/repository-connector).

It is also possible to manage the exchange of asset metadata with other types of third party technologies using the [Integration Connectors](/concepts/integraiton-connector) running in an [integration daemon](/concepts/integration-daemon). Using this pattern is simpler to integrate but involves maintaining a copy of the third party technology's metadata in a [metadata access store](/concepts/metadata-access-store) that can then join one or more open metadata repository cohorts to share this metadata more broadly. The mapping between the third party technology's APIs and the open metadata APIs in this case is implemented in an [integration connector](/connectors/integration-connector).
It is also possible to manage the exchange of asset metadata with other types of third party technologies using the [Integration Connectors](/concepts/integration-connector) running in an [integration daemon](/concepts/integration-daemon). Using this pattern is simpler to integrate but involves maintaining a copy of the third party technology's metadata in a [metadata access store](/concepts/metadata-access-store) that can then join one or more open metadata repository cohorts to share this metadata more broadly. The mapping between the third party technology's APIs and the open metadata APIs in this case is implemented in an [integration connector](/connectors/integration-connector).

--8<-- "snippets/abbr.md"
2 changes: 1 addition & 1 deletion site/docs/concepts/audit-log-record-structure.svg
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
29 changes: 29 additions & 0 deletions site/docs/concepts/governance-definition.md
Original file line number Diff line number Diff line change
Expand Up @@ -4,11 +4,40 @@

# Governance Definition

Governance definitions are used to record precise requirements, decisions, intents around governance to help people understand what is being asked of them, why, and how their activity will be measured.

They are hard to define and are often revised as a [governance domain](/concepts/governance-domain) matures. However, they keep the investment and metrics targeted, making it easier to measure, justify and improve.

There are three levels of governance definitions: drivers, policies and controls.

## Governance Driver

A [governance driver](/types/4/0405-Governance-Drivers) describes a motivation to act. It may be a regulation, threat, or an aspect of the business strategy (called a business imperative) that requires a change in the organization.

## Governance Policy

[Governance policies](/types/4/0415-Governance-Responses) describe what is required of the organization. Typically they describe the desired future state (or at least a desirable intermediate state :). There are three main types of policies:

* Principles define the overall values and guidelines for the organization. These help people decide how to act in a situation where there are no rules laid down.
* Obligations define specific actions or processing that must be accommodated.
* Approaches define new initiatives, best practices and ways of working that will be adopted.

They are said to be the "responses to the challenges proposed by the governance drivers" and so they are linked to their relevant governance drivers using the [GovernanceResponse](/types/4/0415-Governance-Responses) relationship.

## Governance Control

Governance is enabled through People, Process and Technology. The governance controls describe actions and activities that support achieving the governance policies that support the drivers. Traditionally governance controls are divided into technical controls (implemented IT function) and organizational controls (training, responsibility, buddy-checking etc). In Egeria we do not formally distinguish between technical and organization controls in the type system since the dividing line between the things that humans do and those that are automated is changing all the time.

[Governance controls](/types/4/0425-Governance-Controls) are linked to their relevant governance policies using the [GovernanceMechanism](/types/4/0425-Governance-Controls) relationship. They include [Governance Actions](/concepts/governance-action) and [Governance Zones](/concepts/governance-zone).

???+ info "Further information"

Definind governance definitions appear in the following Coco Pharmaceuticals scenarios:

* [Creating a data governance program](/practices/coco-pharmaceuticals/scenarios/creating-data-governance-program/overview)
* [Creating a sustainability initiative](/practices/coco-pharmaceuticals/scenarios/sustainability-initiative/overview)

You can browse the governance definitions using [egeria-explorer](/user-interfaces/egeria-explorer/overview) under the **Governance Definitions** tile. They are organized by type and can be filtered by **domainIdentifier**. Setting up a new governance definition is through the [Governance Officer API](/services/omvs/governance-officer/overview).


--8<-- "snippets/abbr.md"
65 changes: 52 additions & 13 deletions site/docs/concepts/information-supply-chain.md
Original file line number Diff line number Diff line change
@@ -1,28 +1,67 @@
---
hide:
- toc
---

<!-- SPDX-License-Identifier: CC-BY-4.0 -->
<!-- Copyright Contributors to the Egeria project. -->


# Information supply chain

Lineage is typically captured from the running technology at a very fine-level of detail. This detail enables automated governance processes to validate that the data pipelines are running as they should be. However, this level of detail is too much for most business users and regulators to comprehend. Therefore, it is necessary to model the flow of data at a level of detail that is meaningful to these users and correlate the fine-grained detail to this business view.
[Lineage](/concepts/lineage) is typically captured at a very fine-level of detail. This detail enables automated governance processes to validate that the data pipelines are running as they should be. However, this level of detail is often too much for most business users and regulators to comprehend. It is necessary to model the flow of data at a level of detail that is meaningful to these users and correlate the fine-grained detail to this business view.

An *InformationSupplyChain* serves as a high-level depiction of data and control movement. It generally outlines how a particular kind of data travels through a digital environment. For instance, one information supply chain might track risk data for a regulatory compliance report, while another illustrates the assembly of data required for carbon accounting.

So the information supply chain is an architectural construct that can provide the focus for data governance or compliance activities that must demonstrate data being delivered where it is needed in a timely and efficient manner.

However, the information supply chain is more than a mere diagram. It is associated with the real lineage producing components that are executing the data flow. It provides a target for roll-ups of activity, errors and volumetrics extracted from the underlying lineage data. The traceability back to the actual components and lineage logs supports validation, investigation and auditability of the information supply chain metrics.

## Information Supply Chain Segments

An information supply chain may be broken down into multiple, nested *segments*. Each segment may represent, for example, the portion of the information supply chain that is owned by a particular team, or the data flows that occur at different phases of the work. This allows drill-down and more focused analysis by the owners of each segment.

## Information Supply Chain Examples

The skill in using information supply chains is to get the right level of abstraction to make its description and analysis meaningful to its intended audience. As such they tend to be designed as part of an initiative or project, where the audience is understood.

In the [Coco Pharmaceuticals](/practices/coco-pharmaceuticals) scenarios we see multiple examples of information supply chains:

* In the [Investigating suspicious activity](/practices/coco-pharmaceuticals/scenarios/investigating-suspicious-activity/overview) scenario, Coco Pharmaceuticals use an information supply chain to model the flow of supplier information and orders through their systems to identify where bogus supplier information could be inserted without detection.
* In the [Sustainability initiative](/practices/coco-pharmaceuticals/scenarios/sustainability-initiative/overview) scenario, Coco Pharmaceuticals model an information supply chain during the development of their new sustainability data pipeline.
* In [Receiving patient data from a hospital](/practices/coco-pharmaceuticals/scenarios/receiving-patient-data-from-a-hospital/overview), we seem various Coco Pharmaceuticals persona using the information supply chains supporting their clinical trials to understand the process of a clinical trial and any issues that need action.
* In the initial set up of the data governance program, we see Coco Pharmaceuticals [identifying the key information supply chains](/practices/coco-pharmaceuticals/scenarios/defining-information-supply-chains/overview) that the program will need to support.

## Working with Information Supply Chains

Information supply chains can be viewed in [Egeria Explorer](/user-interfaces/egeria-explorer/overview) by clicking on the **Information Supply Chains** card. The [Quickstart](/egeria-workspaces/quick-start/overview) environment includes multiple examples of information supply chains, both from Coco Pharmaceuticals and those that are part of the base Egeria implementation.

The [Lineage Explorer](/user-interfaces/lineage-explorer/overview) supports linking from a technical component in a lineage graph to its associated information supply chains in *Egeria Explorer*.

The [Solution Architect API](/services/omvs/solution-architect/overview) is where you can define, query and maintain your information supply chains. This includes the standard OMVS REST API, python interface and Java client libraries for connectors. They are also fully supported by [Dr.Egeria](/user-interfaces/dr-egeria/overview) in the *Solution Architect* family.

## Modelling Information Supply Chains

An information supply chain is represented by an [*InformationSupplyChain*](/types/7/0720-Information-Supply-Chains) entity. This is a type of [collection](/types/0/0021-Collections). So, its nested segments, and any parent information supply chain are organised into a hierarchy using the [*CollectionMembership*](/types/0/0021-Collections) relationship. The handoff of control/data flow between information supply chains is captured using the [*InformationSupplyChainLink*](/types/7/0720-Information-Supply-Chains) relationship. This relationship can also be used to link the business elements (typically [business capability](/concepts/business-capability) or [actor](/concepts/actor)) that initiates or consumes the data/control flow.

### Information Supply Chain Design

The high-level components illustrating the flow of data/control are captured by [Solution Components](/concepts/solution-component). Providing a design is optional, but recommented since it creates a pictorial view of the flow for people that find the detail overwhelming and/or do not have access to the technical assets that are involved.

A solution component may represent an implementation that has multiple information supply chains flowing through it. The linkage between the *InformationSupplyChain* and a relevant solution component uses a [CollectionMembership](/types/0/0021-Collections) relationship. The flows between these solution component in the information supply chain's design are [SolutionLinkingWire](/types/7/0735-Solution-Ports-and-Wires) relationships that have the qualified name of the information supply chain stored in their *iscQualifiedNames* attribute.

An *InformationSupplyChain* is the highest level of abstraction for lineage. It typically identifies the flow of a particular type of data across a digital landscape. For example, there may be an information supply chain capturing risk data for a regulatory compliance report, and another one showing how the data necessary for carbon accounting is assembled.
An information supply chain segment is also represented by an *InformationSupplyChain* entity. It is also linked to its parent using the *CollectionMembership* relationship.

It is an architectural construct that can provide the focus for data governance activities that ensure data is delivered where it is needed in a timely and efficient manner.
### Lineage Implementation

The information supply chain is typically broken down into segments represented by *InformationSupplyChainSegment* entities that each define a particular exchange of data. For example, it may be the flow of data through a specific business unit or country. The segments are linked to their information supply chain via the *InformationSupplyChainComposition* relationship. Their relationship to one another, and its logical sources and destinations are captured using *InformationSupplyChainLink* relationships.
The unique identifiers/names of the information supply chain segments, or information supply chain itself, are encoded in the *iscQualifiedName* attribute of the lineage relationships captured in the lower levels of lineage detail.

The unique identifiers/names of the information supply chain segments, or information supply chain itself, are encoded in the lineage relationships captured in the lower levels of lineage detail:
The lineage relationships that support information supply chains include:

* The architectural view of the flow of data that the information supply chain represents is described using [solution components](/concepts/solution-somponent) that are linked via [ports and wires](/types/7/0735-Solution-Ports-and-Wires). The [*SolutionLinkingWire*](/types/7/0735-Solution-Ports-and-Wires) contains the unique identifiers (GUIDs) of the information supply chain segments that it implements. Typically, each identified segment is for a different information supply chain that happen to share a part of the data flow.
* The [data passing](/types/7/0750-Data-Passing) relationships: DataFlow, ControlFlow and ProcessCall.
* The [lineage mapping](/types/7/0770-Lineage-Mapping) relationships: LineageMapping and DataMapping.
* The [ultimate edges](/types/7/0755-Ultimate-Source-Destination) relationships: UltimateSource and UltimateDesination.
* The [DataSetContent](/types/2/0210-Data-Stores) relationship.
* The [DerivedSchemaTypeQueryTarget](/types/5/0512-Derived-Schema-Elements) relationship.
* The [ImplementedBy](/types/7/0737-Solution-Implementation) relationship.

* The fine-grained view captured from the implementation and operation of the deployed technology is described using the [data passing](/types/7/0750-Data-Passing) and [lineage mapping](/types/7/0770-Lineage-Mapping) relationships.

If additional processing is required to move the data between solution components *along the wire*, for example via an ETL process, this is shown by an *ImplementedBy* relationship between the information supply chain segment and the [Process](/types/0/0010-Base-Model) that represents the implementation. Alternatively, the component that provides the data movement can be show explicitly as a solution component in the [solution blueprint](/types/7/0740-Solution-Blueprints).
???+ info "Further information"
* [Lineage Management](/features/lineage-management/overview)

--8<-- "snippets/abbr.md"
2 changes: 1 addition & 1 deletion site/docs/concepts/integration-daemon.md
Original file line number Diff line number Diff line change
Expand Up @@ -24,7 +24,7 @@ The specific interface that the integration connector needs to implement is defi

The integration connectors that run inside the integration daemon are managed dynamically as open metadata using [integration groups](/concepts/integration-group). Integration connectors are able to modify the integration groups dynamically, and also initiate [governance actions](/concepts/governance-action), create [to dos](/concepts/to-do) and raise [incident reports](/concepts/incident-report).

Each time the integration connector is called (refreshed) by the integration daemon, it produces an [integration report](/concpets/integraiton-report) listing the metadata elements it has created, updated or deleted. The integration report is stored in the open metadata repository. It is possible to disable this function in the integration connector's configuration.
Each time the integration connector is called (refreshed) by the integration daemon, it produces an [integration report](/concpets/integration-report) listing the metadata elements it has created, updated or deleted. The integration report is stored in the open metadata repository. It is possible to disable this function in the integration connector's configuration.

!!! info "Further information"

Expand Down
5 changes: 3 additions & 2 deletions site/docs/concepts/lineage.md
Original file line number Diff line number Diff line change
Expand Up @@ -11,7 +11,8 @@ Lineage shows how data and control flows from its origins to its various destina

* whether the operational processes that implement the data flows are executing correctly (known as *governance by expectation*).

[Lineage Management](/features/lineage-management/overview) describes how lineage is collected, managed and used in Egeria.
[Lineage Management](/features/lineage-management/overview) describes how lineage is collected, managed and used in Egeria. [Information Supply Chains](/concepts/information-supply-chain) are used to create business-friendly views of lineage.


???+ info "Further information"

Expand All @@ -22,7 +23,7 @@ Lineage shows how data and control flows from its origins to its various destina
* [Model 0750](/types/7/0750-Data-Passing) for the DataFlow, ProcessCall and ControlFlow relationships.
* [Model 0755](/types/7/0755-Ultimate-Source-Destination) for the UltimateSource and UltimateDestination relationships.
* [Model 0760](/types/7/0760-Business-Lineage) for the BusinessLineage classification.
* [Model 0770](/types/7/0770-Lineage-Mapping) for the LineageMapping and DataMapping relaitonships.
* [Model 0770](/types/7/0770-Lineage-Mapping) for the LineageMapping and DataMapping relationships.

Also see how to set up an [information supply chain](/concepts/information-supply-chain).

Expand Down
Loading
Loading