Skip to content
Draft
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
221 changes: 103 additions & 118 deletions solutions/observability/streams/management/extract.md

Large diffs are not rendered by default.

Original file line number Diff line number Diff line change
Expand Up @@ -24,7 +24,7 @@ To use an append processor:
1. Set **Source Field** to the field you want append values to.
1. Set **Target field** to the values you want to append to the **Source Field**.

This functionality uses the {{es}} [append processor](elasticsearch://reference/enrich-processor/append-processor.md) internally, but you configure it in Streamlang. Streamlang doesn't always have 1:1 parity with the ingest processor options and behavior. Refer to [Processor limitations and inconsistencies](../extract.md#streams-processor-inconsistencies).
This functionality uses the {{es}} [append processor](elasticsearch://reference/enrich-processor/append-processor.md) internally, but you configure it in Streamlang. Streamlang doesn't always have 1:1 parity with the ingest processor options and behavior. Refer to [Processor limitations and inconsistencies](../streamlang.md#streams-processor-inconsistencies).

## YAML reference [streams-append-yaml-reference]

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -29,7 +29,7 @@ To convert a field to a different data type:
If you add a **Convert** processor inside a condition group (a **WHERE** block), you must set a **Target field**.
::::

This functionality uses the {{es}} [Convert processor](elasticsearch://reference/enrich-processor/convert-processor.md) internally, but you configure it in Streamlang. Streamlang doesn't always have 1:1 parity with the ingest processor options and behavior. Refer to [Processor limitations and inconsistencies](../extract.md#streams-processor-inconsistencies).
This functionality uses the {{es}} [Convert processor](elasticsearch://reference/enrich-processor/convert-processor.md) internally, but you configure it in Streamlang. Streamlang doesn't always have 1:1 parity with the ingest processor options and behavior. Refer to [Processor limitations and inconsistencies](../streamlang.md#streams-processor-inconsistencies).

## YAML reference [streams-convert-yaml-reference]

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -25,7 +25,7 @@ To extract a timestamp field using the date processor:
1. Set the **Source Field** to the field containing the timestamp.
1. Set the **Format** field to one of the accepted date formats (ISO8601, UNIX, UNIX_MS, or TAI64N) or use a Java time pattern. Refer to the [example formats](#streams-date-examples) for more information.

This functionality uses the {{es}} [Date processor](elasticsearch://reference/enrich-processor/date-processor.md) internally, but you configure it in Streamlang. Streamlang doesn't always have 1:1 parity with the ingest processor options and behavior. Refer to [Processor limitations and inconsistencies](../extract.md#streams-processor-inconsistencies).
This functionality uses the {{es}} [Date processor](elasticsearch://reference/enrich-processor/date-processor.md) internally, but you configure it in Streamlang. Streamlang doesn't always have 1:1 parity with the ingest processor options and behavior. Refer to [Processor limitations and inconsistencies](../streamlang.md#streams-processor-inconsistencies).

## Example formats [streams-date-examples]

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -26,7 +26,7 @@ To parse a log message with a dissect processor:
1. Set the **Source Field** to the field you want to dissect.
1. Set the delimiters you want to use in the **Pattern** field. Refer to the [example pattern](#streams-dissect-example) for more information on setting delimiters.

This functionality uses the {{es}} [Dissect processor](elasticsearch://reference/enrich-processor/dissect-processor.md) internally, but you configure it in Streamlang. Streamlang doesn’t always have 1:1 parity with the ingest processor options and behavior. Refer to [Processor limitations and inconsistencies](../extract.md#streams-processor-inconsistencies).
This functionality uses the {{es}} [Dissect processor](elasticsearch://reference/enrich-processor/dissect-processor.md) internally, but you configure it in Streamlang. Streamlang doesn’t always have 1:1 parity with the ingest processor options and behavior. Refer to [Processor limitations and inconsistencies](../streamlang.md#streams-processor-inconsistencies).

## Example dissect pattern [streams-dissect-example]

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -28,7 +28,7 @@ To configure a condition for dropping documents:
The default is the `always` condition. Not setting a specific condition results in every document that matches the drop condition getting dropped from indexing.
:::

This functionality uses the {{es}} [Drop processor](elasticsearch://reference/enrich-processor/drop-processor.md) internally, but you configure it in Streamlang. Streamlang doesn't always have 1:1 parity with the ingest processor options and behavior. Refer to [Processor limitations and inconsistencies](../extract.md#streams-processor-inconsistencies).
This functionality uses the {{es}} [Drop processor](elasticsearch://reference/enrich-processor/drop-processor.md) internally, but you configure it in Streamlang. Streamlang doesn't always have 1:1 parity with the ingest processor options and behavior. Refer to [Processor limitations and inconsistencies](../streamlang.md#streams-processor-inconsistencies).

## YAML reference [streams-drop-yaml-reference]

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -27,7 +27,7 @@ To enrich documents:
1. Select an **Enrich policy** from the list of available policies.
1. Set **Target field** to the field where the enriched data is stored.

This functionality uses the {{es}} [Enrich processor](elasticsearch://reference/enrich-processor/enrich-processor.md) internally, but you configure it in Streamlang. Streamlang doesn't always have 1:1 parity with the ingest processor options and behavior. Refer to [Processor limitations and inconsistencies](../extract.md#streams-processor-inconsistencies).
This functionality uses the {{es}} [Enrich processor](elasticsearch://reference/enrich-processor/enrich-processor.md) internally, but you configure it in Streamlang. Streamlang doesn't always have 1:1 parity with the ingest processor options and behavior. Refer to [Processor limitations and inconsistencies](../streamlang.md#streams-processor-inconsistencies).

## YAML reference [streams-enrich-yaml-reference]

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -30,7 +30,7 @@ To parse a log message with a grok processor:
1. Set the **Source Field** to the field you want to search for grok matches.
1. Set the patterns you want to use in the **Grok patterns** field. Refer to the [example pattern](#streams-grok-example) for more information on patterns.

This functionality uses the {{es}} [Grok processor](elasticsearch://reference/enrich-processor/grok-processor.md) internally, but you configure it in Streamlang. Streamlang doesn’t always have 1:1 parity with the ingest processor options and behavior. Refer to [Processor limitations and inconsistencies](../extract.md#streams-processor-inconsistencies).
This functionality uses the {{es}} [Grok processor](elasticsearch://reference/enrich-processor/grok-processor.md) internally, but you configure it in Streamlang. Streamlang doesn’t always have 1:1 parity with the ingest processor options and behavior. Refer to [Processor limitations and inconsistencies](../streamlang.md#streams-processor-inconsistencies).

## Example grok pattern [streams-grok-example]

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -25,7 +25,7 @@ To redact sensitive information:
1. Set the **Source Field** to the field containing text you want to redact.
1. Set the **Patterns** to one or more grok patterns that match sensitive data (for example, IP addresses or email addresses).

This functionality uses the {{es}} [Redact processor](elasticsearch://reference/enrich-processor/redact-processor.md) internally, but you configure it in Streamlang. Streamlang doesn't always have 1:1 parity with the ingest processor options and behavior. Refer to [Processor limitations and inconsistencies](../extract.md#streams-processor-inconsistencies).
This functionality uses the {{es}} [Redact processor](elasticsearch://reference/enrich-processor/redact-processor.md) internally, but you configure it in Streamlang. Streamlang doesn't always have 1:1 parity with the ingest processor options and behavior. Refer to [Processor limitations and inconsistencies](../streamlang.md#streams-processor-inconsistencies).

## YAML reference [streams-redact-yaml-reference]

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -24,7 +24,7 @@ To remove a field:
1. From the **Processor** menu, select **Remove** to remove a field or **Remove by prefix** to remove a field and all its nested fields.
1. Set the **Source Field** to the field you want to remove.

This functionality uses the {{es}} [Remove processor](elasticsearch://reference/enrich-processor/remove-processor.md) internally, but you configure it in Streamlang. Streamlang doesn't always have 1:1 parity with the ingest processor options and behavior. Refer to [Processor limitations and inconsistencies](../extract.md#streams-processor-inconsistencies).
This functionality uses the {{es}} [Remove processor](elasticsearch://reference/enrich-processor/remove-processor.md) internally, but you configure it in Streamlang. Streamlang doesn't always have 1:1 parity with the ingest processor options and behavior. Refer to [Processor limitations and inconsistencies](../streamlang.md#streams-processor-inconsistencies).

## YAML reference [streams-remove-yaml-reference]

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -24,7 +24,7 @@ To use a rename processor:
1. Set **Source Field** to the field you want to rename.
1. Set **Target field** to the new name you want to use for the **Source Field**.

This functionality uses the {{es}} [Rename processor](elasticsearch://reference/enrich-processor/rename-processor.md) internally, but you configure it in Streamlang. Streamlang doesn't always have 1:1 parity with the ingest processor options and behavior. Refer to [Processor limitations and inconsistencies](../extract.md#streams-processor-inconsistencies).
This functionality uses the {{es}} [Rename processor](elasticsearch://reference/enrich-processor/rename-processor.md) internally, but you configure it in Streamlang. Streamlang doesn't always have 1:1 parity with the ingest processor options and behavior. Refer to [Processor limitations and inconsistencies](../streamlang.md#streams-processor-inconsistencies).

## YAML reference [streams-rename-yaml-reference]

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -26,7 +26,7 @@ To use the **Replace** processor:
1. Set the **Pattern** to the regular expression or text that you want to replace.
1. Set the **Replacement** to the value that will replace the portion of the string matching your pattern. Replacements can be text, an empty value, or a capture group reference.

This functionality uses the {{es}} [Gsub processor](elasticsearch://reference/enrich-processor/gsub-processor.md) internally, but you configure it in Streamlang. Streamlang doesn't always have 1:1 parity with the ingest processor options and behavior. Refer to [Processor limitations and inconsistencies](../extract.md#streams-processor-inconsistencies).
This functionality uses the {{es}} [Gsub processor](elasticsearch://reference/enrich-processor/gsub-processor.md) internally, but you configure it in Streamlang. Streamlang doesn't always have 1:1 parity with the ingest processor options and behavior. Refer to [Processor limitations and inconsistencies](../streamlang.md#streams-processor-inconsistencies).

## YAML reference [streams-replace-yaml-reference]

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -24,7 +24,7 @@ To use a set processor:
1. Set **Source Field** to the field you want to insert, upsert, or update.
1. Set **Value** to the value you want the source field to be set to.

This functionality uses the {{es}} [Set processor](elasticsearch://reference/enrich-processor/set-processor.md) internally, but you configure it in Streamlang. Streamlang doesn't always have 1:1 parity with the ingest processor options and behavior. Refer to [Processor limitations and inconsistencies](../extract.md#streams-processor-inconsistencies).
This functionality uses the {{es}} [Set processor](elasticsearch://reference/enrich-processor/set-processor.md) internally, but you configure it in Streamlang. Streamlang doesn't always have 1:1 parity with the ingest processor options and behavior. Refer to [Processor limitations and inconsistencies](../streamlang.md#streams-processor-inconsistencies).

## YAML reference [streams-set-yaml-reference]

Expand Down
10 changes: 10 additions & 0 deletions solutions/observability/streams/management/streamlang.md
Original file line number Diff line number Diff line change
Expand Up @@ -78,6 +78,16 @@
| [`trim`](./extract/trim.md) | {applies_to}`stack: ga 9.4+` Removes leading and trailing whitespace from a string field. |
| [`uppercase`](./extract/uppercase.md) | {applies_to}`stack: ga 9.4+` Converts a string field to uppercase. |

### Processor limitations and inconsistencies [streams-processor-inconsistencies]

Streams exposes a Streamlang configuration, but internally it relies on {{es}} ingest pipeline processors and {{esql}}. Streamlang doesn't always have 1:1 parity with the ingest processors because it needs to support options that work in both ingest pipelines and {{esql}}. In most cases, you won't need to worry about these details, but the underlying design decisions still affect the UI and available configuration options. The following are some limitations and inconsistencies when using Streamlang processors:

Check notice on line 83 in solutions/observability/streams/management/streamlang.md

View workflow job for this annotation

GitHub Actions / build / vale

Elastic.Wordiness: Consider using 'usually' instead of 'In most cases'.

- **Consistently typed fields**: {{esql}} requires one consistent type per column, so workflows that produce mixed types across documents won't transpile.
- **Conversion of types**: {{esql}} and ingest pipelines accept different conversion combinations and strictness (especially for strings), so `convert` can behave differently across targets.
- **Multi-value commands/functions**: Fields can contain one or multiple values. {{esql}} and ingest processors don't always handle these cases the same way. For example, grok in {{esql}} handles multiple values automatically, while the grok processor does not.
- **Conditional execution**: {{esql}}'s enforced table shape limits conditional casting, parsing, and wildcard field operations that ingest pipelines can do per-document.
- **Arrays of objects / flattening**: Ingest pipelines preserve nested JSON arrays, while {{esql}} flattens to columns, so operations like rename and delete on parent objects can differ or fail.

## Conditions [streams-streamlang-conditions]

Conditions are Boolean expressions that control when processors run and how wired streams route data into partitions. They appear in `where` clauses on processors, in [condition blocks](#streams-streamlang-condition-blocks), and in stream [partitioning](#streams-streamlang-partition-conditions).
Expand Down
Loading