The OpenTelemetry project (OTEL) defines standard vocabularies for traces, metrics, and logs. Korrel8r can correlate OTEL data with other data formats for observability signals.

Korrel8r complements OTEL by:

A domain represents one type of signal or resource. For example: - k8s domain: Kubernetes resources (Pods, Services, Deployments, etc.) - log domain: Application and system logs - metric domain: Prometheus metrics - trace domain: Distributed traces - alert domain: Prometheus alerts

Available domains are described in the Domain Reference.

Each domain defines the following abstractions, allowing Korrel8r to treat domains uniformly:

Rules express relationships between classes, possibly in different domains.
For example: - "Pods belong to a Deployment" - relates k8s:Pod to k8s:Deployment - "Pods generate logs" - relates k8s:Pod to log:application - "Applications emit metrics" - relates k8s:Pod to metric:metric

Rules are templates that generate a goal query using data from a start object. The start and goal can be different classes, possibly from different domains. If a rule cannot be applied to an object it may return a blank string, or raise an error.

The set of rules forms a graph connecting all the classes of data that korrel8r knows about. Korrel8r works by traversing this graph: applying rules to some initial objects, executing the resulting queries, retrieving more objects and so on.

You can run Korrel8r locally for development and testing. This connects to cluster stores via routes or ingress.

Your local Korrel8r service is now available at http://localhost:8080.

Once your Korrel8r service is running, it provides a REST API. You can interact with it in several ways:

Every entry in the stores section has a domain field to identify the domain. Other fields depend on the domain, see Domain Reference.

Store fields may contain templates that expand to URLs.

Korrel8r comes with a comprehensive set of rules by default, but you can modify them or add your own.

A rule has the following key elements:

The query template should generate a string of the form:

The query-details part depends on the domain, see Domain Reference

Korrel8r rules and store configuration can include Go templates.

Korrel8r provides additional template functions to simplify writing rules and configurations:

There is a single class alert:alert.

An alert object is represented by this Go type. Rules starting from an alert should use the capitalized Go field names rather than the lowercase JSON names. Object

A JSON map of string names to string values, matched against alert labels, for example:

To query mutiple alerts at the same time, it’s possible to provide an array of maps:

A client of Prometheus and/or AlertManager. Store configuration:

Either or both of metrics or alertmanager may be present.

There is a single class incident:incident.

An incident object contains id and mapping to the sources (alert is the only supported source type at the moment).

One can query the incident by its id.

It’s also possible to provide labels from an alert to get a corresponding incident.

A client of Prometheus. Store configuration:

Class represents a kind of kubernetes resource. The format of a class name is: "k8s:KIND\[.VERSION]\[.GROUP]".

Missing VERSION implies "v1", if present VERSION must follow the Kubernetes version patterns. Missing GROUP implies the core group.

Examples: k8s:Pod, ks8:Pod/v1, k8s:Deployment.apps, k8s:Deployment.apps/v1, k8s:Route.route.openshift.io/v1

Object represents a kubernetes resource as a map, map keys are serialized field names. Rule templates should use the JSON (lowerCase) field names, NOT the UpperCase Go struct field names.

Example:

The k8s domain automatically connects to the current cluster (as determined by kubectl), no additional configuration is needed.

The following template functions are available to rules.

There are 3 classes corresponding to the 3 openshift logging log types:

A log object is a set of attributes in a map with string keys and values.
Attribute names contain only ASCII letters, digits, underscores, and (required for Loki labels). Other characters are replaced with "_".

For stored logs: Loki stream labels and structured metadata labels become attributes. If the log body is a JSON object, all nested field paths are flattened into attribute names.

For direct logs: the pod, namespace, and label meta-data are added as attributes using the same label names as for stored logs.

Special attributes:

A query starts with the log class, followed by one of the following:

The store configuration must include domain: log. It may include an optional loki: URL or lokiStack: URL, but not both. It may include an optional direct: true to enable direct API server logs.

The default configuration is:

Korrel8r log queries will do the following:

You can also configure a store with only direct: true to use only API server access, or only loki or lokiStack to use only stored logs.

The following functions can be used in rule templates when the log domain is available:

There is only one class: metric:metric

A Metric is a time series identified by a label set. Korrel8r does not consider load the sample data for a time series, or use it in rules. If a korrel8r search time constraints, then metrics that have no values that meet the constraint are ignored.

Query is a PromQL query string.

Korrel8r uses metric labels for correlation, it does not use time-series data values. The PromQL query is analyzed to identify series it uses, labels of those series are used for correlation.

Prometheus is the store, store configuration:

There is a single class netflow:network

A log object is a JSON map\[string]any in NetFlow format.

A query is a LogQL query string, prefixed by netflow:network:, for example:

To connect to a netflow lokiStack store use this configuration:

To connect to plain loki store use:

Class singleton trace:span representing an OpenTelemetry span

A trace is simply a set of spans with the same trace-id. There is no explicit class or object representing a trace.

Object represents an OpenTelemetry span

A trace is simply a set of spans with the same trace-id. There is no explicit class or object representing a trace.

Query selector has two forms: a TraceQL query string, or a list of trace IDs.

A TraceQL query selects spans from many traces that match the query criteria. Example:

A trace-id query is a list of hexadecimal trace IDs. It returns all the spans included by each trace. Example:

The trace domain accepts an optional "tempostack" field with a URL for tempostack. If absent, connect to the default location for the trace store on an Openshift cluster.

Constrains the objects that will be included in search results.

Domain configuration information.

Directed edge in the result graph, from Start to Goal classes.

Error result containing an error message.

Starting point for a goals search.

Graph resulting from a correlation search.

Options controlling the form of the returned graph.

Starting point for a neighbors search.

Node in the result graph, contains results for a single class.

Query with number of results.

Rule is a correlation rule with a list of queries and results counts found during navigation.

Start identifies a set of starting objects for correlation.

Store is a map string keys and values used to connect to a store.