A korrel8r domain represents one type of signal or resource. Available domains are described in the Domain Reference.

Each domain defines the following abstractions, allowing Korrel8r to treat domains uniformly:

Rules express relationships between classes, possibly in different domains.

Rules are templates that generate a goal query using data from a start object. The start and goal can be different classes, possibly from different domains. If a rule cannot be applied to an object it may return a blank string, or raise an error.

The set of rules forms a graph connecting all the classes of data that korrel8r knows about. Korrel8r works by traversing this graph: applying rules to some initial objects, executing the resulting queries, retrieving more objects and so on.

There are two types of search:

The OpenTelemetry project (OTEL) defines a standard vocabulary for traces, metrics, and logs. The Korrel8r [_trace_domain] domain uses OTEL attributes, other OTEL-based domains may be added future.

As OTEL is adopted, it will provide a more consistent vocabulary for Korrel8r rules, but it does not replace Korrel8r:

The Red Hat Cluster Observability Operator automatically installs and configures korrel8r on a Red Hat OpenShift cluster.

For testing and development you can also:

To connect to your service from outside the cluster, you can expose it with an 'ingress' or 'route'.

You can use the Clients to test your service at $HOST.

Korrel8r can run outside the cluster, using routes or ingress URLs to connect to observability stores inside the cluster. This can be useful for development and testing.

Run korrel8r --help to get command line help.

See Clients for ways to test your service at http://localhost:8080.

Once the Korrel8r service is running, there are several ways you can interact with the REST API.

Every entry in the stores section has a domain field to identify the domain. Other fields depend on the domain, see Domain Reference.

Store fields may contain templates that expand to URLs.

Korrel8r comes with a comprehensive set of rules by default, but you can modify them or add your own.

A rule has the following key elements:

The query template should generate a string of the form:

The query-details part depends on the domain, see Domain Reference

Korrel8r rules and store configuration can include Go templates.

Korrel8r provides additional template functions to simplify writing rules and configurations:

There is a single class alert:alert.

An alert object is represented by this Go type. Rules starting from an alert should use the capitalized Go field names rather than the lowercase JSON names. Object

A JSON map of string names to string values, matched against alert labels, for example:

To query mutiple alerts at the same time, it’s possible to provide an array of maps:

A client of Prometheus and/or AlertManager. Store configuration:

Either or both of metrics or alertmanager may be present.

-e = Incident domain

Package incident provides mapping to https://github.com/openshift/cluster-health-analyzer incidents.

There is a single class incident:incident.

An incident object contains id and mapping to the sources (alert is the only supported source type at the moment).

One can query the incident by its id.

It’s also possible to provide labels from an alert to get a corresponding incident.

A client of Prometheus. Store configuration:

-e = K8s domain

Kubernetes resources stored in a Kube API server.

Class represents a kind of kubernetes resource. The format of a class name is: "k8s:KIND\[.VERSION]\[.GROUP]".

Missing VERSION implies "v1", if present VERSION must follow the Kubernetes version patterns. Missing GROUP implies the core group.

Examples: k8s:Pod, ks8:Pod/v1, k8s:Deployment.apps, k8s:Deployment.apps/v1, k8s:Route.route.openshift.io/v1

Object represents a kubernetes resource as a map, map keys are serialized field names. Rule templates should use the JSON (lowerCase) field names, NOT the UpperCase Go struct field names.

Example:

The k8s domain automatically connects to the current cluster (as determined by kubectl), no additional configuration is needed.

The following template functions are available to rules.

-e = Log domain :logql: LogQL

Logs can be stored on the cluster in LokiStack or in an external Loki server. They can also be retrieved directly from the Kubernetes API server. Direct API server access does not provide long-term log storage, but it gives short-term access when there is no long term log store available.

There are 3 classes corresponding to the 3 openshift logging log types:

A log object is a set of attributes in a map with string keys and values.
Attribute names contain only ASCII letters, digits, underscores, and (required for Loki labels). Other characters are replaced with "_".

For stored logs: Loki stream labels and structured metadata labels become attributes. If the log body is a JSON object, all nested field paths are flattened into attribute names.

For direct logs: the pod, namespace, and label meta-data are added as attributes using the same label names as for stored logs.

Special attributes:

A query starts with the log class, followed by one of the following:

The store configuration must include domain: log. It may include an optional loki: URL or lokiStack: URL, but not both. It may include an optional direct: true to enable direct API server logs.

The default configuration is:

Korrel8r log queries will do the following:

You can also configure a store with only direct: true to use only API server access, or only loki or lokiStack to use only stored logs.

The following functions can be used in rule templates when the log domain is available:

-e = Metric domain

Prometheus metric time-series.

There is only one class: metric:metric

A Metric is a time series identified by a label set. Korrel8r does not consider load the sample data for a time series, or use it in rules. If a korrel8r search time constraints, then metrics that have no values that meet the constraint are ignored.

Query is a PromQL query string.

Korrel8r uses metric labels for correlation, it does not use time-series data values. The PromQL query is analyzed to identify series it uses, labels of those series are used for correlation.

Prometheus is the store, store configuration:

-e = Netflow domain

Package netflow is a domain for network observability flow events stored in Loki or LokiStack.

There is a single class netflow:network

A log object is a JSON map\[string]any in NetFlow format.

A query is a LogQL query string, prefixed by netflow:network:, for example:

To connect to a netflow lokiStack store use this configuration:

To connect to plain loki store use:

-e = Trace domain :copyright: This file is part of korrel8r, released under https://github.com/korrel8r/korrel8r/blob/main/LICENSE :description: Correlation of observability signal data :keywords: correlation, observability, resource, signal, kubernetes :icons: font :sectanchors: :sectlinks: :project: https://github.com/korrel8r/korrel8r :pages: https://korrel8r.github.io/korrel8r :raw-etc-korrel8r: https://raw.githubusercontent.com/korrel8r/korrel8r/main/etc/korrel8r :etc-korrel8r: https://github.com/korrel8r/korrel8r/blob/main/etc/korrel8r :korrel8r-help: xref:gen/cmd/korrel8r.adoc :korrel8rcli-help: xref:gen/cmd/korrel8rcli.adoc :otel-signals: https://opentelemetry.io/docs/concepts/signals/ :pkg-korrel8r: https://pkg.go.dev/github.com/korrel8/korrel8/pkg/korrel8 :curl: curl :goref-badge: image:https://pkg.go.dev/badge/github.com/korrel8r/korrel8r.svg :pkg-dev: https://pkg.go.dev/github.com/korrel8r/korrel8r/ :pkg-dev-domains: https://pkg.go.dev/github.com/korrel8r/korrel8r/pkg/domains :icon: :operator: Korrel8r Community Operator :otel-icon: :otel: OpenTelemetry :red-hat: Red Hat :rh-logging: logging subsystem for Red Hat OpenShift :rh-ocp: Red Hat OpenShift :rh-coo: Red Hat Cluster Observability Operator :go-templates: Go templates :sprig: sprig

Package trace implements OpenTelemetry traces stored in the Grafana Tempo data store.

Class singleton trace:span representing an OpenTelemetry span

A trace is simply a set of spans with the same trace-id. There is no explicit class or object representing a trace.

Object represents an OpenTelemetry span

A trace is simply a set of spans with the same trace-id. There is no explicit class or object representing a trace.

Query selector has two forms: a TraceQL query string, or a list of trace IDs.

A TraceQL query selects spans from many traces that match the query criteria. Example:

A trace-id query is a list of hexadecimal trace IDs. It returns all the spans included by each trace. Example:

The trace domain accepts an optional "tempostack" field with a URL for tempostack. If absent, connect to the default location for the trace store on an Openshift cluster.

Constrains the objects that will be included in search results.

Domain configuration information.

Directed edge in the result graph, from Start to Goal classes.

Error result containing an error message.

Starting point for a goals search.

Graph resulting from a correlation search.

Options controlling the form of the returned graph.

Starting point for a neighbours search.

Node in the result graph, contains results for a single class.

Query with number of results.

Rule is a correlation rule with a list of queries and results counts found during navigation.

Start identifies a set of starting objects for correlation.

Store is a map string keys and values used to connect to a store.