Fluent Operator
Fluent Operator provides great flexibility in building a logging layer based on Fluent Bit and Fluentd.
Once installed, the Fluent Operator provides the following features:
- Fluent Bit Management: Deploy and destroy Fluent Bit DaemonSet automatically.
- Fluentd Management: Deploy and destroy Fluentd StatefulSet automatically.
- Custom Configuration: Select input/filter/output plugins via labels.
- Dynamic Reloading: Update configuration without rebooting Fluent Bit and Fluentd pods.
- Multi-tenant log isolation: Fluentd supports multi-tenant log isolation through the label-router plugin.
- Pluggable deployment components: Either Fluent Bit or Fluentd can be deployed separately.
Table of contents
Overview
Although both Fluent Bit and Fluentd can collect, process(parse and filter) and then forward log to the final destinations, still they have strengths in different aspects.
Fluent Bit is a good choice as a logging agent because of its lightweight and efficiency, while Fluentd is more powerful to perform advanced processing on logs because of its rich plugins.
- Fluent Bit only mode: If you just need to collect logs and send logs to the final destinations, all you need is Fluent Bit.
- Fluent Bit + Fluentd mode: If you also need to perform some advanced processing on the logs collected or send to more sinks, then you also need Fluentd.
- Fluentd only mode: If you need to receive logs through networks like HTTP or Syslog and then process and send the log to the final sinks, you only need Fluentd.
Fluent Operator includes CRDs and controllers for both Fluent Bit and Fluentd which allows you to config your log processing pipelines in the 3 modes mentioned above as you wish.
Fluent Bit will be deployed as a DaemonSet while Fluentd will be deployed as a StatefulSet. The whole workflow could be described as below:
Fluent Bit
The following CRDs are defined for Fluent Bit:
FluentBit: Defines the Fluent Bit DaemonSet and its configs. A custom Fluent Bit image ghcr.io/fluent/fluent-operator/fluent-bit is required to work with FluentBit Operator for dynamic configuration reloading.ClusterFluentBitConfig: Select cluster-level input/filter/output plugins and generates the final config into a Secret.FluentBitConfig: Selects namespace-level filter/output/parser and cluster-level parser plugins and generates final config into a Secret.ClusterInput: Defines cluster-level input config sections.Parser: Defines namespace-level parser config sections.ClusterParser: Defines cluster-level parser config sections.MultilineParser: Defines namespace-level multiline-parser config sectionsClusterMultilineParser: Defines cluster-level multiline-parser config sectionsFilter: Defines namespace-level filter config sections.ClusterFilter: Defines cluster-level filter config sections.Output: Defines namespace-level output config sections.ClusterOutput: Defines cluster-level output config sections.
Each ClusterInput, ClusterParser, ClusterFilter, ClusterMultilineParser, ClusterOutput represents a Fluent Bit config section, which are selected by ClusterFluentBitConfig via label selectors. Fluent Operator watches those objects, constructs the final config, and finally creates a Secret to store the config which will be mounted into the Fluent Bit DaemonSet. The entire workflow looks like below:
To enable Fluent Bit to pick up and use the latest config whenever the Fluent Bit config changes, a wrapper called Fluent Bit watcher is added to restart the Fluent Bit process as soon as Fluent Bit config changes are detected. This way, the Fluent Bit pod needn't be restarted to reload the new config. The Fluent Bit config is reloaded in this way because there is no reloading interface in Fluent Bit itself. Please refer to this known issue for more details.
Fluentd
The following CRDs are defined for Fluentd:
This is the beginning of the README. Read the rest at the original. Read the full README ↗
