Evaluate fluent-operator to deploy collector/aggregator log architecture

[ricsanfre]

Scope

Evaluate fluent-operator to deploy collector/aggregator log architecture based on fluent-bit and fluentd

Fluent Operator includes CRDs and controllers for both Fluent Bit and Fluentd which allows to control logs data pipelines in 3 modes:

1. Fluent Bit only mode: Deploy just Fluentbit as collection and distribution layer.
2. Fluent Bit + Fluentd mode: Deploy Fluentbit as collector and Fluentd as distribution layer to perform more complicated processing of logs.
3. Fluentd only mode: Deploy just Fluentd as collection and distribution layer

Moreover, Fluentd helm chart does not seem to be maintained anymore.

References

• Fluent Operator Github repo
• Fluent Operator helm chart

[ricsanfre]

Fluentd/ fluent-bit Plugins support

Operator uses ad-hoc docker images kubesphere/fluentd and kubesphere/fluent-bit that comes preconfigured with a set of supported plugins

• List of fluent-bit supported plugins: https://github.com/fluent/fluent-operator/blob/master/docs/plugins/fluentbit/index.md
• List of fluentd supported plugins: https://github.com/fluent/fluent-operator/blob/master/docs/plugins/fluentd/index.md

Required Plugins

Fluentd

Currently I am using for Fluentd aggregator layer a customized fluentd image containing following plugins:

• fluent-plugin-elasticsearch
• fluent-plugin-prometheus
• fluent-plugin-record-modifier
• fluent-plugin-grafana-loki

All of them supported by Operator's fluentd image except fluent-plugin-prometheus

There is an open issue asking to include support for that plugin: fluent/fluent-operator#725

This missing capability is a stopper. Current implementation cannot be migrated to Fluent Operator

Fluent-bit

Currently I am using the following fluent-bit plugins

• Input
  • tail
• Filter
  • mutiline
  • kubernetes
  • modify
  • lua
• Parser
  • regex
• Output
  • forward

All of them supported currently by the Operator's image

Prometheus integration is also supported: https://github.com/fluent/fluent-operator/blob/master/docs/best-practice/monitoring.md