Contributing to fluentbit-operator

[alexandre-allard]

Hello,

My team and I are working on a Kubernetes distribution named MetalK8s and we're currently working on our log centralization system composed of Fluent Bit and Loki.

Your project seems really interesting for us and instead of developing yet another operator (or forking your repository), we would like to contribute to your project to help you integrate new plugins support such as Loki Output.

First, do you accept contribution?
If yes, do you have guidelines?

Otherwise, we would like to know what motivated your choice for building this operator instead of using already existing things like Logging Operator?
But above all, why basing the log collection only on Fluent Bit rather than the Fluent Bit + Fluentd couple as we can find in most log centralization architecture using Fluent Bit?
On our side, we chose this to have a minimal resources consumption, but is there anything else motivating your choice?
We're still thinking of our implementation choice and it would help us a lot to have your thought.

Thanks,

MetalK8s team.

[benjaminhuo]

Hi @alexandre-allard-scality , yes sure we'll definitely happy to accept your contribution. The dev guide is here, https:/kubesphere/fluentbit-operator#contributing , it's a bit simple though. Let us know if you need anything else.

We did investigate logging operator for a while, we finally decided to develop FluentBit operator by ourselves for a few reasons:

• For logging operator, all logs has to flow from FluentBit to Fluentd, and Fluentd might become a single point of failure if the cluster size is large.

• Actually, we developed Fluentbit forward output crd https:/kubesphere/fluentbit-operator/blob/master/docs/plugins/output/forward.md which can output logs from to Fluentd . So if user want to use Fluentd to forward logs to more sinks, he can use the forward output crd to integrate with Fluentd for now.

We do have plan to develop crds for Fluentd to deploy Fluentd as a deployment, ideally it will help users to deploy Fluentd with ease into K8s. The Fluentd crds should be an optional choice instead of a mandatory one in a logging stack in our opinion.

• FluentBit's minimal resource consumption is indeed an important factor that we choose to support it by default.

• Another reason we develop FluentBit operator is that we found that we have to restart FluentBit pod to reload new FluentBit configs, there is no reload interface when we develop FluentBit operator. Pls refer to Restart when config directory has changes fluent-bit#842 and Support dynamic configuration fluent-bit#365 . With FluentBit operator, FluentBit pod needn't to be recreated to reload config anymore.

• And we do have plan to integrate our logging sidecar function into FluentBit operator which helps to collect logs files on Pod's volume instead of stdout

@huanggze is the core maintainer of FluentBit Operator, @huanggze do you have any comments?

Regards,
Benjamin Huo

[huanggze]

Thanks for your message. We are currently working on fluentbit operator v0.2.0 which will be released very soon. plz see the master branch.

Our vision is to make a tool for cluster operators to build k8s logging layer flexibly and efficiently. As @benjaminhuo mentioned, we start with lightweight Fluent Bit , but will incorporate Fluentd in the future.

Here is our broad features or roadmap:

• Automate the deployment of Fluentd and Fluent Bit (as daemonset/deployment)
• Configure and reload Fluentd and Fluent Bit dynamically
• Switch on/off input, filter and output plugins
• Select and group multiple plugins for different logging purposes
• Inject sidecar container to collect log files on pod volume
• Support full Fluent Bit/Fluentd plugins (Loki would definitely be included.)

[benjaminhuo]

@alexandre-allard-scality we developed FluentBit Operator using kubebuilder.
@huanggze which kubebuilder version we use? Maybe we coud put kubebuilder related info into contribute guide as well as an env setup guide.

[huanggze]

The project is build with v2.3.0

[alexandre-allard]

Thanks for the quick answers,

Ok, that's cool, we totally agree on the fact that Fluentd must be optional, that's the reason we didn't chose logging-operator.

As I said, we're currently working on our log centralization system, so we will probably start working on the Loki output plugin integration soon. We will let you know if we have question regarding the good practices or anything else.

[benjaminhuo]

Thanks for the quick answers,

Ok, that's cool, we totally agree on the fact that Fluentd must be optional, that's the reason we didn't chose logging-operator.

As I said, we're currently working on our log centralization system, so we will probably start working on the Loki output plugin integration soon. We will let you know if we have question regarding the good practices or anything else.

We're looking forward to your loki output plugin :)
Let us know if you need any other info

Regards,
Ben

[benjaminhuo]

@alexandre-allard-scality , you can join our slack channel #sig-observability anytime you have questions

[benjaminhuo]

Loki plugin has already been added.
Close this issue for now