Add support for managing OTel eBPF Instrumentation (aka OBI)

[cyrille-leclerc]

Component(s)

No response

Is your feature request related to a problem? Please describe.

Make OpenTelemetry eBPF Instrumentation (aka OBI) manageable with the OpenTelemetry Operator similarly to the way the OpenTelemetry Collector and OpenTelemetry language instrumentation SDKs are manageable using the OTel Operator.

Describe the solution you'd like

I would like to define OBI's configuration through a CRD that looks similar to the OTelCollector and OTel Instrumentation CRDs and then get the OTel Operator manage OBI accordingly so that all the OTel instrumentation of my K8s clusters are managed through one single mchanism that ensures proper correlation between all the telemetry types and layers.

Describe alternatives you've considered

Today I manage OBI with a distinct deployment model. This causes downsides such as

• Risk of gaps, overlaps, or conflicts between the OTel instrumentation
• Challenge to correlate OTel telemetry types and layers like
  • OBI metrics and traces
  • OTel SDKs metrics, traces, and logs
  • K8s pod logs and metrics

Additional context

This Enhancement request was discussed with @grcevski

Tip

_{React with 👍 to help prioritize this issue. Please use comments to provide useful context, avoiding +1 or me too, to help us triage it. Learn more here.}

[cforce]

Related issues

open-telemetry/opentelemetry-collector-contrib#34321

open-telemetry/opentelemetry-collector-contrib#32446

open-telemetry/opentelemetry-network#168

open-telemetry/opentelemetry-network#241

open-telemetry/opentelemetry-ebpf-instrumentation#27

[swiatekm]

Does the EBPF instrumentation publish a container image we could use directly? That would considerably simplify the implementation.

[iblancasa]

I think is this one https://github.com/open-telemetry/opentelemetry-ebpf-instrumentation/pkgs/container/opentelemetry-ebpf-instrumentation%2Febpf-instrument

[swiatekm]

Nice! The next thing I'd like to see before agreeing to implement this is a simple PoC, a set of K8s manifests showing a sample application instrumented with this image. I want to understand what it is that the operator needs to do as part of the injection process.

The next step after that would be an RFC document, as this is a major enough change to warrant it.

[grcevski]

I can help on that :). From the operator perspective, would it possible to run OBI as a daemonset or you'd prefer that we do sidecars? Daemonset is more resource efficient, but I don't know if that will fit the current model well.

[swiatekm]

My understanding is that the execution model would be similar to the way Go instrumentation works right now, and that uses a sidecar. How resource efficient that is really comes down to how many instrumented applications are running in the cluster, and how they're distributed among Nodes. If you wanted to do DaemonSet, you'd need a completely different deployment mechanism - our Instrumentations all use a mutating webhook - so that's an argument in favor of sidecars too.

[cyrille-leclerc]

All deployment forms supported by the OTel Collector are available through the OTel Operator: deployment, daemonset, statefulset or sidecar (here).
I would assume that the OTel eBPF Instrumentation would follow the sam. flexibility of offering through the OTel Operator all the deployment modes it supports.
Does this help?

[iblancasa]

All deployment forms supported by the OTel Collector are available through the OTel Operator: deployment, daemonset, statefulset or sidecar (here). I would assume that the OTel eBPF Instrumentation would follow the sam. flexibility of offering through the OTel Operator all the deployment modes it supports. Does this help?

The instrumentation is injected using a mutating webhook as @swiatekm pointed. For other languages, some init containers and environment variables are injected in the workload pod to instrument. But, in the case of go instrumentation, it is a sidecar.

The suggestion is following a similar approach to the go instrumentation. It has nothing to do with the supported modes for the Collector.

[grcevski]

My understanding is that the execution model would be similar to the way Go instrumentation works right now, and that uses a sidecar. How resource efficient that is really comes down to how many instrumented applications are running in the cluster, and how they're distributed among Nodes. If you wanted to do DaemonSet, you'd need a completely different deployment mechanism - our Instrumentations all use a mutating webhook - so that's an argument in favor of sidecars too.

Yeah I know, the one difference is that unlike Go auto instrumentation OBI supports instrumenting multiple processes, which allows us to share all internal maps, datastructures etc. When you add a sidecar, both OBI and Go auto instrumentation use a number of BPF maps, which have hardcoded number of elements. So in a sense, if you have 100 pods in the node, you'll effectively be using 100 times more memory than if you had OBI as a single daemonset.

The suggestion is following a similar approach to the go instrumentation. It has nothing to do with the supported modes for the Collector.

I think Cyrille is maybe refering to work we have ongoing to make OBI a receiver in the collector, so in that case it will get deployed with the collector, just like the eBPF profiler.

What I'm trying to say, the most efficient way to deploy OBI would be to deploy it as a daemonset with the operator (or as part of the collector) and set a config option to make it instrument based on deployment labels, just like the regular instrumentations. We can use a sidecar, but it will be very memory inefficient.

[grcevski]

I apologise that I'm not up to date on this, but does the OTel operator support the OTel eBPF profiler at the moment? If it does in any form, I think the best way is to use the same approach for OBI.

[cyrille-leclerc]

I think Cyrille is maybe referring to work we have ongoing to make OBI a receiver in the collector, so in that case it will get deployed with the collector, just like the eBPF profiler.

I just wanted to say that I see the OTel Operator not being prescriptive about the deployment mode of components it manages like the OTel Collector and that it supports many K8s deployment modes for the OTel Collector, including the sidecar and daemonset modes you listed.

I apologise that I'm not up to date on this, but does the OTel operator support the OTel eBPF profiler at the moment? If it does in any form, I think the best way is to use the same approach for OBI.

I searched before bringing this topic of adding OBI in the OTel Operator and I couldn't find anything about adding OTel eBPF Instr in the OTel Operator.

[iblancasa]

I apologise that I'm not up to date on this, but does the OTel operator support the OTel eBPF profiler at the moment? If it does in any form, I think the best way is to use the same approach for OBI.

It doesn't.

Yeah I know, the one difference is that unlike Go auto instrumentation OBI supports instrumenting multiple processes, which allows us to share all internal maps, datastructures etc. When you add a sidecar, both OBI and Go auto instrumentation use a number of BPF maps, which have hardcoded number of elements. So in a sense, if you have 100 pods in the node, you'll effectively be using 100 times more memory than if you had OBI as a single daemonset.

In that case, for me, it makes sense to just support daemonset. Later, if a new use case appears, we can support others.

What I'm trying to say, the most efficient way to deploy OBI would be to deploy it as a daemonset with the operator (or as part of the collector) and set a config option to make it instrument based on deployment labels, just like the regular instrumentations. We can use a sidecar, but it will be very memory inefficient.

I think it would make sense to do it as something separate. Otherwise, I expect to end up with the same situation we had for the target allocator.

I just wanted to say that I see the OTel Operator not being prescriptive about the deployment mode of components it manages like the OTel Collector and that it supports many K8s deployment modes for the OTel Collector, including the sidecar and daemonset modes you listed.

Can you elaborate? I think with what we have in https://github.com/open-telemetry/opentelemetry-operator/tree/main/docs/api we are covered but feedback is always welcomed.

[swiatekm]

To be clear, I'm not against deploying a DaemonSet. My point is simply that the implementation will be much more involved, and you'll likely need to add a new CRD for it. Our Instrumentation CRD currently only acts via a Mutating Pod webhook, which means that it has no effect until the user actually annotates a Pod to have instrumentation injected. If you want to use a DaemonSet, you'd be deploying an application to each Node in the cluster even if the user hasn't actually annotated any of their Pods. That's not necessarily a problem, but the semantics are different enough from our existing instrumentations, that you'd need to implement it from scratch.

Even if you don't want to use a sidecar in the end, it might be a good idea to do a PoC that way, because it's going to be much simpler.