v2.18.0 includes breaking changes

[bcdady]

A clear and concise description of what the bug is.

Expected Behavior

Per DEPRECATIONS Guidance

The Helm charts that KEDA provides has a separate release & version lifecycle. They follow Semantic Versioning but specifically around the Helm charts and is versioned independently from the KEDA runtime.
A minor version update to keda-operator chart should Not include breaking changes

Actual Behavior

v2.18.0 includes breaking changes

Steps to Reproduce the Problem

1. Deploy keda-operator chart version 2.17.2
2. Configure a ScaledJob with triggers.type: cpu
3. Update to version 2.18.1

Specifications

• KEDA Version: 2.18.0
• Platform & Version: EKS 1.32
• Kubernetes Version: 1.32
• Scaler(s): CPU, Memory

[bcdady]

Hi @JorTurFer, you seem to be most active in Discussions, so I'm hoping you can chime in here.
WDYT of this issue? Is there anyone else that could provide feedback if this is a valid problem?

Despite running it through our Chainsaw tests, updating this chart caused us some difficulties in our production environment. So, I'm seeking feedback from others that have recently updated on if they had symptoms from this "breaking" change in a minor release. If not, we'd love to learn how we can improve our test-update processes.

[JorTurFer]

Hello
Sorry for the inconvenience generated :(

The change is a breaking change indeed, but not of the KEDA chart but from KEDA itself.

All the deprecations that we do in KEDA are announced with at least 6 months (2 versions at least) and this specific case was announced here -> kedacore/keda#6348 and KEDA logs should have something like trigger.metadata.type is deprecated in favor of trigger.metricType or The 'type' setting is DEPRECATED and will be removed in v2.18 - Use 'metricType' instead. depending on the version you have.
Also, it has announced in the release -> https://github.com/kedacore/keda/releases/tag/v2.18.0

I really apologize for the inconveniences, we try to reduce these things as much as we can, but sometimes we need to remove some code.

In the short term, I can only suggest following the deprecations channel or reviewing the change log, this only happens from "minor" to "minor" (KEDA is v2, but k8s is v1 to...)

In the mid-term, I'm thinking that we can include a metric about deprecated features usage which can be monitored, @kedacore/keda-core-maintainers @kedacore/keda-core-contributors ?

[bcdady]

I appreciate the apology and understand the need remove code some times. I'd like to reiterate that this project's DEPRECATIONS Guidance states "minor version update[s] to keda-operator chart should Not include breaking changes", and yet this minor version update to keda-operator chart did, in fact, include breaking changes.

In our case, we did review the change log messages for 2.17 to 2.18, as you suggested, but we did not detect that the change to the triggers.type would cause an implementation/usage of KEDA in our environment to fail.

If the maintainers agree that this was/is an expected and acceptable Breaking change in a minor version update, then the DEPRECATIONS Guidance should be edited to clarify that minor version updates very well may include breaking changes. In this update, I'd like to see clear recommendations to the KEDA chart community that, as you said above, we should more closely follow the deprecations channel AND carefully test minor updates for breaking changes.

[wozniakjan]

I would like to also extend my apology. At the time of 2.18.0 release, since this particular field was marked for deprecation in 2.7.0 released in 2022, I had an incorrect belief that all users had stopped using it. I realize my mistake now and will do better in the future.

In the short term, I can only suggest following the deprecations channel or reviewing the change log, this only happens from "minor" to "minor" (KEDA is v2, but k8s is v1 to...)

while this is where the announcements are made, the information flow feels distant from the KEDA deployment that users have. It's easy to miss release notes, changelog entries and discussion channels. We should definitely keep doing the announcements there but KEDA has ways to inform the users in clusters and be louder.

In the mid-term, I'm thinking that we can include a metric about deprecated features usage which can be monitored, @kedacore/keda-core-maintainers @kedacore/keda-core-contributors ?

that is a pretty good idea. A metric like keda_scaler_detail_errors_total or keda_scaled_object_errors_total could contain information about all current deprecations as well as if the user is currently using it or not. Also, since kedacore/keda#6330, the TypedConfig now throws a Kubernetes event on the ScaledObject when parsing trigger metadata if there are any deprecation announcements. We could make that more frequent so it's harder to miss.

[rickbrouwer]

My apologies too.
I also want to look back and see how we can improve this process.

What @JorTurFer and @wozniakjan suggest is a good idea, I think. I might want a new metric that counts the deprecation announcements so you know how many exist and in which namespace.