Add guidelines for metric graduation #8709

richabanker · 2025-12-11T21:56:12Z

Which issue(s) this PR fixes:

k8s-ci-robot · 2025-12-11T21:56:22Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: richabanker

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details

Needs approval from an approver in each of these files:

~~contributors/devel/sig-instrumentation/OWNERS~~ [richabanker]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

richabanker · 2025-12-11T21:56:27Z

cc @dgrisonnet @dashpole

k8s-triage-robot · 2025-12-12T00:02:34Z

Unknown CLA label state. Rechecking for CLA labels.

Send feedback to sig-contributor-experience at kubernetes/community.

/check-cla
/easycla

dashpole · 2025-12-12T01:26:41Z

contributors/devel/sig-instrumentation/metric-instrumentation.md

+1. **Testing requirement**: The metric must have comprehensive tests that validate:
+   - The metric is registered and emitted correctly
+   - The metric has the expected labels and values under known conditions
+   - The metric is included in the [stable metrics list](https://github.com/kubernetes/kubernetes/blob/master/test/instrumentation/testdata/stable-metrics-list.yaml). See the [instrumentation test README](https://github.com/kubernetes/kubernetes/tree/master/test/instrumentation/README.md) for steps on how to generate this file correctly.


Is this something we can enforce with a presubmit? E.g. a hack verify script?

there is already a script for that in https://github.com/kubernetes/kubernetes/blob/master/hack/verify-generated-stable-metrics.sh, but I don't know if we updated that to work for Beta

This script should only check for (BETA, STABLE) levels as those are checked by default here unless the --allstabilityclasses flag is passed, which we dont when generating the stable metrics, ref

So looks like we are covered here for BETA and STABLE. Though the name "verify-generated-stable-metrics.sh" is a little misleading since it doesnt contain beta. I'll see if I can improve that in a follow up.

dashpole · 2025-12-12T01:28:11Z

contributors/devel/sig-instrumentation/metric-instrumentation.md

+
+2. **Stability validation**: The metric should have been at Beta stability for at least one release to ensure it has been sufficiently validated in production environments.
+
+3. **API Review**: Graduating a metric to Stable requires an API review by SIG Instrumentation, as it represents a contractual API agreement. See the [API Review](/contributors/devel/sig-instrumentation/metric-stability.md#api-review) section in the metrics stability documentation.


I think ideally we want to be part of beta graduation? Most of the time by the time something is going stable it is too late to make changes without breaking existing users.

Or maybe we should do API review when it is added and for beta graduation?

I think it is fine to only do it during the Beta graduation process, otherwise we will need to have a look at all the changes to Alpha metrics and that could take quite a toll

What Damien said makes sense, I'll change to add the API review at the time of graduation to beta as well. We can keep reviewing for stable graduation too, and if/when turns out to be quite unactionable from our end - we can remove ourselves from stable graduation review process at that time ?

Marking metric as stable is a commitment by SIG to maintain stability guarantees. I think it's more important for the owning SIG leads to review it.

Ok I added that for STABLE graduation, primarily a review from owning SIG is required along with an approval from SIG inst. Lmk if that sounds good or if we think that SIG Inst doesnt need to be present in the review process at all for graduation to STABLE.

contributors/devel/sig-instrumentation/metric-instrumentation.md

dgrisonnet · 2025-12-12T15:44:05Z

cc @rexagod

dgrisonnet · 2025-12-19T08:56:50Z

contributors/devel/sig-instrumentation/metric-instrumentation.md

+1. **Testing requirement**: The metric must have comprehensive tests that validate:
+   - The metric is registered and emitted correctly
+   - The metric has the expected labels and values under known conditions
+   - The metric is included in the [stable metrics list](https://github.com/kubernetes/kubernetes/blob/master/test/instrumentation/testdata/stable-metrics-list.yaml). See the [instrumentation test README](https://github.com/kubernetes/kubernetes/tree/master/test/instrumentation/README.md) for steps on how to generate this file correctly.


This is the case even for BETA metrics, otherwise our approval will not be required to validate the graduation.

dgrisonnet · 2025-12-19T08:59:25Z

contributors/devel/sig-instrumentation/metric-instrumentation.md

+1. **Testing requirement**: The metric must have comprehensive tests that validate:
+   - The metric is registered and emitted correctly
+   - The metric has the expected labels and values under known conditions
+   - The metric is included in the [stable metrics list](https://github.com/kubernetes/kubernetes/blob/master/test/instrumentation/testdata/stable-metrics-list.yaml). See the [instrumentation test README](https://github.com/kubernetes/kubernetes/tree/master/test/instrumentation/README.md) for steps on how to generate this file correctly.


there is already a script for that in https://github.com/kubernetes/kubernetes/blob/master/hack/verify-generated-stable-metrics.sh, but I don't know if we updated that to work for Beta

dgrisonnet · 2025-12-19T09:02:41Z

contributors/devel/sig-instrumentation/metric-instrumentation.md

+
+2. **Stability validation**: The metric should have been at Beta stability for at least one release to ensure it has been sufficiently validated in production environments.
+
+3. **API Review**: Graduating a metric to Stable requires an API review by SIG Instrumentation, as it represents a contractual API agreement. See the [API Review](/contributors/devel/sig-instrumentation/metric-stability.md#api-review) section in the metrics stability documentation.


I think it is fine to only do it during the Beta graduation process, otherwise we will need to have a look at all the changes to Alpha metrics and that could take quite a toll

richabanker · 2026-01-08T23:34:06Z

Ping for another look

@dashpole @dgrisonnet @rexagod

contributors/devel/sig-instrumentation/metric-instrumentation.md

contributors/devel/sig-instrumentation/metric-stability.md

rexagod · 2026-01-13T18:26:12Z

contributors/devel/sig-instrumentation/metric-instrumentation.md

+
+When graduating a metric from Alpha to Beta or from Beta to Stable, the following requirements must be met. For more information on stability levels and their guarantees, see [Metrics Stability](#metrics-stability).
+
+#### Graduating to Beta


Maybe a section on cardinality requirements as well, that states that the owning SIG must review and propose, to the best of their ability, the "least cardinal" version of the metric being introduced that also fulfills their use-case.

Im not sure if we want to differentiate between ALPHA / BETA based on cardinality. Ideally thats a concern that should be thought through at the time when a new metric is being introduced (i.e. right from ALPHA). Plus "least cardinal" is kinda subjective too.. so not sure how to frame this clearly to avoid confusion.

contributors/devel/sig-instrumentation/metric-instrumentation.md

dgrisonnet · 2026-01-23T15:41:06Z

ping @dgrisonnet

richabanker · 2026-01-30T22:30:48Z

I think we should prioritize merging this since there are a bunch of metric graduation PRs open and likely there will be more in future.

(sorry for the continuous pings!)

cc @dashpole @rexagod @dgrisonnet

richabanker · 2026-01-30T23:07:15Z

contributors/devel/sig-instrumentation/metric-instrumentation.md

+
+Wherever possible, ensure that metrics graduating to Beta follow the [Prometheus metric naming best practices](https://prometheus.io/docs/practices/naming/).
+
+1. **Testing requirement**: The metric must have a corresponding test that validates:


Btw I slightly revised the testing requirements here. Ideally we would like for tests to validate that the metric behaves as expected when the relevant code instrumented with that metric is executed (instead of just verifying that the metric is registered with the right name / stability level / labels etc.)

Does this sound reasonable?

cc @dashpole @dgrisonnet @rexagod

Co-authored-by: Marek Siarkowicz <[email protected]>

k8s-ci-robot added the area/developer-guide Issues or PRs related to the developer guide label Dec 11, 2025

k8s-ci-robot added size/S Denotes a PR that changes 10-29 lines, ignoring generated files. sig/contributor-experience Categorizes an issue or PR as relevant to SIG Contributor Experience. labels Dec 11, 2025

k8s-ci-robot requested review from dashpole and serathius December 11, 2025 21:56

k8s-ci-robot added approved Indicates a PR has been approved by an approver from all required OWNERS files. sig/instrumentation Categorizes an issue or PR as relevant to SIG Instrumentation. labels Dec 11, 2025

k8s-ci-robot added the cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. label Dec 12, 2025

dashpole reviewed Dec 12, 2025

View reviewed changes

dgrisonnet reviewed Dec 19, 2025

View reviewed changes

richabanker force-pushed the metrics-graduation-guidelines branch from 13b5bb0 to e80ec21 Compare January 8, 2026 23:32

k8s-ci-robot added size/M Denotes a PR that changes 30-99 lines, ignoring generated files. and removed size/S Denotes a PR that changes 10-29 lines, ignoring generated files. labels Jan 8, 2026

richabanker force-pushed the metrics-graduation-guidelines branch from e80ec21 to 9de615d Compare January 8, 2026 23:32

k8s-ci-robot added size/S Denotes a PR that changes 10-29 lines, ignoring generated files. and removed size/M Denotes a PR that changes 30-99 lines, ignoring generated files. labels Jan 8, 2026

serathius reviewed Jan 9, 2026

View reviewed changes

contributors/devel/sig-instrumentation/metric-instrumentation.md Outdated Show resolved Hide resolved

richabanker force-pushed the metrics-graduation-guidelines branch from d3fe14e to a33d767 Compare January 9, 2026 22:11

k8s-ci-robot added size/M Denotes a PR that changes 30-99 lines, ignoring generated files. and removed size/S Denotes a PR that changes 10-29 lines, ignoring generated files. labels Jan 9, 2026

richabanker force-pushed the metrics-graduation-guidelines branch from a33d767 to 0aa1588 Compare January 9, 2026 22:11

rexagod reviewed Jan 13, 2026

View reviewed changes

richabanker force-pushed the metrics-graduation-guidelines branch 2 times, most recently from 7b754e3 to 091e2db Compare January 16, 2026 05:55

richabanker force-pushed the metrics-graduation-guidelines branch from 091e2db to 4d97b9e Compare January 30, 2026 22:28

richabanker mentioned this pull request Jan 30, 2026

Promote apiserver and aggregator metrics to beta kubernetes/kubernetes#136196

Open

richabanker force-pushed the metrics-graduation-guidelines branch 2 times, most recently from 870e12b to 1147118 Compare January 30, 2026 23:04

richabanker commented Jan 30, 2026

View reviewed changes

Add guidelines for metric graduation

f4e5045

Co-authored-by: Marek Siarkowicz <[email protected]>

richabanker force-pushed the metrics-graduation-guidelines branch from 1147118 to f4e5045 Compare January 30, 2026 23:08


		2. Stability validation: The metric should have been at Beta stability for at least one release to ensure it has been sufficiently validated in production environments.

		3. API Review: Graduating a metric to Stable requires an API review by SIG Instrumentation, as it represents a contractual API agreement. See the [API Review](/contributors/devel/sig-instrumentation/metric-stability.md#api-review) section in the metrics stability documentation.


		When graduating a metric from Alpha to Beta or from Beta to Stable, the following requirements must be met. For more information on stability levels and their guarantees, see [Metrics Stability](#metrics-stability).

		#### Graduating to Beta


		Wherever possible, ensure that metrics graduating to Beta follow the [Prometheus metric naming best practices](https://prometheus.io/docs/practices/naming/).

		1. Testing requirement: The metric must have a corresponding test that validates:

Add guidelines for metric graduation #8709

Are you sure you want to change the base?

Add guidelines for metric graduation #8709

Conversation

richabanker commented Dec 11, 2025

Uh oh!

k8s-ci-robot commented Dec 11, 2025

Uh oh!

richabanker commented Dec 11, 2025

Uh oh!

k8s-triage-robot commented Dec 12, 2025

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

dgrisonnet commented Dec 12, 2025

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

richabanker commented Jan 8, 2026

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

dgrisonnet commented Jan 23, 2026

Uh oh!

richabanker commented Jan 30, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

7 participants