add new config file entry to extend containerd configuration

[lavigne958]

What would you like to be added:

I would like to have a way to extend containerd configuration without having to manually create a new file and mount it onto the host, please.

Why is this needed:

This is needed in my case to setup containerd to use my local registry instead of the regular docker.io that is rate limited.

Extra context:

This containerd configuration changes used to be located in the file /etc/containerd/config.toml
and that file can be extended using the kind cluster configuration entry containerdConfigPatches before containerd v2.2

now with containerd v2.2 we must manually place a file in the node under /etc/containerd/certs.d/<old registry> with the registry override configuration for containerd to take it into account.

This requires the user to create 2 files

1. the kind cluster configuration file
2. the containerd extended configuration

Now that this new configuration format is made by placing a new file, we could add a new entry in the kind configuration and take the entire string from that entry and put it in the file at the right place.

as opposed as before where some merging was necessary.

I'm willing to help and work on this feature if it gets accepted.

[BenTheElder]

https://kind.sigs.k8s.io/docs/user/local-registry/ shows one way to do this at runtime, containerd will load/reload these files on the fly so it's not necessary that we specially handle restarting containerd when these are created, unlike modifying the core containerd config file.

There are various other ways you can populate a file currently e.g. kind extraMounts, or a trivial pod with hostPath (typical for things like CNI installers).

Now that this new configuration format is made by placing a new file, we could add a new entry in the kind configuration and take the entire string from that entry and put it in the file at the right place.

I think it will be a bit of a complex footgun to add a general "create files" API beyond the tools that already exist, and containerd has already made breaking changes to this before so I'm not sure we want to add a "containerd registry config" API.

beyond contents, files have mode/permissions, extended attributes, owner / group ....

[lavigne958]

Hi thanks for the prompt reply.

yes that new way of configuring a registry is making it more complex.
yes the kind extraMount is what I use now to mount my new config file.

I think it will be a bit of a complex footgun to add a general "create files" API

That's not my idea.
My proposal is to provide a new configuration entry that looks like that:

kind: Cluster
apiVersion: kind.x-k8s.io/v1alpha4
containerdConfigCertsFile:
  "host.toml" : |-
     server = "https://docker.io"

     [host."https://my-mirror-for-docker.io"]
       capabilities = ["pull", "resolve"]

and this will only take file content to be placed under /etc/containerd/certs.d/ so it dedicated to pass these configurations only.

I understand that they made breaking change before, they just did it again here too 😅

it should not prevent us from finding ways around that, this config change should stick around for some time I think, like the previous one was working for a long time (meaning this new config layout won't change for a few years I expect).

[BenTheElder]

If we add this, people will want binary cert files next, I'm not sure this will be more reasonable than just using a relative extraMount.

Also, your example doesn't really make sense (?), these files need to be placed in sub-directories for specific hosts.

it should not prevent us from finding ways around that, this config change should stick around for some time I think, like the previous one was working for a long time (meaning this new config layout won't change for a few years I expect).

That's not clear, and we're actually already still preparing to ease v2 vs v3 config format, and #3795 , I don't know that we want to add tighter coupling to the current state of containerd.

As a general rule, we also dissuade users from depending on the node filesystem, we've gotten complaints from updating internal details of the node images before (e.g. which linux distro) and it's just not sustainable, but we do permit depending on the containerd standard paths with the understanding that containerd may still have breaking changes.

I'm not saying a definite no, but I think this will actually require a fair bit of thought, and if we introduce it for this purpose, then why not for other files on the node? And then all those other details certainly matter and the API gets complex and messy.

In the local registry case, you already need to run additional actions on the host anyhow, so having a script step to write the file out seems like an unimportant distinction. Other use cases so far have been pretty uncommon.

https://kind.sigs.k8s.io/docs/design/node-image/

[lavigne958]

If we add this, people will want binary cert files next, I'm not sure this will be more reasonable than just using a relative extraMount.
agreed, opening one door opens the next one.

Also, your example doesn't really make sense (?), these files need to be placed in sub-directories for specific hosts.

you're right I forgot to add the sub folder under certs.d which is expected to be named after your target endpoint you wish to override.

agreed, it does increase the API complexity, but we can scope it to only containerd files for example ?

The main target I have is: having a single file that describes my cluster and let kind create necessary extra files if need be from the YAML cluster configuration.

So this could be scoped to certain folders in containerd to make it restrictive and prevent having a use case that is too large.

for my current purpose and I believe others are in the same setup, we have a docker cache on our running CI and we configure containerd to use that registry instead of the direct public url to prevent rate limiting.
so the registry is always here and configured.

we only have to deploy the kind cluster.