Kicbase: update cri-o packaging sources #19504

JanGutter · 2024-08-25T14:51:23Z

cri-o packages have moved to pkgs.k8s.io: https://kubernetes.io/blog/2023/10/10/cri-o-community-package-infrastructure/
The new packaging format depends on some k8s distro packages. This makes the assumption that the most recent stable k8s is expected to work with the most recent stable cri-o.
Since cri-o is also installed in the Kicbase/ISO via a different method, further work might still be needed for that case: Add matching Kubernetes Version for crictl to the preload #18359

* cri-o packages have moved to pkgs.k8s.io: https://kubernetes.io/blog/2023/10/10/cri-o-community-package-infrastructure/ * The new packaging format depends on some k8s distro packages. This makes the assumption that the most recent stable k8s is expected to work with the most recent stable cri-o. * Since cri-o is also installed in the Kicbase/ISO via a different method, further work might still be needed for that case: kubernetes#18359

k8s-ci-robot · 2024-08-25T14:51:31Z

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: JanGutter
Once this PR has been reviewed and has the lgtm label, please assign spowelljr for approval. For more information see the Kubernetes Code Review Process.

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these files:

OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

k8s-ci-robot · 2024-08-25T14:51:31Z

Welcome @JanGutter!

It looks like this is your first PR to kubernetes/minikube 🎉. Please refer to our pull request process documentation to help your PR have a smooth ride to approval.

You will be prompted by a bot to use commands during the review process. Do not be afraid to follow the prompts! It is okay to experiment. Here is the bot commands documentation.

You can also check if kubernetes/minikube has its own contribution guidelines.

You may want to refer to our testing guide if you run into trouble with your tests not passing.

If you are having difficulty getting your pull request seen, please follow the recommended escalation practices. Also, for tips and tricks in the contribution process you may want to read the Kubernetes contributor cheat sheet. We want to make sure your contribution gets all the attention it needs!

Thank you, and welcome to Kubernetes. 😃

k8s-ci-robot · 2024-08-25T14:51:32Z

Hi @JanGutter. Thanks for your PR.

I'm waiting for a kubernetes member to verify that this patch is reasonable to test. If it is, they should reply with /ok-to-test on its own line. Until that is done, I will not automatically test new commits in this PR, but the usual testing commands by org members will still work. Regular contributors should join the org to skip this step.

Once the patch is verified, the new status will be reflected by the ok-to-test label.

I understand the commands that are listed here.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

minikube-bot · 2024-08-25T14:57:09Z

Can one of the admins verify this patch?

JanGutter · 2024-08-25T15:09:46Z

I'd be happy to chat on slack, or open an issue, if needed.

afbjorklund · 2024-08-26T06:40:05Z

Note that the new packages are only available for "new" versions of Kubernetes.

https://kubernetes.io/blog/2023/10/10/cri-o-community-package-infrastructure/

So versions 1.24-1.27 still use the old repo, versions 1.28-1.31 use the new repo(s).

New packages contain the same binaries as the github releases / GCS cri-o/artifacts.

Replacing the other packages such as podman will take more effort, and to fix the bugs...

CRIO: add docker.io to the unqualified-search-registries #19396

trying to overwrite '/usr/libexec/crio/conmon', which is also in package conmon

It would also be good if podman could be updated from 3.4.x to 4.9.x, even it not required.

afbjorklund · 2024-08-26T06:45:50Z

I'd be happy to chat on slack, or open an issue, if needed.

Podman needs a way out of "experimental" in the roadmap.

This sort-of involves also the CRI-O container runtime, at least as long as they share image storage
(Similar to how docker and containerd are connected by optionally using the same image storage...)

So that is something for https://minikube.sigs.k8s.io/community/

There is also some discussion in the upstream Kubernetes project SIGs, if it is "included" in the k8s release.
The moving of the packages and the release script is one step into make it more standard, but it missed 1.31.

afbjorklund · 2024-08-26T06:57:35Z

hack/update/kubernetes_version/update_kubernetes_version.go

@@ -101,6 +101,11 @@ var (
 `kubernetesVersion:.*`: `kubernetesVersion: {{.LatestVersionP0}}`,
 },
 },
+ "deploy/kicbase/Dockerfile": {
+ Replace: map[string]string{
+ `ARG KUBERNETES_VERSION=.*`: `ARG KUBERNETES_VERSION="{{.StableVersion}}"`,


The kicbase cannot have a single kubernetes version, if it is going to be reused for multiple versions.

Would it work for more cases if this kubernetes sources.list is then replaced by later tooling, when the specific version of k8s is installed?

The easiest would probably be to install the CRI adapter with the Kubernetes components (like cri-tools/cni-plugins/kubeadm/kubelet/kubectl) - possibly split it out into per container runtime, like the "preload" is now?

It seems like CRI-O now uses an internal bundling of crun and conmon, so it doesn't use the system installation of those anymore,,, Otherwise such shared things could go with the OS installation still, if not k8s-version specific.

Heh, the yak shaving continues (the path that led me here is quite convoluted already!), I'll check out the preload alternative.

Added CRI-O to the discuss items for today's meeting, since there was already some WIP PR for it

Experimented with building static podman with Nix, similar to the cri-o packaging... (discussion)

afbjorklund · 2024-08-26T07:00:35Z

afbjorklund · 2024-08-26T09:06:13Z

The new packaging format depends on some k8s distro packages.

That sounds like a bug, the cri-o package should not have any hard dependencies towards kubernetes packages.
It does depend on CRI and CNI during runtime, but those tools do not have to be provided by the other repository.

Depends: systemd-sysv,systemd,iptables
Recommends: kubernetes-cni

This makes the assumption that the most recent stable k8s is expected to work with the most recent stable cri-o.

That is not currently the case, the stable k8s was tested with a prerelease of cri-o. Users are supposed to use stable.
This means that they will (for some weeks after the release) need to run the new k8s with the old (stable) cri-o.

KUBERNETES_VERSION=v1.31
CRIO_VERSION=v1.30

JanGutter · 2024-08-26T10:47:16Z

The new packaging format depends on some k8s distro packages.

That sounds like a bug, the cri-o package should not have any hard dependencies towards kubernetes packages. It does depend on CRI and CNI during runtime, but those tools do not have to be provided by the other repository.

Yeah, that's probably due to my misinterpretation of the instructions at https:/cri-o/packaging

This makes the assumption that the most recent stable k8s is expected to work with the most recent stable cri-o.

That is not currently the case, the stable k8s was tested with a prerelease of cri-o. Users are supposed to use stable. This means that they will (for some weeks after the release) need to run the new k8s with the old (stable) cri-o.

My assumption stemmed from the same source: it looks like they list the latest stable series of k8s (1.31) plus the latest stable series of cri-o (1.30) in the example, leading to my assumption that this is expected to work.

JanGutter · 2024-08-26T10:50:07Z

Note that the new packages are only available for "new" versions of Kubernetes.

https://kubernetes.io/blog/2023/10/10/cri-o-community-package-infrastructure/

So versions 1.24-1.27 still use the old repo, versions 1.28-1.31 use the new repo(s).

The old repos went through a bit of instability a month or so ago - incidentally, this was one of the reasons I started going down this rabbit hole.

afbjorklund · 2024-08-26T10:51:34Z

The old repos went through a bit of instability a month or so ago

There has been a lot of stability issues with OBS packages over the years.

I think that the new kicbase packaging is unlikely to use the pkgs.k8s.io.

Probably go for something similar to dl.k8s.io instead, also for cri-o artifacts.

BIN_URL="https://storage.googleapis.com/cri-o/artifacts/$COMMIT/$ARCH"

See https:/cri-o/cri-o/releases for the available versions in GCS.

afbjorklund · 2024-08-26T15:36:27Z

Added this PR for discussion in the community meeting and the office hours, if cri-o should be prioritized...

Last time, we added improved support for Podman to the minikube roadmap - maybe also add CRI-O too ?

2021 datadog: https://www.datadoghq.com/container-report-2021/

2023 datadog: https://www.datadoghq.com/container-report/

You can clearly see cri-o being utilized more, right ? Like the Linux desktop...

https://linux.slashdot.org/story/24/08/01/2029219/linux-hits-another-desktop-market-share-record

JanGutter · 2024-08-26T16:09:33Z

Oh man, slashdot is still going!

This whole rabbithole started when I tried to get a minimal cluster running with centos 9, podman and an ingress. The nice feature of cri-o was that I could configure registry mirrors to rewrite both docker.io and quay.io fqdn paths, so with a CI job and no manifest rewrites, I could do a test of a kubernetes operator and a number of containers.

JanGutter · 2024-08-26T16:16:33Z

(This open issue put me on the path to podman moby/moby#18818 ), I'll have to check if containerd's rewrite options are similar. Honestly, if containerd for rootful and rootless podman works better, I'm much less interested in getting cri-o to work.

medyagh · 2024-08-26T21:24:32Z

deploy/kicbase/Dockerfile

+# cri-o depends on some kubernetes packages, the latest stable series
+# is a reasonable choice.
+ARG KUBERNETES_VERSION="v1.31.0"
+ARG CRIO_VERSION="v1.30"


I dont think we are ready to bump to the latest crio version yet (till we make a preload tarbal to include the critctl and other binaries)

k8s-ci-robot · 2024-09-15T19:09:03Z

PR needs rebase.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

k8s-ci-robot added the cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. label Aug 25, 2024

k8s-ci-robot requested review from medyagh and spowelljr August 25, 2024 14:51

k8s-ci-robot added the needs-ok-to-test Indicates a PR that requires an org member to verify it is safe to test. label Aug 25, 2024

k8s-ci-robot added the size/M Denotes a PR that changes 30-99 lines, ignoring generated files. label Aug 25, 2024

afbjorklund reviewed Aug 26, 2024

View reviewed changes

afbjorklund added co/runtime/crio CRIO related issues triage/discuss Items for discussion labels Aug 26, 2024

medyagh reviewed Aug 26, 2024

View reviewed changes

k8s-ci-robot added the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Sep 15, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Kicbase: update cri-o packaging sources #19504

Kicbase: update cri-o packaging sources #19504

JanGutter commented Aug 25, 2024

k8s-ci-robot commented Aug 25, 2024

k8s-ci-robot commented Aug 25, 2024

k8s-ci-robot commented Aug 25, 2024

minikube-bot commented Aug 25, 2024

JanGutter commented Aug 25, 2024

afbjorklund commented Aug 26, 2024 •

edited

Loading

afbjorklund commented Aug 26, 2024

afbjorklund Aug 26, 2024

JanGutter Aug 26, 2024

afbjorklund Aug 26, 2024

afbjorklund Aug 26, 2024

JanGutter Aug 26, 2024

afbjorklund Aug 26, 2024 •

edited

Loading

afbjorklund Aug 26, 2024

afbjorklund commented Aug 26, 2024

afbjorklund commented Aug 26, 2024

JanGutter commented Aug 26, 2024

JanGutter commented Aug 26, 2024

afbjorklund commented Aug 26, 2024 •

edited

Loading

afbjorklund commented Aug 26, 2024 •

edited

Loading

JanGutter commented Aug 26, 2024

JanGutter commented Aug 26, 2024

medyagh Aug 26, 2024

k8s-ci-robot commented Sep 15, 2024

Kicbase: update cri-o packaging sources #19504

Are you sure you want to change the base?

Kicbase: update cri-o packaging sources #19504

Conversation

JanGutter commented Aug 25, 2024

k8s-ci-robot commented Aug 25, 2024

k8s-ci-robot commented Aug 25, 2024

k8s-ci-robot commented Aug 25, 2024

minikube-bot commented Aug 25, 2024

JanGutter commented Aug 25, 2024

afbjorklund commented Aug 26, 2024 • edited Loading

afbjorklund commented Aug 26, 2024

afbjorklund Aug 26, 2024

Choose a reason for hiding this comment

JanGutter Aug 26, 2024

Choose a reason for hiding this comment

afbjorklund Aug 26, 2024

Choose a reason for hiding this comment

afbjorklund Aug 26, 2024

Choose a reason for hiding this comment

JanGutter Aug 26, 2024

Choose a reason for hiding this comment

afbjorklund Aug 26, 2024 • edited Loading

Choose a reason for hiding this comment

afbjorklund Aug 26, 2024

Choose a reason for hiding this comment

afbjorklund commented Aug 26, 2024

afbjorklund commented Aug 26, 2024

JanGutter commented Aug 26, 2024

JanGutter commented Aug 26, 2024

afbjorklund commented Aug 26, 2024 • edited Loading

afbjorklund commented Aug 26, 2024 • edited Loading

JanGutter commented Aug 26, 2024

JanGutter commented Aug 26, 2024

medyagh Aug 26, 2024

Choose a reason for hiding this comment

k8s-ci-robot commented Sep 15, 2024

afbjorklund commented Aug 26, 2024 •

edited

Loading

afbjorklund Aug 26, 2024 •

edited

Loading

afbjorklund commented Aug 26, 2024 •

edited

Loading

afbjorklund commented Aug 26, 2024 •

edited

Loading