Release Proposal v2.28.0

[tico88612]

Announcement

⚠️ This is the last version of RHEL 8 that we support, and it will be deprecated in the next release, see #11872 for a discussion and reasons why.
⚠️ We have removed the Weave CNI test in previous versions and will remove it in the next release because the project has been deprecated.

Urgent Upgrade Notes

(No, really, you MUST read this before you upgrade)

• Action required
  Krew installation support is removed (Remove krew support #11824, @VannTen)
• Action required
  You should remove the leading 'v' of all explicit version of components deployed by kubespray (most notably kube_version) (Adapt checksums and versions to new hashes updater #11890, @VannTen)
• Action required
  etcd_kubeadm_enabled (was deprecated) is removed. You should remove it from your inventory (Cleanup of preinstall assertions #11901, @VannTen)
• gateway_api_experimental_channel is deprecated, please use gateway_api_channel and set experimental. (Refactor Gateway API installation process and bump Gateway API v1.2.1 #11763, @tico88612)

Changes by Kind

Feature

• Add Kubernetes 1.32.x hash (Make kubernetes 1.32.4 default #12161, @tmurakam) ([kubernetes] Support Kubernetes v1.32.0 with RHEL8 #11885, @yankay) (Make kubernetes 1.32.2 default #12003, @mzaian) (Bump kube patch versions #12052, @0ekk)
• Add containerd 2.0.x hash ([containerd] Support containerd v2.0.x #11845, @mzaian) ([containerd] Make containerd v2.0.3, update hashes for other releases #12011, @mzaian)
• Update runc binary to v1.2.4
  Set containerd_limit_open_file_num to 1048576 so it's configurable. ([containerd] Support containerd v2.0.x #11845, @mzaian)
• Update runc binary to v1.2.5 ([containerd] Make containerd v2.0.3, update hashes for other releases #12011, @mzaian)
• Make nerdctl 2.0.3 default (Bump nerdctl to v2.0.3 #11913, @mzaian)
• Add deploy_coredns: bool (true by default), to let kubespray deploy or not coredns in kube-system (Add option to [not] install coredns via Kubespray #12218, @ant31)
• Add option ubuntu_stop_unattended_upgrades to stop Ubuntu unattended upgrades (Allow stopping ubuntu unattended-upgrades #12174, @0ekk)
• Add support for ranges: (start‑stop or single start) as an additional way to define Cilium LoadBalancer IP pools, alongside the existing cidrs: field. (Feat: add Cilium LB IP Pool configuration to support ranges #12140, @Kimcheolhui)
• Adds the script controb/offline/upload2artifactory.py for offline environments. (Contrib: upload2artifactory.py #11886, @bbaassssiiee)
• ArgoCD updated to version 2.14.5 to maintain compatibility with Kubernetes version 1.31. ([argocd] Bump ArgoCD version to 2.14.5 #12041, @farshadasadpour)
• Automatically publish ingress-nginx service address if manual address is not specified and ingress-nginx is not using host network (Publish ingress-nginx service address if manual address not specified and not using host network #11879, @ThisIsQasim)
• Bump node-local-dns (k8s-dns-node-cache) image (Bump node-local-dns (k8s-dns-node-cache) image #11981, @sathieu)
• Cilium CNI installation replaces Jinja template with Cilium CLI
  cilium_agent_custom_args and cilium_operator_custom_args are deprecated, please use cilium_agent_extra_args and cilium_operator_extra_args.
  cilium_identity_allocation_mode default change to crd.
  cilium_enable_host_legacy_routing default change to false.
  Add CIlium hubble export advanced flow log settings (cilium_hubble_export_file_max_backups, cilium_hubble_export_file_max_size_mb, cilium_hubble_export_dynamic_enabled and cilium_hubble_export_dynamic_config_content)
  Deprecated cilium_ipsec_node_encryption, replace it with cilium_encryption_node_encryption (Refactor Cilium CNI installation #12101, @tico88612)
• Default etcd snapshot count to 10000 (update etcd snapshot count #11997, @ErikJiang)
• Enable_dual_stack_networks deprecated, refact network stack with separate ipv4 and ipv6 (refact ip stack #11953, @borislitv)
• Ensure metrics port exists for nodelocaldns/nodelocaldns-second daemonsets (Ensure metrics port exists for nodelocaldns/nodelocaldns-second daemonsets #11998, @Rickkwa)
• Fix cilium network plugin config issue deploying cilium 1.17 ( cilium 1.17: fix etcd trusted-ca-file config var #11986, @pedro-peter)
• For RHEL hosts, checking for subscription status timeout after rh_subscription_check_timeout (default to 3 minutes) (Timeout on RHEL subscription check #12115, @VannTen)
• Gateway API can be brought forward before the CNI installation. (Feat: Gateway API early installation #12189, @tico88612)
• Improve ntp package conflict handling (improve ntp package conflict handling #12212, @ErikJiang)
• Increase the control plane memory requirement to 2GB (Increase the control plane memory requirement to 2GB #11864, @yankay)
• Network: Fix calico-kube-controller can't list the tiers resources (Fix calico-kube-controller can't list the tiers resources #12169, @cyclinder)
• Setting up a Docker image service for offline installation on a Mac (Feat: offline installation registry support macOS #11960, @diguage)
• Support containerd registry mirror certificate configuration (add containerd registry mirror certificate configuration #11857, @KubeKyrie)
• Support kube-proxy nftables mode (Support kube-proxy nftables mode #12060, @yankay)
• Terraform upcloud: Add possibility to setup cluster using nodes with no public IPs (Upcloud: Add possibility to setup cluster using nodes with no public IPs #11696, @Xartos)
• Terraform: Added support for UpCloud routers and gateways (Upcloud: Added support for routers and gateways #11386, @Xartos)
• The external_cloud_provider support manual option lets users install the cloud controller manager themselves. (Add manual option to the external_cloud_provider variable #11883, @tico88612)
• Tolerations of cilium-operator deployments can be defined using the cilium_operator_tolerations group_var (create cilium_operator_tolerations variable in group_var #12200, @felipe88alves)
• Update default crio capabilities to allow rancher to start (Feature: configure crio default capabilities #11989, @jvkassi)
• Update CI test from AlmaLinux8 to AlmaLinux9 (Update CI test from AlmaLinux8 to AlmaLinux9 #11889, @yankay)
• Update kube-vip to v0.8.9 (Update kube-vip to v0.8.9 #11983, @sathieu)
• Upgrade OpenStack Cloud Controller Manager to v1.32.0 (Bump: OpenStack Cloud Controller Manager upgrade to v1.32.0 #12121, @tico88612)
• Upgrade ingress-nginx to version v1.12.1 to resolve critical vulnerabilities (CVE-2025-1974 and others) and webhook certgen to v1.5.2. (fix(ingress-nginx): Upgrade ingress-nginx to v1.12.1 and webhook certgen image to v1.5.2 #12075, @farshadasadpour)
• Upgrade kube-router to 2.1.1 (Upgrade kube-router #12066, @VannTen)
• Upgrade load balancers image version to Nginx 1.27, Haproxy 3.1. (Update load balancers versions to Nginx 1.27, Haproxy 3.1 #11928, @guoard)
• Upgrade the default Docker version to 28.0 (Bump Docker default version to 28.0 #12070, @tico88612)
• Users can now configure hubble-export-file-max-backups and hubble-export-file-max-size-mb through the Kubespray inventory. (Add support for hubble-export-file-max-backups and max-size-mb variables #12072, @ErmolenkoMaxim)
• [calico] Update default calico to v3.29.2 ([calico] Make calico v3.29.2, update hashes for other releases #12012, @mzaian)
• [kubernetes/control-plane] Added support for structured AuthorizationConfiguration files. (Structured AuthorizationConfiguration #11852, @chadswen)

Documentation

• Fix documentation for offline usage by adding the 'v' prefix in download urls (docs: Fix offline-environment.md to add 'v' prefix of some versions #12166, @tmurakam)
• Fix path to facts.yml in node facts refresh section (Fix path to facts.yml in node facts refresh section #12177, @guoard)
• Fix sample inventory for the reserved resource (Updated sample in inventory #11895, @anshuman-agarwala)
• No longer reserve outdated cephfs-provisioner installation and documentation ([cephfs-provisioner] deprecate outdated application and documentation #12113, @tico88612)
• No longer reserve outdated rbd-provisioner installation and documentation ([rbd-provisioner] deprecate outdated application and documentation #12114, @tico88612)
• Our CRI-O default capabilities remove NET_RAW and SYS_CHROOT. (Fix: CRI-O default capabilities follow with the upstream #12018, @tico88612)

Failing Test

• Add dns_autoscaler_affinity and remove in-place values. (Feat: add dns_autoscaler_affinity and remove in-place values #12165, @tico88612)
• Fix CI by exclude the .ansible in .ansible-lint
  Remove ctr image pull workaround for nerdctl (Fix CI by exclude the .ansible in .ansible-lint & remove ctr image pull workaround #11948, @yankay)

Bug or Regression

• Add support for control plane reconfiguration on upgrades
  Add support for kubeadm-config v1beta4 UpgradeConfiguration.apply and UpgradeConfiguration.node
  Use kubeadm upgrade node during secondary control plane node upgrades (Refactor control plane upgrades with reconfiguration support #12015, @chadswen)
• Enable NRI by default on containerd (following containerd defaults) (fix: Enable NRI for containerd and disable plugin when nri_enabled is false #12152, @ShinyaIshitobi)
• File download.url's are masked unless the extra var unsafe_show_logs is true. (Fix information disclosure #11959, @bbaassssiiee)
• Fix a bug where kubeadm_certificate_key was not defined if control plane nodes were not in correct order (Changed to use first_kube_control_plane to parse kubeadm_certificate_key #11875, @Xartos)
• Fix a bug where custom TCP/UDP ports were not exposed by the ingress-nginx-controller container and service. ([ingress-nginx] expose custom tcp and udp ports in ingress-nginx-controller #11850, @commx)
• Fix broken calico Typha template when using both calico_ipam_host_local and typha_secure (fix Calico typha deployment issue: #11916 #11917, @c-romeo)
• Fix broken dhclient hooks when using resolvconf (Fix quotations in dhclient hooks #11946, @kyrbrbik)
• Fix control plane pods deletion with proper shell quoting (Fix control plane pods forced deletion (proper shell quoting) #11943, @iptizer)
• Fix coredns deployment with coredns_pod_disruption_budget: true or enable_nodelocaldns_secondary (Fix incorrect syntax for secondary nodelocaldns manifest #11952, @RaulButuc)
• Fix hubble-ui deployment to not renders tls volume when the cilium_hubble_tls_generate option not configured. (fix: do not mount hubble-ui tls volume when cilium_hubble_tls_generate is false #12143, @atobaum)
• Fix scale.yml problems with cached IP facts (Make main_access_ip cacheable in facts #12020, @0ekk)
• Fix: Using the ./manage-offline-container-images.sh register command does not create a new container but registers the image in the existing container registry. (Fix: Using the ./manage-offline-container-images.sh register command does not create a new container but registers the image in the existing container registry. #11964, @DearJey)
• Fix: arm64 checksums for youki and kata-containers (fix: arm64 checksums for youki and kata-containers #12173, @ErikJiang)
• Fix: missing 'v' prefix in offline image tags (fix: missing 'v' prefix in offline image tags #12086, @ErikJiang)
• Fix: prevent kubeadm to override coredns configuration/deployment on upgrade (fix: ensure CoreDNS is kept disabled on kubeadm upgrade #12028, @sathieu)
• Fixed an issue where the second and subsequent parameters in kubelet_cpu_manager_policy_options were ignored due to incorrect indentation. (fix: correct indent of cpuManagerPolicyOptions #12123, @HoKim98)
• Fixed kube-vip to use kube-vip/kube-vip-iptables image instead of kube-vip/kube-vip when lb_fwdmethod or kube_vip_lb_fwdmethod is set to masquerade ([Bug] use kube-vip-iptables image when kube_vip_lb_fwdmethod is masquerade #12145, @aviral-agarwal)
• Install symlinks parroting as other control plane nodes etcd certificates (and key) on all control plane nodes, to make kubeadm works (Workaround missing etcd certds on control plane node #12181, @VannTen)
• Kubelet-csr-approver moves to regular application installation (Fix: kubelet-csr-approver moves to regular application installation #12141, @tico88612)
• New Boolean default variable leave_etc_backup_files: true, set to false for uncluttered /etc directory on target nodes. (Bugfix/11936 - backup: "{{ leave_etc_backup_files }}" #11937, @bbaassssiiee)
• [calico] Fix kubecontrollersconfigurations list permission ([calico] fix: kubecontrollersconfigurations list permission #12035, @darkobas2)

Other (Cleanup or Flake)

• Binary checksums are no longer overridable from inventories or host facts (Move checksums to kubespray_defaults/vars #12234, @VannTen)
• Calico-node pods no longer have a cpu limit by default (calico: don't set calico-node cpu limits by default #11914, @VannTen)
• Enhance safety and validation mechanisms in the node removal process (fix(remove-node): Ensure safety and validation for node removal process #12085, @farshadasadpour)
• Heketi playbook (contrib) is removed. ([contrib] Remove heketi and glusterfs #12091, @VannTen)
• Kubectl bash completion and alias available for Suse operation systems family (enable bash completion tasks for Suse OS family #11860, @noama-nv)
• Kubespray-defaults role is renamed to kubespray_defaults (Rename kubespray-defaults to kubespray_defaults #12202, @VannTen)
• Remove contrib/kvm-setup and contrib/mitogen. (Cleanup old things in contrib/ #12093, @VannTen)
• Rename role bootstrap-os to bootstrap_os (Rename bootstrap-os to bootstrap_os #12203, @VannTen)
• Update KUBESPRAY_VERSION for v2.27.0 (Update KUBESPRAY_VERSION in CI and Readme for v2.27.0 #11854, @yankay)
• Update containerd.options key name (update containerd.options key name #12170, @flpanbin)
• Upgrade CI for openSuse 15.6 (Upgrade CI for openSuse 15.6 #12074, @yankay)

Component versions

• kubernetes 1.32.5
• etcd 3.5.16
• docker 28.0
• containerd 2.0.5
• cri-o 1.32.0
• cni-plugins 1.4.1
• calico 3.29.3
• cilium 1.17.3
• flannel 0.22.0
• kube-ovn 1.12.21
• kube-router 2.1.1
• multus 4.1.0
• weave 2.8.7
• kube-vip 0.8.0
• cert-manager 1.15.3
• coredns 1.11.3
• ingress-nginx 1.12.1
• argocd 2.14.5
• helm 3.16.4
• metallb 0.13.9
• registry 2.8.1
• aws-ebs-csi-plugin 0.5.0
• azure-csi-plugin 1.10.0
• cinder-csi-plugin 1.30.0
• gcp-pd-csi-plugin 1.9.2
• local-path-provisioner 0.0.24
• local-volume-provisioner 2.5.0
• node-feature-discovery 0.16.4

[tmurakam]

Will you notify the deprecation of RHEL8 support of #11872 in advance?

[tico88612]

Will you notify the deprecation of RHEL8 support of #11872 in advance?

Absolutely, we've pinned the issue and slack discussion threads, and no other user has suggested a better solution in the meantime, so yes, it's scheduled to be removed in the next release and users will be notified.

[toomer]

When do you plan to release 2.28.0?

[VannTen]

Ideally we should remove the repetitive version bump (kubernetes to 1.32.0, 1, etc)
I don't think we'll have time for this release, but a script leveraging the git diff between two tags (+ some massaging to have a readable format would be nice) on the README or defaults could be nice.

Also added a blocking issue.

[tico88612]

@VannTen, I'll be cleaning the version before I release it.

There's quite a bit of PR going on, we shouldn't be releasing a version anytime soon. (but we'll asap)

[TheMatrix97]

Hi! Calico just released 3.30.0 fixing some issues and introducing highly demanding features. Do you think this version could make it into this new release? At least, updating the CRD Checksums file to allow users to upgrade their deployments to 3.30 would be nice :)
Thanks!

[tico88612]

@TheMatrix97 Calico just released a new version, which should go into the next release (2.29)

@VannTen, I added two blocking PRs. Should we have other PRs that need to be merged in this release (2.28)?

cc @mzaian @ant31 @yankay

[lengrongfu]

@tico88612 Can we add this pr #12170 to the release?

[tico88612]

@lengrongfu If this PR is merged in master branch, it will in 2.28.

[chadswen]

@TheMatrix97 Calico just released a new version, which should go into the next release (2.29)

@VannTen, I added two blocking PRs. Should we have other PRs that need to be merged in this release (2.28)?

cc @mzaian @ant31 @yankay

@tico88612 #12197 (opened yesterday) will be needed in this release to support Structured AuthorizationConfiguration. It's an XS change.

[mzaian]

@TheMatrix97 Calico just released a new version, which should go into the next release (2.29)

@VannTen, I added two blocking PRs. Should we have other PRs that need to be merged in this release (2.28)?

cc @mzaian @ant31 @yankay

Yes the new Calico version has a lot of features and needs to be tested carefully we can only include the hashes into this release but not the default version. e.g https://github.com/projectcalico/calico/blob/release-v3.30/release-notes/v3.30.0-release-notes.md
For the release I would say we freeze any features PR, we only merge fixes if needed.

[VannTen]

#12219 (fix for #12220) is a blocker