Unable to upgrade helm chart

[Giaco9NN]

Spegel version

from 0.3.0 to 0.4.0

Kubernetes distribution

EKS

Kubernetes version

v1.34.1-eks-113cf36

CNI

Amazon CNI

Describe the bug

We use the Helm chart to install Spegel on our clusters. The Helm chart is managed by Terraform using the helm_release resource.
The procedure to upgrade from version 0.3.0 to version 0.4.0 fails because of the following error:

│ Error: Error upgrading chart
│ 
│   with module.eks.helm_release.spegel,
│   on modules/eks/spegel.tf line 1, in resource "helm_release" "spegel":
│    1: resource "helm_release" "spegel" {
│ 
│ Upgrade failed: failed to replace object: DaemonSet.apps
│ "spegel" is invalid: spec.selector: Invalid value:
│ {"matchLabels":{"app.kubernetes.io/component":"spegel","app.kubernetes.io/instance":"spegel","app.kubernetes.io/name":"spegel"}}:
│ field is immutable

I think the commit that introduced the error is: 8c5f7f4

The code to install the chart is:

resource "helm_release" "spegel" {
  name             = "spegel"
  namespace        = "spegel"
  repository       = "oci://ghcr.io/spegel-org/helm-charts"
  chart            = "spegel"
  version          = "0.4.0"
  max_history      = 3
  timeout          = 1800
  create_namespace = true
  values           = [local.spegel_values]
}

These are the values applied:

service:
  registry:
    nodeIp: 127.0.0.1

spegel:
  mirrorResolveTimeout: 10s

I already tried to use a couple of utilities offered by the Helm provider (doc):
force = true. (Boolean) Force resource update through delete/recreate if needed. Defaults to false.
replace = true. Re-use the given name, even if that name is already used. This is unsafe in production. Defaults to false.

Thank you for your help and for this fantastic tool!

[nogweii]

Yeah, running spegel 0.3.0 on my homelab Talos cluster (running k8s 1.33) and trying to upgrade to 0.4.0 throws the same error. And indeed helm doesn't know how to handle the immutable field error, resulting in it not replacing the daemonset.

[phillebaba]

I should have caught this when reviewing the PR, I will look into if I should revert the selector in the next release. I will also add a CI test to this to verify that this does not happen in the future. In the meantime the easiest way forward is to recreate the Daemonset.

[weisdd]

If the change is reverted, then those who have already upgraded to 0.4.0 would have to apply the workaround again. So, not sure if that's a good idea.
One of the potential ways to handle it would be to extend the function that generates match labels to conditionally include / omit the label that was added to 0.4.0. With that, users can decide whether they want to continue using the old set of labels or just go with the new one.

[hanlin-openai]

It might be great if we have a flag to optin that label in selector?

[phillebaba]

I agree with @weisdd, reverting now would cause even more confusion and cause even more problems. I am sorry I merged this without catching the breaking change. Really there isn't a way to go back from this anymore. Long term having the additional component label will be beneficial. Bringing the tech debt forward is not something I want either as we would eventually face the selector problems sometime in the future. I am still looking for a good tool to detect these types of breaking changes in the future, so any suggestions would be appreciated.

Most CD tools like Flux have a force option that recreates the resource by deleting the daemonset and then recreates it. If that is not possible another option would be to manually delete the DS and orphan the pods kubectl delete ds <name> -n <namespace> --cascade=orphan then create then upgrade the chart. The steps are very dependent on requirements and infrastructure setup. If someone else wants to do this I am happy to take a contribution.

Enterprise support is available if anyone needs critical help with specific zero downtime fixes for this.

I will leave this issue open and close it once I have a GitHub action that catches future issues.