CRD Validation Cost Exceeds Kubernetes Budget

[mateo-moon]

CRD Validation Cost Exceeds Kubernetes Budget

Description

The gcenodeclasses.karpenter.k8s.gcp CustomResourceDefinition has validation rules that exceed Kubernetes API server's cost budget, causing the CRD to fail installation/update.

Error Message

CustomResourceDefinition.apiextensions.k8s.io "gcenodeclasses.karpenter.k8s.gcp" is invalid: 
[
  spec.validation.openAPIV3Schema.properties[spec].properties[networkTags].x-kubernetes-validations[0].rule: 
    Forbidden: estimated rule cost exceeds budget by factor of 18.1x 
    (try simplifying the rule, or adding maxItems, maxProperties, and maxLength where arrays, maps, and strings are declared),
  
  spec.validation.openAPIV3Schema.properties[spec].properties[networkTags].x-kubernetes-validations[0].rule: 
    Forbidden: contributed to estimated rule cost total exceeding cost limit for entire OpenAPIv3 schema,
  
  spec.validation.openAPIV3Schema.properties[spec].properties[kubeletConfiguration].properties[evictionHard].x-kubernetes-validations[0].rule: 
    Forbidden: contributed to estimated rule cost total exceeding cost limit for entire OpenAPIv3 schema,
  
  spec.validation.openAPIV3Schema.properties[spec].properties[kubeletConfiguration].properties[evictionSoftGracePeriod].x-kubernetes-validations[0].rule: 
    Forbidden: contributed to estimated rule cost total exceeding cost limit for entire OpenAPIv3 schema,
  
  spec.validation.openAPIV3Schema.properties[spec].properties[kubeletConfiguration].properties[evictionSoft].x-kubernetes-validations[0].rule: 
    Forbidden: contributed to estimated rule cost total exceeding cost limit for entire OpenAPIv3 schema,
  
  spec.validation.openAPIV3Schema: 
    Forbidden: x-kubernetes-validations estimated rule cost total for entire OpenAPIv3 schema exceeds budget by factor of 2.3x 
    (try simplifying the rule, or adding maxItems, maxProperties, and maxLength where arrays, maps, and strings are declared)
]

Steps to Reproduce

1. Install the karpenter-provider-gcp Helm chart from the main branch
2. The CRD gcenodeclasses.karpenter.k8s.gcp fails to install/update
3. Kubernetes API server rejects the CRD due to validation cost exceeding budget

Expected Behavior

The CRD should install successfully without validation cost errors.

Actual Behavior

The CRD installation fails with validation cost errors, preventing the chart from deploying.

Affected Fields

1. networkTags - Validation rule exceeds budget by 18.1x

   • Current: maxItems: 64 with validation rule self.all(x, x.matches('^[a-z]([-a-z0-9]{0,61}[a-z0-9])?$'))
   • The rule iterates over all items (up to 64), making it expensive

2. kubeletConfiguration.evictionHard - Contributes to exceeding cost limit

3. kubeletConfiguration.evictionSoft - Contributes to exceeding cost limit

4. kubeletConfiguration.evictionSoftGracePeriod - Contributes to exceeding cost limit

Suggested Fix

As recommended by the error message, consider:

1. For networkTags: Reduce maxItems from 64 to a smaller value (e.g., 10) to lower validation cost
2. Simplify validation rules: Break down complex validation rules or remove them if not critical
3. Add constraints: Ensure maxItems, maxProperties, and maxLength are properly set for arrays, maps, and strings

Workaround

Currently pinning to commit 0a270d61b1cd768635f7cccab26f0aa123b81919 which doesn't have this issue.

Environment

• Kubernetes Version: GKE (version varies)
• Chart Version: Latest from main branch
• Helm Version: v4.x

Additional Context

This issue was discovered when deploying the chart using Pulumi. The CRD validation cost is a Kubernetes API server limitation introduced to prevent DoS attacks through expensive validation rules.

Reference: Kubernetes CEL Validation Cost Budget

[jwcesign]

Hi, @mateo-moon, could you submit a PR to fix this issue?

[mateo-moon]

Hi, @mateo-moon, could you submit a PR to fix this issue?

Hi) Done

[roeishuster]

Can you please update the chart as well?
I'm using version: 0.1.0-rc.0 under repository: "https://cloudpilot-ai.github.io/karpenter-provider-gcp"