Determine how errors in xds-translator are handled

[guydc]

Description

Currently, the XDS Translator may encounter several types of failures that would lead to an error being returned from the Translate function:

• Failure to update Routes, Network/HTTP filters, Clusters and Secrets due to unexpected missing XDS resources or unexpected existing conflicting XDS resource (existing route config, filters, default filter chain, ... ).
• Failure to translate IR config to XDS due to missing/unsupported IR input, which was not properly verified in earlier validation phases (CEL, gateway-api translator validations, ... )
• Failure to apply Envoy Patch Policies: errors in input validation of patches or proto validation of the patched resources
• Failures in validation of EG-generated envoy protos during translation, leading to resources being dropped from the xds resource table.
• Failures in validation of all XDS resources post-translation, leading to errors being reported without resources being dropped.
• Failures in translation by EG Extension Server.

The XDS translators works in a "best-effort" manner: when errors are encountered, some resources may be omitted/changed due to errors. However, the resulting configuration is still published and saved to the XDS cache. This configuration may be partial or invalid. As a result:

• The proxy may not reflect the user's intent.
• The proxy may reject some or all of the config. This may be due to partial configuration that contains invalid resource references or invalid resources. Rejection of config creates a risk that new proxy instances would not be programmed at all.
• Users may not notice that translation is partially failing, if the impact is limited.
• There's no way for users to understand which part of their desired state is "really" applied.
• The proxies remain in an "incorrect" while users work to identify which issue/configuration change triggered this state.

In #4155, users requested the following:

Ideally, similar to #3873, there would be an option added to ExtensionManager which would allow either "fail open" (current behavior of best effort) or "fail closed" (alternate behavior of disabling the resource associated with the failed hook).

This option was implemented in #4936. The FailClosed option in #4936 provides greater reliability that traffic is either handled according to configuration provided by the extension server or rejected entirely.

#4936 is a partial solution:

• It only addresses the extension server use case
• users may prefer to handle translation errors by pausing XDS publishes until the issue is resolved and the translator produces error-free xds resources.
  • Existing proxies are serving a recent version of XDS config which was valid at the time of the translation.
  • By persisting a valid XDS snapshot, there is a guarantee that new proxies can still be programmed.

An alternative approach, supported by other projects, is to store and use the last successful translation results, and continue using them when translation errors occur:

• Kong: https://docs.konghq.com/kubernetes-ingress-controller/latest/guides/high-availability/last-known-good-config/
• Gloo Mesh: https://docs.solo.io/gloo-mesh-gateway/main/traffic_management/concepts/route-tables/route-failure-modes/#freeze-configuration

Already today, EG behaves this way in some edge cases:

• If an empty translation result being produced by the translator, EG falls back to the last known translation result:
  

  gateway/internal/xds/translator/runner/runner.go

  Line 91 in d5872be

  if result == nil {

• If a panic is raised, EG translation is halted and the existing cache is used.

By avoiding the publish of snapshots when any xds translator error occurs, EG can offer a similar error handling strategy that might be preferred by users.

Concerns

• How would this interact with existing strategies for handling failures in the XDS translation layer?
• What would happen if EG starts up and produces an empty cache - is there an impact on live proxies that have an existing config?
• There's not guarantee that the last "successful" translation is preferable (from an end user perspective)

Tracking related work:

• Avoid updating XDS snapshot in case of translator error: Skip snapshot updates on XDS translator error #5540
• Improve Configurability of extension server retry policy: Improve configurability of retry policy for extension server #5612
• Avoid missing route configs when moving from FailClosed to recovered state: fix: use rds instead of inline route config in fail closed mode #5611

[optional Relevant Links:]

Any extra documentation required to understand the issue.

[Chever-John]

I'd like to discuss the design architecture in detail. Please feel free to identify any potential flaws or areas for improvement in my approach.:

Plan to implement a flexible persistence framework for Envoy Gateway that:

1. Prioritizes ConfigMap as the primary storage mechanism while supporting filesystem and PersistentVolume alternatives
2. Incorporates robust version control with automatic fallback capability to Last Known Good (LKG) configurations
3. Ensures proper initialization of the LKG state during Envoy Gateway startup

Also, extend the EnvoyGateway Custom Resource Definition to:

• Support comprehensive LKG feature configuration
  • Choose a primary storage mechanism
  • Set Mornitioring metrics
• Allow administrators to specify version retention policies
  • How long the version is retained
• Configure fallback behavior preferences

For configurations exceeding the 1MB ConfigMap limit, I'll implement a transparent access layer that abstracts the underlying storage implementation, providing both automatic sharding of configurations into multiple ConfigMap slices and seamless transition to PersistentVolume storage for exceptionally large datasets.

[guydc]

Hi @Chever-John! glad to see that you're interested in this topic.
Some things that come to mind immediately:

• We may need to offer additional forms of storage to address requirements of non-K8s deployments.
• PVs could pose a challenge for a control plane running with multiple replicas. We would maybe need a PV that supports the ReadWriteMany mode which is not a trivial requirement.
• When running the control plane in multiple replicas, if we hold a single synchronized translation result, we may want the "follower" pods to consume the LKG as-well, instead of running their own translation process.

There are also domain-specific solutions in the envoy ecosystem to consider, like this one: https://github.com/envoyproxy/xds-relay.

IMHO, it would be nice to start small with by using the existing xds snapshot as an in-memory LKG, and maybe later adding more storage modes, synchronizations options, etc.

[github-actions]

This issue has been automatically marked as stale because it has not had activity in the last 30 days.

[guydc]

XDS translator errors will now lead to xds update pause, as implemented in #5540. Closing this issue for now.

LKG can be addressed in a dedicated issue.