How to achive tracing per gateway?

[pkalemba]

Hi!

I'm currently testing the ai-gateway, and I found documentation about OTEL tracing, but it describes global tracing. Is there any option to enable it per gateway?

I'd like to configure tracing selectively for specific gateway instances rather than enabling it across the entire setup. Could you point me to the relevant configuration options or documentation?

Thanks in advance!

[sqaisar]

This will be really nice if we could add OTEL_EXPORTER_OTLP_ENDPOINT in the CRDs rather than defining it in the ai gateway controller's helm chart configs.

It'll be a really nice feature. I don't think it exists today.

[sqaisar]

@pkalemba I've opened the above PR #1561 that adds that support to define environment variables per route giving it the ability to define tracing config per ai gateway deployment.
let's see if that gets approved and merged.

[mathetake]

yeah this is a valid request. however, unfortunately, #1561 is not the way to go. Because of the how request-body-based (model name) routing is performed, the filter configuration (extproc config) is gateway-wide side effect, and the env var like this should be a gateway-scoped configuration instead of router level (as you see, the PR has appending multiple env var configs from multiple routes). The "ai gateway filter config" filterConfig will be deprecated soon anyways (see below), and the existing cpu/memory config shouldn't exist there too (historically before we could only attach one route to a gateway, hence the resource config is there).

The path forward IMO is introduce "GatewayConfig" CRD in our API where we allow the config like this as well as the resource (cpu/memory) config, and use the annotation to indicate the configuration attached to a Gateway object. Then, when we promote the existing v1alpha1 API to beta, I would suggest to drop the filterConfig field from AIGatewayRoute cc @johnugeorge @yuzisun

Meanwhile, you can already deploy AIGW in any namespace and specify the watch target namespace, and that would allow you to configure the different tracing config for multiple Gateways. I hope this helps

[nacx]

Agree that routes are not the right places to configure such things. This is an infrastructure concern and, as such, should be configured outside the scope of a route.

Having a custom CRD to customize it seems reasonable, but I think we should also think about how to better honor the EnvoyGateway configuration. Right now the AIGW has its own telemetry configuration that is not honoring what is defined in the EnvoyGateway, and I think if we go for a custom CRD we should also consider the unified experience and how to leverage those settings. EG doesn't support (AFAIK) environment variable-based configuration for OTEL today (we should probably add support for that), but we should try to provide an experience that is as consistent and cohesive as possible.

[pkalemba]

"Meanwhile, you can already deploy AIGW in any namespace and specify the watch target namespace, and that would allow you to configure the different tracing config for multiple Gateways. I hope this helps"
@mathetake

Sorry , maybe I'm not understanding it well, but it's about deploying the whole stack X times to send telemetry properly?

[sqaisar]

@mathetake @nacx

I can contribute support for this, and here’s the direction I’m considering:

Introduce a GatewayConfig CRD that provides gateway-scoped configuration for the external processor (environment variables + CPU/memory). Gateways would reference it through an annotation:

aigateway.envoyproxy.io/gateway-config: <name>

Gateway would inherit the configuration, allowing the existing route-level filterConfig fields to be deprecated and avoiding the gateway-wide side effects mentioned above.

The controller would merge global defaults with the GatewayConfig (with GatewayConfig taking precedence), and the implementation would follow the same patterns as existing CRDs. This keeps the configuration surface isolated to AI Gateway while allowing future alignment with Envoy Gateway Telemetry if needed.

apiVersion: aigateway.envoyproxy.io/v1alpha1
kind: GatewayConfig
metadata:
  name: my-gateway-config
spec:
  externalProcessor:
    env:
      - name: OTEL_EXPORTER_OTLP_HEADERS
        value: "api-key=secret"
      - name: LOG_LEVEL
        value: "debug"

    resources:
      requests:
        cpu: "200m"
        memory: "256Mi"
      limits:
        cpu: "500m"
        memory: "512Mi"
---
apiVersion: gateway.networking.k8s.io/v1
kind: Gateway
metadata:
  name: my-gateway
  namespace: default
  annotations:
    aigateway.envoyproxy.io/gateway-config: my-gateway-config
spec:
  gatewayClassName: aigateway
  listeners:
    - name: http
      protocol: HTTP
      port: 80

adding externalProcessor in the CRD for GatewayConfig would allow us to add more configs in future.

Let me know if this direction aligns with what you have in mind, and I can start working on it.

[sqaisar]

Or @nacx @mathetake if you prefer to keep it similar to https://gateway.envoyproxy.io/docs/api/extension_types/#proxytelemetry not sure if we would need all of the configs defined in that ProxyTelemetry

apiVersion: aigateway.envoyproxy.io/v1alpha1
kind: GatewayConfig
metadata:
  name: comprehensive-config
  namespace: default
spec:
  # Structured telemetry configuration (preferred for tracing/metrics/logs)
  telemetry:
    tracing:
      ....
  
  # Custom environment variables for non-telemetry purposes
  env:
    - name: LOG_LEVEL
      value: "debug"
    - name: API_SECRET
      valueFrom:
        ...
  
  # Resource limits
  resources:
    requests:
      memory: "128Mi"
      cpu: "100m"
    limits:
      memory: "512Mi"
      cpu: "500m"

[mathetake]

@pkalemba fantastic, thank you for taking a shot at it already. I think the question @nacx was to not only to align API with EG but also maybe we should think about how to honoring the existing EG's config about telemetry etc. For example, if the otel telemetry is configured, then simply inherit the config to be used here etc.

But i think your proposed config API looks like a good start, resources should be under top level extProc though. wdyt @nacx The deprecation of route level extproc config should happen sooner than later anyways

[sqaisar]

@mathetake I've opened the above initial PR, please review and we'll go form there if you have any concerns I'll be able to update.