[Feature]: Fallback for only some errors

[superpoussin22]

The Feature

there's only 3 options for fallback what is limited and can introduce some noises and a complexcity in dev

@ishaan-jaff @krrishdholakia

Motivation, pitch

We are struggling to manage the fallback.
The option today is too limited. We need to fallback only for the 429 and 500.
The full fallback is generating to many errors on the application side.
The best will be to select the code error for a specific fallback, or if too complex to be able to fallback on 429 and 500

Are you a ML Ops Team?

Yes

Twitter / LinkedIn details

No response

[ishaan-jaff]

This is the fallback policy request, adding to march 2025 roadmap

[colesmcintosh]

scoping

[colesmcintosh]

hey @superpoussin22, I’ve been digging into the fallback options and think we can solve this with a more targeted approach. How about adding an error_type_fallbacks parameter? Something like:

SDK:

response = litellm.completion(
    model="gpt-4o",
    messages=[{"role": "user", "content": "Hello"}],
    error_type_fallbacks={
        "rate_limit": ["gpt-4o-mini", "claude-3-haiku"],
        "server_error": ["claude-3-opus"]
    }
)

Config:

router_settings:
  error_type_fallbacks:
    rate_limit: ["gpt-4o-mini", "claude-3-haiku"]
    server_error: ["claude-3-opus"]

Thoughts? Anything you’d change?

[yuyuma]

@colesmcintosh @superpoussin22 I'm also interested in more granular controls for errors. I like the direction that @colesmcintosh is going, but I believe the proxy config example he provided operates at the global level rather than on a per-model level which is the level of control I'm seeking.

The current docs (https://docs.litellm.ai/docs/proxy/reliability#fallbacks--retries--timeouts--cooldowns) mentions that per-model fallbacks are defined under litellm_settings, so a sample config might look like this:

litellm_settings:
  fallbacks:
    zephyr-beta:
      rate_limit: ["gpt-4o-mini", "claude-3-haiku"]
      server_error: ["claude-3-opus"]

That said, I realize that implementing this above might be backwards incompatible. I'm not sure what the optimal path forward is, but an alternative might be to separate different fallbacks out into its own configuration, similar to how context window and content policy fallbacks can be configured today (see: https://docs.litellm.ai/docs/proxy/reliability#context-window-fallbacks-pre-call-checks--fallbacks).

Example:

litellm_settings:
  fallbacks:     # preserves existing behavior, but can be overridden by more specific fallbacks below
    zephyr-beta: ["claude-3-opus"]
  rate_limit_fallbacks:
    zephyr-beta: ["gpt-4o-mini", "claude-3-haiku"]
  server_error_fallbacks:
    zephyr-beta: ["claude-3-opus"]
  context_window_fallbacks:
    zephyr-beta: ["gemini-2-flash"]

Preserving the ability to specify specific fallback options at the global level would be great as well.

[colesmcintosh]

@yuyuma @superpoussin22 I think a good short-term fix is to focus our fallback mechanism on just the most troublesome errors—specifically, the 429 (rate limit) and 500 (server error) cases. This way, we only intervene when it's really needed, reducing the extra noise that a broader fallback can cause.

That said, since the current setup applies fallbacks globally, @yuyuma I think you're totally correct, have to be cautious about any changes that might break existing configurations. I think one way to handle more granular fallbacks would be to to introduce an optional error_type_fallbacks parameter alongside our current settings. This allows users to opt into these more granular controls without forcing everyone to change their setup overnight. Essentially, the new settings would take precedence when provided, but otherwise, the system would continue to work as it always has.

So, in short, we could:

1. Focus fallback handling on 429 and 500 errors.
2. Offer this as an optional parameter (error_type_fallbacks) to keep current setups intact.
3. Carefully scope the change and provide clear migration steps to address any backward compatibility issues.

What do you think?

[kresimirfijacko]

suggestion: focus fallback on 503 ServiceUnavailableError as well

[github-actions]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs.