[Feature]: Support cost mapping for OpenAI-compatible API

[xingyaoww]

The Feature

Support a custom field like actual_model that allows an OpenAI-compatible API to point to an existing model in https://github.com/BerriAI/litellm/blob/main/model_prices_and_context_window.json, so that the user can get price and context info about the model.

Motivation, pitch

When using LiteLLM Proxy Server, we create OpenAI-compatible APIs.
However, when users directly call it using litellm via OpenAI-compatible API, they might not fully benefit from LiteLLM, including cost tracking and model info.

Twitter / LinkedIn details

No response

[krrishdholakia]

hi @xingyaoww you can already set custom pricing - https://docs.litellm.ai/docs/proxy/custom_pricing

What am i missing here?

[xingyaoww]

Yeah i think that's one way to go, but it requires the user to manually look up the pricing for different models. I guess is it possible for user to say this "openai/claude-3-5" corresponds to "claude-3-5-sonnet" in the existing file (https://github.com/BerriAI/litellm/blob/main/model_prices_and_context_window.json) that LiteLLM maintained, and will inherit all the settings from there?

[krrishdholakia]

hey @xingyaoww i haven't seen this before - why is your claude an openai-compatible endpoint but then being mapped to another model on the model cost map?

Can you help me understand what you're doing?

[xingyaoww]

Yep - so I'm setting a LiteLLM LLM proxy which hosts Claude (and other models like Gemini) I got from a different LLM provider (e.g., vertex AI, bedrock) into an OpenAI compatible API, so the end user can just use these models from different provider through a single LLM-proxy OpenAI-compatible base_url.

But if the end-user needs to track costs, they need to manually set things like per token prices for different models, which can be somewhat complicated. Is there a way from either the llm-proxy side, or the client side to map the model to a cost&context info of a known model (e.g., LiteLLM already have cost and context info for claude and gemini)?

[krrishdholakia]

hey @xingyaoww the cost for the call is already returned to the end-user in the response headers. they can also calculate the cost ahead of time via /spend/calculate https://litellm-api.up.railway.app/#/Budget%20%26%20Spend%20Tracking/calculate_spend_spend_calculate_post

What am i missing?

[krrishdholakia]

How are you getting successful claude responses, if you're passing model as openai/claude.. for vertex/bedrock?

Those are called differently - https://docs.litellm.ai/docs/providers/vertex, https://docs.litellm.ai/docs/providers/bedrock

[xingyaoww]

How are you getting successful claude responses, if you're passing model as openai/claude.. for vertex/bedrock?

Because i deployed the LiteLLM Proxy (https://docs.litellm.ai/docs/simple_proxy), and access the proxied model through LiteLLM again following this: https://docs.litellm.ai/docs/proxy/quick_start#using-litellm-proxy---curl-request-openai-package-langchain

Is there a better way to do this?

Basically

[VertexAI / BedRock] <----> LiteLLM Proxy <---OpenAI compatible API ---> Local LiteLLM completion('openai/XXX')

[github-actions]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs.

[ishaan-jaff]

not stale

[github-actions]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs.