[Bug]: no billing reported using rerank endpoint and amazon bedrock

[ladrians]

What happened?

I am testing the liteLLM rerank endpoint against bedrock; for us-west-2 region I got access to cohere.rerank-v3-5:0 and amazon.rerank-v1:0. Noticed that no billing is reported; both models the meta section details

Relevant log output

"meta": {
        "billed_units": {},
        "tokens": {}
    }

Are you a ML Ops Team?

No

What LiteLLM version are you on ?

main-v1.55.2

Twitter / LinkedIn details

No response

[krrishdholakia]

I don't believe bedrock returns these values. I think they charge just on number of requests.

Let me know if I'm mistaken @ladrians

[madhukar01]

They charge based on requests and the response doesn't include usage info.
Ref: example response here, and from the pricing page

[ladrians]

got it, I though it would be nice to still generate a usage based on the usage and to keep the "unified response".
Feel free to close it, thanks!

[krrishdholakia]

What value would you expect there? @ladrians

[ladrians]

to fill-in the billing with something like this?

"billed_units": {
      "search_units": 1
  }

[ilarischeinin]

It would be nice to have the costs tracked, but I understand that is not trivial when the response doesn't include any usage stats. Both the Amazon and Cohere rerank models are priced per request, but if there are more than 100 documents, it'll count as multiple requests. And documents longer than 512 tokens will be broken down into multiple documents.

Amazon:

Cohere:

[krrishdholakia]

Adding this to our feb 2025 roadmap - #8375

cc: @vibhavbhat as an fyi

[krrishdholakia]

picking this up