[Feature]: Improve import speed

[lsorber]

The Feature

Simply importing LiteLLM can take up to 1 second, which is rather slow. Could the import speed be improved?

Minimal reproducible example:

uv run --python 3.10 --with litellm python -X importtime -c "import litellm"

Output:

import time: self [us] | cumulative | imported package
...
import time:    353694 |    1098960 | litellm

From the logs, you can see that litellm.utils already takes 250ms to load for example.

Motivation, pitch

If a project has 3 such dependencies, it would take 3 seconds to import that project.

Are you a ML Ops Team?

No

Twitter / LinkedIn details

@laurentsorber

[yachty66]

For me, it's super slow as well (even slower then OP):

uv run --python 3.10 --with litellm python -X importtime -c "import litellm"

import time: 20107192 | 20857950 | litellm

[krrishdholakia]

Hey @lsorber @yachty66 i tried looking into this but didn't have a good way to trace the issue - if you / anyone else can help us identify the bottleneck, we'd welcome a PR / some help here!

[yachty66]

Hey Krish. The problem from my side was actually the network I was in and not the library itself, so all is okay from my side. Regards, litellm @krrishdholakia .

[lsorber]

@krrishdholakia I don’t have the bandwidth to contribute a fix unfortunately. But the command I shared in the OP outputs a full trace of all imports and so should be a good starting point to analyze what’s causing it to be slow.

[krrishdholakia]

cc: @ishaan-jaff in case this is helpful in your perf work

[tcapelle]

This would be an amazing upgrade, it should be extremely fast to import litellm

[krrishdholakia]

@tcapelle if you want to contribute this fix - that would be helpful.

I wasn't able to make much headway in my investigation.

[tcapelle]

I don't have much time at the moment but o3-mini-high thinks this about the output of the uv ...

The timing output shows that importing litellm takes nearly a second—which is a lot for a module import. A closer look at the trace reveals that litellm pulls in a very large dependency graph. In particular, heavy libraries such as OpenAI’s SDK, Pydantic, tokenizers (tiktoken, regex, etc.), and even parts of HTTP libraries are all being loaded up front.

Some key points:
• Deep Dependency Tree: Litellm imports many submodules and third‐party libraries. For example, the OpenAI library and Pydantic are known to perform lots of work at import time.
• Eager Imports: All these modules and submodules are imported immediately on startup—even if not all of them are used in every run. This “eager” import strategy adds up.
• Initialization Overhead: Some of these libraries do nontrivial initialization (e.g. caching, compiling regex patterns, reading metadata) which further delays the import.

Possible Remedies:
• Lazy Imports: Consider restructuring the package so that some of the heavy imports occur only when needed (i.e. inside functions rather than at module level).
• Modularization: Split the functionality into smaller submodules so that users only import the parts they need.
• Optimize Dependencies: Evaluate if all dependencies are necessary on startup, or if some can be delayed or replaced with lighter alternatives.

In summary, the slowness is less an issue with litellm itself and more a side effect of a deep and heavy dependency graph. Addressing it will likely require a combination of lazy-loading and refactoring the package’s import structure.

[tcapelle]

The 1000 lines init.py is probably the culprit...

[fmmoret]

Here's an issue from huggingface diffusers where they are dealing with the same problem: huggingface/diffusers#4260 -- they are copying the lazy load patterns from hf's transformers package.
Ya'll might be able to adopt the same kind of strat as these packages

[fmmoret]

And here's that repo's PR for their first sets of changes for that effort: huggingface/diffusers#4829
10x speedup & there was other low hanging fruit for them still after that pr

[willbelr]

Indeed it feels very heavy for a 'lite' lib;

import time: 594194 | 2800485 | litellm

[krrishdholakia]

Thanks @fmmoret, this approach looks promising - https://github.com/huggingface/diffusers/pull/4829/files#diff-b828b278a068d46526a52d0facbdff41298fc776db6b6a5a8c04cd264ea3043f