[Bug]: Inconsistent stream output between OpenAI and LiteLLM clients during tool calling

[VicentGJ]

What happened?

I have been working on modifying the Swarm agents library to integrate with LiteLLM, allowing for the use of multiple other LLMs instead of being limited to OpenAI's. The modification required was minimal; I only needed to change line 70 in core.py from return self.client.chat.completions.create(**create_params) to return completion(**create_params).

However, I encountered an issue when using streaming to run the agents. The error is as follows:

Traceback (most recent call last):
  File "/Users/vicente/Developer/agents-test/swarm_test/main.py", line 53, in <module>
    for chunk in stream:
                 ^^^^^^
  File "/Users/vicente/Developer/agents-test/swarm_test/swarm/core.py", line 215, in run_and_stream
    tool_call_object = ChatCompletionMessageToolCall(
                       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/vicente/Developer/agents-test/.venv/lib/python3.12/site-packages/pydantic/main.py", line 214, in __init__
    validated_self = self.__pydantic_validator__.validate_python(data, self_instance=self)
                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
pydantic_core._pydantic_core.ValidationError: 1 validation error for ChatCompletionMessageToolCall
type
  Input should be 'function' [type=literal_error, input_value='functionfunctionfunction...unctionfunctionfunction', input_type=str]

The issue appears to be related to the fact that the OpenAI client outputs the tool's type only once in the stream. Consequently, only the first chunk of the tool call contains that property, as illustrated below:

{'content': None, 'function_call': None, 'refusal': None, 'role': None, 'tool_calls': [{'index': 1, 'id': 'call_hFCNIDFOnRANhrZsXH8V2fhy', 'function': {'arguments': '', 'name': 'web_search'}, 'type': 'function'}]}
{'content': None, 'function_call': None, 'refusal': None, 'role': None, 'tool_calls': [{'index': 1, 'id': None, 'function': {'arguments': '{"qu', 'name': None}, 'type': None}]}

In contrast, LiteLLM seems to output the tool type in every chunk of the stream:

{'id': 'call_Yy84QjsA2lMv66k9R0tEcCDW', 'function': {'arguments': '', 'name': 'web_search'}, 'type': 'function', 'index': 2}], 'audio': None, 'sender': 'Researcher'}
{'refusal': None, 'content': None, 'role': 'assistant', 'function_call': None, 'tool_calls': [{'id': None, 'function': {'arguments': '{"qu', 'name': None}, 'type': 'function', 'index': 2}], 'audio': None, 'sender': 'Researcher'}

This discrepancy was straightforward to address on my end. However, considering that LiteLLM aims to provide an OpenAI-compatible client, resolving this minor compatibility issue could be advantageous.
Interestingly, the 'assistant' property also appears only once in the first chunk of the OpenAI stream, whereas it is present in every chunk of the LiteLLM stream. While this behavior did not cause issues in my specific use case, it may be worth investigating further.

Relevant log output

Are you a ML Ops Team?

Yes

What LiteLLM version are you on ?

v1.59.8

Twitter / LinkedIn details

No response

[ishaan-jaff]

hi @VicentGJ can you give me a litellm.completion request to repro this problem ?

[ishaan-jaff]

this is being picked up by @Kaushikdkrikhanu

[Kaushikdkrikhanu]

I think the observation here is that, if function id is none then type is also none.

[github-actions]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs.