fix(openai): handle null choices from model_dump() for vLLM compatibility#34791
Conversation
Merging this PR will not alter performance
Comparing Footnotes
|
|
vLLM requires NVIDIA GPU. The Docker image cannot run on CPU-only systems. I had one provisioned on GCP from work when I ran into this issue but that instance has since been decommissioned after project completion. |
312d5f5 to
a1db4d6
Compare
2fb91bd to
ba0dc52
Compare
ba0dc52 to
91ca677
Compare
…lity Made-with: Cursor
91ca677 to
1f06e9b
Compare
|
Closing -- I can no longer reproduce/test this since the GCP instance with vLLM has been decommissioned. If someone else encounters null choices from model_dump() with vLLM, feel free to re-open or reference this fix. |
Summary
Fixes #32252
When using OpenAI-compatible APIs like vLLM,
model_dump()can returnnullforchoiceseven though the response object has valid choices accessible via direct attribute access. This occurs because the OpenAI SDK's Pydantic model may serialize differently when handling non-OpenAI API responses.Changes
_create_chat_resultthat extractschoicesdirectly from the response object whenmodel_dump()returnsnullTest plan
test_create_chat_result_with_null_choices_in_model_dumpthat simulates the vLLM scenario