-
Notifications
You must be signed in to change notification settings - Fork 943
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Add stream_options support according to OpenAI API #1552
base: main
Are you sure you want to change the base?
Conversation
@tpfau thank you for starting on this, I'll review in more depth but my initial request would be that instead of |
@abetlen I hope this change is what you had in mind. |
I would really like to see this in production, as I think this feature can be quite important for a lot of applications that need some kind of usage control or just let the users know about their usage. Of course people can work around this kind of approach but it requires additional code in the code base that does tokenization and thus computation which is completely unnecessary as it is already calculated by the model. |
@abetlen Any news on this? |
771721e
to
dcca56a
Compare
Updated again to fix conflicts. Would be great to see this feature in the codebase. If there is anything missing, please let me know |
…g token generation into it's own function to avoid replicated statements
Did some tests again, seems to be working. Please let me know if you have any updates which you want to have in. |
This PR adds the
stream_options
andinclude_usage
fields as defined in the OpenAI API reference to the server.Set to false by default they can be activated which leads to the last chunk being returned during streaming to supply the usage information.
This solves the Question raised in #1461 and the request indicated in #1498.
I decided to put the chunk generation code into its own function since not doing so would have lead to blown up code.
I did not update the places where
stream
is clearly set tofalse
.I'm happy to update the PR with any necessary/requested modifications wrt code style or if you think all chunk generation places should be changed to use functions (or add more options/update the chunk generation function). Just let me know.
Closes #1498