Switch narration app to use quarkus-langchain4j
edeandrea opened this issue · 12 comments
Currently the rest-narration application uses the Microsoft Semantic Kernel for Java library for connecting to the OpenAI providers.
There are some drawbacks to using this library in Quarkus:
- It does not support native compilation (see microsoft/semantic-kernel#2885)
- It is "heavy" and is built on top of project reactor and not Java 9 Flows
- It is very difficult to test. You can not use WireMock for mocking calls due to the fact the Azure client thats baked in requires the use of specific Microsoft certificates that aren't provided. Because of this we have to revert to object-based mocking, which means having to know/understand the internals of the Semantic Kernel library to know & understand how to mock it
Instead, we should use the Quarkus Langchain4J extension. This would also make it super easy to use a local LLM instead of invoking a remode instance. Its also more advanced than Semantic Kernel, has more functionality, the API is nicer (no project reactor), and it has a larger community behind it.
@edeandrea would you be able to give me the URL the application uses when it hits OpenAI?
Asking because I tried something really quick with Langchain4j but I am getting 404s
I really have no idea what url it hits currently. It's baked into the semantic kernel API somewhere.....
Or do you mean when it hits the Azure OpenAI service?
The URL. I was hoping that semantic kernel has some debugging you can use
Which URL though? The URL it uses for the Azure OpenAI service or for OpenAI.net?
The former
It would be something like https://<cognitive-service-endpoint>/deployments/<deployment-name>/completions?api-version=<api-version>
Once quarkiverse/quarkus-langchain4j#39 is in and we do a 0.2.0 release, the switch should be possible and the move to quarkus-langchain4j-azure-openai should make the narration service a lot simpler.
Per the conversation in quarkiverse/quarkus-langchain4j#39, unfortunately for now we can not bring that extension into the mix within this application due to the fact that this app currently supports integration with both Azure OpenAI and OpenAI.net. The Quarkus Langchain4j extension currently only supports a single provider.
Perhaps in the future it might support multiple providers via configuration, in which case we can re-examine it.
This is blocked by quarkiverse/quarkus-langchain4j#56
@geoand If I include both the openai & azure-open-ai extensions, how do I tell it (at build time) which one to use? Would I use the chat-model.enabled property?
Nevermind I found quarkus.langchain4j.chat-model.provider!