Allow the outputFormatInstructions to be included or not in the prompt

[andreadimaio]

Today, when an AIService method returns an object, the schema is included in the prompt. It might be useful to create a property to give the developer the ability to choose whether this must be included or not, because in some cases (when the developer uses LLM-specific tags) the place where the schema is added (at the end of the prompt) can cause unexpected behavior.

Another improvement might be to reserve a tag for the schema and give the developer the ability to place it anywhere in the prompt.

[geoand]

Interesting.

Do you have specific examples where the current approach might fail?

[andreadimaio]

Let's say I'm using watsonx with the llama3 model.

This is my prompt:

<|begin_of_text|><|start_header_id|>system<|end_header_id|>

You are poet<|eot_id|><|start_header_id|>user<|end_header_id|>

Generate a poem of 5 lines about {topic}<|eot_id|><|start_header_id|>assistant<|end_header_id|>

And I want to put the poem in a specific java object

@RegisterAiService
public interface LLMService {

	@SystemMessage(...)
	@UserMessage(...)
	public Poem poem(String topic)
}

In this particular case, the schema of the Poem object is appended at the end of the prompt, but the correct place should be in the system or user "tags".

Now, in this particular example, this behavior may not cause a problem, but having the ability to enable/disable the schema, or better yet, give the developer the ability to choose where to put it, would be great.

[geoand]

I see, thanks.

I think it makes sense.

[andreadimaio]

What I have in mind is to have a property quarkus.langchain4j.response-schema with values of true (default) or false, which will be inherited by all LLM providers. This configuration allows the developer to decide whether or not to create the schema for each object returned in the AIService methods.

With the new property, there is also a new reserved placeholder called {response_schema} that can be used by the developer to choose where to place the schema in the prompt (could be in SystemMessage, UserMessage, or both).

If quarkus.langchain4j.response-schema is set to true and there is no {response_schema}, the schema will be placed at the end of the prompt (same behavior as today), otherwise if there is at least one {response_schema}, it will only be inserted where specified.

If quarkus.langchain4j.response-schema is set to false, no schemas will be generated and included in prompts.
If quarkus.langchain4j.response-schema is set to false and a {response_schema} exists, an exception will be thrown.

Make sense?

[geoand]

Sure, we can have that :)

[andreadimaio]

I have some code in my local environment to work with the new placeholder, the only thing missing is the default behavior (the schema is not added by default).