google-gemini/cookbook

Enabling JSON Format Responses for Image Inputs in Gemini 1.5

kihapper opened this issue · 5 comments

kihapper commented

The new Gemini 1.5 model has the capability to enforce JSON responses, but there is limited documentation available on how to implement this, particularly for obtaining JSON responses from image inputs.

Could you provide an example of how to achieve this?
The REST example here and the official doc provided did not give me clarity on the process.

kihapper commented

I am calling it like below in python but is getting the error message
ValueError: Protocol message GenerationConfig has no "response_mime_type" field.

# Configure JSON response and safety settings
generation_config = {
    'response_mime_type': 'application/json'  # Add this line for JSON response
}

# Call generate_content with the updated configuration
response = model.generate_content(
    [prompt_variable, image_data],
    generation_config=generation_config,
    stream=False
)

 response.resolve()
TYMichaelChen commented

Hi @kihapper thanks for reporting the issue. We are working on this - This should be fixed in the upcoming SDK release. Will update this once the SDK is published.

kihapper commented

@TYMichaelChen
Thanks for this! Do you know roughly when these SDK will be rolled out for python and node.js?

markmcd commented

Like Michael says - we're still working on the feature, though some support in the Python SDK is available at HEAD. You can pip install git+https://github.com/google/generative-ai-python to get the latest version and try it out, but caveat emptor.

For the specific proto field you're trying to use, as long as you have google-ai-generativelanguage>=0.6.2, you should be good to go.

TYMichaelChen commented

The newest SDK on PyPI should now have the changes: https://pypi.org/project/google-generativeai/0.5.1/. Let us know if you are still seeing this issue