langchain-ai/langchain

langchain_ollama not able to do structured outputs for gpt-oss

Opened this issue Β· 11 comments

ChrisRawstone commented

Checked other resources

  • This is a bug, not a usage question.
  • I added a clear and descriptive title that summarizes this issue.
  • I used the GitHub search to find a similar question and didn't find it.
  • I am sure that this is a bug in LangChain rather than my code.
  • The bug is not resolved by updating to the latest stable version of LangChain (or the specific integration package).
  • This is not related to the langchain-community package.
  • I read what a minimal reproducible example is (https://stackoverflow.com/help/minimal-reproducible-example).
  • I posted a self-contained, minimal, reproducible example. A maintainer can copy it and run it AS IS.

Example Code

The following code does not work

import dotenv
from typing import Optional
from pydantic import BaseModel, Field
from rich.console import Console
from langchain_ollama import ChatOllama
from langchain_openai import ChatOpenAI

# dotenv.load_dotenv()



# llm = ChatOpenAI(model="gpt-4o-mini") # works
llm = ChatOllama(model="gpt-oss:20b", reasoning=True) # does not work
# llm = ChatOllama(model="llama3.1:8b") # works

console = Console()

class Joke(BaseModel):
    """Joke to tell user."""

    setup: str = Field(description="The setup of the joke")
    punchline: str = Field(description="The punchline to the joke")
    rating: Optional[int] = Field(
        default=None, description="How funny the joke is, from 1 to 10"
    )


structured_llm = llm.with_structured_output(Joke, method="json_schema")
response = structured_llm.invoke("Tell me a joke about cats")
console.print(response)

Error Message and Stack Trace (if applicable)

test_repo on ξ‚  full_clean_refactor_langchain_switch [$!?] is πŸ“¦ v0.1.0 via  v22.20.0 via 🐍 v3.12.0 (test_repoenv) ❯ /home/test_user/Repos/test_repo/.venv/bin/python /home/test_user/Repos/test_repo/backend/services/ask_test_repo/src/langchain_structured_outputs_test.py Traceback (most recent call last): File "/home/test_user/Repos/test_repo/.venv/lib/python3.12/site-packages/langchain_core/output_parsers/json.py", line 82, in parse_result return parse_json_markdown(text) ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/test_user/Repos/test_repo/.venv/lib/python3.12/site-packages/langchain_core/utils/json.py", line 153, in parse_json_markdown return _parse_json(json_str, parser=parser) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/test_user/Repos/test_repo/.venv/lib/python3.12/site-packages/langchain_core/utils/json.py", line 169, in parse_json return parser(json_str) ^^^^^^^^^^^^^^^^ File "/home/test_user/Repos/test_repo/.venv/lib/python3.12/site-packages/langchain_core/utils/json.py", line 126, in parse_partial_json return json.loads(s, strict=strict) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/test_user/.local/share/uv/python/cpython-3.12.0-linux-x86_64-gnu/lib/python3.12/json/init.py", line 359, in loads return cls(**kw).decode(s) ^^^^^^^^^^^^^^^^^^^ File "/home/test_user/.local/share/uv/python/cpython-3.12.0-linux-x86_64-gnu/lib/python3.12/json/decoder.py", line 337, in decode obj, end = self.raw_decode(s, idx=w(s, 0).end()) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/test_user/.local/share/uv/python/cpython-3.12.0-linux-x86_64-gnu/lib/python3.12/json/decoder.py", line 355, in raw_decode raise JSONDecodeError("Expecting value", s, err.value) from None json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0) The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/home/test_user/Repos/test_repo/backend/services/ask_test_repo/src/langchain_structured_outputs_test.py", line 32, in response = structured_llm.invoke("Tell me a joke about cats") ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/test_user/Repos/test_repo/.venv/lib/python3.12/site-packages/langchain_core/runnables/base.py", line 3245, in invoke input = context.run(step.invoke, input, config) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/test_user/Repos/test_repo/.venv/lib/python3.12/site-packages/langchain_core/output_parsers/base.py", line 200, in invoke return self._call_with_config( ^^^^^^^^^^^^^^^^^^^^^^^ File "/home/test_user/Repos/test_repo/.venv/lib/python3.12/site-packages/langchain_core/runnables/base.py", line 2089, in _call_with_config context.run( File "/home/test_user/Repos/test_repo/.venv/lib/python3.12/site-packages/langchain_core/runnables/config.py", line 430, in call_func_with_variable_args return func(input, **kwargs) # type: ignore[call-arg] ^^^^^^^^^^^^^^^^^^^^^ File "/home/test_user/Repos/test_repo/.venv/lib/python3.12/site-packages/langchain_core/output_parsers/base.py", line 201, in lambda inner_input: self.parse_result( ^^^^^^^^^^^^^^^^^^ File "/home/test_user/Repos/test_repo/.venv/lib/python3.12/site-packages/langchain_core/output_parsers/pydantic.py", line 65, in parse_result json_object = super().parse_result(result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/test_user/Repos/test_repo/.venv/lib/python3.12/site-packages/langchain_core/output_parsers/json.py", line 85, in parse_result raise OutputParserException(msg, llm_output=text) from e langchain_core.exceptions.OutputParserException: Invalid json output: For troubleshooting, visit: https://python.langchain.com/docs/troubleshooting/errors/OUTPUT_PARSING_FAILURE

Description

When I am trying to use langchain with ollama with gpt-oss:20b with structured outputs then I get the error above.
I tried to downgrade and upgrade to various different package versions with no luck.
I had no issues with "llama3.1:8b" on ollama producing structured outputs and also no issues with openai api.
So the problem seems isolated to gpt-oss:20b when running structured outputs. I have no issues when not running structured outputs with gpt-oss:20b.

Hardware:
5070ti

System Info

TestRepo on ξ‚  full_clean_refactor_langchain_switch [$!?] is πŸ“¦ v0.1.0 via  v22.20.0 via 🐍 v3.12.0 (testenv)
❯ /home/test_user/Repos/TestRepo/.venv/bin/python /home/test_user/TestRepo/Alexsis/backend/quick_testing/hardware_info.py

System Information

OS: Linux
OS Version: #83~22.04.1-Ubuntu SMP PREEMPT_DYNAMIC Tue Sep 9 18:19:47 UTC 2
Python Version: 3.12.0 (main, Oct 3 2023, 01:27:23) [Clang 17.0.1 ]

Package Information

langchain_core: 0.3.76
langsmith: 0.4.31
langchain_ollama: 0.3.8
langchain_openai: 0.3.33

Optional packages not installed

langserve

Other Dependencies

httpx<1,>=0.23.0: Installed. No version info available.
jsonpatch<2.0,>=1.33: Installed. No version info available.
langchain-core<1.0.0,>=0.3.76: Installed. No version info available.
langsmith-pyo3>=0.1.0rc2;: Installed. No version info available.
langsmith>=0.3.45: Installed. No version info available.
ollama<1.0.0,>=0.5.3: Installed. No version info available.
openai-agents>=0.0.3;: Installed. No version info available.
openai<2.0.0,>=1.104.2: Installed. No version info available.
opentelemetry-api>=1.30.0;: Installed. No version info available.
opentelemetry-exporter-otlp-proto-http>=1.30.0;: Installed. No version info available.
opentelemetry-sdk>=1.30.0;: Installed. No version info available.
orjson>=3.9.14;: Installed. No version info available.
packaging>=23.2: Installed. No version info available.
pydantic<3,>=1: Installed. No version info available.
pydantic>=2.7.4: Installed. No version info available.
pytest>=7.0.0;: Installed. No version info available.
PyYAML>=5.3: Installed. No version info available.
requests-toolbelt>=1.0.0: Installed. No version info available.
requests>=2.0.0: Installed. No version info available.
rich>=13.9.4;: Installed. No version info available.
tenacity!=8.4.0,<10.0.0,>=8.1.0: Installed. No version info available.
tiktoken<1,>=0.7: Installed. No version info available.
typing-extensions>=4.7: Installed. No version info available.
vcrpy>=7.0.0;: Installed. No version info available.
zstandard>=0.23.0: Installed. No version info available.

TestRepo on ξ‚  full_clean_refactor_langchain_switch [$!?] is πŸ“¦ v0.1.0 via  v22.20.0 via 🐍 v3.12.0 (testenv)
❯

mahimairaja commented

This might be due to harmony response format

mahimairaja commented

Are you willing to work on this issue @ChrisRawstone?

heyits-manan commented

@ChrisRawstone If u want a simple work around for now, u can just use ChatOllama(model="gpt-oss:20b", reasoning=False). ChatOllama(model="gpt-oss:20b", reasoning=True) fail because the model prepends non-JSON β€œreasoning” tokens. with_structured_output(..., method="json_schema") expects JSON at the start and throws OutputParserException. The same code works when reasoning=False or with models that don’t prepend reasoning.

ChrisRawstone commented

@heyits-manan
Just tried ChatOllama(model="gpt-oss:20b", reasoning=False) and got the same error.

I did try that before I made this github issue as well. I am unsure, what you have done to get it working?

ChrisRawstone commented

Are you willing to work on this issue @ChrisRawstone?

@mahimairaja

Don't think I have the skills to fix this issue, sorry.

heyits-manan commented

@ChrisRawstone hmmm. I will look into it. I thought it will fix it

heyits-manan commented

@ChrisRawstone Yes the issue is with the Langchain itself not your code. There is already an active pull request #33042. U can also use the below code to make it work

import re
from langchain_core.output_parsers import PydanticOutputParser
from langchain_core.prompts import ChatPromptTemplate
from langchain_core.output_parsers import StrOutputParser

parser = PydanticOutputParser(pydantic_object=Joke)
prompt = ChatPromptTemplate.from_messages([
("system", "Return ONLY valid JSON matching this schema:\n" + parser.get_format_instructions()),
("user", "{input}")
])

llm = ChatOllama(model="gpt-oss:20b", reasoning=True)

def strip_reasoning(text:str) -> str:
# remove blocks and any leading non-JSON noise
text = re.sub(r".?", "", text, flags=re.DOTALL|re.IGNORECASE)
# try to extract the first JSON object
m = re.search(r"{.}", text, flags=re.DOTALL)
return m.group(0) if m else text # best effort

chain = prompt | llm | StrOutputParser() | (lambda s: strip_reasoning(s)) | parser
print(chain.invoke({"input": "Tell me a joke about cats"}))

The strip_reasoning() method will remove the (think tags) blocks that appears when u add "reasoning=True" or "reasoning=False" which is causing the problem.

AkibDa commented

@ChrisRawstone Here is the corrected code. It uses json_mode and removes the extra parameter, which will force the model to output valid JSON that matches your Joke class.

import dotenv
from typing import Optional
from pydantic import BaseModel, Field
from rich.console import Console
from langchain_ollama import ChatOllama
from langchain_openai import ChatOpenAI

# dotenv.load_dotenv()

# llm = ChatOpenAI(model="gpt-4o-mini") # works

# ❌ Issue: `reasoning=True` is not supported in ChatOllama β†’ caused init error
# βœ… Fix: remove `reasoning=True`
llm = ChatOllama(model="gpt-oss:20b")  # fixed

# llm = ChatOllama(model="llama3.1:8b") # works

console = Console()

class Joke(BaseModel):
    """Joke to tell user."""

    setup: str = Field(description="The setup of the joke")
    punchline: str = Field(description="The punchline to the joke")
    rating: Optional[int] = Field(
        default=None, description="How funny the joke is, from 1 to 10"
    )

# ❌ Issue: Ollama models often failed with `json_schema` β†’ invalid JSON parsing
# βœ… Fix: switch to `json_mode` which is better supported in ChatOllama
structured_llm = llm.with_structured_output(Joke, method="json_mode")

response = structured_llm.invoke("Tell me a joke about cats")
console.print(response)
ChrisRawstone commented

Thanks @AkibDa @heyits-manan for coming up with suggestions. However the problem is that both suggestions/approaches avoids using the ollama's native structured output. Sadly "json_mode" is not as reliable as "json_schema" and tend to break at some point.
Before creating this issue, I did notice that both "json_mode" and "function calling" works, but it is a shame that json_schema does not work...

According to Ollama blogpost when they introduced gpt-oss they mentioned they support native structured outputs. So I assume the problem is in the langchain implementation. https://ollama.com/library/gpt-oss

The json_schema works with every other reasoning and llm model I have tested so it would be great if we could get this working with the state of the art llm model "gpt-oss" as well..-

Please correct me if I am wrong.

AkibDa commented

You are absolutely correct in your reasoning. The issue almost certainly lies in the LangChain integration layer for Ollama, not with the gpt-oss model or Ollama itself.The problem isn't that gpt-oss can't produce structured output; it's that the specific, highly-constrained json_schema method in LangChain has an integration gap with this particular model via Ollama.

mdrxy commented

This is an upstream issue ollama/ollama#11691

Happy to address it once released in ollama and ollama-python