`ValueError: No data received from Ollama stream` when using Ollama
Closed this issue · 2 comments
When I use Ollama via Langchain I get the error:
/paper_searching/env/lib/python3.11/site-packages/langchain/llms/ollama.py:185: RuntimeWarning: coroutine 'AsyncCallbackManagerForLLMRun.on_llm_new_token' was never awaited
run_manager.on_llm_new_token(
RuntimeWarning: Enable tracemalloc to get the object allocation traceback
Traceback (most recent call last):
File "/paper_searching/issue.py", line 19, in <module>
File "/paper_searching/env/lib/python3.11/site-packages/paperqa/docs.py", line 584, in query
return loop.run_until_complete(
^^^^^^^^^^^^^^^^^^^^^^^^
File "/paper_searching/env/lib/python3.11/asyncio/base_events.py", line 653, in run_until_complete
return future.result()
^^^^^^^^^^^^^^^
File "/paper_searching/env/lib/python3.11/site-packages/paperqa/docs.py", line 621, in aquery
answer = await self.aget_evidence(
^^^^^^^^^^^^^^^^^^^^^^^^^
File "/paper_searching/env/lib/python3.11/site-packages/paperqa/docs.py", line 540, in aget_evidence
results = await gather_with_concurrency(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/paper_searching/env/lib/python3.11/site-packages/paperqa/utils.py", line 87, in gather_with_concurrency
return await asyncio.gather(*(sem_coro(c) for c in coros))
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/paper_searching/env/lib/python3.11/site-packages/paperqa/utils.py", line 85, in sem_coro
return await coro
^^^^^^^^^^
File "/paper_searching/env/lib/python3.11/site-packages/paperqa/docs.py", line 508, in process
raise e
File "/paper_searching/env/lib/python3.11/site-packages/paperqa/docs.py", line 497, in process
context = await summary_chain.arun(
^^^^^^^^^^^^^^^^^^^^^^^^^
File "/paper_searching/env/lib/python3.11/site-packages/langchain/chains/base.py", line 588, in arun
await self.acall(
File "/paper_searching/env/lib/python3.11/site-packages/langchain/chains/base.py", line 381, in acall
raise e
File "/paper_searching/env/lib/python3.11/site-packages/langchain/chains/base.py", line 375, in acall
await self._acall(inputs, run_manager=run_manager)
File "/paper_searching/env/lib/python3.11/site-packages/langchain/chains/llm.py", line 275, in _acall
response = await self.agenerate([inputs], run_manager=run_manager)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/paper_searching/env/lib/python3.11/site-packages/paperqa/chains.py", line 38, in agenerate
return await super().agenerate(input_list, run_manager=run_manager)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/paper_searching/env/lib/python3.11/site-packages/langchain/chains/llm.py", line 142, in agenerate
return await self.llm.agenerate_prompt(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/paper_searching/env/lib/python3.11/site-packages/langchain_core/language_models/llms.py", line 516, in agenerate_prompt
return await self.agenerate(
^^^^^^^^^^^^^^^^^^^^^
File "/paper_searching/env/lib/python3.11/site-packages/langchain_core/language_models/llms.py", line 825, in agenerate
output = await self._agenerate_helper(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/paper_searching/env/lib/python3.11/site-packages/langchain_core/language_models/llms.py", line 712, in _agenerate_helper
raise e
File "/paper_searching/env/lib/python3.11/site-packages/langchain_core/language_models/llms.py", line 699, in _agenerate_helper
await self._agenerate(
File "/paper_searching/env/lib/python3.11/site-packages/langchain_core/language_models/llms.py", line 476, in _agenerate
return await asyncio.get_running_loop().run_in_executor(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/paper_searching/env/lib/python3.11/concurrent/futures/thread.py", line 58, in run
result = self.fn(*self.args, **self.kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/paper_searching/env/lib/python3.11/site-packages/langchain/llms/ollama.py", line 241, in _generate
final_chunk = super()._stream_with_aggregation(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/paper_searching/env/lib/python3.11/site-packages/langchain/llms/ollama.py", line 190, in _stream_with_aggregation
raise ValueError("No data received from Ollama stream.")
ValueError: No data received from Ollama stream.
/paper_searching/env/lib/python3.11/site-packages/langchain/llms/ollama.py:185: RuntimeWarning: coroutine 'AsyncCallbackManagerForLLMRun.on_llm_new_token' was never awaited
run_manager.on_llm_new_token(
You can reproduce it using
from paperqa import Docs
from langchain.callbacks.manager import CallbackManager
from langchain.callbacks.streaming_stdout import StreamingStdOutCallbackHandler
from langchain.llms import Ollama
from langchain.embeddings import OllamaEmbeddings
# have tried a few models
model = "llama2"
llm = Ollama(
model=model,callback_manager=CallbackManager([StreamingStdOutCallbackHandler()])
)
# Demonstrate Ollama and langchain are working
print(llm("Who was the first US President?"))
embeddings = OllamaEmbeddings(base_url="http://localhost:11434", model=model)
docs = Docs(llm=llm, embeddings=embeddings)
docs.add("I Pencil.pdf")
answer = docs.query("Are pencils made of wood?")The pdf I'm using here the I Pencil essay, but I've had this behaviour will all the PDFs I've tried.
As the print(llm("Who... works I believe the error is to do with paper-qa and/or its interaction with langchain
I ran into the same issue when I have 2 calls to chain.invoke()
The first chain always work. Only the second chain failed.
What I found out is that if I used LLMChain and it would work but the placement of the import statement is important.
If the import is in the beginning of the script then it would failed the same way.
But the import of LLMChain right before the second chain.invoke and works fine
This is using:
langchain Version: 0.0.340
ollama version 0.1.13
Note it used to all works fine with: ollama version 0.1.8
Here is how it failed.
File "/home/kenneth/learning/venv/lib/python3.10/site-packages/langchain/llms/ollama.py", line 191, in _stream_with_aggregation
raise ValueError("No data received from Ollama stream.")
ValueError: No data received from Ollama stream.
from langchain.llms import Ollama
from langchain.callbacks.manager import CallbackManager
from langchain.callbacks.streaming_stdout import StreamingStdOutCallbackHandler
from langchain.prompts import PromptTemplate
#from langchain.chains import LLMChain
customer_email = """
Arrr, I be fuming that me blender lid
flew off and splattered me kitchen walls
with smoothie! And to make matters worse,
the warranty don't cover the cost of
cleaning up me kitchen. I need yer help
right now, matey!
"""
style = """American English
in a calm and respectful tone
"""
customer_template = """Translate the text
that is delimited by triple backticks
into a style that is {style}.
text: {customer_email}
"""
customer_prompt = PromptTemplate(input_variables=["style", "customer_email"], template=customer_template)
print(f"prompt: {customer_prompt}")
llm = Ollama(model="mistral:instruct")
chain1 = customer_prompt | llm
response = chain1.invoke({"style": style, "customer_email": customer_email})
Service reply
service_reply = """Hey there customer,
the warranty does not cover
cleaning expenses for your kitchen
because it's your fault that
you misused your blender
by forgetting to put the lid on before
starting the blender.
Tough luck! See ya!
"""
service_style_pirate = """
a polite tone
that speaks in Spanish
"""
service_template = """Translate the text
that is delimited by triple backticks
into a style that is {style}.
text: {service_reply}
"""
service_prompt = PromptTemplate(input_variables=["style", "service_reply"], template=service_template)
#chain2 = service_prompt | llm
from langchain.chains import LLMChain
chain2 = LLMChain(llm=llm, prompt=service_prompt, verbose=True)
print(f"222 chain2: {chain2}")
response = chain2.invoke({"style": service_style_pirate, "service_reply": service_reply})
print(f"Service response: >>>{response}<<<")
(venv) kenneth@kpc:~/learning/deeplearning/deeplearning_langchain_llm_course$ python 1_chain.py
prompt: input_variables=['customer_email', 'style'] template='Translate the text that is delimited by triple backticks \ninto a style that is {style}.\ntext: {customer_email}\n'
222 chain2: verbose=True prompt=PromptTemplate(input_variables=['service_reply', 'style'], template='Translate the text that is delimited by triple backticks \ninto a style that is {style}.\ntext: {service_reply}\n') llm=Ollama(model='mistral:instruct')
Entering new LLMChain chain...
Prompt after formatting:
Translate the text that is delimited by triple backticks
into a style that is a polite tone that speaks in Spanish.
text: ```Hey there customer, the warranty does not cover cleaning expenses for your kitchen because it's your fault that you misused your blender by forgetting to put the lid on before starting the blender. Tough luck! See ya!
> Finished chain.
Service response: >>>{'style': 'a polite tone that speaks in Spanish', 'service_reply': "Hey there customer, the warranty does not cover cleaning expenses for your kitchen because it's your fault that you misused your blender by forgetting to put the lid on before starting the blender. Tough luck! See ya!\n", 'text': ' Hola cliente, la garantía no cubre los costos de limpieza de tu cocina porque es tu culpa que has maltratado tu licuadora al olvidar poner la tapa antes de empezar a usarla. Lo siento mucho! ¡Hasta luego!'}<<<
(venv) kenneth@kpc:~/learning/deeplearning/deeplearning_langchain_llm_course$
Hello everyone, we have just released version 5, which completely outsources all LLM management to https://github.com/BerriAI/litellm and drops LangChain.
If your issue persists, please reopen a new issue using paper-qa>=5