Future-House/paper-qa

Timout error

Opened this issue · 7 comments

Snikch63200 commented

Hello,

I recently updated paper-qa and now get this error :

Give Feedback / Get Help: https://github.com/BerriAI/litellm/issues/new
LiteLLM.Info: If you need to debug this error, use `litellm.set_verbose=True'.

InMemoryCache: set_cache. current size= 5

Give Feedback / Get Help: https://github.com/BerriAI/litellm/issues/new
LiteLLM.Info: If you need to debug this error, use `litellm.set_verbose=True'.

InMemoryCache: set_cache. current size= 6

Give Feedback / Get Help: https://github.com/BerriAI/litellm/issues/new
LiteLLM.Info: If you need to debug this error, use `litellm.set_verbose=True'.

InMemoryCache: set_cache. current size= 6

Give Feedback / Get Help: https://github.com/BerriAI/litellm/issues/new
LiteLLM.Info: If you need to debug this error, use `litellm.set_verbose=True'.

InMemoryCache: set_cache. current size= 6

Give Feedback / Get Help: https://github.com/BerriAI/litellm/issues/new
LiteLLM.Info: If you need to debug this error, use `litellm.set_verbose=True'.

InMemoryCache: set_cache. current size= 6
Traceback (most recent call last):
  File "C:\ProgramData\anaconda3\envs\PaperQA_env\Lib\site-packages\httpx\_transports\default.py", line 72, in map_httpcore_exceptions
    yield
  File "C:\ProgramData\anaconda3\envs\PaperQA_env\Lib\site-packages\httpx\_transports\default.py", line 377, in handle_async_request
    resp = await self._pool.handle_async_request(req)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\ProgramData\anaconda3\envs\PaperQA_env\Lib\site-packages\httpcore\_async\connection_pool.py", line 216, in handle_async_request
    raise exc from None
  File "C:\ProgramData\anaconda3\envs\PaperQA_env\Lib\site-packages\httpcore\_async\connection_pool.py", line 196, in handle_async_request
    response = await connection.handle_async_request(
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\ProgramData\anaconda3\envs\PaperQA_env\Lib\site-packages\httpcore\_async\connection.py", line 101, in handle_async_request
    return await self._connection.handle_async_request(request)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\ProgramData\anaconda3\envs\PaperQA_env\Lib\site-packages\httpcore\_async\http11.py", line 143, in handle_async_request
    raise exc
  File "C:\ProgramData\anaconda3\envs\PaperQA_env\Lib\site-packages\httpcore\_async\http11.py", line 113, in handle_async_request
    ) = await self._receive_response_headers(**kwargs)
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\ProgramData\anaconda3\envs\PaperQA_env\Lib\site-packages\httpcore\_async\http11.py", line 186, in _receive_response_headers
    event = await self._receive_event(timeout=timeout)
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\ProgramData\anaconda3\envs\PaperQA_env\Lib\site-packages\httpcore\_async\http11.py", line 224, in _receive_event
    data = await self._network_stream.read(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\ProgramData\anaconda3\envs\PaperQA_env\Lib\site-packages\httpcore\_backends\anyio.py", line 32, in read
    with map_exceptions(exc_map):
         ^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\ProgramData\anaconda3\envs\PaperQA_env\Lib\contextlib.py", line 158, in __exit__
    self.gen.throw(value)
  File "C:\ProgramData\anaconda3\envs\PaperQA_env\Lib\site-packages\httpcore\_exceptions.py", line 14, in map_exceptions
    raise to_exc(exc) from exc
httpcore.ReadTimeout

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "C:\ProgramData\anaconda3\envs\PaperQA_env\Lib\site-packages\openai\_base_client.py", line 1554, in _request
    response = await self._client.send(
               ^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\ProgramData\anaconda3\envs\PaperQA_env\Lib\site-packages\httpx\_client.py", line 1674, in send
    response = await self._send_handling_auth(
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\ProgramData\anaconda3\envs\PaperQA_env\Lib\site-packages\httpx\_client.py", line 1702, in _send_handling_auth
    response = await self._send_handling_redirects(
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\ProgramData\anaconda3\envs\PaperQA_env\Lib\site-packages\httpx\_client.py", line 1739, in _send_handling_redirects
    response = await self._send_single_request(request)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\ProgramData\anaconda3\envs\PaperQA_env\Lib\site-packages\httpx\_client.py", line 1776, in _send_single_request
    response = await transport.handle_async_request(request)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\ProgramData\anaconda3\envs\PaperQA_env\Lib\site-packages\httpx\_transports\default.py", line 376, in handle_async_request
    with map_httpcore_exceptions():
         ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\ProgramData\anaconda3\envs\PaperQA_env\Lib\contextlib.py", line 158, in __exit__
    self.gen.throw(value)
  File "C:\ProgramData\anaconda3\envs\PaperQA_env\Lib\site-packages\httpx\_transports\default.py", line 89, in map_httpcore_exceptions
    raise mapped_exc(message) from exc
httpx.ReadTimeout

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "C:\ProgramData\anaconda3\envs\PaperQA_env\Lib\site-packages\litellm\llms\OpenAI\openai.py", line 867, in acompletion
    headers, response = await self.make_openai_chat_completion_request(
                        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\ProgramData\anaconda3\envs\PaperQA_env\Lib\site-packages\litellm\llms\OpenAI\openai.py", line 580, in make_openai_chat_completion_request
    raise e
  File "C:\ProgramData\anaconda3\envs\PaperQA_env\Lib\site-packages\litellm\llms\OpenAI\openai.py", line 568, in make_openai_chat_completion_request
    await openai_aclient.chat.completions.with_raw_response.create(
  File "C:\ProgramData\anaconda3\envs\PaperQA_env\Lib\site-packages\openai\_legacy_response.py", line 370, in wrapped
    return cast(LegacyAPIResponse[R], await func(*args, **kwargs))
                                      ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\ProgramData\anaconda3\envs\PaperQA_env\Lib\site-packages\openai\resources\chat\completions.py", line 1412, in create
    return await self._post(
           ^^^^^^^^^^^^^^^^^
  File "C:\ProgramData\anaconda3\envs\PaperQA_env\Lib\site-packages\openai\_base_client.py", line 1821, in post
    return await self.request(cast_to, opts, stream=stream, stream_cls=stream_cls)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\ProgramData\anaconda3\envs\PaperQA_env\Lib\site-packages\openai\_base_client.py", line 1515, in request
    return await self._request(
           ^^^^^^^^^^^^^^^^^^^^
  File "C:\ProgramData\anaconda3\envs\PaperQA_env\Lib\site-packages\openai\_base_client.py", line 1573, in _request
    raise APITimeoutError(request=request) from err
openai.APITimeoutError: Request timed out.

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "C:\ProgramData\anaconda3\envs\PaperQA_env\Lib\site-packages\litellm\main.py", line 435, in acompletion
    response = await init_response
               ^^^^^^^^^^^^^^^^^^^
  File "C:\ProgramData\anaconda3\envs\PaperQA_env\Lib\site-packages\litellm\llms\OpenAI\openai.py", line 898, in acompletion
    raise OpenAIError(
litellm.llms.OpenAI.openai.OpenAIError: Request timed out.

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "C:\Users\XXXXXXXXXXXXXX\Desktop\PaperQA\paper_qa_question_sync.py", line 106, in <module>
    my_query()
  File "C:\Users\XXXXXXXXXXXXXX\Desktop\PaperQA\paper_qa_question_sync.py", line 93, in my_query
    answer = docs.query(
             ^^^^^^^^^^^
  File "C:\ProgramData\anaconda3\envs\PaperQA_env\Lib\site-packages\paperqa\docs.py", line 636, in query
    return get_loop().run_until_complete(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\ProgramData\anaconda3\envs\PaperQA_env\Lib\asyncio\base_events.py", line 687, in run_until_complete
    return future.result()
           ^^^^^^^^^^^^^^^
  File "C:\ProgramData\anaconda3\envs\PaperQA_env\Lib\site-packages\paperqa\docs.py", line 677, in aquery
    answer = await self.aget_evidence(
             ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\ProgramData\anaconda3\envs\PaperQA_env\Lib\site-packages\paperqa\docs.py", line 603, in aget_evidence
    results = await gather_with_concurrency(
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\ProgramData\anaconda3\envs\PaperQA_env\Lib\site-packages\paperqa\utils.py", line 112, in gather_with_concurrency
    return await asyncio.gather(*(sem_coro(c) for c in coros))
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\ProgramData\anaconda3\envs\PaperQA_env\Lib\site-packages\paperqa\utils.py", line 110, in sem_coro
    return await coro
           ^^^^^^^^^^
  File "C:\ProgramData\anaconda3\envs\PaperQA_env\Lib\site-packages\paperqa\core.py", line 75, in map_fxn_summary
    llm_result = await prompt_runner(
                 ^^^^^^^^^^^^^^^^^^^^
  File "C:\ProgramData\anaconda3\envs\PaperQA_env\Lib\site-packages\paperqa\llms.py", line 335, in run_prompt
    return await self._run_chat(
           ^^^^^^^^^^^^^^^^^^^^^
  File "C:\ProgramData\anaconda3\envs\PaperQA_env\Lib\site-packages\paperqa\llms.py", line 388, in _run_chat
    chunk = await self.achat(messages)
            ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\ProgramData\anaconda3\envs\PaperQA_env\Lib\site-packages\paperqa\llms.py", line 540, in wrapper
    result = await func(self, *args, **kwargs)  # type: ignore[misc]
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\ProgramData\anaconda3\envs\PaperQA_env\Lib\site-packages\paperqa\llms.py", line 721, in achat
    response = await self.router.acompletion(self.name, list(messages))
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\ProgramData\anaconda3\envs\PaperQA_env\Lib\site-packages\litellm\router.py", line 740, in acompletion
    raise e
  File "C:\ProgramData\anaconda3\envs\PaperQA_env\Lib\site-packages\litellm\router.py", line 728, in acompletion
    response = await self.async_function_with_fallbacks(**kwargs)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\ProgramData\anaconda3\envs\PaperQA_env\Lib\site-packages\litellm\router.py", line 3044, in async_function_with_fallbacks
    raise original_exception
  File "C:\ProgramData\anaconda3\envs\PaperQA_env\Lib\site-packages\litellm\router.py", line 2898, in async_function_with_fallbacks
    response = await self.async_function_with_retries(*args, **kwargs)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\ProgramData\anaconda3\envs\PaperQA_env\Lib\site-packages\litellm\router.py", line 3174, in async_function_with_retries
    raise original_exception
  File "C:\ProgramData\anaconda3\envs\PaperQA_env\Lib\site-packages\litellm\router.py", line 3088, in async_function_with_retries
    response = await self.make_call(original_function, *args, **kwargs)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\ProgramData\anaconda3\envs\PaperQA_env\Lib\site-packages\litellm\router.py", line 3181, in make_call
    response = await original_function(*args, **kwargs)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\ProgramData\anaconda3\envs\PaperQA_env\Lib\site-packages\litellm\router.py", line 879, in _acompletion
    raise e
  File "C:\ProgramData\anaconda3\envs\PaperQA_env\Lib\site-packages\litellm\router.py", line 851, in _acompletion
    response = await _response
               ^^^^^^^^^^^^^^^
  File "C:\ProgramData\anaconda3\envs\PaperQA_env\Lib\site-packages\litellm\utils.py", line 1573, in wrapper_async
    raise e
  File "C:\ProgramData\anaconda3\envs\PaperQA_env\Lib\site-packages\litellm\utils.py", line 1388, in wrapper_async
    result = await original_function(*args, **kwargs)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\ProgramData\anaconda3\envs\PaperQA_env\Lib\site-packages\litellm\main.py", line 457, in acompletion
    raise exception_type(
          ^^^^^^^^^^^^^^^
  File "C:\ProgramData\anaconda3\envs\PaperQA_env\Lib\site-packages\litellm\litellm_core_utils\exception_mapping_utils.py", line 2033, in exception_type
    raise e
  File "C:\ProgramData\anaconda3\envs\PaperQA_env\Lib\site-packages\litellm\litellm_core_utils\exception_mapping_utils.py", line 209, in exception_type
    raise Timeout(
litellm.exceptions.Timeout: litellm.Timeout: APITimeoutError - Request timed out.
error_str: Request timed out.
Received Model Group=openai/mixtral:8x7b
Available Model Group Fallbacks=None LiteLLM Retried: 2 times, LiteLLM Max Retries: 3

There is a timout error with LLM i guess.

Is there a solution for this ?

Best regards @dosu.

dosubot commented

The timeout error you're encountering seems to be related to the litellm library's handling of API requests. Here are a few steps you can take to address this issue:

  1. Increase Timeout Settings: If possible, try increasing the timeout settings for the API requests. This might involve adjusting the timeout parameters in the litellm or httpx configurations, though specific details on how to do this in the paper-qa codebase are not available in the provided information.

  2. Check Retry Configuration: The litellm library appears to have a retry mechanism in place, as indicated by the "LiteLLM Retried: 2 times, LiteLLM Max Retries: 3" message. You might want to check if you can increase the number of retries or adjust the retry strategy to better handle transient network issues.

  3. Debugging: Enable verbose logging by setting litellm.set_verbose=True as suggested in the error message. This can provide more detailed information about what might be causing the timeout.

  4. Network Stability: Ensure that your network connection is stable and that there are no interruptions that could be causing the timeouts.

  5. Library Updates: Make sure that all related libraries (litellm, httpx, etc.) are up to date, as updates might include fixes for known issues.

If these steps do not resolve the issue, you may need to look into the specific implementation details of the litellm library or consult its documentation for further guidance on handling timeouts.

To continue talking to Dosu, mention @dosu.

Praffful0507 commented

assign

Snikch63200 commented

How can I increase (or turn off) timeout in the litellm or httpx configurations ?

@dosu.

dosubot commented

To increase or turn off the timeout in the litellm configurations, you can use the get_litellm_retrying_config function. This function allows you to set a custom timeout value. For example, to set the timeout to 120 seconds, you can do the following:

config = get_litellm_retrying_config(timeout=120.0)

This will configure litellm to use a timeout of 120 seconds for its operations. If you want to effectively turn off the timeout, you might set it to a very high value, although completely disabling it might not be directly supported [1].

To continue talking to Dosu, mention @dosu.

Snikch63200 commented

Where can I set up litellm timeout in paper-qa API ?

(aka. passing 'config' argument config = get_litellm_retrying_config(timeout=120.0))

Best regards.

@dosu.

dosubot commented

You can set up the litellm timeout in the paper-qa API by passing the 'config' argument using the get_litellm_retrying_config function. This is done in the LiteLLMEmbeddingModel class, specifically in the set_up_default_config method:

@field_validator("config")
@classmethod
def set_up_default_config(cls, value: dict[str, Any]) -> dict[str, Any]:
    if "kwargs" not in value:
        value["kwargs"] = get_litellm_retrying_config(
            timeout=120,  # 2-min timeout seemed reasonable
        )
    return value

This method sets a default configuration for the config field, where the timeout is set to 120 seconds by calling get_litellm_retrying_config(timeout=120.0) [1].

To continue talking to Dosu, mention @dosu.

Snikch63200 commented

Solved the problem by changing timeout in 'llms.py' API script :

lines 121 - 128 :

    @field_validator("config")
    @classmethod
    def set_up_default_config(cls, value: dict[str, Any]) -> dict[str, Any]:
        if "kwargs" not in value:
            value["kwargs"] = get_litellm_retrying_config(
                timeout=600,  # Changed 120 to 600
            )
        return value

And lines 578 - 580 :

def get_litellm_retrying_config(timeout: float = 600.0) -> dict[str, Any]: Changed 60 to 600 think the problem was here...
    """Get retrying configuration for litellm.acompletion and litellm.aembedding."""
    return {"num_retries": 3, "timeout": timeout}

So, issu solved... Guess there was changes between two versions.

Best regards.