api_base settings not useful
Opened this issue · 4 comments
I'm running in jupyter notebook with paper-qa==5.3.0, tyring to use an alternative OPENAI compatiable api.
here is my script
# make sure to run in jupyter
import os
import nest_asyncio
nest_asyncio.apply()
# then I import paperqa and set litellm verbose
from paperqa import Docs, Settings
import litellm
litellm.set_verbose=True
# add docs for testing
doc_paths = ("Cui2021.pdf", "Ghosh2018.pdf")
docs = Docs()
for doc in doc_paths:
docs.add(doc)
I follow the instruction in readme and #492, with the config like this:
llm_config = dict(
model_list=[
dict(
model_name='gpt-4o-2024-08-06',
litellm_params=dict(
model='gpt-4o-2024-08-06',
api_key=os.environ["OPENAI_API_KEY"],
api_base="https://api.deepbricks.ai/v1/"
)
),
],
)
and then make docs.query:
answer = docs.query(
"What challenges are unique to bispecific antibodies?",
settings=Settings(llm="gpt-4o-2024-08-06",
llm_config=llm_config,
summary_llm="gpt-4o-2024-08-06",
summary_llm_config=llm_config)
)
print(answer.formatted_answer)
Unfortunatedly I got following output:
Token Counter - using OpenAI token counter, for model=gpt-4o-2024-08-06
LiteLLM: Utils - Counting tokens for OpenAI model=gpt-4o-2024-08-06
Token Counter - using OpenAI token counter, for model=gpt-4o-2024-08-06
LiteLLM: Utils - Counting tokens for OpenAI model=gpt-4o-2024-08-06
set cache: key: b622c96ddd11ae0ab2d5badac10abf2bb7977f0b5de7d310a0dc617035bb4e25_async_client; value: <openai.AsyncOpenAI object at 0x00000257096CDA90>
InMemoryCache: set_cache. current size= 0
set cache: key: b622c96ddd11ae0ab2d5badac10abf2bb7977f0b5de7d310a0dc617035bb4e25_client; value: <openai.OpenAI object at 0x00000257096CF9B0>
InMemoryCache: set_cache. current size= 1
set cache: key: b622c96ddd11ae0ab2d5badac10abf2bb7977f0b5de7d310a0dc617035bb4e25_stream_async_client; value: <openai.AsyncOpenAI object at 0x00000257096CF020>
InMemoryCache: set_cache. current size= 2
14:40:21 - LiteLLM:WARNING: utils.py:325 - `litellm.set_verbose` is deprecated. Please set `os.environ['LITELLM_LOG'] = 'DEBUG'` for debug logs.
set cache: key: b622c96ddd11ae0ab2d5badac10abf2bb7977f0b5de7d310a0dc617035bb4e25_stream_client; value: <openai.OpenAI object at 0x00000257096D2720>
InMemoryCache: set_cache. current size= 3
get cache: cache key: b622c96ddd11ae0ab2d5badac10abf2bb7977f0b5de7d310a0dc617035bb4e25; local_only: True
get cache: cache result: None
set cache: key: b622c96ddd11ae0ab2d5badac10abf2bb7977f0b5de7d310a0dc617035bb4e25; value: 1
InMemoryCache: set_cache. current size= 4
get cache: cache key: b622c96ddd11ae0ab2d5badac10abf2bb7977f0b5de7d310a0dc617035bb4e25_async_client; local_only: True
get cache: cache result: <openai.AsyncOpenAI object at 0x00000257096CDA90>
get cache: cache key: b622c96ddd11ae0ab2d5badac10abf2bb7977f0b5de7d310a0dc617035bb4e25_max_parallel_requests_client; local_only: True
get cache: cache result: None
Request to litellm:
litellm.acompletion(model='gpt-4o-2024-08-06', temperature=0.0, messages=[{'role': 'user', 'content': '
_Too Long So I removed the content_
'}], caching=False, client=<openai.AsyncOpenAI object at 0x00000257096CDA90>, timeout=60.0, stream=False, metadata={'model_group': 'gpt-4o-2024-08-06', 'model_group_size': 1,
'deployment': 'gpt-4o-2024-08-06', 'model_info': {'id': 'b622c96ddd11ae0ab2d5badac10abf2bb7977f0b5de7d310a0dc617035bb4e25', 'db_model': False},
**'api_base': None**, 'caching_groups': None}, model_info={'id': 'b622c96ddd11ae0ab2d5badac10abf2bb7977f0b5de7d310a0dc617035bb4e25', 'db_model': False}, max_retries=0)
ASYNC kwargs[caching]: False; litellm.cache: None; kwargs.get('cache'): None
Final returned optional params: {'temperature': 0.0, 'stream': False, 'max_retries': 0, 'extra_body': {}}
POST Request Sent from LiteLLM:
curl -X POST \
**https://api.openai.com/v1/** \
-H 'Authorization: Bearer sk-eE9N********************************************' \
-d '{'model': 'gpt-4o-2024-08-06', 'messages': [{'role': 'user', 'content': '_Also Too Long_'}], 'temperature': 0.0, 'stream': False, 'extra_body': {}}'
...
Give Feedback / Get Help: https://github.com/BerriAI/litellm/issues/new
LiteLLM.Info: If you need to debug this error, use `litellm.set_verbose=True'.
InMemoryCache: set_cache. current size= 5
Output is truncated. View as a [scrollable element](command:cellOutput.enableScrolling?6400e798-a497-47f1-ae13-294c29d9ba0d) or open in a [text editor](command:workbench.action.openLargeOutput?6400e798-a497-47f1-ae13-294c29d9ba0d). Adjust cell output [settings](command:workbench.action.openSettings?%5B%22%40tag%3AnotebookOutputLayout%22%5D)...
[d:\ProgramData\anaconda3\envs\paperqa\Lib\site-packages\executing\executing.py:171](file:///D:/ProgramData/anaconda3/envs/paperqa/Lib/site-packages/executing/executing.py:171): RuntimeWarning: coroutine 'Docs.aadd' was never awaited
self._nodes_by_line[lineno].append(node)
RuntimeWarning: Enable tracemalloc to get the object al
I wonder what have i done wrong in this situation, litellm obviously got 'api_base': None
and send request to api.openai.com
by default. How should I config for this api_base
?
Hello, @ganansuan647! I'm here to assist you with any bugs, questions, or concerns you may have. Let's work together to resolve your issue.
I found a similar closed issue regarding the use of an alternative API base with paperqa
and litellm
. The issue suggested that version 5 of paperqa
outsources all LLM management to BerriAI/litellm
[1].
To ensure that litellm
uses the provided api_base
, you should verify that the api_base
parameter is correctly set in the litellm_params
dictionary within your llm_config
. Your configuration seems correct, but the issue might be related to how litellm
processes this parameter internally.
Here are a few steps you can try to troubleshoot and resolve the issue:
-
Check Environment Variables: Ensure that there are no conflicting environment variables that might override your
api_base
setting. -
Update
litellm
andpaperqa
: Make sure you are using the latest versions of bothlitellm
andpaperqa
, as updates might have fixed related issues. -
Debugging: Set the environment variable
LITELLM_LOG
toDEBUG
to get more detailed logs, which might help identify why theapi_base
is not being used. -
Directly Check
litellm
Configuration: If possible, inspect thelitellm
library's source code or documentation to ensure that theapi_base
parameter is being correctly passed and utilized.
If these steps do not resolve the issue, it might be necessary to look deeper into the litellm
library's handling of the api_base
parameter, as there might be an internal issue or a specific configuration requirement not documented.
I actually don't know if this works with litellm, but you can try just setting the environment variable:
export OPENAI_API_BASE=https://api.deepbricks.ai/v1/
before executing.
I actually don't know if this works with litellm, but you can try just setting the environment variable:
export OPENAI_API_BASE=https://api.deepbricks.ai/v1/
before executing.
Thanks for your reply, but I'm afraid it doesn‘t work for me, I've tried setting env variables like OPENAI_API_BASE
,ANTHROPIC_API_BASE
,LITELLM_API_BASE
to deepbricks.ai. Unfortunately they all fails.
I actually don't know if this works with litellm, but you can try just setting the environment variable:
export OPENAI_API_BASE=https://api.deepbricks.ai/v1/
before executing.
Thanks for your reply, but I'm afraid it doesn‘t work for me, I've tried setting env variables like
OPENAI_API_BASE
,ANTHROPIC_API_BASE
,LITELLM_API_BASE
to deepbricks.ai. Unfortunately they all fails.
You may not have set the api of agent_llm
, like
from paperqa AgentSettings
settings=Settings(
llm="gpt-4o-2024-08-06",
llm_config=llm_config,
summary_llm="gpt-4o-2024-08-06",
summary_llm_config=llm_config,
agent=AgentSettings(
agent_llm='gpt-4o-2024-08-06',
agent_llm_config=llm_config)
))