ReplicateError input: prompt is required
kzaho opened this issue · 4 comments
HI!
I am exploring the capabilities of Llama 2 70b and found a bug.
I wrote the temp solution, but I think many people are extremely curious about llama-2-70b,
so it would be great to get a working fix (no temp one).
replicate==0.11.0
langchain==0.0.250
python 3.10
I tried to run that code and it worked.
But, with:
model="meta/llama-2-70b:8570a72c77eeae584bbeded5b0928704a44246ba77728bd5223fbc3d6ef2888e"
I got "ReplicateError input: prompt is required".
So, I debugged and found that in the file:
/usr/local/anaconda3/envs/llm/lib/python3.10/site-packages/replicate/prediction.py
at class "PredictionCollection(Collection)", inside the method "create" we have such part:
resp = self._client._request(
"POST",
"/v1/predictions",
json=```
,
)
where replicate_weights value (inside input, which is inside body) is replaced with promt.
So, after I added that code before "resp = self._client._request" - the error is gone:
if 'prompt' not in body['input']:
body['input']['prompt'] = body['input']['replicate_weights']
del body['input']['replicate_weights']
Hi @kzaho. Thanks for the detailed write-up. This is a problem in LangChain's integration with Replicate, rather than the Python library itself (even though you can work around this issue by patching the client library).
The bug comes from the following lines, where the prompt
field is derived by selecting the first ordered field in the schema rather than, say, a field named "prompt":
Working on a solution for that now.
@kzaho A workaround was introduced upstream by langchain-ai/langchain#10512. You can now set the prompt_key
argument to "prompt"
:
llm = Replicate(
model="a16z-infra/llama13b-v2-chat:df7690f1994d94e96ad9d568eac121aecf50684a0b0963b25a41cc40061269e5",
model_kwargs={"temperature": 0.75, "max_length": 500, "top_p": 1},
prompt_key="prompt"
)
prompt = """
User: Answer the following yes/no question by reasoning step by step. Can a dog drive a car?
Assistant:
"""
llm(prompt)
Hopefully this unblocks you. I don't personally like this extra step, so I'll try to contribute a different fix upstream to make the parameter unnecessary in most cases.
On Replicate's side, replicate_weights
will now show up last in the list of inputs, so this won't be an issue for new models created as of a few days ago.