ReplicateError input: prompt is required

[kzaho]

HI!
I am exploring the capabilities of Llama 2 70b and found a bug.
I wrote the temp solution, but I think many people are extremely curious about llama-2-70b,
so it would be great to get a working fix (no temp one).
replicate==0.11.0
langchain==0.0.250
python 3.10

I tried to run that code and it worked.
But, with:

model="meta/llama-2-70b:8570a72c77eeae584bbeded5b0928704a44246ba77728bd5223fbc3d6ef2888e"

I got "ReplicateError input: prompt is required".

So, I debugged and found that in the file:
/usr/local/anaconda3/envs/llm/lib/python3.10/site-packages/replicate/prediction.py
at class "PredictionCollection(Collection)", inside the method "create" we have such part:

resp = self._client._request(
            "POST",
            "/v1/predictions",
            json=```
,
        )

where replicate_weights value (inside input, which is inside body) is replaced with promt.
So, after I added that code before "resp = self._client._request" - the error is gone:

if 'prompt' not in body['input']:
    body['input']['prompt'] = body['input']['replicate_weights']
    del body['input']['replicate_weights']

[mattt]

Hi @kzaho. Thanks for the detailed write-up. This is a problem in LangChain's integration with Replicate, rather than the Python library itself (even though you can work around this issue by patching the client library).

The bug comes from the following lines, where the prompt field is derived by selecting the first ordered field in the schema rather than, say, a field named "prompt":

https://github.com/langchain-ai/langchain/blob/01e9d7902d4bdd0de88bc8d63c96d8d8894b8c4c/libs/langchain/langchain/llms/replicate.py#L118-L125

Working on a solution for that now.

[kzaho]

@mattt, thanks for a quick response!
I will follow up on updates in integration)

[mattt]

@kzaho A workaround was introduced upstream by langchain-ai/langchain#10512. You can now set the prompt_key argument to "prompt":

llm = Replicate(
    model="a16z-infra/llama13b-v2-chat:df7690f1994d94e96ad9d568eac121aecf50684a0b0963b25a41cc40061269e5",
    model_kwargs={"temperature": 0.75, "max_length": 500, "top_p": 1},
    prompt_key="prompt"
)
prompt = """
User: Answer the following yes/no question by reasoning step by step. Can a dog drive a car?
Assistant:
"""
llm(prompt)

Hopefully this unblocks you. I don't personally like this extra step, so I'll try to contribute a different fix upstream to make the parameter unnecessary in most cases.

On Replicate's side, replicate_weights will now show up last in the list of inputs, so this won't be an issue for new models created as of a few days ago.

[kzaho]

@mattt Thanks so much for the update!
I am closing the issue because it is solved for now)
Thank you for contributing; I hope you have time to do what you mentioned.