Human: / Assistant: formatting
Opened this issue · 0 comments
llm-claude currently follows the advice at https://docs.anthropic.com/claude/reference/getting-started-with-the-api#prompt-formatting to:
when using the API you must format the prompts like:
\n\nHuman: Why is the sky blue?\n\nAssistant:
For reference, these are the prompt constants from https://github.com/tomviner/anthropic-sdk-python/blob/d9b7e30b26235860fe9e7e3053171615173e32ca/src/anthropic/_constants.py#L1
HUMAN_PROMPT = "\n\nHuman:"
AI_PROMPT = "\n\nAssistant:"
However, it's allowed to include text starting the AI assistant's answer for it, which isn't currently possible with llm-claude.
On the Ask Claude for rewrites docs page, they explain how starting the AI answer with special text, can assist with formatting it's response:
Human: {prompt listing some text and requesting a rewrite according to some rules}
Please put your rewrite in <rewrite></rewrite> tags.
Assistant: <rewrite>
this starts the assistant already in rewrite mode.
Couple example of starting the AI response here, when you only want a specific format:
- https://docs.anthropic.com/claude/docs/content-moderation
- https://docs.anthropic.com/claude/docs/multiple-choice-and-classification
And another starting in <thinking>
mode: https://docs.anthropic.com/claude/docs/roleplay-dialogue#complex-customer-support-agent
Another:
Assistant: Here is a summary of the document:
from https://docs.anthropic.com/claude/docs/claude-is-hallucinating#document-summary
Claude has been trained and fine-tuned using RLHF (reinforcement learning with human feedback) methods on \n\nHuman: and \n\nAssistant: data like this, so you will need to use these prompts in the API in order to stay “on-distribution” and get the expected results. It's important to remember to have the two newlines before both Human and Assistant, as that's what it was trained on.
However, I don't think llm-claude should leave the human/assistant prefixes complete up to the end user, because their presence is actually enforced at the API level:
$ curl -s --request POST \
--url https://api.anthropic.com/v1/complete \
--header "anthropic-version: 2023-06-01" \
--header "content-type: application/json" \
--header "x-api-key: $ANTHROPIC_API_KEY" \
--data '
{
"model": "claude-instant-1",
"prompt": "\n\nHuman: 1+1",
"max_tokens_to_sample": 10
}
'
{"error":{"type":"invalid_request_error","message":"prompt must end with \"\n\nAssistant:\" turn"}}
From playing around like this, the rules appear to be:
- prompt must start with
{HUMAN_PROMPT}
, well almost. Only 1 new line needed at the start. So"\nHuman:"
- prompt may contain multiple human and AI prompt prefixes. This is how continued conversation are sent.
- prompt must include at least one
{AI_PROMPT}
that doesn't have a{HUMAN_PROMPT}
after it.
These rules are similarly listed at https://docs.anthropic.com/claude/docs/prompt-troubleshooting-checklist#the-prompt-is-formatted-correctly