guidance-ai/guidance

[Phi3] System role missing

Closed this issue · 1 comments

The bug

Trying to use with system(): with lm set to a Phi3 model, I got an UnsupportedRoleException. We have copied the template supplied by Phi3:

# [05/08/24] https://huggingface.co/microsoft/Phi-3-mini-4k-instruct/blob/main/tokenizer_config.json#L119

This indeed does not mention system. However:
https://github.com/microsoft/Phi-3CookBook/blob/main/md/02.QuickStart/Huggingface_QuickStart.md#2-learn-about-phi-3-prompt-template

To Reproduce

See above

System info (please complete the following information):

  • OS (e.g. Ubuntu, Windows 11, Mac OS, etc.): Windows, but this will be universal
  • Guidance Version (guidance.__version__): Working from synced fork

The phi-3 cookbook is out of date, they initially started with a system role but removed it later because the model wasn't trained for it. Latest official recommendation is to not use system prompts.