Investigate Microsoft Guidance

[Dodorotata]

https://github.com/guidance-ai/guidance/blob/main/notebooks/guaranteeing_valid_syntax.ipynb
https://github.com/guidance-ai/guidance?tab=readme-ov-file#guidance-acceleration

Structured data -> json works fairly well

[Dodorotata]

---

CONCLUSIONS:

• Llama-index guidance pydantic program dicarded due to LLM not outputting correct json schema even with mixtral. # -> OutputParserException: Failed to parse pydantic object from guidance program. Probably the LLM failed to produce data with right json schema
• Extract data to json works well for select from a list of choices and generation funcion with llama-cpp-python==0.2.26
• Content of the generated output from longer text or scientific article good quality. Not all instructions in prompt are followed. Output can possibly be improved with prompting: propmpt engineering, chain-of-thought, few-shot examples.
• Extraction from full article might be too slow
• Guidance documentation not great

---