patrickfleith/astro-llms-notebooks

Collection of notebooks showcasing usage of LLMs for Space Engineering

Notebooks for LLMs used in Space Mission Engineering

A collection of notebooks showcasing the usage of Large Language Models in the domain of Space, Space Engineering, Space Mission Design, Space operations and Astronautics at large.

1. Evaluations of LLMs

1.1 Evaluating an open-source Mistral model on a benchmark task dataset (AstroMCQA)

• Formatting of an existing dataset hosted on huggingface to feed an LLM for evaluation
• Load an huggingface LLM and collect its outputs
• Evaluating the LLM based on multiple scoring strategies
  • string character-based distance
  • embedding-based distance
  • f1 and exact match
  • LLM-as-a-Judge