os-chat allows you to chat with your computer, retrieving all kind of information from log files, CLI commands and an osquery interface.
Setup the project with
pdm installSetup OpenAI API key:
export OPENAI_API_KEY=sk-***pdm run startThe goal of the benchmark is to measure the ability of different LLMs to retriev the relevant information from the available interfaces.
| Model Name | Model Type | Retrieval Accuracy |
|---|---|---|
| gpt-3.5-turbo | API | 1.0 |
| nous-hermes2 | local | 0.2 |
| llama2 7b | local | 0.0 |
| llama2 7b-chat | local | 0.0 |
| openhermes | local | 0.0 |
Currently local LLMs are not able to use the interface correctly and therefore retriev no information.
To run the benchmark use the following pdm script:
pdm run benchmarkNote: Currently the benchmark targets are hardcoded for one specific test system and need to be adapted to run on any other system.
