Multilingual Analysis of LLMs for Clinical Observations
Built using the PhEval runner template (see instructions below).
Let us start by documenting how to run the current version in a new folder. This has to be changed!
poetry install
poetry shell
mkdir myinputdirectory
mkdir myoutputdirectory
cp -r /path/to/promptdir myinputdirectory/
cp inputdir/config.yaml myinputdirectory
pheval run -i myinputdirectory -r "malcorunner" -o myoutputdirectory -t tests
This serves as a template repository designed for crafting a personalised PhEval runner. Presently, the runner executes a mock predictor found in src/pheval_template/run/fake_predictor.py
. Nevertheless, the primary objective is to leverage this repository as a starting point to develop your own runner for your tool, allowing you to customise and override existing methods effortlessly, given that it already encompasses all the necessary setup for integration with PhEval. There are exemplary methods throughout the runner to provide an idea on how things could be implemented.
git clone https://github.com/yaseminbridges/pheval.template.git
cd pheval.template
poetry install
poetry shell
A config.yaml
should be located in the input directory and formatted like so:
tool: template
tool_version: 1.0.0
variant_analysis: False
gene_analysis: True
disease_analysis: False
tool_specific_configuration_options:
The testdata directory should include the subdirectory named phenopackets
- which should contain phenopackets.
pheval run --input-dir /path/to/input_dir \
--runner templatephevalrunner \
--output-dir /path/to/output_dir \
--testdata-dir /path/to/testdata_dir
You can benchmark the run with the pheval-utils benchmark
command:
pheval-utils benchmark --directory /path/to/output_directoy \
--phenopacket-dir /path/to/phenopacket_dir \
--output-prefix OUTPUT_PREFIX \
--gene-analysis \
--plot-type bar_cumulative
The path provided to the --directory
parameter should be the same as the one provided to the --output-dir
in the pheval run
command
If overriding this template to create your own runner implementation. There are key files that should change to fit with your runner implementation.
- The name of the Runner class in
src/pheval_template/runner.py
should be changed. - Once the name of the Runner class has been customised, line 15 in
pyproject.toml
should also be changed to match the class name, then runpoetry lock
andpoetry install
The runner you give on the CLI will then change to the name of the runner class.
You should also remove the src/pheval_template/run/fake_predictor.py
and implement the running of your own tool. Methods in the post-processing can also be altered to process your own tools output.