IBM/unitxt

More meaningful error message when catalog consistency fails

Closed this issue · 1 comments

Today, the when the catalog consistency fails one gets this error message.

Comparing generated and old catalog...
***** Directories has differences ******
{'file': '/home/runner/work/unitxt/unitxt/src/unitxt/catalog/metrics/string_containment', 'diff': 'old only'}
  File "/home/runner/work/unitxt/unitxt/utils/prepare_all_artifacts.py", line 165, in <module>
    main()
  File "/home/runner/work/unitxt/unitxt/utils/prepare_all_artifacts.py", line 160, in main
    raise RuntimeError("Directories has differences")
RuntimeError: Directories has differences

It should return a more meaningful error such as.

Catalog consistency check failed.  The catalog json files were regenerated from the prepare/*/*.py files.  

* Some json files (or complete directories) which were committed in the past were not regenerated:

 home/runner/work/unitxt/unitxt/src/unitxt/catalog/metrics/string_containment
 
This can happen when renaming to catalog names of artifacts in  prepare/*/*.py files , but not deleting the old files.

* Some files which were generated were not committed.

 home/runner/work/unitxt/unitxt/src/unitxt/catalog/metrics/new_string_containment.json
 
 This can happen when not committing json files together with prepare/*/*.py.
 
 
 To fix recreate and fix this in your local environment:

 rm -fr  src/unitxt/catalog
 python utils/prepare_all_artifacts.py
 
This will cause all artifacts to be regenerated and you can use git status to see the changes and then commit them.

Resolved with PR #1201