kids-first/kf-lib-data-ingest

Report skipped entities with warning log level

Opened this issue · 1 comments

Christina-J-Diaz commented

When values are missing from an entity and skip the loading stage, no warning sign is displayed. You can use debug mode to see which records are being skipped, but the message is misleading:

2022-11-01 12:49:00,975 - LoadStage - Thread: MainThread - DEBUG - Skip phenotype. Missing key components. Failed to construct unique key from record:
{'PARTICIPANT|ID': '28652',
 'PHENOTYPE|EVENT_AGE_DAYS': '0.6598220397',
 'PHENOTYPE|NAME': 'macroglossia',
 'PHENOTYPE|OBSERVED': 'Reported Unknown',
 'PROJECT|ID': 'SD_PREASA7S'}

The above record was skipped because PHENOTYPE.EVENT_AGE_DAYS is of type float when it should be of type int. A warning log should be added to clearly state the reason why these records skipped loading.

znatty22 commented

This

above record was skipped because PHENOTYPE.EVENT_AGE_DAYS is of type float when it should be of type int

is due to a bug that will be fixed in #649.

I do like the idea of reporting skipped entities with the warning log level tho 👍

  • 👍1