AbsaOSS/spark-metadata-tool

fixPath breaks idempotence

Closed this issue · 0 comments

Equality check in metadata tool causes it to fail when run on files with already fixed paths. This is not intended behaviour as it
makes reruns after partial failure impossible.

The check was originally added as validation for cases in which the top level partition key is not found in the old path, signifying corrupted path. If the validation can't be kept without the tool failing on already fixed paths, it should be removed.