AbsaOSS/spark-metadata-tool

HDFS Support

Closed this issue · 0 comments

Implement IO operations on HDFS so the tool can operate on it.

I suggest using FileSystem class from the Hadoop library which is already included in the project, as it provides most of the required operations. Need to make sure it can load Hadoop config without issues(it might also be a good idea to allow user to provide config explicitly)