WindowsLibsUtils hard-codes relative path to DLLs
Closed this issue · 8 comments
The WindowsLibsUtils initialization uses a hard-coded relative path to find the DLLs needed to run Hadoop/HDFS. If the hadoop-mini-clusters dependency is used in a multi-module project, I need to have the DLLs available in each project (or actually, in the parent of each project, which is complex when the module graph is a tree).
Would it be possible to make this an injectable system property the same way HBase uses HADOOP_HOME to find the location for winutils.exe?
Thanks!
@mlschechter - Sorry for the delay. This is absolutely doable. Do you want to see HADOOP_HOME reused here? or should this be its own system property, WINDOWS_HADOOP_LIBS perhaps?
Either would be a good solution; I think I'm in favor of consolidation if possible, and just re-using HADOOP_HOME. HBase assumes a bin directory underneath HADOOP_HOME, so you could assume a lib directory as well. This could also remove the need for the hdp.release.version and hadoop.home.dir properties.
The main thing is to provide a path with minimal hardcoding, and I think going for HADOOP_HOME/lib would definitely do that.
@mlschechter - Thanks for the feedback. I'll work to get this implemented and a new release cut by next week, hopefully sooner.
@mlschechter - I have made the change to allow setting HADOOP_HOME to whatever directory you'd like. If set, that value will be used. It was still necessary to set hadoop.home.dir and hdp.release.version is used to handle multiple hdp versions.
Let me know if the latest changes work for you or if you have additional suggestions and I'll work to cut a new release with this change.
Thanks!
It sounds like we're going in the right direction. Could you please let me know where I could take a quick look at the changes?
Thanks!
Looks great - thanks!
How soon can you have a release?
@mlschechter - 0.1.2 (d5eae24) has been released which contains this feature. Closing out the issue. Please open a new one if you have any issues.
Thanks again for the review and feedback!!