GoogleCloudPlatform/oozie-to-airflow

Support for Oozie lib .so files

potiuk opened this issue · 0 comments

Is your feature request related to a problem? Please describe.
Ooozie adds all the .so files which are in the lib folder of the workflow application to the LD_LIBRARY_PATH automatically. This is currently only implemented in Java Mapper.

Describe the solution you'd like
All the mappers should have .jar files added to the classpath.

Describe alternatives you've considered
No alternatives.

Additional context
Each Oozie operator has its own way of adding .so files to LD_LIBRARY_PATH so it should be done in mapper-specific way. Currently in workflow.py the lib folder is scanned for all .jar files and they are stored in Workflow.

For all the Map-Reduce jobs that are not executed as single-node mapper (on master), the .so files should be added to Local Cache.