Yelp/mrjob

Support --conf-path when using mrjob spark-submit

mj3c opened this issue · 0 comments

mj3c commented

Am I doing something wrong or is the --conf-path / -c option not supported when trying to run a PySpark script using mrjob spark-submit? For example, I have an EMR runner configuration specified in the custom.conf file, and I try to run the script with:

mrjob spark-submit -r emr -c custom.conf main.py

In the logs, it clearly says:

Looking for configs in /home/<my_home>/.mrjob.conf
Looking for configs in /etc/mrjob.conf
No configs found; falling back on auto-configuration

Why does it only check those two locations, and not consider the passed argument?