Can the same jar file be used by multiple spark submit commands?

Question

Can the same jar file be used by multiple spark submit commands?

Closed this issue 5 years ago · 3 comments

Hello,

Is it possible to run multiple spark submit commands simultaneously using the same metorikku.jar file?
I tried running 2 spark submit commands, pointing to the jar file, on an emr cluster but only the first one was syncing.

Thank you

Answer 1 · 2020-02-19T14:01:22.000Z

It really depends on your cluster configuration, for example, if you're running using yarn and your cluster has dynamicallocation on then you can run multiple jobs under the same cluster.
So it's not a metorikku feature, but a spark feature.

Answer 2 · 2020-02-19T14:02:23.000Z

If you're using EMR, you need to make sure the job is not getting all the resources (executor/driver.memory)

Answer 3 · 2020-02-19T21:19:40.000Z

Thank you, lyogev.
You're right, the resources were being consumed by the first spark job.