how to run it on yarn

Question

how to run it on yarn

tonywang-sh opened this issue 2 years ago · 12 comments

I didn't find submarine usage guide for running on yarn. Whether its ver 0.7 does support to run task on yarn? Any guide documents? Thanks.

Answer 1 · 2022-07-20T05:32:28.000Z

@tonywang-sh
Hi~ You can try 0.6.0. Here is the document https://submarine.apache.org/docs/0.6.0/adminDocs/yarn/

Answer 2 · 2022-07-21T08:57:31.000Z

From v0.6.0 yarn document, there is the cmd line, "java org.apache.submarine.client.cli.Cli job run --name tf-job-001 \ ....". Where is the 'java org.apache.submarine.client.cli.Cli' implementation? I didn't find its package with Cli interface. Thanks.

Answer 3 · 2022-07-22T00:28:27.000Z

@tonywang-sh
It is in submarine-client, and you need to change tag to release-0.6.0

Answer 4 · 2022-07-22T00:44:11.000Z

Does release v0.7 or later version support to run on yarn?

Answer 5 · 2022-07-22T03:13:19.000Z

In addition, Does release-0.6.0 support hadoop 3.2.2? It reported errors when compiling the source codes of release-0.6.0 as the below.

Answer 6 · 2022-07-22T08:13:32.000Z

@tonywang-sh
Can you show your build command and mvn/java version?
There is a profile hadoop-3.2 in pom.xml, hadoop 3.2 should be compiled if it is configured correctly.

Answer 7 · 2022-07-22T10:37:00.000Z

release-0.6.0 showed the errors with hadoop 3.2 as the below.

Answer 8 · 2022-07-23T02:16:46.000Z

@tonywang-sh
Can you show your build command and mvn/java version?
I run mvn clean install -DskipTests -Phadoop-3.2 and build succeeded.

Answer 9 · 2022-07-23T10:32:52.000Z

I tried the cmd line, mvn clean install -DskipTests -Phadoop-3.2.2, but it failed. Another issue, release-0.7.0 or later versions don't have java code about yarn, how to run release-0.7.0 or later version on yarn?

Answer 10 · 2022-07-23T13:15:36.000Z

@tonywang-sh
You can not use hadoop-3.2.2, because there is no hadoop-3.2.2 profile in pom.xml. If you want to use hadoop 3.2.2, I think you need to change hadoop.version to 3.2.2.
From 0.7.0, we only support k8s, so that you may use 0.6.0 for yarn as latest version.

Answer 11 · 2022-07-25T06:56:38.000Z

Thanks for your reply. because submarine v0.6.0 on yarn needs TonY framework, how to set application conf for TonY to meet kerberos verification of hadoop? I didn't find TonY conf setting from the website, https://submarine.apache.org/docs/0.6.0/userDocs/yarn/YARNRuntimeGuide/.

Answer 12 · 2022-07-25T12:40:11.000Z

@tonywang-sh
I have not tried kerberos, but in my experience, if you have logged into kerberos, hadoop will recognize kerberos authentication information by default.
If you encounter problems when trying kerberos you can leave a comment here and I or someone else may be able to help you out.