Installation Script Action Failure

Question

Installation Script Action Failure

Closed this issue 8 years ago · 15 comments

I've tested this on both HDInsight 3.5 and 3.6. The execution fails on both head nodes with this error:

slider package --install --name presto1 --package build/presto-yarn-package.zip --replacepkg
2017-05-04 01:54:27,144 [main] INFO  service.AbstractService - Service org.apache.hadoop.yarn.client.api.impl.YarnClientImpl failed in state INITED; cause: java.net.BindException: Invalid yarn.resourcemanager.address value:0.0.0.0:8032 - see https://wiki.apache.org/hadoop/UnsetHostnameOrPort
java.net.BindException: Invalid yarn.resourcemanager.address value:0.0.0.0:8032 - see https://wiki.apache.org/hadoop/UnsetHostnameOrPort
	at org.apache.slider.client.SliderYarnClientImpl.serviceInit(SliderYarnClientImpl.java:81)
	at org.apache.hadoop.service.AbstractService.init(AbstractService.java:163)
	at org.apache.slider.client.SliderClient.initHadoopBinding(SliderClient.java:490)
	at org.apache.slider.client.SliderClient.serviceInit(SliderClient.java:318)
	at org.apache.hadoop.service.AbstractService.init(AbstractService.java:163)
	at org.apache.slider.core.main.ServiceLauncher.launchService(ServiceLauncher.java:182)
	at org.apache.slider.core.main.ServiceLauncher.launchServiceRobustly(ServiceLauncher.java:475)
	at org.apache.slider.core.main.ServiceLauncher.launchServiceAndExit(ServiceLauncher.java:403)
	at org.apache.slider.core.main.ServiceLauncher.serviceMain(ServiceLauncher.java:630)
	at org.apache.slider.Slider.main(Slider.java:49)
2017-05-04 01:54:27,148 [main] INFO  service.AbstractService - Service Slider Client failed in state INITED; cause: org.apache.hadoop.service.ServiceStateException: java.net.BindException: Invalid yarn.resourcemanager.address value:0.0.0.0:8032 - see https://wiki.apache.org/hadoop/UnsetHostnameOrPort
org.apache.hadoop.service.ServiceStateException: java.net.BindException: Invalid yarn.resourcemanager.address value:0.0.0.0:8032 - see https://wiki.apache.org/hadoop/UnsetHostnameOrPort
	at org.apache.hadoop.service.ServiceStateException.convert(ServiceStateException.java:59)
	at org.apache.hadoop.service.AbstractService.init(AbstractService.java:172)
	at org.apache.slider.client.SliderClient.initHadoopBinding(SliderClient.java:490)
	at org.apache.slider.client.SliderClient.serviceInit(SliderClient.java:318)
	at org.apache.hadoop.service.AbstractService.init(AbstractService.java:163)
	at org.apache.slider.core.main.ServiceLauncher.launchService(ServiceLauncher.java:182)
	at org.apache.slider.core.main.ServiceLauncher.launchServiceRobustly(ServiceLauncher.java:475)
	at org.apache.slider.core.main.ServiceLauncher.launchServiceAndExit(ServiceLauncher.java:403)
	at org.apache.slider.core.main.ServiceLauncher.serviceMain(ServiceLauncher.java:630)
	at org.apache.slider.Slider.main(Slider.java:49)
Caused by: java.net.BindException: Invalid yarn.resourcemanager.address value:0.0.0.0:8032 - see https://wiki.apache.org/hadoop/UnsetHostnameOrPort
	at org.apache.slider.client.SliderYarnClientImpl.serviceInit(SliderYarnClientImpl.java:81)
	at org.apache.hadoop.service.AbstractService.init(AbstractService.java:163)
	... 8 more
Exception: java.net.BindException: Invalid yarn.resourcemanager.address value:0.0.0.0:8032 - see https://wiki.apache.org/hadoop/UnsetHostnameOrPort
2017-05-04 01:54:27,149 [main] ERROR main.ServiceLauncher - Exception: java.net.BindException: Invalid yarn.resourcemanager.address value:0.0.0.0:8032 - see https://wiki.apache.org/hadoop/UnsetHostnameOrPort
org.apache.hadoop.service.ServiceStateException: java.net.BindException: Invalid yarn.resourcemanager.address value:0.0.0.0:8032 - see https://wiki.apache.org/hadoop/UnsetHostnameOrPort
	at org.apache.hadoop.service.ServiceStateException.convert(ServiceStateException.java:59)
	at org.apache.hadoop.service.AbstractService.init(AbstractService.java:172)
	at org.apache.slider.client.SliderClient.initHadoopBinding(SliderClient.java:490)
	at org.apache.slider.client.SliderClient.serviceInit(SliderClient.java:318)
	at org.apache.hadoop.service.AbstractService.init(AbstractService.java:163)
	at org.apache.slider.core.main.ServiceLauncher.launchService(ServiceLauncher.java:182)
	at org.apache.slider.core.main.ServiceLauncher.launchServiceRobustly(ServiceLauncher.java:475)
	at org.apache.slider.core.main.ServiceLauncher.launchServiceAndExit(ServiceLauncher.java:403)
	at org.apache.slider.core.main.ServiceLauncher.serviceMain(ServiceLauncher.java:630)
	at org.apache.slider.Slider.main(Slider.java:49)
Caused by: java.net.BindException: Invalid yarn.resourcemanager.address value:0.0.0.0:8032 - see https://wiki.apache.org/hadoop/UnsetHostnameOrPort
	at org.apache.slider.client.SliderYarnClientImpl.serviceInit(SliderYarnClientImpl.java:81)
	at org.apache.hadoop.service.AbstractService.init(AbstractService.java:163)
	... 8 more
2017-05-04 01:54:27,150 [main] INFO  util.ExitUtil - Exiting with status 56
Traceback (most recent call last):
  File "/var/lib/ambari-agent/cache/custom_actions/scripts/run_customscriptaction.py", line 194, in <module>
    ExecuteScriptAction().execute()
  File "/usr/lib/python2.6/site-packages/resource_management/libraries/script/script.py", line 306, in execute
    method(env)
  File "/var/lib/ambari-agent/cache/custom_actions/scripts/run_customscriptaction.py", line 179, in actionexecute
    ExecuteScriptAction.execute_bash_script(bash_script, scriptpath, scriptparams)
  File "/var/lib/ambari-agent/cache/custom_actions/scripts/run_customscriptaction.py", line 149, in execute_bash_script
    raise Exception("Execution of custom script failed with exit code",exitcode)
Exception: ('Execution of custom script failed with exit code', 56)

As best as I can tell, yarn.resourcemanager.address is not set on HDInsight and the installation is using a default. Instead yarn.resourcemanager.address.rm1 and yarn.resourcemanager.address.rm2 are set on HDInsight for HA mode.

Would configuring yarn.resourcemanager.address to the address of rm1 fix the issue, or is there something more comprehensive with the installation process that should be done?

Answer 1 · 2017-05-04T02:53:03.000Z

I think you are running this on Spark cluster. Can you try it on a hadoop cluster ?

Answer 2 · 2017-05-04T22:56:23.000Z

You were correct. This error came from running the installation action on a Spark cluster.

Answer 3 · 2017-07-05T20:50:15.000Z

Hi,

I'm getting the same error, also with Spark on HDInsights. Is it not possible to somehow change the default port during presto installation? Readme suggests that for any customizations there needs to be existing presto cluster running.

Can we maybe add some environment variable before we run the script?

Answer 4 · 2017-07-05T21:57:50.000Z

The script is not supported on Spark cluster. You will need to run this on a hadoop cluster.
You can change the default port by changing site.global.presto_server_port in createconfigs.sh. Does customizing after installation (as described in readme) not work for you ? What is the scenario you are trying to achieve ?

Answer 5 · 2017-07-06T06:44:45.000Z

Why is this? Spark on HDInsights runs on yarn, not standalone. Only difference that I see is the port on which yarn is listening (8050 insted of 8032). Are there other issues when Spark and Presto are running side by side?
I think I mislead you with the port question. I was asking if it's possible to give the script a different port for yarn. When spark is installed, yarn is on 8050.
I understood from the readme that presto should already be installed on the cluster prior to customization. Since installation failed, I did not even try.

Answer 6 · 2017-07-06T22:02:18.000Z

The limitation of Presto running on Spark cluster is not due to the port. This presto installation is managed via Apache slider. We don't ship slider on Spark cluster. That's why you need a Hadoop cluster.

Answer 7 · 2017-07-07T05:37:07.000Z

Hmm, now I'm a bit confused. If I connect to my cluster over SSH and run slider version this is what i get:

sshuser@hn0-cigspa:~$ slider version
2017-07-07 05:20:24,119 [main] INFO client.SliderClient - Slider Core-0.92.0.2.6.0.10-29 Built against commit# 790c3934c8 on Java 1.7.0_21 by jenkins
2017-07-07 05:20:24,121 [main] INFO client.SliderClient - Compiled against Hadoop 2.7.3.2.6.0.10-29
2017-07-07 05:20:24,128 [main] INFO client.SliderClient - Hadoop runtime version (HEAD detached at aaf0730) with source checksum 2e8e51d932beeff9cb25224a3a758e2 and build date 2017-05-15T18:52Z

Same thing on one of the nodes.

Answer 8 · 2017-07-07T20:29:40.000Z

On Spark cluster only the slider client binary is installed, but Slider is not configured properly. That is why you see "Invalid yarn.resourcemanager.address value:0.0.0.0:8032".

Answer 9 · 2017-07-07T20:49:27.000Z

Ok. Thank you for the info. Is Presto support maybe coming in the future? I find it very useful to have both spark and presto on the same cluster. If for no other reason then for Dev/Test. From: Dharmesh Kakadia Sent: Friday, July 7, 22:30 Subject: Re: [dharmeshkakadia/presto-hdinsight] Installation Script Action Failure (#7) To: dharmeshkakadia/presto-hdinsight Cc: Ivan Grbavac, Comment On Spark cluster only the slider client binary is installed, but Slider is not configured properly. That is why you see "Invalid yarn.resourcemanager.address value:0.0.0.0:8032". — You are receiving this because you commented. Reply to this email directly, view it on GitHub<#7 (comment)>, or mute the thread<https://github.com/notifications/unsubscribe-auth/ABPaZCAtLwEO9uFs3QMGh6-__KU6Zmkwks5sLpU1gaJpZM4NQKWu>.

Answer 10 · 2017-07-07T20:55:36.000Z

Spark and presto running on a same cluster will also compete for resources, since both require significant amount of memory. I highly recommend using separate presto and spark cluster.

Answer 11 · 2017-07-07T21:04:26.000Z

I'm aware of that and I definitely agree that separate clusters is the way to go for production workload, but when testing/developing, it's both easier and cheaper to run it on one cluster.

…

________________________________ From: Dharmesh Kakadia <notifications@github.com> Sent: Friday, July 7, 2017 10:55:37 PM To: dharmeshkakadia/presto-hdinsight Cc: Ivan Grbavac; Comment Subject: Re: [dharmeshkakadia/presto-hdinsight] Installation Script Action Failure (#7) Spark and presto running on a same cluster will also compete for resources, since both require significant amount of memory. I highly recommend using separate presto and spark cluster. — You are receiving this because you commented. Reply to this email directly, view it on GitHub<#7 (comment)>, or mute the thread<https://github.com/notifications/unsubscribe-auth/ABPaZL_xzBhvf3bvrmePDXrQa9gKlHjiks5sLptJgaJpZM4NQKWu>.

Answer 12 · 2017-07-07T21:50:26.000Z

I am afraid you have to configure presto yourself for that scenario.

Answer 13 · 2017-07-08T06:39:14.000Z

Ok. Tnx for you help.

…

________________________________ From: Dharmesh Kakadia <notifications@github.com> Sent: Friday, July 7, 2017 11:50:27 PM To: dharmeshkakadia/presto-hdinsight Cc: Ivan Grbavac; Comment Subject: Re: [dharmeshkakadia/presto-hdinsight] Installation Script Action Failure (#7) I am afraid you have to configure presto yourself for that scenario. — You are receiving this because you commented. Reply to this email directly, view it on GitHub<#7 (comment)>, or mute the thread<https://github.com/notifications/unsubscribe-auth/ABPaZGNkxA7JdGtFePp_kS4uaJtEY1WTks5sLqgigaJpZM4NQKWu>.

Answer 14 · 2017-07-09T18:26:53.000Z

@grbinho, if you need a less expensive presto development environment, another option to consider is just running presto standalone on a vm and avoid a second cluster + hdinsight markup.

Answer 15 · 2017-07-09T19:01:43.000Z

@grbinho Also, if you want a local docker based environment, consider using https://github.com/arsenvlad/docker-presto-adls-wasb which has support for Azure storage support.
cc @arsenvlad