dcos spark not installed
leftluoyi opened this issue · 8 comments
leftluoyi commented
I have a dcos 1.10 cluster on centos 7.3, where hdfs has been successfully installed. When I want to install the Spark through the Catalog in the Web UI, meet the same issue as this, which is solved by change the username from nobody to root. Now, the permission denied issue is gone, but I'm still unable to run Spark with Spark logs showing
I1227 12:44:50.796188 16423 exec.cpp:162] Version: 1.4.0
I1227 12:44:50.806447 16426 exec.cpp:237] Executor registered on agent 27915a1f-052e-47b7-8db1-4ee24fe1a16b-S0
I1227 12:44:50.807636 16426 executor.cpp:120] Registered docker executor on 192.168.200.116
I1227 12:44:50.807708 16426 executor.cpp:160] Starting task spark.aaf1e752-eac0-11e7-85aa-da29fd6579a8
+ export DISPATCHER_PORT=6211
+ DISPATCHER_PORT=6211
+ export DISPATCHER_UI_PORT=6212
+ DISPATCHER_UI_PORT=6212
+ export SPARK_PROXY_PORT=6213
+ SPARK_PROXY_PORT=6213
+ SCHEME=http
+ OTHER_SCHEME=https
+ [[ '' == true ]]
+ export DISPATCHER_UI_WEB_PROXY_BASE=/service/spark
+ DISPATCHER_UI_WEB_PROXY_BASE=/service/spark
+ grep -v '#https#' /etc/nginx/conf.d/spark.conf.template
+ sed s,#http#,,
+ sed -i 's,<PORT>,6213,' /etc/nginx/conf.d/spark.conf
+ sed -i 's,<DISPATCHER_URL>,http://192.168.200.116:6211,' /etc/nginx/conf.d/spark.conf
+ sed -i 's,<DISPATCHER_UI_URL>,http://192.168.200.116:6212,' /etc/nginx/conf.d/spark.conf
+ sed -i 's,<PROTOCOL>,,' /etc/nginx/conf.d/spark.conf
+ [[ '' == true ]]
+ [[ -f hdfs-site.xml ]]
+ [[ -n '' ]]
+ exec runsvdir -P /etc/service
+ mkdir -p /mnt/mesos/sandbox/nginx
+ exec
+ mkdir -p /mnt/mesos/sandbox/spark
+ exec svlogd /mnt/mesos/sandbox/spark
+ exec svlogd /mnt/mesos/sandbox/nginx
+ exec
... (repeats for about a hundred time)
+ exec
+ exec
I1227 12:48:53.038286 16427 executor.cpp:269] Received killTask for task spark.aaf1e752-eac0-11e7-85aa-da29fd6579a8
+ exec
+ exec
+ exec
+ exec
+ exec
+ exec
+ exec
I1227 12:49:03.042243 16427 executor.cpp:269] Received killTask for task spark.aaf1e752-eac0-11e7-85aa-da29fd6579a8
+ exec
+ exec
+ exec
+ exec
+ exec
W1227 12:49:03.042243 16423 logging.cpp:91] RAW: Received signal SIGTERM from process 4581 of user 0; exiting
here is the option file:
{
"id": "/spark",
"backoffFactor": 1.15,
"backoffSeconds": 1,
"cmd": "/sbin/init.sh",
"container": {
"type": "DOCKER",
"volumes": [],
"docker": {
"image": "mesosphere/spark:2.1.0-2.2.0-1-hadoop-2.6",
"forcePullImage": true,
"privileged": false,
"parameters": [
{
"key": "user",
"value": "root"
}
]
}
},
"cpus": 1,
"disk": 0,
"env": {
"DCOS_SERVICE_NAME": "spark",
"NO_BOOTSTRAP": "true",
"SPARK_DISPATCHER_MESOS_ROLE": "*",
"SPARK_USER": "root",
"SPARK_LOG_LEVEL": "INFO"
},
"healthChecks": [
{
"gracePeriodSeconds": 5,
"ignoreHttp1xx": false,
"intervalSeconds": 60,
"maxConsecutiveFailures": 3,
"portIndex": 2,
"timeoutSeconds": 10,
"delaySeconds": 15,
"protocol": "HTTP",
"path": "/"
}
],
"instances": 1,
"labels": {
"DCOS_PACKAGE_OPTIONS": "eyJzZXJ2aWNlIjp7Im5hbWUiOiJzcGFyayIsImNwdXMiOjEsIm1lbSI6MTAyNCwicm9sZSI6IioiLCJ1c2VyIjoicm9vdCIsImRvY2tlci1pbWFnZSI6Im1lc29zcGhlcmUvc3Bhcms6Mi4xLjAtMi4yLjAtMS1oYWRvb3AtMi42IiwibG9nLWxldmVsIjoiSU5GTyJ9LCJzZWN1cml0eSI6eyJrZXJiZXJvcyI6e30sInNzbCI6eyJlbmFibGVkIjpmYWxzZSwicHJvdG9jb2wiOiJUTFN2MS4yIiwiZW5hYmxlZEFsZ29yaXRobXMiOiJTU0xfREhfYW5vbl9FWFBPUlRfV0lUSF9ERVM0MF9DQkNfU0hBLFNTTF9ESF9hbm9uX1dJVEhfM0RFU19FREVfQ0JDX1NIQSxTU0xfREhfYW5vbl9XSVRIX0RFU19DQkNfU0hBLFNTTF9ESEVfRFNTX0VYUE9SVF9XSVRIX0RFUzQwX0NCQ19TSEEsU1NMX0RIRV9EU1NfV0lUSF8zREVTX0VERV9DQkNfU0hBLFNTTF9ESEVfRFNTX1dJVEhfREVTX0NCQ19TSEEsU1NMX0RIRV9SU0FfRVhQT1JUX1dJVEhfREVTNDBfQ0JDX1NIQSxTU0xfREhFX1JTQV9XSVRIXzNERVNfRURFX0NCQ19TSEEsU1NMX0RIRV9SU0FfV0lUSF9ERVNfQ0JDX1NIQSxTU0xfUlNBX0VYUE9SVF9XSVRIX0RFUzQwX0NCQ19TSEEsU1NMX1JTQV9XSVRIXzNERVNfRURFX0NCQ19TSEEsU1NMX1JTQV9XSVRIX0RFU19DQkNfU0hBLFNTTF9SU0FfV0lUSF9OVUxMX01ENSxTU0xfUlNBX1dJVEhfTlVMTF9TSEEsVExTX0RIX2Fub25fV0lUSF9BRVNfMTI4X0NCQ19TSEEsVExTX0RIX2Fub25fV0lUSF9BRVNfMTI4X0NCQ19TSEEyNTYsVExTX0RIX2Fub25fV0lUSF9BRVNfMTI4X0dDTV9TSEEyNTYsVExTX0RIX2Fub25fV0lUSF9BRVNfMjU2X0NCQ19TSEEsVExTX0RIX2Fub25fV0lUSF9BRVNfMjU2X0NCQ19TSEEyNTYsVExTX0RIX2Fub25fV0lUSF9BRVNfMjU2X0dDTV9TSEEzODQsVExTX0RIRV9EU1NfV0lUSF9BRVNfMTI4X0NCQ19TSEEsVExTX0RIRV9EU1NfV0lUSF9BRVNfMTI4X0NCQ19TSEEyNTYsVExTX0RIRV9EU1NfV0lUSF9BRVNfMTI4X0dDTV9TSEEyNTYsVExTX0RIRV9EU1NfV0lUSF9BRVNfMjU2X0NCQ19TSEEsVExTX0RIRV9EU1NfV0lUSF9BRVNfMjU2X0NCQ19TSEEyNTYsVExTX0RIRV9EU1NfV0lUSF9BRVNfMjU2X0dDTV9TSEEzODQsVExTX0RIRV9SU0FfV0lUSF9BRVNfMTI4X0NCQ19TSEEsVExTX0RIRV9SU0FfV0lUSF9BRVNfMTI4X0NCQ19TSEEyNTYsVExTX0RIRV9SU0FfV0lUSF9BRVNfMTI4X0dDTV9TSEEyNTYsVExTX0RIRV9SU0FfV0lUSF9BRVNfMjU2X0NCQ19TSEEsVExTX0RIRV9SU0FfV0lUSF9BRVNfMjU2X0NCQ19TSEEyNTYsVExTX0RIRV9SU0FfV0lUSF9BRVNfMjU2X0dDTV9TSEEzODQsVExTX0VDREhfYW5vbl9XSVRIXzNERVNfRURFX0NCQ19TSEEsVExTX0VDREhfYW5vbl9XSVRIX0FFU18xMjhfQ0JDX1NIQSxUTFNfRUNESF9hbm9uX1dJVEhfQUVTXzI1Nl9DQkNfU0hBLFRMU19FQ0RIX2Fub25fV0lUSF9OVUxMX1NIQSxUTFNfRUNESF9FQ0RTQV9XSVRIXzNERVNfRURFX0NCQ19TSEEsVExTX0VDREhfRUNEU0FfV0lUSF9BRVNfMTI4X0NCQ19TSEEsVExTX0VDREhfRUNEU0FfV0lUSF9BRVNfMTI4X0NCQ19TSEEyNTYsVExTX0VDREhfRUNEU0FfV0lUSF9BRVNfMTI4X0dDTV9TSEEyNTYsVExTX0VDREhfRUNEU0FfV0lUSF9BRVNfMjU2X0NCQ19TSEEsVExTX0VDREhfRUNEU0FfV0lUSF9BRVNfMjU2X0NCQ19TSEEzODQsVExTX0VDREhfRUNEU0FfV0lUSF9BRVNfMjU2X0dDTV9TSEEzODQsVExTX0VDREhfRUNEU0FfV0lUSF9OVUxMX1NIQSxUTFNfRUNESF9SU0FfV0lUSF8zREVTX0VERV9DQkNfU0hBLFRMU19FQ0RIX1JTQV9XSVRIX0FFU18xMjhfQ0JDX1NIQSxUTFNfRUNESF9SU0FfV0lUSF9BRVNfMTI4X0NCQ19TSEEyNTYsVExTX0VDREhfUlNBX1dJVEhfQUVTXzEyOF9HQ01fU0hBMjU2LFRMU19FQ0RIX1JTQV9XSVRIX0FFU18yNTZfQ0JDX1NIQSxUTFNfRUNESF9SU0FfV0lUSF9BRVNfMjU2X0NCQ19TSEEzODQsVExTX0VDREhfUlNBX1dJVEhfQUVTXzI1Nl9HQ01fU0hBMzg0LFRMU19FQ0RIX1JTQV9XSVRIX05VTExfU0hBLFRMU19FQ0RIRV9FQ0RTQV9XSVRIXzNERVNfRURFX0NCQ19TSEEsVExTX0VDREhFX0VDRFNBX1dJVEhfQUVTXzEyOF9DQkNfU0hBLFRMU19FQ0RIRV9FQ0RTQV9XSVRIX0FFU18xMjhfQ0JDX1NIQTI1NixUTFNfRUNESEVfRUNEU0FfV0lUSF9BRVNfMTI4X0dDTV9TSEEyNTYsVExTX0VDREhFX0VDRFNBX1dJVEhfQUVTXzI1Nl9DQkNfU0hBLFRMU19FQ0RIRV9FQ0RTQV9XSVRIX0FFU18yNTZfQ0JDX1NIQTM4NCxUTFNfRUNESEVfRUNEU0FfV0lUSF9BRVNfMjU2X0dDTV9TSEEzODQsVExTX0VDREhFX0VDRFNBX1dJVEhfTlVMTF9TSEEsVExTX0VDREhFX1JTQV9XSVRIXzNERVNfRURFX0NCQ19TSEEsVExTX0VDREhFX1JTQV9XSVRIX0FFU18xMjhfQ0JDX1NIQSxUTFNfRUNESEVfUlNBX1dJVEhfQUVTXzEyOF9DQkNfU0hBMjU2LFRMU19FQ0RIRV9SU0FfV0lUSF9BRVNfMTI4X0dDTV9TSEEyNTYsVExTX0VDREhFX1JTQV9XSVRIX0FFU18yNTZfQ0JDX1NIQSxUTFNfRUNESEVfUlNBX1dJVEhfQUVTXzI1Nl9DQkNfU0hBMzg0LFRMU19FQ0RIRV9SU0FfV0lUSF9BRVNfMjU2X0dDTV9TSEEzODQsVExTX0VDREhFX1JTQV9XSVRIX05VTExfU0hBLFRMU19FTVBUWV9SRU5FR09USUFUSU9OX0lORk9fU0NTVixUTFNfS1JCNV9FWFBPUlRfV0lUSF9ERVNfQ0JDXzQwX01ENSxUTFNfS1JCNV9FWFBPUlRfV0lUSF9ERVNfQ0JDXzQwX1NIQSxUTFNfS1JCNV9XSVRIXzNERVNfRURFX0NCQ19NRDUsVExTX0tSQjVfV0lUSF8zREVTX0VERV9DQkNfU0hBLFRMU19LUkI1X1dJVEhfREVTX0NCQ19NRDUsVExTX0tSQjVfV0lUSF9ERVNfQ0JDX1NIQSxUTFNfUlNBX1dJVEhfQUVTXzEyOF9DQkNfU0hBLFRMU19SU0FfV0lUSF9BRVNfMTI4X0NCQ19TSEEyNTYsVExTX1JTQV9XSVRIX0FFU18xMjhfR0NNX1NIQTI1NixUTFNfUlNBX1dJVEhfQUVTXzI1Nl9DQkNfU0hBLFRMU19SU0FfV0lUSF9BRVNfMjU2X0NCQ19TSEEyNTYsVExTX1JTQV9XSVRIX0FFU18yNTZfR0NNX1NIQTM4NCxUTFNfUlNBX1dJVEhfTlVMTF9TSEEyNTYifX0sImhkZnMiOnt9fQ==",
"DCOS_SERVICE_SCHEME": "http",
"DCOS_PACKAGE_SOURCE": "https://universe.mesosphere.com/repo",
"DCOS_PACKAGE_METADATA": "eyJwYWNrYWdpbmdWZXJzaW9uIjoiMy4wIiwibmFtZSI6InNwYXJrIiwidmVyc2lvbiI6IjIuMS4wLTIuMi4wLTEiLCJtYWludGFpbmVyIjoic3VwcG9ydEBtZXNvc3BoZXJlLmlvIiwiZGVzY3JpcHRpb24iOiJTcGFyayBpcyBhIGZhc3QgYW5kIGdlbmVyYWwgY2x1c3RlciBjb21wdXRpbmcgc3lzdGVtIGZvciBCaWcgRGF0YS4gIERvY3VtZW50YXRpb246IGh0dHBzOi8vZG9jcy5tZXNvc3BoZXJlLmNvbS9zZXJ2aWNlLWRvY3Mvc3BhcmsvIiwidGFncyI6WyJiaWdkYXRhIiwibWFwcmVkdWNlIiwiYmF0Y2giLCJhbmFseXRpY3MiXSwic2VsZWN0ZWQiOnRydWUsInNjbSI6Imh0dHBzOi8vZ2l0aHViLmNvbS9hcGFjaGUvc3BhcmsuZ2l0Iiwid2Vic2l0ZSI6Imh0dHBzOi8vZG9jcy5tZXNvc3BoZXJlLmNvbS9jdXJyZW50L3VzYWdlL3NlcnZpY2UtZ3VpZGVzL3NwYXJrLyIsImZyYW1ld29yayI6ZmFsc2UsInBvc3RJbnN0YWxsTm90ZXMiOiJEQy9PUyBTcGFyayBpcyBiZWluZyBpbnN0YWxsZWQhXG5cblx0RG9jdW1lbnRhdGlvbjogaHR0cHM6Ly9kb2NzLm1lc29zcGhlcmUuY29tL3NlcnZpY2UtZG9jcy9zcGFyay9cblx0SXNzdWVzOiBodHRwczovL2RvY3MubWVzb3NwaGVyZS5jb20vc3VwcG9ydC8iLCJwb3N0VW5pbnN0YWxsTm90ZXMiOiJUaGUgQXBhY2hlIFNwYXJrIERDL09TIFNlcnZpY2UgaGFzIGJlZW4gdW5pbnN0YWxsZWQgYW5kIHdpbGwgbm8gbG9uZ2VyIHJ1bi5cblBsZWFzZSBmb2xsb3cgdGhlIGluc3RydWN0aW9ucyBhdCBodHRwczovL2RvY3MubWVzb3NwaGVyZS5jb20vY3VycmVudC91c2FnZS9zZXJ2aWNlLWd1aWRlcy9zcGFyay91bmluc3RhbGwvIHRvIGNsZWFuIHVwIGFueSBwZXJzaXN0ZWQgc3RhdGUuIiwibGljZW5zZXMiOlt7Im5hbWUiOiJBcGFjaGUgTGljZW5zZSBWZXJzaW9uIDIuMCIsInVybCI6Imh0dHBzOi8vcmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbS9hcGFjaGUvc3BhcmsvbWFzdGVyL0xJQ0VOU0UifV0sImltYWdlcyI6eyJpY29uLXNtYWxsIjoiaHR0cHM6Ly9kb3dubG9hZHMubWVzb3NwaGVyZS5pby9zcGFyay9hc3NldHMvaWNvbi1zZXJ2aWNlLXNwYXJrLXNtYWxsLnBuZyIsImljb24tbWVkaXVtIjoiaHR0cHM6Ly9kb3dubG9hZHMubWVzb3NwaGVyZS5pby9zcGFyay9hc3NldHMvaWNvbi1zZXJ2aWNlLXNwYXJrLW1lZGl1bS5wbmciLCJpY29uLWxhcmdlIjoiaHR0cHM6Ly9kb3dubG9hZHMubWVzb3NwaGVyZS5pby9zcGFyay9hc3NldHMvaWNvbi1zZXJ2aWNlLXNwYXJrLWxhcmdlLnBuZyJ9fQ==",
"DCOS_SERVICE_NAME": "spark",
"DCOS_PACKAGE_FRAMEWORK_NAME": "spark",
"DCOS_SERVICE_PORT_INDEX": "2",
"DCOS_PACKAGE_DEFINITION": "eyJtZXRhZGF0YSI6eyJDb250ZW50LVR5cGUiOiJhcHBsaWNhdGlvbi92bmQuZGNvcy51bml2ZXJzZS5wYWNrYWdlK2pzb247Y2hhcnNldD11dGYtODt2ZXJzaW9uPXYzIiwiQ29udGVudC1FbmNvZGluZyI6Imd6aXAifSwiZGF0YSI6Ikg0c0lBQUFBQUFBQUFMMWJlWFBpU0xML0tucjhOZlBDQm9uRHRqcDJONTQ1eE5FZ0cxMWdkaWNjSmFrRU1xVWpKQUhHRS8zZFgxWkpZQVFDM0RNZEd4MTI2OGpLNDVkSFpWWEpmNVpDWkMzUjNQWG5CbzVpTi9CTDMwcTFNbCs2S2ZuSXczQVRoeWhhd3UxNi83cGFGc3I4YmJWY2hkOEN2SWt3d1NqRysvRTEvcWJrSWRkUDRBZEhsTVVxRElNbytUOFB4MEVjTG5DRXkyNEFBMjBjVzVFYkppbGJsUXJpM0poRG5JUGloRU8remMweGNFQ0VzOGdxVG5ERVdZRVhyaEpRbG91MzhNRGpuQ0RpbXU2Y2E2TUVsVG11SFZnckQ0Tmt5dk1idDBpU01QNVdxZGlCRlpjUHBBT2JTb3lqdFd2aFcvcXV3b3lzZ0VvSm1zZWxiLzh1bWU3Y0JwWWxha2tZWVh0bFliZzJVV0l0NEgva0k3Sk5YQ3N1L1hGVGlzRjhLOEYyNlZzU3JURGNXeDVZc3hNOWQ1UEZ5bVFTRVVDOXdLbXNNandIUmh0c3htNkNEK2lMVkxWV1VRUkdWVll4bXVPOTR2T1ZDd0IrcXU1RTRMQk5BTTc2NXE4SXVTbUIzbjAvVGhBaGNwRGdlUDg0aUpQODgxSzdWWGxTdWIwRFRFd1JkbE1hYlAvUGYzejRsL3c5YkNtSGZoeXZjSHhsYUJvcjFDQ3FxTzY3ZVZXMUJlWWVHWkNadnBudXFVUnVnYWo2Mk9kVy9sNS9Ga2tibHhET0R6Z1MrSE1JcEdqbGwvL2pQN1BBaFNBaUpOaHdDZkNrZzZLVlJVMkVRRXd1cXZvRnQreTFxSEJKQUZHTUVXZ1dna0piTHFUcEFqRnNjMENRNERJWVRHQzhIMU16Ly8zbkx2MHlXNGZwS3k1TE1xN0tVblFWa1lQUWlkQ21uSWJiQ3BTeEFqOEI3VTRpcitJaG1rcVZZYi9Wa2RWTzZRZUVzT2Y2YlN1SWxhTkVMZ25sQjVZQkVVb1c5TUdmcFhYMU1ReEhrSXlVb1lhOWtDQVd2aEI0ODM3cmNkNzN5TEwvQnRkK2NtOVZqYTN0a2JlWjJsaTlUQVRpVEhsMytMaWpremYyZE9RK3VjMTdYSk9KNWMvQ2w2cStlcWsxQmFzbWlKOTBBakhWL2wyL2wrT1h6Q2FDNktpYjc0eW1OWGd6SjJQZzllZ09hN0tMSnZVMW1qUkN1OVg0UUsxKzNISXB6YU03bXpTcVZEZThDZS94VnY2WVRXVEI4Z2lQMVVZMG13NWMrRmxiMjBaa2VYM2hwU3F1WnI0Z1ptUFRIeUtQeDhaZ09PVUZTZWRGYmNvbjZ2ak5jTWE4K0tRUXNUVTJaRWwrUy9YQXRlUkwvUHZ1NWp1dXhXdXc3YzJlRGtLN3Qxd2hzUFBGTTdabWJiUkN0WUVyVCtTMTZjMUVaeHg4NnRJYVBHbWduc2FMdWk0bzZ0aDRaSEx0M2tDWXFXQXo0TjhIWEY5cXhoWk5sY2JRbHovTVZvT1ludVNhWFdNSnZ2ait5ZXZSMVlWbVUxOG1qaTdJb3lsdlBJMzF3VWpSQzIzNXNLcWJGV0M1ZVBIZXlheVgybkNlVnlKTkRGblhoSUUwWGtxYW9yM3dCL0d4dDltcXlmSFFTd2cyWk42c0RjallrejVtUUp1TG1kUnVUZTlJcWlxSW1zNi9PNFkwTU1CK3plaUlxcUlQbWpxdlYrVVd4RXRyd09McXlIN2U4bzBQVy9vNUdTcHZ6RURHczc0MHdQK3ladkFnNjd3TThEWFI3SzY0blJuTkJkaFZBMW5MTDhub1NKb3UyQ0RuUE80UUd3MjlLMzFZTlh0dGVRcmt3RDYrOHpFcXlKb21RV3dRUXpPa0ZQOHY4Njhwd05zWXYwemxEN3NLZGx6enNkQlVOY0FIOG1CMDFyZCtFK0paV1VQY3g5ZXdVUFJHYzd4OGw1U09ORklBYTlWUStwb3h1aDZMSFNtZWdiN2dpOENjeWlrMm9NdndTaDdzY21WWGcvWjF4aEFYTDFWNWJVOGEvSlR4R0JDN2QxUVRsa3BiRTJSSDV3M1ZXSksyWW9qTk1TOC9HM3BEbi9JeXhLUWlhVVFaalBVTnEzVmdLOVYvTzV1bVB0akwwdUhaUkNHbXI0UXZrNDNiNy9GNXZ5NTNOVWQ0VmlSanBPendFRFpnQjB3Y25yUkZFK05qU3VOU1N1Nm5yWUdISnU4a3ZVOXBabDE1YlFsaURqOFR4ZzliVFF2cTdIcldNK0xaVkxhb0R5ZTFCSjQxRnViRVlQZFRXbU9yZ2dQM0gyWk4zcUN1QWZFdExzR2ZIMkJuVEdNRlRXUlY3OGhORy9BSG4vQnd2N0M3QnVnd0VLZFR3VDZKMWIxTmttRjBGaWxPdktRcnVzNXNBOTdKdENvUUMyUlkzUVdCbUhReUc5b216WUVxZWROSnN6MmVHanpvczdLNzVPMWxxcERaZ2QrLzdOTjVOaWRNQjFYUW0wQ2NiRjZtZzJnNFdZUVd6UWQvbWFROGpPMXdDclg1T0E3MmM4Tml3R29GSVZsTUdPcFVNRlR0b0Y3c2RhcEJmZkJnS3UyU0Qxb3ZzQ293K2JOcFA0SGN5ZktFNmlYK25GN3pYRzUxeGhSalFaWjBNZ044RFVkYlNyS2ludW9ET1pITXB1ZHpYZTBvWFIydzEvaEdWOVZ0eHpBRyszeEhYY1dEdVExaXBBRnhaeDNvWDVqbkJzV21vRThRNlBXRmNSM1ZvRGdyYmJVRHVPN21ZV0h3ckhYMHovcXd4MnF3aGh3L1h5TjVaVUJ0TXpweVh6RUdqcVliR3N0bGFURFFsdklBWkk0S2ROenpuMWFsTjRoUkFYTDI1Mlh3Umh2bUU3MlFQNnNQbDNpK1B5dUNDRFZnSmhYRmxka1YvU0hrUHNRRTgwSEtvMGw3S3RwcnZGMnh4YUcxRzNLTi8rd1JkalZMK1VqN3A1eHY5anpHa0V1ek5BOFhKdFdyRngvR1VmcmpRKzVOQnlDN0lCK3U1R2ovTUs1N21XNjBUdjUwcm1kOC9DYmtUMnJUNU5DbTltTU85MUVybHcrYkRFLzdvSmRkQTI0TDZEbUo1WjdZRFQwWjJXVDJkalJlSGlwRy96aStsMlpWaHI2aHorYUk0L2xvMWhYZmFFODRuSkRrWldLVG85NXQxM2V6ZHdYeHRPT2RvSW13bUZXTjNMeDRTUWJrRXI3SUo5ZVBwcmFtNy9hNUNER3o4M2VEOWtLQS8yaEZlMSs3SzRVbXJWa2U3UWtTd0EzbThpN01ZWUtZeVJsOFdjOVRMQm9naC9aSE1kTkQ3WWlhMGZyc2lYZTg3Q205UHFyanFUOW9qYVZ6SHc5Nm5NYkgvZ2Q4ZFRKMnB3UDBzcW5QV1UwN3FxazV6R3hQaXUxSlVmMHlQakljQ3NlZTRrOWo4bmkreUhnVStHcm1pZHVYcWpHMkorK3h1dmRiay9iSTVOQ3UzZGlEOVJiTTc2QnpkOUdHZWZjTjFkZzZ6andjVTdER09sazNGT0Z1ZVNJUC9sK2JhVjJEdVVhUXgzcWpVNGdmelR2UTIvVEd1OTZxK2pKNUY3SmNvLzFuM0c4Skh2eEUvZTZDdDN2Tmp5ZjNZVDNvTEo1MVlYdzNrSm93VDQrM3cyMmYyWGE4RGt2WEpBYXNYUWN4OUdhQUlkZ2tOSDdhSnVoSkFwYVg3c05SM3FkK3NHRHV0enNFWXR1b1U3cFJrYTBIL1RQanRWem9odlQ0MlIvOG5FNCs5SXRRSDV2UVU1SDFUSUxhU05lalBkWlRDa1U2UW0yanVVRHgxV1pzUFpyV1Rma3RYeXN6L3REN3dicTlac0JxektBOXdkTGFnay9hblUwUmIzTWkxY2NnUDhWZUNXSGU2cjVNU0F4K0lPbTQwUUVlVGZHMC9yNkRyVWJNYUtIMjBqbnVURC95Qy9zYWhuc3c2ODQraGg1Ym40ZXpyUUErZUQ5ZXAzLzJQTXRsUVk3ditqa0Yrcmt4NVI4VzlEOVpEOWVFdFVYU1ZLQ0hVOGlnQ2IzRUZOWjEwQTgwNExxZ2ZxVDkrMmtmY2JwbWdmVytJSDFwMytaTVQ2bkQrdHVRUk9pYkZHbkNjcmUvemZZMjNpN24xQlcrUFBSTk85MjhOSWY3OHkvRVBOZ3g5YisrLzZGMFpLZ0pvZ1k5MmtUVlpRbDBvRDNtcms4TzdKNnkrWkxjZmE5MXZKNDRxc1Bzdm4rcDVwL01mVWZ6bFVzeFpURXZaZjFHRCthMzZtSTU4OU01ZHVZUi8yaXRRbVVWeDZ5YXhmSlVEdTJ1dnNKZElmNnN1ZENubmVFN2hGcHRlVHFzUVFnUDhWSTNZZTJhWVhBMVA2YmpFSHF4Zi82ejlJT2VJY1RCS3JJdzNXTkVjWXlUbUY2dEluZS9iVTMzTTdNRGhUOUxkbUF0MHl1MnAvbTZ1eTk5YnRHbXU1M2Zjb2NWdHd0a0IwRUl0M2VsSHo5QXJPdWhPV2FpWE9CL0czdUlrTnhtL01ZbkFiSnpXNzl1a0cya3BucFcwcEhaN2k5N2svSXBoLzY4ZEpNeTlyRHRycnhmd0RsbGRNaWFvR2lPZndGbnhpZGpIRnNSeG42OENKSU1mb0RLSWk3RnlYUjlGTGtwWmh2WEIybnM4djNoN3ZhdVRxK1c4SkJ1Q0w5amE1VWdrK0NUZmVwQ0RkbmUvNkdLT2NkVmJDdUlVMFhMd0xtVWhnUDJreDZLRjJ5N0hKRjVRSSthRnFqYXVLUEhWWWlzS0N4Mzlmc0d1cnV2V1dialhyemorVWExTGphY2UvemdQSmdpcmxwMXNXN2RpN1U3c2ZZZ2lMd2pQRHpZeUVFUDJIRVFiemFRVS9yeEJ3MFU0dnFyOS8rdW9iZXB6SytiV3IrdjgzYlZNb1dIUnIxUnN4czJxdHAzTlZFVXFnMnIwV2hnMnpGdDBieC9zT3YzOXIxbzNnblZ1bDAzQmNmaExmdWhYbTlrcHRvb0FzZitsMjNOaEg3ZFdFRkFOZDZ1ZzZNYS9JTjVaL0dtYmRjRjNoSEZSaDFab3NuWFJNdXBZeVR5Z2dsREhYUXZOZ1RIZEhpTWJPZnVYbVRHc2hvQUVoMTNUcTFNdGlGbEhaaHYyS0tuYzJFVWhEaEtzbWpQMG9XVm45ekI1ZUhwV2Nwc0ZiRkRNdTZBd2MwVjd1bHhEMlh0b0JWSkRnNWU4OEswejRNdkZ5Z1NDNEJPRDdjaVBIZlpJZW5HVFJiY2lIb2h2VXdXOUZDVm5hdnVqZ2c1S3E3TWNScDlsUm1XY2pFeGg5YklKZFRKdXdNd2NPOC9xSzlld2VYLzJoMTBWZjVCZWZ5cjhtbFpuRVF1bEE4S2FiaUtjOFlJeDJhMG5uVU9mQm94WkR6WGR6MWFIdmt5ditmbXJ6d1RTanB3ODdDWFo4Wlg2NVF3ejNHRXZTRGFjcitObXI4ZnN0d1JuM0NOQWxMZ3k3OEtMK1ZXUG9IaTVzQ2QvMHVGWnRpOUlzc0tWbjd5NitTSElNOXlRMFF1SzFHZ3cydU1vZHIvRFZXOEkxVlNmdGYxb0llVXAxSTdyTVFFVVp5Sld2azBkQmxqT3VBeVd6OHdBM3RMbWFjTndTMmIzdzlTKzJETXNhbnBDSTZOb0tKc2VtSkx4ZE5qWVh1UHdBMW5SeTc5RnVLR0hTdmpuYnFRVEgzbmhuMlBnTitSRnhKOHcyMkRGZWRqZWdDZEFXbXVYSktrVUNFdURySGxPcTdGOVZoVHdtVWZXTndBZ0VscThCcG1XcGFIb0VuZ1l5NXcwak5xMXJSd3RNSitucVhUand4U0UxalZqU3JIblZDRmZ0cFF5Y0gxRTgwU3pJREIvSmJnTlNiWDRRVFMraHNIdnprMmdJRlNGRTJBMlFodGFjbGhKZ2ZzWEJ3S3ZHdHpSeXcrN1lSSDlHT1ZjbnFrWFE2aWVZWFJWb1J5dFlKQ2wzMXlRSjltWjk3cHl5RmxVbDRrSHNrQjBKZWxKNVlTYkE0Q3lNR1RXOVlYUWV5dytlMmFxYm95cEc3NXRLNlhNbUVmSkVBNC9ZYkw4L0tGSXBxVG5QYWtrRC9ROXliYjA5eElKZXplLy9XNUJvSUVpbUFRbjVQd1BYdWZsMUMrempjeUczVElLZDhtaXZGZG5jTytGZGlRRDVTd1RDazV4eVYwSG1vRm9adG1ITXUyTk1QeUNjYkY5Q1dpcVlHM25BZHhBd1VNeHpITnNvajczbTRCbTBjQ1JDeDN6UzNqbEFITHBTNjlWcE1ZK2pFNUI0dXFEam1MV2d0Sml4S2M2Z2VGZHcwMzNCSnZmeFl1N05Qa3RuTnptNE5JakUrK2pFcXpnMzVlUlA5UHgxRjFEbVNZUVVDL0xxSFJES3FvWURWT1FjODNGU2NCZk9RWmt3UW1seTIxNkVkQWlCdEFPMEN0bzBBV1RIRWdEL1NLazE4dGtURTlMeE1VZW9hZUVub1orNHE0UjU4TDJEVWlYSmdOWVJXSnhzZWg5OXkwMUY4MGRRZnRyNUM5RThUOVJ1TW9lOEs1RG5kZ0cvMGF5dzhTYmc0SjRmOStCZjFmb2RRVjJDR0FrOEFLU0Y2RU5sVFhVSDRMcDFWNHQ1dloyRFRtT0VWNUNLeXpiSGlFZmgrcTI4TExONUFsaVBiWGR1OFYrWUgvMnBrK1B5bmE2NlN2OVY3YkhiWE92N2FhclZlMTkzaHpTTVZlMStEOWE2ZmRPVTlCQ2ZJdk82OXRWYjBxSktXNkpPU0Fva2lJb2o1K1FRaWx1aXhrVDNFczVMcUF5OHd2TW1ZdlpIMDRmQjIxR3dWUEtTMzRQby8xSXpBU3FnOTdadGNJWU1sM25xYmJHbDJrZ1JlWEJSMFFYR1NTQ2FvOTFET2FBOGNXVzNTZTRGTlFBYzJKUlVjMHB4YWRKN2pJcE1DaXZmZk9XWFNPNEZEUUNVMkJSVG1hSW92T0VWeGtjbVJScDNXNUNoVFFGTmxkUUhLczhSRkpMdmJaTy9oMU5zdUtpTTdxY1o1bWg4dzVzaU12RkpFVm1uV2VKb2R6QVZtUk93N0lUbEU2WDRsT1NjNGlkQzFFaTRtSzBMa1lwQmNwY2laL0tVeUw2eVo5MWZsUzhIUytFajJYaUE0dEw2UXJRT2lFcmdpa1MwU0hFQlRTRlVEVnVSaERsK2JLQXByelNIMGxqSzdYdWdLcVlveStFa2xmcTNpZGM3RTBldFplWHBXTzNPaythZjFIcmY4a3Y5S2w3NnZhVWcxRzhWMXBObzU3QmFZSE5BeDBmcjlHczVQRWFFNWRrR054M2tPZnIzZXNUd2NlOWlMMHpVVi9YdlBrTlI5ZTlONjFlZXFheDA1OGxXNXdIelhyOUU5ZVBIUWJZMWlDb2dqV1N3U1d0SFRMQWRHL1dZQjd5NlU3T0hGQlQ1MXVieTlzNTl4Qy83YlhsdFMvdm8yUUR2emFQZ2xkOUhGMHM0UXQ0ZGx5UE9ZT3hMT05nSmo3aldwN1MvOGtwdnp1a1J0NEdlSDk3ZTlzajQydGgyS2MzSER0MW43Ym5lMFZ3c0tmYzVPWXl4WU5SNGJSUlZhRUVTeDFvc0JMeDFMNTVkTCtHQUJ3OXRuaUtYUkQramRBZitkb3czVDkxMDJFUW9EcmxpL3paZUUyM0ZiTDRiWjJDNjBDdmtYK3RyeFprTklmUDM3OFA0alNYcmFDTlFBQSJ9",
"DCOS_PACKAGE_VERSION": "2.1.0-2.2.0-1",
"DCOS_PACKAGE_NAME": "spark"
},
"maxLaunchDelaySeconds": 3600,
"mem": 1024,
"gpus": 0,
"networks": [
{
"mode": "host"
}
],
"portDefinitions": [
{
"protocol": "tcp",
"port": 10001
},
{
"protocol": "tcp",
"port": 10002
},
{
"protocol": "tcp",
"port": 10003
}
],
"requirePorts": false,
"upgradeStrategy": {
"maximumOverCapacity": 1,
"minimumHealthCapacity": 1
},
"user": "root",
"killSelection": "YOUNGEST_FIRST",
"unreachableStrategy": {
"inactiveAfterSeconds": 0,
"expungeAfterSeconds": 0
},
"fetch": [],
"constraints": []
}
The Spark service is also showing as Unhealth in the web UI. When checking from Marathon, it says Task was killed since health check failed. Reason: 502 Bad Gateway.
Any idea how to fix it?
dprslt commented
Same problem here.
DCOS : 1.10.1
CoreOs : 1576.4.0
dprslt commented
In my case this issue seems to be due to a DNS problem.
While booting, spark is trying to solve the hostname of the host, and if this entry is neither in the /etc/hosts file or in the DNS, the startup will fail.
You should be able to check it by connecting to the node hosting the spark docker container via ssh.
Then I've openened a shell into the docker :
- Get the dockerr id from :
docker ps docker exec -it <container_id> /bin/bash)
And try yo launch./bin/spark-shell
After a while, I got an error saying he was unable to resolve the hostname.
Any node or master in your cluster should be able to run correctly ping $(hostname) for get the thing work.
When installing, DCOS crush the existing configuration existing in /etc/resolve.conf on the nodes and masters.
We're deploying on AWS, and we're able to solve the issues by adding the Amazon DNS address in the dcos config file :
...
resolvers:
- 192.168.0.2
- 8.8.8.8
- 8.8.4.4
...
hantuzun commented
@dprslt's fix works for me as well. Thanks!
Which project is responsible for this issue though? DC/OS?
dprslt commented
In fact, i don't thing is really an "issue". It's a configuration issue because the dns or the hosts file is unable to resolve the hostname. The issue is mainly a lack of logging in the DC/OS web view.
grohan2002 commented
hantuzun commented
@grohan2002, if you could modify your /etc/hosts it would be enough for an workaround.
If $HOSTNAME is set, here's my script for 1. removing the line from /etc/hosts if it exists, and 2. adding a $IP $HOSTNAME line to /etc/hosts:
sudo sed -i '/'$HOSTNAME'/d' /etc/hosts
export IP=$(hostname -I | awk '{print $1}')
echo $IP' '$HOSTNAME | sudo tee -a /etc/hostsTry it first.
grohan2002 commented
@hantuzun Thanks a lot for the workaround , however i did input the Amazon DNS in the config.yaml file and rebuilt the DC/OS stack .
Thank you again for the workaround.
hantuzun commented
@grohan2002, building DC/OS takes a long time and could fail randomly according to my experience. I'm more cautious on touching DC/OS now.