Exception: Java gateway process exited before sending the driver its port number

Question

Exception: Java gateway process exited before sending the driver its port number

Closed this issue 9 years ago · 53 comments

wlsherica commented 9 years ago

I got this Spark connection issue, and SparkContext didn't work for sc.

The command to initialize ipython notebook:

ipython notebook --profile=pyspark

Environment:
Mac OS
Python 2.7.10
Spark 1.4.1
java version "1.8.0_65"

Answer 1 · 2015-12-09T14:42:17.000Z

any solution to this. experiencing same thing.

Answer 2 · 2015-12-14T13:22:49.000Z

How did you install Spark? You may need to set SPARK_HOME for it to find it properly.

Answer 3 · 2015-12-15T00:39:54.000Z

@minrk I have set SPARK_HOME already. I downloaded spark-1.4.1-bin-hadoop2.6.tgz from official website then tar it.

Answer 4 · 2015-12-15T09:32:29.000Z

Do you have any other spark env variables defined, such as PYSPARK_SUBMIT_ARGS?

Answer 5 · 2015-12-16T01:54:59.000Z

@minrk Yes, I followed the step here, https://gist.github.com/ololobus/4c221a0891775eaa86b0

Answer 6 · 2015-12-16T12:09:48.000Z

Try removing the PYSPARK_SUBMIT_ARGS env.

Answer 7 · 2016-02-03T23:52:39.000Z

I was having the same problem with spark 1.6.0 but removing PYSPARK_SUBMIT_ARGS env from my bash solved the problem. In my bashrc i have set only SPARK_HOME and PYTHONPATH and launching the jupyter notebook I am using the default profile not the pyspark profile.

Answer 8 · 2016-02-04T01:57:07.000Z

@Hanuman26 Thanks for passing along details of your success for others. cc/ @wlsherica @minrk

I'm going to mark this closed but feel free to reopen if needed.

Answer 9 · 2016-08-12T14:00:57.000Z

I was experiencing the same error .Removing the PYSPARK_SUBMIT_ARGS env did the trick.
Thanks

Answer 10 · 2016-09-20T12:12:28.000Z

You actually have to define "pyspark-shell" in PYSPARK_SUBMIT_ARGS if you define this.

For instance:

import os
os.environ['PYSPARK_SUBMIT_ARGS'] = "--master mymaster --total-executor 2 --conf "spark.driver.extraJavaOptions=-Dhttp.proxyHost=proxy.mycorp.com-Dhttp.proxyPort=1234 -Dhttp.nonProxyHosts=localhost|.mycorp.com|127.0.0.1 -Dhttps.proxyHost=proxy.mycorp.com -Dhttps.proxyPort=1234 -Dhttps.nonProxyHosts=localhost|.mycorp.com|127.0.0.1 pyspark-shell"

works

Answer 11 · 2016-12-01T19:08:21.000Z

@stibbons , where do I put this in the .bash_profile
thanks.

Answer 12 · 2016-12-01T23:48:59.000Z

in your jupiter notebook juste before importing findspark

Answer 13 · 2017-09-29T06:18:37.000Z

I removed removed it by doing os.unsetenv("PYSPARK_SUBMIT_ARGS") but still same error happens

Answer 14 · 2017-11-08T19:04:13.000Z

I am using the python 3 and I installed the pyspark library with
pip install pyspark
its installed successfully , and the library is imported successfully,
but when I use this command
sc1 = sp.SparkContext.getOrCreate()

so I am getting the same error

Exception: Java gateway process exited before sending the driver its port number
I saw comments but didn't help me, any solution how can I fix this issue.?

Answer 15 · 2017-12-02T16:13:02.000Z

@raviladhar The only thing that helped me fix this was to install a prebuilt spark version from https://spark.apache.org/downloads.html
Hope this works for you!!

Answer 16 · 2017-12-22T12:47:12.000Z

The only way to get more info on this is by modifying the java_gateway.py file like this :

(add the line print(command, env))

        ....
        from subprocess import STDOUT

        # Launch the Java gateway.
        # We open a pipe to stdin so that the Java gateway can die when the pipe is broken
        if not on_windows:
            # Don't send ctrl-c / SIGINT to the Java gateway:
            def preexec_func():
                signal.signal(signal.SIGINT, signal.SIG_IGN)
            proc = Popen(command, stdin=PIPE, stdout=PIPE, stderr=STDOUT, preexec_fn=preexec_func, env=env)
        else:
            # preexec_fn not supported on Windows
            proc = Popen(command, stdin=PIPE, stdout=PIPE, stderr=STDOUT, env=env)

        gateway_port = None
        # We use select() here in order to avoid blocking indefinitely if the subprocess dies
        # before connecting
        print(command, env)

        while gateway_port is None and proc.poll() is None:
            timeout = 1  # (seconds)
            readable, _, _ = select.select([callback_socket], [], [], timeout)
            if callback_socket in readable:
                gateway_connection = callback_socket.accept()[0]
                # Determine which ephemeral port the server started on:
                gateway_port = read_int(gateway_connection.makefile(mode="rb"))
                gateway_connection.close()
                callback_socket.close()
        if gateway_port is None:
            debug = proc.communicate()
            raise Exception("Java gateway process exited before sending the driver its port number", debug)
        ....

Then for example you will get the following stacktrace :


Java gateway process exited before sending the driver its port number
 (b'command hdp-select is not found
 please manually export HDP_VERSION in spark-env.sh or current environment

Indicating that some env variables are missing!

Answer 17 · 2018-02-20T21:35:28.000Z

I am getting the same error from just doing a fresh install from apache spark (the pre-built prepackaged version) on Mac OSX
/usr/local/spark/spark-2.2.1-bin-hadoop2.7
in this directory running
$ ./bin/pyspark
not 100% sure what is going on

Answer 18 · 2018-03-22T20:30:14.000Z

Make sure you use the correct protocol when specifying the master.

Answer 19 · 2018-03-27T05:39:49.000Z

Worked hours on this. My problem was with Java 10 installation. I uninstalled it and installed Java 8, and now Pyspark works.

Answer 20 · 2018-03-27T08:04:39.000Z

@amnghd

Don't use the new Java 9 or 10 with Spark!
A new version of Java just came out daily - Java 9 and 10. Spark is not compatible with Java 9 + versions....
You should make sure you install a Java 8 JDK and set JAVA_HOME points to it. do not install a Java 9 or JAVA 10 JDK.

Answer 21 · 2018-04-18T17:51:03.000Z

I figured out the problem in Windows system. The installation directory for Java must not have blanks in the path such as in "C:\Program Files". I re-installed Java in "C\Java". I set JAVA_HOME to C:\Java and the problem went away.

Answer 22 · 2018-05-10T14:55:19.000Z

I had similar problem and setting JAVA_HOME helped.

Answer 23 · 2018-06-13T02:01:44.000Z

在项目中添加os.environ['JAVA_HOME'] = “/the/path/to/java home directory”, then ok!

Answer 24 · 2018-08-18T12:50:51.000Z

set JAVA_HOME=C:\Progra~2\Java\jdk1.8.0_181 worked for me

Answer 25 · 2018-08-22T05:52:51.000Z

Hi there,
I solved the issue:
Exception: Java gateway process exited before sending the driver its port number #743

with installing this package
conda install -c conda-forge findspark

and importing it:

import findspark as fs
fs.init()

Answer 26 · 2018-09-09T08:31:15.000Z

@SujanMukherjee You don't necessarily have to reinstall it, you can simply escape the spaces:

import os
os.environ["JAVA_HOME"] = "C:/Program\ Files/Java/jdk1.8.0_60"

This fixed the error on Windows 10.

Answer 27 · 2018-10-10T04:43:05.000Z

( downloading openjdk 8 ) sudo apt-get install openjdk-8-jdk
( adding following lines in .bashrc )
export JAVA_HOME="/usr/lib/jvm/java-8-openjdk-amd64"
export PATH=$PATH:$JAVA_HOME/bin

java 9 or 10 is not supported for me.
(OS : ubuntu 16.04)

Answer 28 · 2018-10-24T15:09:46.000Z

On the folder in whick I was working, I had a log (text file) with an error of Java environment, I deleted that file and the problem was solved.

Answer 29 · 2018-12-13T17:11:39.000Z

So, this is the top result if you google this exception (but I am not using jupyter). For me, the root cause of the problem was I had used pip to install the latest version of pyspark, but the version of spark I had installed was different (specifically i had 2.3.0 installed) uninstalling the python module, and then using an explicit version "pip install pyspark==2.3.0" fixed my error.

Answer 30 · 2019-02-04T20:55:58.000Z

I had the same... just type
"module load spark" in command line. This issue will be resolved

Answer 31 · 2019-03-14T21:18:47.000Z

Not sure if this will help anyone, but I ran into the same error while running Pyspark using the spark-without-hadoop installation and a separate, specific Hadoop version.
I was able to get things working with the help of some not-easy-to-find hadoop docs

Basically,
export SPARK_DIST_CLASSPATH=$(/home/ec2-user/hadoop-3.1.2/bin/hadoop classpath)

Also, setting $JAVA_HOME, as mentioned above was also necessary.

Answer 32 · 2019-03-22T17:53:34.000Z

Ran into the exact same error few minutes back - I had multiple versions of JAVA installed on my machine (all > 8 + versions). Solution was to just keep Java8 and remove rest of them, worked like charm ! Hope this helps.

Answer 33 · 2019-05-06T11:28:24.000Z

在项目中添加os.environ['JAVA_HOME'] = “/the/path/to/java home directory”, then ok!

我佛了.It works, even here I can meet Chinese

Answer 34 · 2019-05-12T07:45:52.000Z

I resolve the issue under OS X Yosemite version 10.10.5
1: make sure you have JAVA8
2: Find your JAVA8's home directory then add those two lines.
import os
os.environ["JAVA_HOME"]="/Library/Java/JavaVirtualMachines/adoptopenjdk-8.jdk/Contents/Home"

Answer 35 · 2019-06-15T16:02:58.000Z

@shekharkoirala Thanks. This was enough to solve on my linux machine.

Answer 36 · 2020-03-22T07:48:24.000Z

Same problem. My JAVA_HOME path was messed up. Once i fixed that it worked.

Answer 37 · 2020-03-30T15:46:16.000Z

I got this Spark connection issue, and SparkContext didn't work for sc.

The command to initialize ipython notebook:

ipython notebook --profile=pyspark

Environment:
Mac OS
Python 2.7.10
Spark 1.4.1
java version "1.8.0_65"

Add this code line helps me solve the same problem:
Help to solve Java gateway process exited before sending the driver its port number
import findspark as fs
fs.init()

Answer 38 · 2020-04-11T16:11:13.000Z

Try restarting the Kernel and Clear all Output in Jupyter notebooks.

It resolved for me.

Answer 39 · 2020-04-11T17:11:23.000Z

@Naveen-kr Thank you so much

Answer 40 · 2020-05-26T07:38:37.000Z

@minrk where is the removing the PYSPARK_SUBMIT_ARGS env?

Answer 41 · 2020-06-05T15:06:40.000Z

I got this Spark connection issue, and SparkContext didn't work for sc.
The command to initialize ipython notebook:

ipython notebook --profile=pyspark

Environment:
Mac OS
Python 2.7.10
Spark 1.4.1
java version "1.8.0_65"

Add this code line helps me solve the same problem:
Help to solve Java gateway process exited before sending the driver its port number
import findspark as fs
fs.init()

Hi, do you know how to do that in Anaconda on Windows?

Answer 42 · 2020-06-05T15:41:05.000Z

Hi
My problem is into VSCode IDE when I’m debugging my code, vscode don’t detect Java installed in my pc(windows)
Now I’m builiding my testing functions with pytest and I want to check if my functions are ok
My solution have done:
Into the UseR directory in my pc: c:/users/xxxx
I created .bash_profile y there I wrote This line:
export path_java_my_pc
With it my pc know where is java when I Started my bash console
After I created virtual enviaronoment and I installed the necessary libraries (pyspark,pytest,etc) and after I Throw the command pytest file.py and It show me the errors and I managed to pass the tests
Thanks
Regards

Answer 43 · 2020-07-29T08:45:55.000Z

Had the same problem. For me, Java and OpenJDK were missing on my system (Ubuntu 20).

Installed the OpenJDK from the link below and after setting the JAVA_HOME variable manually, it got solved.
https://docs.datastax.com/en/jdk-install/doc/jdk-install/installOpenJdkDeb.html

Cheers!

Answer 44 · 2020-08-05T07:53:53.000Z

running Window 10. having this issue. alrdy set HOME and stuffs but nth is helping. need to make pyspark so i can continue with my lessons.......... pls help.....

Answer 45 · 2020-08-05T16:47:54.000Z

My solution was start over but with python, not anaconda. Anaconda was the problem, the set of the variables it is ok.

running Window 10. having this issue. alrdy set HOME and stuffs but nth is helping. need to make pyspark so i can continue with my lessons.......... pls help.....

Answer 46 · 2020-08-06T06:35:41.000Z

hi mikesneider, thx for the reply! Does that mean I have to uninstall and remove all the paths related to Java, Anaconda, etc etc and start the installation all over starting from Python?

Answer 47 · 2020-08-06T13:28:12.000Z

do not change the paths, that's it is ok, you only need to uninstall anaconda and start over with Python3. Actually, in the Microsoft store, you can install it. If after that do you have trouble with sklearn reply.

Answer 48 · 2020-11-23T08:02:23.000Z

I solved this problem by updating my java version(java7->java8). I think it could be helpful for some of you.

Answer 49 · 2021-01-10T08:06:08.000Z

How I solved my similar problem

Prerequisite:

anaconda already installed
Spark already installed (https://spark.apache.org/downloads.html)
pyspark already installed (https://anaconda.org/conda-forge/pyspark)

Steps I did (NOTE: set the folder path accordingly to your system)

set the following environment variables.
SPARK_HOME to 'C:\spark\spark-3.0.1-bin-hadoop2.7'
set HADOOP_HOME to 'C:\spark\spark-3.0.1-bin-hadoop2.7'
set PYSPARK_DRIVER_PYTHON to 'jupyter'
set PYSPARK_DRIVER_PYTHON_OPTS to 'notebook'
add 'C:\spark\spark-3.0.1-bin-hadoop2.7\bin;' to PATH system variable.
Change the java installed folder directly under C: (Previously java was installed under Program files, so I re-installed directly under C:)
so my JAVA_HOME will become like this 'C:\java\jdk1.8.0_271'

now. it works !

Answer 50 · 2021-05-06T10:04:43.000Z

Had the same situation under MAC. After installing spark with the anaconda environment tool, open a notebook and

from pyspark import SparkContext
sc = SparkContext()

then I got an error message with a lot of stuff and finally:

Java gateway process exited before sending the driver its port number

I solved the situation with the next steps:

check the java version you are using and the compatibility with spark.
to see the compatibility access: https://spark.apache.org/docs/latest/index.html

(base) MacBook-Pro-de-daniel:~ danielvegamartinez$ java -version
java version "16.0.1" 2021-04-20

(base) MacBook-Pro-de-daniel:~ danielvegamartinez$ ls /Library/Java/JavaVirtualMachines/
1.6.0.jdk	jdk-16.0.1.jdk

So I removed the 16 version

(base) MacBook-Pro-de-daniel:~ danielvegamartinez$ sudo rm -fr /Library/Internet\ Plug-Ins/JavaAppletPlugin.plugin
(base) MacBook-Pro-de-daniel:~ danielvegamartinez$ sudo rm -fr /Library/PreferencesPanes/JavaControlPanel.prefPane
(base) MacBook-Pro-de-daniel:~ danielvegamartinez$ sudo rm -fr ~/Library/Application\ Support/Oracle/Java
(base) MacBook-Pro-de-daniel:~ danielvegamartinez$ sudo rm -fr /Library/Java/JavaVirtualMachines/jdk-16.0.1.jdk/
(base) MacBook-Pro-de-daniel:~ danielvegamartinez$ ls /Library/Java/JavaVirtualMachines/
1.6.0.jdk

Then went to Oracle and downloaded a java 10 version which seems to be compatible to spark 3.1.1

https://www.oracle.com/java/technologies/oracle-java-archive-downloads.html

Installed it with the installer and now I am running the 10 verison

(base) MacBook-Pro-de-daniel:~ danielvegamartinez$ ls /Library/Java/JavaVirtualMachines/
1.6.0.jdk	jdk-10.0.2.jdk

(base) MacBook-Pro-de-daniel:~ danielvegamartinez$ java -version
java version "10.0.2" 2018-07-17

Re-started the notebook and now it is running and fine.

Hope my case helps. Good luck,

Answer 51 · 2021-05-18T16:52:05.000Z

@dvegamar
Same issue i was facing and worked for hours,resolved now.Thanks a lot

Answer 52 · 2021-11-03T18:30:13.000Z

Had the same situation under MAC. After installing spark with the anaconda environment tool, open a notebook and
from pyspark import SparkContext
sc = SparkContext()
then I got an error message with a lot of stuff and finally:

Java gateway process exited before sending the driver its port number

I solved the situation with the next steps:

check the java version you are using and the compatibility with spark.

to see the compatibility access: https://spark.apache.org/docs/latest/index.html
(base) MacBook-Pro-de-daniel:~ danielvegamartinez$ java -version
java version "16.0.1" 2021-04-20
(base) MacBook-Pro-de-daniel:~ danielvegamartinez$ ls /Library/Java/JavaVirtualMachines/
1.6.0.jdk	jdk-16.0.1.jdk
So I removed the 16 version
(base) MacBook-Pro-de-daniel:~ danielvegamartinez$ sudo rm -fr /Library/Internet\ Plug-Ins/JavaAppletPlugin.plugin
(base) MacBook-Pro-de-daniel:~ danielvegamartinez$ sudo rm -fr /Library/PreferencesPanes/JavaControlPanel.prefPane
(base) MacBook-Pro-de-daniel:~ danielvegamartinez$ sudo rm -fr ~/Library/Application\ Support/Oracle/Java
(base) MacBook-Pro-de-daniel:~ danielvegamartinez$ sudo rm -fr /Library/Java/JavaVirtualMachines/jdk-16.0.1.jdk/
(base) MacBook-Pro-de-daniel:~ danielvegamartinez$ ls /Library/Java/JavaVirtualMachines/
1.6.0.jdk
Then went to Oracle and downloaded a java 10 version which seems to be compatible to spark 3.1.1

https://www.oracle.com/java/technologies/oracle-java-archive-downloads.html

Installed it with the installer and now I am running the 10 verison
(base) MacBook-Pro-de-daniel:~ danielvegamartinez$ ls /Library/Java/JavaVirtualMachines/
1.6.0.jdk	jdk-10.0.2.jdk
(base) MacBook-Pro-de-daniel:~ danielvegamartinez$ java -version
java version "10.0.2" 2018-07-17
Re-started the notebook and now it is running and fine.

Hope my case helps. Good luck,

Thank you so much!!!
I checked my java version on the terminal and uninstalled the problematic one and now it works just fine.

Answer 53 · 2021-12-02T06:33:14.000Z

check JAVA_HOME
check SPARK_HOME
check if PATHONPATH contains this path $SPARK_HOME/python/lib/py4j-0.10.4-src.zip
chech python libs pip list | grep spark only contains findspark
findspark.init($SPARK_HOME)