big-data-europe/docker-hadoop-spark-workbench

Error when accessing hdfs file from remote spark driver

jrabary opened this issue · 1 comments

Thank you for sharing the result of your project. I'm trying play the hadoop spark workbench and I got some trouble when I want to access the data stored in the hdfs file.

My setting is as follow:

  • I set up the hadoop-spark-workbench in a physical server inside our internal network.
  • I run a remote spark-shell from my laptop by specifying the master address.

When I try to load a data on the hdfs for example using the following line:

val data = sparkContext.textFile("hdfs://namenode/data.txt")

I get a connection refused error even if the port 8082 is forwarded in the docker.

Finally get it work. I forwarded the wrong port!