Error: Java heap space

[ObadaM]

HI everyone,

I am facing a problem while I am trying to load a tif file with the help from spark. It gives me the error:
Caused by: java.lang.OutOfMemoryError: Java heap space

import thunder as td
sc = SparkContext("local","app1")
img = td.images.fromtif('20160511_HuC_H2B_G6s_F1_00002.tif','tif',None,None,False,None,None,None,sc,None).npartitions(100)

1- I tried to update "spark-env.sh" and "spark-defaults.conf" with

spark.executor.memory 2000m

but I got the same error.

2- I tried to make it like this :
sc = SparkContext("local","app1","--executor-memory 20G")

I got the same error.

3- I tried to add a config variable

conf = SparkConf().setAppName("Simple App").setMaster("local").set("spark.executor.memory 2000m")
sc = SparkContext(conf)

I got error with conf, _AttributeError: 'SparkConf' object has no attribute 'get_object_id'

I would appreciate any kind of help.

[freeman-lab]

@ObadaM are you running this on a local machine or on a cluster? If on a local machine, you are unlikely to see much speed up from using Spark, and we recommend local mode.

But if on a cluster, or if you are running Spark on a single machine just to try it out, you can try adding this line to your bash profile (assuming you're on Mac OS X or Linux)

export _JAVA_OPTIONS="-Xms512m -Xmx4g"

[ObadaM]

@freeman-lab I tried your solution but it did not work as well.
actually it is always 511 MB from RAM

16/05/23 08:45:09 INFO MemoryStore: MemoryStore started with capacity 511.5 MB

and the only way that it changed was from running Spark from terminal.

./pyspark --driver-memory 5g

Yes I am running it on a single machine just to be familiar with the whole spark thunder environment then I will apply it on a cluster.

[ObadaM]

I found the solution. first I downloaded the source code of spark then I built it again with sbt, after than i created spark-env.sh in conf file with the following:

SPARK_EXECUTOR_INSTANCES=14
SPARK_EXECUTOR_MEMORY=4G
SPARK_DRIVER_MEMORY=4G

I tried the pre-built version of spark with this before but it did not work.