BitwiseInc/Hydrograph

spark tests fail with out of memory while building the package

Closed this issue · 0 comments

i have tried to set the spark.driver.memory=4g property @ "Hydrograph\hydrograph.engine\hydrograph.engine.spark\src\main\resources\runtime_service" but it doesnt seem to work.

ERROR spark.SparkContext: Error initializing SparkContext.
java.lang.IllegalArgumentException: System memory 259522560 must be at least 471859200. Please increase heap size using the --driver-memory option or spark.driver.memory in Spark configuration.
at org.apache.spark.memory.UnifiedMemoryManager$.getMaxMemory(UnifiedMemoryManager.scala:212)
at org.apache.spark.memory.UnifiedMemoryManager$.apply(UnifiedMemoryManager.scala:194)
at org.apache.spark.SparkEnv$.create(SparkEnv.scala:308)
at org.apache.spark.SparkEnv$.createDriverEnv(SparkEnv.scala:165)
at org.apache.spark.SparkContext.createSparkEnv(SparkContext.scala:256)
at org.apache.spark.SparkContext.(SparkContext.scala:420)
at org.apache.spark.SparkContext$.getOrCreate(SparkContext.scala:2258)
at org.apache.spark.sql.SparkSession$Builder$$anonfun$8.apply(SparkSession.scala:831)
at org.apache.spark.sql.SparkSession$Builder$$anonfun$8.apply(SparkSession.scala:823)
at scala.Option.getOrElse(Option.scala:121)
at org.apache.spark.sql.SparkSession$Builder.getOrCreate(SparkSession.scala:823)
at hydrograph.engine.testing.wrapper.DataBuilder.build(DataBuilder.scala:145)
at hydrograph.engine.spark.components.CumulateComponentTest.CumulateCountOfResultsAndMapFields(CumulateComponentTest.scala:42)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)