spark-root/laurelin

java.lang.IllegalArgumentException in slice function of RawArray.java

tomcornelis opened this issue · 5 comments

tomcornelis commented

I am trying to use laurelin, but even this minimal example results in a crash I can't explain.

This is the example:

#!/usr/bin/env python3
from pyspark.sql import SparkSession

local_jars = ','.join([
    './laurelin-1.1.1.jar',
    './log4j-api-2.13.0.jar',
    './log4j-core-2.13.0.jar',
])

spark = SparkSession\
    .builder\
    .appName("TnP")\
    .config("spark.jars", local_jars)\
    .config("spark.driver.extraClassPath", local_jars)\
    .config("spark.executor.extraClassPath", local_jars)\
    .getOrCreate()

sc = spark.sparkContext
print(sc.getConf().toDebugString())
rootfile = spark.read.format("root").option('tree', 'tpTree/fitter_tree').load('hdfs://analytix/user/tomc/tnpTuples_muons/MC_Moriond17_DY_tranch4Premix_part10_0.root')
rootfile.printSchema()
rootfile.show(1) # crash

spark.stop()

resulting in the following crash:

java.lang.RuntimeException: java.util.concurrent.ExecutionException: java.lang.IllegalArgumentException
	at edu.vanderbilt.accre.laurelin.array.ArrayBuilder.getArray(ArrayBuilder.java:202)
	at edu.vanderbilt.accre.laurelin.spark_ttree.TTreeColumnVector.getFloats(TTreeColumnVector.java:188)
	at edu.vanderbilt.accre.laurelin.spark_ttree.TTreeColumnVector.getFloat(TTreeColumnVector.java:106)
	at org.apache.spark.sql.catalyst.expressions.GeneratedClass$GeneratedIteratorForCodegenStage1.processNext(Unknown Source)
	at org.apache.spark.sql.execution.BufferedRowIterator.hasNext(BufferedRowIterator.java:43)
	at org.apache.spark.sql.execution.WholeStageCodegenExec$$anonfun$13$$anon$1.hasNext(WholeStageCodegenExec.scala:636)
	at org.apache.spark.sql.execution.SparkPlan$$anonfun$2.apply(SparkPlan.scala:255)
	at org.apache.spark.sql.execution.SparkPlan$$anonfun$2.apply(SparkPlan.scala:247)
	at org.apache.spark.rdd.RDD$$anonfun$mapPartitionsInternal$1$$anonfun$apply$24.apply(RDD.scala:858)
	at org.apache.spark.rdd.RDD$$anonfun$mapPartitionsInternal$1$$anonfun$apply$24.apply(RDD.scala:858)
	at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:52)
	at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:346)
	at org.apache.spark.rdd.RDD.iterator(RDD.scala:310)
	at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:52)
	at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:346)
	at org.apache.spark.rdd.RDD.iterator(RDD.scala:310)
	at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:90)
	at org.apache.spark.scheduler.Task.run(Task.scala:123)
	at org.apache.spark.executor.Executor$TaskRunner$$anonfun$10.apply(Executor.scala:408)
	at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:1360)
	at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:414)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)
Caused by: java.util.concurrent.ExecutionException: java.lang.IllegalArgumentException
	at java.util.concurrent.FutureTask.report(FutureTask.java:122)
	at java.util.concurrent.FutureTask.get(FutureTask.java:192)
	at edu.vanderbilt.accre.laurelin.array.ArrayBuilder.getArray(ArrayBuilder.java:198)
	... 23 more
Caused by: java.lang.IllegalArgumentException
	at java.nio.Buffer.limit(Buffer.java:275)
	at edu.vanderbilt.accre.laurelin.array.RawArray.slice(RawArray.java:29)
	at edu.vanderbilt.accre.laurelin.interpretation.AsDtype.fromroot(AsDtype.java:169)
	at edu.vanderbilt.accre.laurelin.array.ArrayBuilder.processBasket(ArrayBuilder.java:90)
	at edu.vanderbilt.accre.laurelin.array.ArrayBuilder.lambda$new$0(ArrayBuilder.java:185)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	... 3 more

The test file which is causing this issue can be found at http://tomc.web.cern.ch/tomc/MC_Moriond17_DY_tranch4Premix_part10_0.root

PerilousApricot commented

Hi Tom,

The offending issue is here:

	at java.nio.Buffer.limit(Buffer.java:275)
	at edu.vanderbilt.accre.laurelin.array.RawArray.slice(RawArray.java:29)

where I try to take a slice of a java bytebuffer. The function limit() only supports numbers between 0 and 2GB, so this symptom is usually due to me decoding the ROOT metadata improperly and ending up trying to request to read 100s of exabytes. I'll take a look at your file and see where the explosion happened.

PerilousApricot commented

@tomcornelis FWIW, Is the UserData subdir in your file something useful to deserialize, or is it just metadata?

tomcornelis commented

@PerilousApricot The UserData is not needed for me, it's just metadata which comes from the tree where I skimmed this subset from.

  • 👍1
PerilousApricot commented

Ah, okay. The issue is that the ROOT file was somehow not closed properly, so the accounting for how many events are stored in the last basket Is stored slightly differently. Let me reverse-engineer how ROOT handles it, and run some tests

PerilousApricot commented