Index extraction fails on larger rosbags

[blubbi321]

Using the provided Dockerfile I cant seem to import a 6GB rosbag. Smaller bags work.

Input (on the Jupyter notebook):

%%bash
java -jar ../lib/rosbaginputformat_2.11-0.9.3.jar -f /srv/data/my.bag

Output:

Exception in thread "main" java.lang.IllegalArgumentException: Size exceeds Integer.MAX_VALUE
	at sun.nio.ch.FileChannelImpl.map(FileChannelImpl.java:869)
	at de.valtech.foss.Main$$anonfun$process$1.apply(Main.scala:77)
	at de.valtech.foss.Main$$anonfun$process$1.apply(Main.scala:76)
	at de.valtech.foss.Main$.use(Main.scala:65)
	at de.valtech.foss.Main$.process(Main.scala:76)
	at de.valtech.foss.Main$.delayedEndpoint$de$valtech$foss$Main$1(Main.scala:70)
	at de.valtech.foss.Main$delayedInit$body.apply(Main.scala:33)
	at scala.Function0$class.apply$mcV$sp(Function0.scala:34)
	at scala.runtime.AbstractFunction0.apply$mcV$sp(AbstractFunction0.scala:12)
	at scala.App$$anonfun$main$1.apply(App.scala:76)
	at scala.App$$anonfun$main$1.apply(App.scala:76)
	at scala.collection.immutable.List.foreach(List.scala:381)
	at scala.collection.generic.TraversableForwarder$class.foreach(TraversableForwarder.scala:35)
	at scala.App$class.main(App.scala:76)
	at de.valtech.foss.Main$.main(Main.scala:33)
	at de.valtech.foss.Main.main(Main.scala)

While googling for this error the results [1] suggest that the spark.default.parallelism setting needs to be adjusted (for > 2GB files). However, the bag file is not even in hdfs at this point in time and I get the same error when trying to create the index from outside the docker container.

Id appreciate any pointers in the right direction as I had hoped to feed in some bigger rosbags.

[1] https://stackoverflow.com/questions/42247630/sql-query-in-spark-scala-size-exceeds-integer-max-value

[Deleted user]

Issue should be solved.

Should you encounter any further problems please reindex your Rosbag file and open an issue.

For an example of Rosbag file larger than 2 GB see ./doc/"Rosbag larger than 2 GB.ipynb" Solved the issue #6 The issue was due to ByteBuffer being limitted by JVM Integer size and has nothing to do with Spark or how the RosbagMapInputFormat works within Spark. It was only problematic to extract the conf index with the jar.

[Deleted user]

@blubbi321 Please test it and let us know if it works as expected so that we can close the issue.

[blubbi321]

Sorry didnt see the comment earlier, will test asap. Thanks for getting to it so quickly!