Pushing large data set results in Too many open files error
yanokwa opened this issue · 3 comments
yanokwa commented
Software versions
Briefcase v1.17.1, macOS 10.15.2
Problem description
Pushing a large data set to Aggregate results in Too many open files error.
I believe that these errors cause the push to stop entirely.
I cannot reproduce this on Ubuntu server, which suggests to me that this is a macOS-specific issue. Documentation would be a fine fix for this, but if we can work around it in code, that'd be better.
Other information
2019-12-16 11:21:39,249 [ForkJoinPool-1-worker-15] ERROR o.o.briefcase.reused.job.JobsRunner - Error running Job
java.io.UncheckedIOException: java.nio.file.FileSystemException: /Blah/instances/uuid80a92285-f4e8-4d27-a420-37c96cc8df80/submission.xml: Too many open files
at org.opendatakit.briefcase.reused.UncheckedFiles.newInputStream(UncheckedFiles.java:300)
at org.opendatakit.briefcase.reused.transfer.AggregateServer.getPushSubmissionRequest(AggregateServer.java:185)
at org.opendatakit.briefcase.push.aggregate.PushToAggregate.pushSubmissionAndAttachments(PushToAggregate.java:185)
at org.opendatakit.briefcase.push.aggregate.PushToAggregate.pushSubmissionAndAttachments(PushToAggregate.java:175)
at org.opendatakit.briefcase.push.aggregate.PushToAggregate.lambda$push$6(PushToAggregate.java:98)
at java.base/java.util.stream.ForEachOps$ForEachOp$OfRef.accept(ForEachOps.java:183)
at java.base/java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1654)
at java.base/java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:484)
at java.base/java.util.stream.ForEachOps$ForEachTask.compute(ForEachOps.java:290)
at java.base/java.util.concurrent.CountedCompleter.exec(CountedCompleter.java:746)
at java.base/java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:290)
at java.base/java.util.concurrent.ForkJoinPool$WorkQueue.helpCC(ForkJoinPool.java:1115)
at java.base/java.util.concurrent.ForkJoinPool.awaitJoin(ForkJoinPool.java:1687)
at java.base/java.util.concurrent.ForkJoinTask.doInvoke(ForkJoinTask.java:411)
at java.base/java.util.concurrent.ForkJoinTask.invoke(ForkJoinTask.java:736)
at java.base/java.util.stream.ForEachOps$ForEachOp.evaluateParallel(ForEachOps.java:159)
at java.base/java.util.stream.ForEachOps$ForEachOp$OfRef.evaluateParallel(ForEachOps.java:173)
at java.base/java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:233)
at java.base/java.util.stream.ReferencePipeline.forEach(ReferencePipeline.java:497)
at java.base/java.util.stream.ReferencePipeline$Head.forEach(ReferencePipeline.java:661)
at org.opendatakit.briefcase.push.aggregate.PushToAggregate.lambda$push$7(PushToAggregate.java:95)
at org.opendatakit.briefcase.reused.job.Job.lambda$thenAccept$8(Job.java:134)
at org.opendatakit.briefcase.reused.job.Job.lambda$thenRun$6(Job.java:109)
at org.opendatakit.briefcase.reused.job.JobsRunner.lambda$launchAsync$1(JobsRunner.java:65)
at java.base/java.util.concurrent.ForkJoinTask$AdaptedRunnableAction.exec(ForkJoinTask.java:1407)
at java.base/java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:290)
at java.base/java.util.concurrent.ForkJoinPool$WorkQueue.topLevelExec(ForkJoinPool.java:1020)
at java.base/java.util.concurrent.ForkJoinPool.scan(ForkJoinPool.java:1656)
at java.base/java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1594)
at java.base/java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:177)
Caused by: java.nio.file.FileSystemException: /Blah/instances/uuid80a92285-f4e8-4d27-a420-37c96cc8df80/submission.xml: Too many open files
at java.base/sun.nio.fs.UnixException.translateToIOException(UnixException.java:100)
at java.base/sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:111)
at java.base/sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:116)
at java.base/sun.nio.fs.UnixFileSystemProvider.newByteChannel(UnixFileSystemProvider.java:215)
at java.base/java.nio.file.Files.newByteChannel(Files.java:370)
at java.base/java.nio.file.Files.newByteChannel(Files.java:421)
at java.base/java.nio.file.spi.FileSystemProvider.newInputStream(FileSystemProvider.java:420)
at java.base/java.nio.file.Files.newInputStream(Files.java:155)
at org.opendatakit.briefcase.reused.UncheckedFiles.newInputStream(UncheckedFiles.java:298)
... 29 common frames omitted
2019-12-16 11:21:39,249 [ForkJoinPool-1-worker-5] ERROR o.o.briefcase.reused.job.JobsRunner - Error running Job
java.io.UncheckedIOException: java.nio.file.FileSystemException: /Blah/instances/uuidaa63b5ff-c5b0-4137-9ed4-aebd28b89f1b: Too many open files
at org.opendatakit.briefcase.reused.UncheckedFiles.list(UncheckedFiles.java:278)
at org.opendatakit.briefcase.push.aggregate.PushToAggregate.getSubmissionAttachments(PushToAggregate.java:114)
at org.opendatakit.briefcase.push.aggregate.PushToAggregate.lambda$push$6(PushToAggregate.java:96)
at java.base/java.util.stream.ForEachOps$ForEachOp$OfRef.accept(ForEachOps.java:183)
at java.base/java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1654)
at java.base/java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:484)
at java.base/java.util.stream.ForEachOps$ForEachTask.compute(ForEachOps.java:290)
at java.base/java.util.concurrent.CountedCompleter.exec(CountedCompleter.java:746)
at java.base/java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:290)
at java.base/java.util.concurrent.ForkJoinPool$WorkQueue.helpCC(ForkJoinPool.java:1115)
at java.base/java.util.concurrent.ForkJoinPool.awaitJoin(ForkJoinPool.java:1687)
at java.base/java.util.concurrent.ForkJoinTask.doInvoke(ForkJoinTask.java:411)
at java.base/java.util.concurrent.ForkJoinTask.invoke(ForkJoinTask.java:736)
at java.base/java.util.stream.ForEachOps$ForEachOp.evaluateParallel(ForEachOps.java:159)
at java.base/java.util.stream.ForEachOps$ForEachOp$OfRef.evaluateParallel(ForEachOps.java:173)
at java.base/java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:233)
at java.base/java.util.stream.ReferencePipeline.forEach(ReferencePipeline.java:497)
at java.base/java.util.stream.ReferencePipeline$Head.forEach(ReferencePipeline.java:661)
at org.opendatakit.briefcase.push.aggregate.PushToAggregate.lambda$push$7(PushToAggregate.java:95)
at org.opendatakit.briefcase.reused.job.Job.lambda$thenAccept$8(Job.java:134)
at org.opendatakit.briefcase.reused.job.Job.lambda$thenRun$6(Job.java:109)
at org.opendatakit.briefcase.reused.job.JobsRunner.lambda$launchAsync$1(JobsRunner.java:65)
at java.base/java.util.concurrent.ForkJoinTask$AdaptedRunnableAction.exec(ForkJoinTask.java:1407)
at java.base/java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:290)
at java.base/java.util.concurrent.ForkJoinPool$WorkQueue.topLevelExec(ForkJoinPool.java:1020)
at java.base/java.util.concurrent.ForkJoinPool.scan(ForkJoinPool.java:1656)
at java.base/java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1594)
at java.base/java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:177)
Caused by: java.nio.file.FileSystemException: /Blah/instances/uuidaa63b5ff-c5b0-4137-9ed4-aebd28b89f1b: Too many open files
at java.base/sun.nio.fs.UnixException.translateToIOException(UnixException.java:100)
at java.base/sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:111)
at java.base/sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:116)
at java.base/sun.nio.fs.UnixFileSystemProvider.newDirectoryStream(UnixFileSystemProvider.java:408)
at java.base/java.nio.file.Files.newDirectoryStream(Files.java:471)
at java.base/java.nio.file.Files.list(Files.java:3694)
at org.opendatakit.briefcase.reused.UncheckedFiles.list(UncheckedFiles.java:276)
... 27 common frames omitted
Error pushing a form: java.nio.file.FileSystemException: /Blah/instances/uuid80a92285-f4e8-4d27-a420-37c96cc8df80/submission.xml: Too many open files (see the logs for more info)2019-12-16 11:21:39,249 [ForkJoinPool-1-worker-3] ERROR o.o.briefcase.reused.job.JobsRunner - Error running Job
java.io.UncheckedIOException: java.nio.file.FileSystemException: /Blah/instances/BC2X5M0HFH25MEHPM28IG8HAH: Too many open files
at org.opendatakit.briefcase.reused.UncheckedFiles.list(UncheckedFiles.java:278)
at org.opendatakit.briefcase.push.aggregate.PushToAggregate.getSubmissionAttachments(PushToAggregate.java:114)
at org.opendatakit.briefcase.push.aggregate.PushToAggregate.lambda$push$6(PushToAggregate.java:96)
at java.base/java.util.stream.ForEachOps$ForEachOp$OfRef.accept(ForEachOps.java:183)
at java.base/java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1654)
at java.base/java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:484)
at java.base/java.util.stream.ForEachOps$ForEachTask.compute(ForEachOps.java:290)
at java.base/java.util.concurrent.CountedCompleter.exec(CountedCompleter.java:746)
at java.base/java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:290)
at java.base/java.util.concurrent.ForkJoinPool$WorkQueue.helpCC(ForkJoinPool.java:1115)
at java.base/java.util.concurrent.ForkJoinPool.awaitJoin(ForkJoinPool.java:1687)
at java.base/java.util.concurrent.ForkJoinTask.doInvoke(ForkJoinTask.java:411)
at java.base/java.util.concurrent.ForkJoinTask.invoke(ForkJoinTask.java:736)
at java.base/java.util.stream.ForEachOps$ForEachOp.evaluateParallel(ForEachOps.java:159)
at java.base/java.util.stream.ForEachOps$ForEachOp$OfRef.evaluateParallel(ForEachOps.java:173)
at java.base/java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:233)
at java.base/java.util.stream.ReferencePipeline.forEach(ReferencePipeline.java:497)
at java.base/java.util.stream.ReferencePipeline$Head.forEach(ReferencePipeline.java:661)
at org.opendatakit.briefcase.push.aggregate.PushToAggregate.lambda$push$7(PushToAggregate.java:95)
at org.opendatakit.briefcase.reused.job.Job.lambda$thenAccept$8(Job.java:134)
at org.opendatakit.briefcase.reused.job.Job.lambda$thenRun$6(Job.java:109)
at org.opendatakit.briefcase.reused.job.JobsRunner.lambda$launchAsync$1(JobsRunner.java:65)
at java.base/java.util.concurrent.ForkJoinTask$AdaptedRunnableAction.exec(ForkJoinTask.java:1407)
at java.base/java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:290)
at java.base/java.util.concurrent.ForkJoinPool$WorkQueue.topLevelExec(ForkJoinPool.java:1020)
at java.base/java.util.concurrent.ForkJoinPool.scan(ForkJoinPool.java:1656)
at java.base/java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1594)
at java.base/java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:177)
Caused by: java.nio.file.FileSystemException: /Blah/instances/BC2X5M0HFH25MEHPM28IG8HAH: Too many open files
at java.base/sun.nio.fs.UnixException.translateToIOException(UnixException.java:100)
at java.base/sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:111)
at java.base/sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:116)
at java.base/sun.nio.fs.UnixFileSystemProvider.newDirectoryStream(UnixFileSystemProvider.java:408)
at java.base/java.nio.file.Files.newDirectoryStream(Files.java:471)
at java.base/java.nio.file.Files.list(Files.java:3694)
at org.opendatakit.briefcase.reused.UncheckedFiles.list(UncheckedFiles.java:276)
... 27 common frames omitted
poketim commented
Depending on the OS, the file systems will be different in the number of opened files allowed.
https://linux.101hacks.com/unix/java-too-many-open-files/
In AggregateServer.java, on method getPushSubmissionRequest(), it's possible new InputStream() is called enough where it surpasses the allowed number of opened files.
To reproduce this problem, is the number of attachments large in the POST request?
One option I can think of is to perform a batch upload with a limit on file/attachments. So if you need 100 attachments uploaded, you can chunk them out and upload 10 at a time or however many at a time until you get through all 100, while being under the # of opened files limit.
Another option would be not to use new InputStream() and instead use classes that support multi-part file upload. https://commons.apache.org/proper/commons-fileupload/apidocs/org/apache/commons/fileupload/FileUpload.html
That way, the files don't have to be opened and can still be submitted to the server. Although, this option may be more difficult because it will require more refactoring in other areas.
lognaturel commented
@timkoon, your explanation sounds just right. I talked to @yanokwa about the data he used to reproduce and he doesn't think that there were many attachments, just many submissions.
The first thing to check, I think, would be that streams are closed immediately upon successful submission.
poketim commented
@opendatakit-bot claim