Change FragPipe temp directory on Linux/Unix
GregStacey opened this issue · 7 comments
I'm successfully running the headless version of fragpipe, but I'm encountering a problem when submitting it via scheduler on a cluster. When submitted, the cache directory is read-only (~/.config/FragPipe/fragpipe/) and the run fails. The full error log is below.
I suspect this can be solved on the cluster side, but I'm also curious if there's a way to change the cache path in headless mode. Thanks!
2022-07-08 11:19:36,512 WARN - The output directory is not empty. Some files might be overwritten in: /scratch/st-ljfoster-1/staceyri/isabela_fragpipe/experiments/fullsearch
2022-07-08 11:19:36,580 ERROR - Error saving ui cache
java.nio.file.FileSystemException: /home/staceyri/.config/FragPipe/fragpipe/fragpipe-ui.cache: Read-only file system
at java.base/sun.nio.fs.UnixException.translateToIOException(UnixException.java:100)
at java.base/sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:111)
at java.base/sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:116)
at java.base/sun.nio.fs.UnixFileSystemProvider.newByteChannel(UnixFileSystemProvider.java:219)
at java.base/java.nio.file.spi.FileSystemProvider.newOutputStream(FileSystemProvider.java:478)
at java.base/java.nio.file.Files.newOutputStream(Files.java:219)
at com.dmtavt.fragpipe.api.PropsFile.save(PropsFile.java:98)
at com.dmtavt.fragpipe.Fragpipe.saveCache(Fragpipe.java:261)
at com.dmtavt.fragpipe.Fragpipe.on(Fragpipe.java:675)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:566)
at org.greenrobot.eventbus.EventBus.invokeSubscriber(EventBus.java:517)
at org.greenrobot.eventbus.EventBus.invokeSubscriber(EventBus.java:511)
at org.greenrobot.eventbus.AsyncPoster.run(AsyncPoster.java:46)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at java.base/java.lang.Thread.run(Thread.java:834)
2022-07-08 11:19:36,594 ERROR - Error saving runtime cache
java.nio.file.FileSystemException: /home/staceyri/.config/FragPipe/fragpipe/fragpipe-runtime.cache: Read-only file system
at java.base/sun.nio.fs.UnixException.translateToIOException(UnixException.java:100)
at java.base/sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:111)
at java.base/sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:116)
at java.base/sun.nio.fs.UnixFileSystemProvider.newByteChannel(UnixFileSystemProvider.java:219)
at java.base/java.nio.file.spi.FileSystemProvider.newOutputStream(FileSystemProvider.java:478)
at java.base/java.nio.file.Files.newOutputStream(Files.java:219)
at com.dmtavt.fragpipe.api.PropsFile.save(PropsFile.java:98)
at com.dmtavt.fragpipe.Fragpipe.saveCache(Fragpipe.java:269)
at com.dmtavt.fragpipe.Fragpipe.on(Fragpipe.java:675)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:566)
at org.greenrobot.eventbus.EventBus.invokeSubscriber(EventBus.java:517)
at org.greenrobot.eventbus.EventBus.invokeSubscriber(EventBus.java:511)
at org.greenrobot.eventbus.AsyncPoster.run(AsyncPoster.java:46)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at java.base/java.lang.Thread.run(Thread.java:834)
[SEVERE] Could not dispatch event: class com.dmtavt.fragpipe.messages.MessageSaveCache to subscribing class class com.dmtavt.fragpipe.Fragpipe
java.lang.IllegalArgumentException: File parameter 'destDir is not writable: '/home/staceyri/.config/FragPipe/fragpipe/workflows'
at org.apache.commons.io.FileUtils.requireCanWrite(FileUtils.java:2619)
at org.apache.commons.io.FileUtils.doCopyDirectory(FileUtils.java:1305)
at org.apache.commons.io.FileUtils.copyDirectory(FileUtils.java:699)
at org.apache.commons.io.FileUtils.copyDirectory(FileUtils.java:630)
at org.apache.commons.io.FileUtils.copyDirectory(FileUtils.java:531)
at org.apache.commons.io.FileUtils.copyDirectory(FileUtils.java:502)
at com.dmtavt.fragpipe.Fragpipe.saveCache(Fragpipe.java:279)
at com.dmtavt.fragpipe.Fragpipe.on(Fragpipe.java:675)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:566)
at org.greenrobot.eventbus.EventBus.invokeSubscriber(EventBus.java:517)
at org.greenrobot.eventbus.EventBus.invokeSubscriber(EventBus.java:511)
at org.greenrobot.eventbus.AsyncPoster.run(AsyncPoster.java:46)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at java.base/java.lang.Thread.run(Thread.java:834)
2022-07-08 11:19:36,629 ERROR - Error delivering events through the bus
java.lang.IllegalArgumentException: File parameter 'destDir is not writable: '/home/staceyri/.config/FragPipe/fragpipe/workflows'
at org.apache.commons.io.FileUtils.requireCanWrite(FileUtils.java:2619)
at org.apache.commons.io.FileUtils.doCopyDirectory(FileUtils.java:1305)
at org.apache.commons.io.FileUtils.copyDirectory(FileUtils.java:699)
at org.apache.commons.io.FileUtils.copyDirectory(FileUtils.java:630)
at org.apache.commons.io.FileUtils.copyDirectory(FileUtils.java:531)
at org.apache.commons.io.FileUtils.copyDirectory(FileUtils.java:502)
at com.dmtavt.fragpipe.Fragpipe.saveCache(Fragpipe.java:279)
at com.dmtavt.fragpipe.Fragpipe.on(Fragpipe.java:675)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:566)
at org.greenrobot.eventbus.EventBus.invokeSubscriber(EventBus.java:517)
at org.greenrobot.eventbus.EventBus.invokeSubscriber(EventBus.java:511)
at org.greenrobot.eventbus.AsyncPoster.run(AsyncPoster.java:46)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at java.base/java.lang.Thread.run(Thread.java:834)
2022-07-08 11:19:36,633 ERROR - java.lang.IllegalArgumentException: File parameter 'destDir is not writable: '/home/staceyri/.config/FragPipe/fragpipe/workflows'
at org.apache.commons.io.FileUtils.requireCanWrite(FileUtils.java:2619)
at org.apache.commons.io.FileUtils.doCopyDirectory(FileUtils.java:1305)
at org.apache.commons.io.FileUtils.copyDirectory(FileUtils.java:699)
at org.apache.commons.io.FileUtils.copyDirectory(FileUtils.java:630)
at org.apache.commons.io.FileUtils.copyDirectory(FileUtils.java:531)
at org.apache.commons.io.FileUtils.copyDirectory(FileUtils.java:502)
at com.dmtavt.fragpipe.Fragpipe.saveCache(Fragpipe.java:279)
at com.dmtavt.fragpipe.Fragpipe.on(Fragpipe.java:675)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:566)
at org.greenrobot.eventbus.EventBus.invokeSubscriber(EventBus.java:517)
at org.greenrobot.eventbus.EventBus.invokeSubscriber(EventBus.java:511)
at org.greenrobot.eventbus.AsyncPoster.run(AsyncPoster.java:46)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at java.base/java.lang.Thread.run(Thread.java:834)
2022-07-08 11:19:37,308 ERROR - Could not save fragger_*.params file to working dir.
Thanks for the quick response. Unfortunately I get the following error about the X11 display
(base) [staceyri@login02 isabela_fragpipe]$ JAVA_OPTS=-Djava.io.tmpdir=/scratch/st-ljfoster-1/staceyri/isabela_fragpipe/tmp /scratch/st-ljfoster-1/staceyri/fragpipe/bin/fragpipe
Exception in thread "main" java.awt.HeadlessException:
No X11 DISPLAY variable was set, but this program performed an operation which requires it.
at java.desktop/java.awt.GraphicsEnvironment.checkHeadless(GraphicsEnvironment.java:208)
at java.desktop/java.awt.Window.(Window.java:548)
at java.desktop/java.awt.Frame.(Frame.java:423)
at java.desktop/java.awt.Frame.(Frame.java:388)
at java.desktop/javax.swing.JFrame.(JFrame.java:180)
at com.dmtavt.fragpipe.FragpipeLoader.(FragpipeLoader.java:61)
at com.dmtavt.fragpipe.Fragpipe.main0(Fragpipe.java:358)
at com.dmtavt.fragpipe.FragPipeMain.main(FragPipeMain.java:40)
You need to add the --headless
flag (and other commands to set the workflow and manifest) so that FragPipe will run in headless mode. Without any flags, FragPipe runs in GUI mode.
Best,
Fengchao
Thanks again. I see my mistake!
Unfortunately export JAVA_OPTS=-Djava.io.tmpdir=/scratch/st-ljfoster-1/staceyri/isabela_fragpipe/tmp/
doesn't fix the error for me. I can see empty folders being made in the new tmp folder:
$ ls /scratch/st-ljfoster-1/staceyri/isabela_fragpipe/tmp/
af6fdd7f-da18-4dd1-b374-2cb499fafd15 d1d6cf4a-2265-4e16-bbaa-d6af9041dfab e109884b-958b-468a-b85c-4f73c44d0e59 f06836aa-9c9f-4e8c-bb02-77224a71b88a
$ ls /scratch/st-ljfoster-1/staceyri/isabela_fragpipe/tmp/af6fdd7f-da18-4dd1-b374-2cb499fafd15/
$
But the same error persists where fragpipe tries to save files in the /home/ directory:
2022-07-08 15:11:15,601 ERROR - Error saving ui cache
java.nio.file.FileSystemException: /home/staceyri/.config/FragPipe/fragpipe/fragpipe-ui.cache: Read-only file system
...
I heard back from the cluster administrators and the problem is that /home/ doesn't exist on the compute nodes. I tried naively redefining $HOME
but that also doesn't solve the issue for me. The submission script is below in case that's helpful. Again I don't see the error when just running /path/to/fragpipe
on the head node.
# change temp folder
export JAVA_OPTS=-Djava.io.tmpdir=/scratch/st-ljfoster-1/staceyri/isabela_fragpipe/tmp/
# confirm temp folder is changed
echo $JAVA_OPTS
# change home folder
HOME=/scratch/st-ljfoster-1/staceyri/
echo ~
# run fragpipe
/scratch/st-ljfoster-1/staceyri/fragpipe/bin/fragpipe --headless --workflow /scratch/st-ljfoster-1/staceyri/isabela_fragpipe/experiments/fullsearch/fullsearch.workflow --manifest /scratch/st-ljfoster-1/staceyri/isabela_fragpipe/experiments/fullsearch/fullsearch.fp-manifest --workdir /scratch/st-ljfoster-1/staceyri/isabela_fragpipe//experiments/fullsearch --ram 0 --config-msfragger /scratch/st-ljfoster-1/staceyri/fragpipe/tools/MSFragger-3.5/MSFragger-3.5.jar --config-philosopher /arc/project/st-ljfoster-1/philosopher/v4.2.2/philosopher --config-python /arc/software/spack/opt/spack/linux-centos7-x86_64/gcc-9.1.0/miniconda3-4.6.14-wsswwucnigbnge5edropjstrirpjx7je/bin/python
Thanks for your feedback.
I think I found the cause of the issue. In Linux and Unix, FragPipe always save cache to HOME
:
For your case, can you try setting a XDG_CONFIG_HOME=<temp path>
environment variable? Details of the code can be found from
Best,
Fengchao
Thanks, Fengchao. Adding the following lines to the submission script fixed the issue for me.
XDG_CONFIG_HOME="/path/to/working/directory/"
export XDG_CONFIG_HOME
Thanks for the positive feedback.
Best,
Fengchao