ezaai calculate issue: temp directories being created read only, java errors out when temp files can't be deleted

Question

ezaai calculate issue: temp directories being created read only, java errors out when temp files can't be deleted

Opened this issue a year ago · 5 comments

I've been trying to run this software on a test dataset. I can convert my protein sequence files to ezaai databases, no trouble. But when I try to actually calculate the AAI :
java -jar ~/Downloads/EzAAI.jar calculate -t 64 -i EzAAI_dbs/CP040925.1_NCHU2014.db -j EzAAI_dbs/CP097207.1_PR08.db -o EzAAI_test_output

I get:
[MAR 07 14:42:18] EzAAI |: EzAAI - v1.2.3 [Feb. 2024] [MAR 07 14:42:18] EzAAI |: Calculating AAI... [Task 1/1] org.apache.commons.io.IOExceptionList: /tmp/ezaai/b72f86d85f7e7f28_tmp at org.apache.commons.io.FileUtils.cleanDirectory(FileUtils.java:331) at org.apache.commons.io.FileUtils.deleteDirectory(FileUtils.java:1192) at leb.process.ProcCalcPairwiseAAI.pairwiseMmseqs(ProcCalcPairwiseAAI.java:399) at leb.process.ProcCalcPairwiseAAI.calculateProteomePairWithDetails(ProcCalcPairwiseAAI.java:67) at leb.main.EzAAI.runCalculate(EzAAI.java:511) at leb.main.EzAAI.run(EzAAI.java:689) at leb.main.EzAAI.main(EzAAI.java:725) Caused by: java.io.IOException: Cannot delete file: /tmp/ezaai/b72f86d85f7e7f28_tmp/9614918276263326714 at org.apache.commons.io.FileUtils.forceDelete(FileUtils.java:1344) at org.apache.commons.io.FileUtils.cleanDirectory(FileUtils.java:324) ... 6 more Caused by: java.nio.file.AccessDeniedException: /tmp/ezaai/b72f86d85f7e7f28_tmp/9614918276263326714/pref_0.dbtype at java.base/sun.nio.fs.UnixException.translateToIOException(UnixException.java:90) at java.base/sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:111) at java.base/sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:116) at java.base/sun.nio.fs.UnixFileSystemProvider.implDelete(UnixFileSystemProvider.java:249) at java.base/sun.nio.fs.AbstractFileSystemProvider.deleteIfExists(AbstractFileSystemProvider.java:110) at java.base/java.nio.file.Files.deleteIfExists(Files.java:1181) at org.apache.commons.io.file.DeletingPathVisitor.visitFile(DeletingPathVisitor.java:161) at org.apache.commons.io.file.DeletingPathVisitor.visitFile(DeletingPathVisitor.java:37) at java.base/java.nio.file.Files.walkFileTree(Files.java:2725) at java.base/java.nio.file.Files.walkFileTree(Files.java:2797) at org.apache.commons.io.file.PathUtils.visitFileTree(PathUtils.java:971) at org.apache.commons.io.file.PathUtils.deleteDirectory(PathUtils.java:434) at org.apache.commons.io.file.PathUtils.delete(PathUtils.java:391) at org.apache.commons.io.FileUtils.forceDelete(FileUtils.java:1341)

Checking the /tmp directory, the directory created (/tmp/ezaai/b72f86d85f7e7f28_tmp/9614918276263326714 in this case) has permissions:
dr-xr-xr-x. 2 myusername myusername 260 Mar 7 14:42 9614918276263326714

So readable and executable, but not writeable, which extends to the files it contains. The folder containing this folder has write permissions just fine, as does the other folder active when the software errors out (the _faa folder). Any ideas?

OS is RockyLinux 9.3

Answer 1 · 2024-03-12T03:11:30.000Z

I suppose your OS doesn't automatically provide write permission to the files created under /tmp/ezaai for some reason.

In the latest version you can provide a custom directory by passing the path with -tmp argument. Could you please run the same command with a directory you certainly own (such as ~/tmp) given as a temp directory?

Answer 2 · 2024-03-12T12:24:25.000Z

Thank you, the process completed no problem specifying my SSD scratch disk as -tmp. Just double checking the system /tmp, some files generated by EzAAI have write access, and some don't. Probably a bug if you get an opportunity to look at it, but the -tmp workaround works great. Thanks!

Answer 3 · 2024-05-10T07:06:56.000Z

Hi @endixk

I alson met this same kind of error even using the -tmp option to set a own directory rather than /tmp. Here is my CML and error messages:

$ /anaconda3/envs/ezaai/bin/EzAAI calculate -i ./EZAAI.test/1806/faa -j ./EZAAI.test/1806/faa -o ./EZAAI.test/1806/faa/ezaai.txt -tmp ./EZAAI.test/1806/tmp -cov 0.95 -id 0.95 -mmseqs /anaconda3/envs/ezaai/bin/mmseqs -t 2
[MAY 10 14:53:35] EzAAI |: EzAAI - v1.2.3 [Feb. 2024]
[MAY 10 14:53:37] EzAAI |: Calculating AAI... [Task 1/9]
[MAY 10 14:54:27] EzAAI |: Calculating AAI... [Task 2/9]
[MAY 10 14:55:16] EzAAI |: Calculating AAI... [Task 3/9]
[MAY 10 14:56:05] EzAAI |: Calculating AAI... [Task 4/9]
[MAY 10 14:56:53] EzAAI |: Calculating AAI... [Task 5/9]
org.apache.commons.io.IOExceptionList: ./EZAAI.test/1806/tmp/d71e082929be4d5f_tmp
at org.apache.commons.io.FileUtils.cleanDirectory(FileUtils.java:331)
at org.apache.commons.io.FileUtils.deleteDirectory(FileUtils.java:1192)
at leb.process.ProcCalcPairwiseAAI.pairwiseMmseqs(ProcCalcPairwiseAAI.java:399)
at leb.process.ProcCalcPairwiseAAI.calculateProteomePairWithDetails(ProcCalcPairwiseAAI.java:67)
at leb.main.EzAAI.runCalculate(EzAAI.java:511)
at leb.main.EzAAI.run(EzAAI.java:689)
at leb.main.EzAAI.main(EzAAI.java:725)
Caused by: java.io.IOException: Cannot delete file: ./EZAAI.test/1806/tmp/d71e082929be4d5f_tmp/9810936425439357669
at org.apache.commons.io.FileUtils.forceDelete(FileUtils.java:1344)
at org.apache.commons.io.FileUtils.cleanDirectory(FileUtils.java:324)
... 6 more
Caused by: java.nio.file.AccessDeniedException: ./EZAAI.test/1806/tmp/d71e082929be4d5f_tmp/9810936425439357669/pref_0.index
at sun.nio.fs.UnixException.translateToIOException(UnixException.java:84)
at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:102)
at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:107)
at sun.nio.fs.UnixFileSystemProvider.implDelete(UnixFileSystemProvider.java:244)
at sun.nio.fs.AbstractFileSystemProvider.deleteIfExists(AbstractFileSystemProvider.java:108)
at java.nio.file.Files.deleteIfExists(Files.java:1165)
at org.apache.commons.io.file.DeletingPathVisitor.visitFile(DeletingPathVisitor.java:161)
at org.apache.commons.io.file.DeletingPathVisitor.visitFile(DeletingPathVisitor.java:37)
at java.nio.file.Files.walkFileTree(Files.java:2670)
at java.nio.file.Files.walkFileTree(Files.java:2742)
at org.apache.commons.io.file.PathUtils.visitFileTree(PathUtils.java
at org.apache.commons.io.file.PathUtils.deleteDirectory(PathUtils.java:434)
at org.apache.commons.io.file.PathUtils.delete(PathUtils.java:391)
at org.apache.commons.io.FileUtils.forceDelete(FileUtils.java:1341)
... 7 more
[MAY 10 14:57:41] ERROR |: Program terminated with error.

I think there is a bug in the -tmp option that causes permission problems when deleting temporary directories.

Answer 4 · 2024-06-03T02:09:07.000Z

Hi, sorry for the late reply.

This doesn't seem to have a consistent circumstance in which the error occurs.

Until I find the exact reason, as a temporary measure, I will soon develop a fallback route where failed deletion attempt emits a warning message without terminating the process.

Answer 5 · 2024-08-08T22:01:19.000Z

Just as a note, we're getting the same above issue, I believe - Trying to calculate aai as follows, following install into the ezaai conda environment as per instructions. Command we used:

ezaai calculate -i Unknown_Rickettsia.db -j rickettsia_db -tmp ~/tmp -t 8 -o rickettsia_aai.tsv

See command and output below:
(ezaai) [usr.name@node.number ~]$ ezaai calculate -i Unknown_Rickettsia.db -j rickettsia_db -tmp ~/tmp -t 8 -o rickettsia_aai.tsv
[AUG 08 15:52:53] EzAAI |: EzAAI - v1.2.3 [Feb. 2024]
[AUG 08 15:53:39] EzAAI |: Calculating AAI... [Task 1/167]
org.apache.commons.io.IOExceptionList: /home/usr.name/tmp/d33d000bb039ea62_tmp
at org.apache.commons.io.FileUtils.cleanDirectory(FileUtils.java:331)
at org.apache.commons.io.FileUtils.deleteDirectory(FileUtils.java:1192)
at leb.process.ProcCalcPairwiseAAI.pairwiseMmseqs(ProcCalcPairwiseAAI.java:399)
at leb.process.ProcCalcPairwiseAAI.calculateProteomePairWithDetails(ProcCalcPairwiseAAI.java:67)
at leb.main.EzAAI.runCalculate(EzAAI.java:511)
at leb.main.EzAAI.run(EzAAI.java:689)
at leb.main.EzAAI.main(EzAAI.java:725)
Caused by: java.io.IOException: Cannot delete file: /home/usr.name/tmp/d33d000bb039ea62_tmp/3326775743774279144
at org.apache.commons.io.FileUtils.forceDelete(FileUtils.java:1344)
at org.apache.commons.io.FileUtils.cleanDirectory(FileUtils.java:324)
... 6 more
Caused by: java.nio.file.AccessDeniedException: /home/usr.name/tmp/d33d000bb039ea62_tmp/3326775743774279144/blastp.sh
at java.base/sun.nio.fs.UnixException.translateToIOException(UnixException.java:90)
at java.base/sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:106)
at java.base/sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:111)
at java.base/sun.nio.fs.UnixFileSystemProvider.implDelete(UnixFileSystemProvider.java:248)
at java.base/sun.nio.fs.AbstractFileSystemProvider.deleteIfExists(AbstractFileSystemProvider.java:110)
at java.base/java.nio.file.Files.deleteIfExists(Files.java:1191)
at org.apache.commons.io.file.DeletingPathVisitor.visitFile(DeletingPathVisitor.java:161)
at org.apache.commons.io.file.DeletingPathVisitor.visitFile(DeletingPathVisitor.java:37)
at java.base/java.nio.file.Files.walkFileTree(Files.java:2811)
at java.base/java.nio.file.Files.walkFileTree(Files.java:2882)
at org.apache.commons.io.file.PathUtils.visitFileTree(PathUtils.java:971)
at org.apache.commons.io.file.PathUtils.deleteDirectory(PathUtils.java:434)
at org.apache.commons.io.file.PathUtils.delete(PathUtils.java:391)
at org.apache.commons.io.FileUtils.forceDelete(FileUtils.java:1341)
... 7 more
[AUG 08 15:53:49] ERROR |: Program terminated with error.