PIPELINE TEST w/ M. genitalium - PermissionError: [Errno 13] Permission denied: '/pgap/output/debug/tmpdir

Question

PIPELINE TEST w/ M. genitalium - PermissionError: [Errno 13] Permission denied: '/pgap/output/debug/tmpdir

raysully opened this issue a year ago · 38 comments

Trying to run the pipeline test w/ M. genitalium and get: PermissionError: [Errno 13] Permission denied: '/pgap/output/debug/tmpdir

OS: Ubuntu 20.04.6 LTS
PGAP version 2023-10-03.build7061
Docker version 24.0.7, build afdd53b

cwltool.log follows:

cwltool.log

Thanks for your support!

Answer 1 · 2023-12-13T16:49:20.000Z

Thanks for your report, Ray!

Could you please post a listing of your mg37_results/ directory?

Answer 2 · 2023-12-13T19:42:09.000Z

Thanks for yr quick response! I sent mg37_results/cwltool.log. The only other dir is mg37_results/debug/log, but it's completely empty.

Answer 3 · 2023-12-13T20:03:41.000Z

Thanks, but all I need is a listing, as in ls -alR

Answer 4 · 2023-12-13T20:07:51.000Z

Sorry. Here it is:

(base) gene@precision6:~/mg37_results$ ls -alR
.:
total 20
drwxrwxr-x 3 gene gene 4096 Dec 13 10:51 .
drwxr-xr-x 81 gene gene 4096 Dec 13 10:51 ..
-rw-rw-r-- 1 gene gene 3108 Dec 13 10:51 cwltool.log
drwxrwxr-x 3 gene gene 4096 Dec 13 10:51 debug
-rw-rw-r-- 1 gene gene 74 Dec 13 10:51 .~lock.cwltool.log#

./debug:
total 12
drwxrwxr-x 3 gene gene 4096 Dec 13 10:51 .
drwxrwxr-x 3 gene gene 4096 Dec 13 10:51 ..
drwxrwxr-x 2 gene gene 4096 Dec 13 10:51 log

./debug/log:
total 8
drwxrwxr-x 2 gene gene 4096 Dec 13 10:51 .
drwxrwxr-x 3 gene gene 4096 Dec 13 10:51 ..
(base) gene@precision6:~/mg37_results$

Answer 5 · 2023-12-13T20:12:27.000Z

Thanks!

Looks like the permissions are fine. Could you please try to run this command and post the response?

/usr/local/bin/docker run -i --rm --user 1000:1000 \
   --volume /home/gene/.pgap/input-2023-10-03.build7061:/pgap/input:ro,z \
   --volume /home/gene:/pgap/user_input:z \
   --volume /home/gene/pgap_input__o732xtc.yaml:/pgap/user_input/pgap_input.yaml:ro,z \
   --volume /home/gene/mytmp:/tmp:rw,z \
   --volume /home/gene/mg37_results:/pgap/output:rw,z \
   --volume /home/gene/mg37_results/debug/log:/log/srv:z \
   ncbi/pgap:2023-10-03.build7061 \
 mkdir /pgap/output/debug/tmpdir

Answer 6 · 2023-12-13T20:15:48.000Z

My bad, command line fixed now.

Answer 7 · 2023-12-13T20:17:21.000Z

(base) gene@precision6:~/mg37_results$ /usr/local/bin/docker run -i --rm --user 1000:1000 \

Answer 8 · 2023-12-13T21:09:17.000Z

Thanks, how about this one:

/usr/local/bin/docker run -i --rm --user 1000:1000 \
   --volume /home/gene/.pgap/input-2023-10-03.build7061:/pgap/input:ro,z \
   --volume /home/gene:/pgap/user_input:z \
   --volume /home/gene/mytmp:/tmp:rw,z \
   --volume /home/gene/mg37_results:/pgap/output:rw,z \
   --volume /home/gene/mg37_results/debug/log:/log/srv:z \
   ncbi/pgap:2023-10-03.build7061 \
 mkdir /pgap/output/debug/tmpdir

Answer 9 · 2023-12-13T21:11:35.000Z

(base) gene@precision6:~/mg37_results$ /usr/local/bin/docker run -i --rm --user 1000:1000 \

Answer 10 · 2023-12-13T21:29:50.000Z

Thanks. Could you please run these commands?

/usr/local/bin/docker run -i --rm --user 1000:1000 \
   --volume /home/gene/.pgap/input-2023-10-03.build7061:/pgap/input:ro,z \
   --volume /home/gene:/pgap/user_input:z \
   --volume /home/gene/mytmp:/tmp:rw,z \
   --volume /home/gene/mg37_results:/pgap/output:rw,z \
   --volume /home/gene/mg37_results/debug/log:/log/srv:z \
   ncbi/pgap:2023-10-03.build7061 \
 ls -lanR /pgap/output/debug/

this:

/usr/local/bin/docker run -i --rm --user 1000:1000 \
   --volume /home/gene/.pgap/input-2023-10-03.build7061:/pgap/input:ro,z \
   --volume /home/gene:/pgap/user_input:z \
   --volume /home/gene/mytmp:/tmp:rw,z \
   --volume /home/gene/mg37_results:/pgap/output:rw,z \
   --volume /home/gene/mg37_results/debug/log:/log/srv:z \
   ncbi/pgap:2023-10-03.build7061 \
  id -u

and

/usr/local/bin/docker run -i --rm --user 1000:1000 \
   --volume /home/gene/.pgap/input-2023-10-03.build7061:/pgap/input:ro,z \
   --volume /home/gene:/pgap/user_input:z \
   --volume /home/gene/mytmp:/tmp:rw,z \
   --volume /home/gene/mg37_results:/pgap/output:rw,z \
   --volume /home/gene/mg37_results/debug/log:/log/srv:z \
   ncbi/pgap:2023-10-03.build7061 \
  id -g

Answer 11 · 2023-12-13T21:32:54.000Z

(base) gene@precision6:~$ /usr/local/bin/docker run -i --rm --user 1000:1000 \

/pgap/output/debug/log:
total 8
drwxrwxr-x 2 0 0 4096 Dec 13 15:51 .
drwxrwxr-x 3 0 0 4096 Dec 13 15:51 ..
(base) gene@precision6:~$ /usr/local/bin/docker run -i --rm --user 1000:1000 \

Answer 12 · 2023-12-13T21:38:48.000Z

Thanks, Ray. Now we are talking.

Now I see what the problem is. docker for some reason does not recognize --user 1000:1000 setting on your machine for user and group ids and uses straight root/root (0/0) settings.

This is peculiar. Normally --user settings are recognized. I am going to research this on the Internet and I suggest that maybe you can request help of your local system folks to clarify what is going on with your docker installation

Answer 13 · 2023-12-13T21:42:55.000Z

Thanks for your help. Unfortunately, It's my own machine and I installed docker myself so you're my only lifeline for this and I'm a novice at this.

Answer 14 · 2023-12-14T09:57:12.000Z

No problem. I opened an internal ticket, now that we identified the specific problem with your installation. The information that it is your personal home computer is useful.

Answer 15 · 2023-12-15T16:21:57.000Z

Thanks Azat! I phoned a friend about this and he noticed the docker id was not showing up and had me restart. It's now showing up:
(base) gene@precision6:~$ id
uid=1000(gene) gid=1000(gene) groups=1000(gene),4(adm),24(cdrom),27(sudo),30(dip),46(plugdev),120(lpadmin),132(lxd),133(sambashare),997(docker),998(minknow)

However, when I reran the M. genitalium test it still errors out, bottom lines follow:
"sqn": {
"location": "file:///var/tmp/t7v5wzep/annot.sqn",
"basename": "annot.sqn",
"nameroot": "annot",
"nameext": ".sqn",
"class": "File",
"checksum": "sha1$92a7bf56b7e6b66069d38c76b84197c7e97777e2",
"size": 5411919,
"http://commonwl.org/cwltool#generation": 0
}
}
[2023-12-13 23:31:40] DEBUG [job Generate_Annotation_Reports_gff_enhanced] Removing input staging directory /var/tmp/ijibnx6m
[2023-12-13 23:31:40] DEBUG [job Generate_Annotation_Reports_gff_enhanced] Removing temporary directory /var/tmp/pgx8bx5_
[2023-12-13 23:31:40] DEBUG Moving /var/tmp/6ifwcjq2/calls.tab to /pgap/output/calls.tab
[2023-12-13 23:31:40] ERROR Unhandled error:
[Errno 13] Permission denied: '/pgap/output/calls.tab'
Traceback (most recent call last):
File "/opt/python-3.9/lib/python3.9/shutil.py", line 825, in move
os.rename(src, real_dst)
OSError: [Errno 18] Invalid cross-device link: '/var/tmp/6ifwcjq2/calls.tab' -> '/pgap/output/calls.tab'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/root/venv/lib/python3.9/site-packages/cwltool/main.py", line 1295, in main
(out, status) = real_executor(
File "/root/venv/lib/python3.9/site-packages/cwltool/executors.py", line 61, in call
return self.execute(process, job_order_object, runtime_context, logger)
File "/root/venv/lib/python3.9/site-packages/cwltool/executors.py", line 147, in execute
self.final_output[0] = relocateOutputs(
File "/root/venv/lib/python3.9/site-packages/cwltool/process.py", line 383, in relocateOutputs
stage_files(pm, stage_func=_relocate, symlink=False, fix_conflicts=True)
File "/root/venv/lib/python3.9/site-packages/cwltool/process.py", line 281, in stage_files
stage_func(entry.resolved, entry.target)
File "/root/venv/lib/python3.9/site-packages/cwltool/process.py", line 356, in _relocate
shutil.move(src, dst)
File "/opt/python-3.9/lib/python3.9/shutil.py", line 845, in move
copy_function(src, real_dst)
File "/opt/python-3.9/lib/python3.9/shutil.py", line 444, in copy2
copyfile(src, dst, follow_symlinks=follow_symlinks)
File "/opt/python-3.9/lib/python3.9/shutil.py", line 266, in copyfile
with open(dst, 'wb') as fdst:
PermissionError: [Errno 13] Permission denied: '/pgap/output/calls.tab'

full .zip attached:
cwltool20231213.zip

The --debug is the same as before: cwltool.log

Answer 16 · 2023-12-18T10:16:23.000Z

I am glad that docker problem was resolved so easily, Ray!

Now to this beast:

I would suggest to point TMPDIR envar to a directory in the same disk as your work directory.

For example:


mkdir -p tmpdir
export TMPDIR=$(readlink -f tmpdir)

and then try to run MG37 example again.

Answer 17 · 2023-12-18T14:27:34.000Z

Azat, reran MG37 AND output is exactly the same as the my post 3 days ago.

Answer 18 · 2023-12-18T14:43:56.000Z

Hmm. "Exactly" immediately makes me think of the same cause: Docker shenanigans. Could you please do the id test for Docker you did before you found necessity to restart it?

Answer 19 · 2023-12-18T14:46:35.000Z

A workaround might be just running this thing as root to use your personal machine owner privilege.

Answer 20 · 2023-12-18T15:25:31.000Z

It's trying to reinstall everything at /root/. I stopped it. I can't afford the space of another instance of pgap.

Answer 21 · 2023-12-18T17:42:50.000Z

You have a massive amount of memory, Ray, seems that this should come with analogously massive amount of space. But you have only

"work disk space (GiB)": 571.5

Anyway, I would suggest to ditch the first instance then. But this is actually Plan B. Plan A is still to fix your docker installation. Could you please test what I asked you earlier?

Answer 22 · 2023-12-18T18:22:16.000Z

Thanks, Azat. Installng on root now and will test.

Answer 23 · 2023-12-18T19:01:22.000Z

I see you are going with Plan B, which seems to be more straightforward.

Answer 24 · 2023-12-18T19:05:37.000Z

Trying to figure out how to get docker running on root. Installation of pgap failed because docker was only running on gene (my user account). Do I have to install another instance of docker on root?

Answer 25 · 2023-12-18T19:07:51.000Z

Probably. Typically software of this kinds is installed by "admin" so more than user can use it. Since this is your personal station, this rule seems to be less obvious, but it makes sense to install it as "root".

Answer 26 · 2023-12-19T13:57:16.000Z

Tried to install docker at root, could not get it to install properly. The desktop came up, but could not get it to turn on at root. Tried sharing dir /root in docker, but it made no difference. Reinstalled docker as user (gene). Reran MG37 AND output is exactly the same as the my post 4 days ago.

(base) gene@precision6:~$ id
uid=1000(gene) gid=1000(gene) groups=1000(gene),4(adm),24(cdrom),27(sudo),30(dip),46(plugdev),120(lpadmin),132(lxd),133(sambashare),997(docker),998(minknow)

Answer 27 · 2023-12-19T14:11:14.000Z

Another option to try is make the top of your directory "rw" for everybody: chmod -R a+rw .

Answer 28 · 2023-12-19T15:53:04.000Z

Still doesn't seem to be able to make dir pgap?

(base) gene@precision6:$ sudo chmod -R a+rw /home
[sudo] password for gene:
(base) gene@precision6:$ ./pgap.py -r -o mg37_results -g $HOME/.pgap/test_genomes/MG37/ASM2732v1.annotation.nucleotide.1.fasta -s 'Mycoplasmoides genitalium' --debug
PGAP version 2023-10-03.build7061 is up to date.
Output will be placed in: /home/gene/mg37_results
PGAP failed, docker exited with rc = 1
Unable to find error in log file.
(base) gene@precision6:~$

cwltool.log

Answer 29 · 2023-12-19T16:16:49.000Z

Could you please post output of this commands again?

Answer 30 · 2023-12-19T16:33:34.000Z

(base) gene@precision6:~$ /usr/local/bin/docker run -i --rm --user 1000:1000 \

/pgap/output/debug/log:
total 8
drwxrwxr-x 2 0 0 4096 Dec 19 15:35 .
drwxrwxr-x 3 0 0 4096 Dec 19 15:35 ..
(base) gene@precision6:~$

(base) gene@precision6:~$ /usr/local/bin/docker run -i --rm --user 1000:1000 \

(base) gene@precision6:~$ /usr/local/bin/docker run -i --rm --user 1000:1000 \

Answer 31 · 2023-12-19T16:44:24.000Z

Thanks.

Notice that for "all" the permissions are still read-only under docker. Let's double check if chmod -R ... you did worked.

could you please run ls -lanR mg37_results (that is, sans docker)?

Answer 32 · 2023-12-19T16:48:13.000Z

(base) gene@precision6:~$ ls -lanR mg37_results
mg37_results:
total 16
drwxrwxr-x 3 1000 1000 4096 Dec 19 10:49 .
drwxrwxrwx 88 1000 1000 4096 Dec 19 10:35 ..
-rw-rw-r-- 1 1000 1000 3107 Dec 19 10:35 cwltool.log
drwxrwxr-x 3 1000 1000 4096 Dec 19 10:35 debug

mg37_results/debug:
total 12
drwxrwxr-x 3 1000 1000 4096 Dec 19 10:35 .
drwxrwxr-x 3 1000 1000 4096 Dec 19 10:49 ..
drwxrwxr-x 2 1000 1000 4096 Dec 19 10:35 log

mg37_results/debug/log:
total 8
drwxrwxr-x 2 1000 1000 4096 Dec 19 10:35 .
drwxrwxr-x 3 1000 1000 4096 Dec 19 10:35 ..
(base) gene@precision6:~$

Answer 33 · 2023-12-19T16:50:56.000Z

Thanks. I know what is happening. chmod -R a... worked, but we also need to set this for the future file creations. Could you please execute umask 000 and run PGAP again?

Thanks!

Answer 34 · 2023-12-19T17:24:20.000Z

YOU'VE DONE IT! Thanks very much, Azat!

(base) gene@precision6:$ ./pgap.py -r -o mg37_results -g $HOME/.pgap/test_genomes/MG37/ASM2732v1.annotation.nucleotide.1.fasta -s 'Mycoplasmoides genitalium' --debug
PGAP version 2023-10-03.build7061 is up to date.
Output will be placed in: /home/gene/mg37_results
PGAP completed successfully.
(base) gene@precision6:$

Answer 35 · 2023-12-19T18:02:29.000Z

You are welcome, Ray. But please have in mind that this was a hacky way, a correct way is to set up docker so it will allow you to recognize user id/group settings from a user outside docker container.

Answer 36 · 2023-12-19T18:04:42.000Z

OK thanks. I'll have to read up on docker.

Answer 37 · 2023-12-28T16:52:15.000Z

Thanks for your help Azat!

Answer 38 · 2024-01-01T15:55:04.000Z

You are welcome, Ray!