Docker run too-many-cells error

Question

Docker run too-many-cells error

lijiacd985 opened this issue 5 years ago · 22 comments

lijiacd985 commented 5 years ago

Hi,

I have this error immediately after run too-many-cells

too-many-cells: readCreateProcess: R "-e" "cat(R.home())" "--quiet" "--slave" (exit -9): failed

Thanks,

Best,

Jia Li

Answer 1 · 2020-04-06T17:59:20.000Z

What command did you use? What OS are you on?

Answer 2 · 2020-04-06T18:29:14.000Z

What command did you use? What OS are you on?

The command I used:
sudo docker run -m 5000000 -it --rm -v "/Users/lijia/Downloads/too-many-cells/" -v "/usr/local/bin" gregoryschwartz/too-many-cells:0.2.2.0 make-tree --matrix-path Veh.matrix.csv --labels-file labels.csv --draw-collection "PieRing" --output out

I am using macOS high Sierra version 10.13.6

Thanks

Answer 3 · 2020-04-07T13:19:31.000Z

Why /usr/local/bin? That might be playing with the R in the docker container. What if you omit that?

Answer 4 · 2020-04-07T14:00:08.000Z

i tried，the same error

…

Sent from my iPhone

On Apr 7, 2020, at 8:19 AM, Gregory Schwartz ***@***.***> wrote: Why /usr/local/bin? That might be playing with the R in the docker container. What if you omit that? — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub, or unsubscribe.

Answer 5 · 2020-04-07T14:40:18.000Z

What stage does it fail? Does it work for other data?

Answer 6 · 2020-04-07T15:29:40.000Z

After I run the command several seconds. This is my first time to run too-many-cells. I have not successfully run too-many-cells yet. The data format is the same as you request.

…

On Apr 7, 2020, at 9:40 AM, Gregory Schwartz ***@***.***> wrote: What stage does it fail? Does it work for other data? — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#19 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AG4D5HBQF2NSPWPHXJQ7RSDRLM3ODANCNFSM4MBRKXYQ>.

Answer 7 · 2020-04-07T18:07:38.000Z

What if you try the workshop (https://gregoryschwartz.github.io/too-many-cells/workshop/workshop.html)?

Answer 8 · 2020-04-07T19:09:31.000Z

I asked other people in our lab to test Docker run too-many-cells, they also have the same error.

Answer 9 · 2020-04-07T20:40:13.000Z

What if you try the workshop (https://gregoryschwartz.github.io/too-many-cells/workshop/workshop.html)?

I tried workshop under Catalina and High Sierra, both have the following errors:

While building package cairo-0.13.5.0 using:
/private/var/folders/xh/k7b18vjx6kz051t7q29r8mfsp28xc5/T/stack75478/cairo-0.13.5.0/.stack-work/dist/x86_64-osx/Cabal-2.2.0.1/setup/setup --builddir=.stack-work/dist/x86_64-osx/Cabal-2.2.0.1 configure --user --package-db=clear --package-db=global --package-db=/Users/jli/.stack/snapshots/x86_64-osx/d96d1c41d18571a54eb85ebae848040464e1476263180e538beb77700203aeb1/8.4.3/pkgdb --libdir=/Users/jli/.stack/snapshots/x86_64-osx/d96d1c41d18571a54eb85ebae848040464e1476263180e538beb77700203aeb1/8.4.3/lib --bindir=/Users/jli/.stack/snapshots/x86_64-osx/d96d1c41d18571a54eb85ebae848040464e1476263180e538beb77700203aeb1/8.4.3/bin --datadir=/Users/jli/.stack/snapshots/x86_64-osx/d96d1c41d18571a54eb85ebae848040464e1476263180e538beb77700203aeb1/8.4.3/share --libexecdir=/Users/jli/.stack/snapshots/x86_64-osx/d96d1c41d18571a54eb85ebae848040464e1476263180e538beb77700203aeb1/8.4.3/libexec --sysconfdir=/Users/jli/.stack/snapshots/x86_64-osx/d96d1c41d18571a54eb85ebae848040464e1476263180e538beb77700203aeb1/8.4.3/etc --docdir=/Users/jli/.stack/snapshots/x86_64-osx/d96d1c41d18571a54eb85ebae848040464e1476263180e538beb77700203aeb1/8.4.3/doc/cairo-0.13.5.0 --htmldir=/Users/jli/.stack/snapshots/x86_64-osx/d96d1c41d18571a54eb85ebae848040464e1476263180e538beb77700203aeb1/8.4.3/doc/cairo-0.13.5.0 --haddockdir=/Users/jli/.stack/snapshots/x86_64-osx/d96d1c41d18571a54eb85ebae848040464e1476263180e538beb77700203aeb1/8.4.3/doc/cairo-0.13.5.0 --dependency=Cabal=Cabal-2.2.0.1 --dependency=array=array-0.5.2.0 --dependency=base=base-4.11.1.0 --dependency=bytestring=bytestring-0.10.8.2 --dependency=gtk2hs-buildtools=gtk2hs-buildtools-0.13.4.0-BeTuALJn73yF2BRI2iGnJc --dependency=mtl=mtl-2.2.2 --dependency=text=text-1.2.3.0 --dependency=utf8-string=utf8-string-1.0.1.1-LRSjvSC6FZkKPp48Qszoj8 -fcairo_pdf -fcairo_ps -fcairo_svg --exact-configuration --ghc-option=-fhide-source-paths
Process exited with code: ExitFailure 1

-- While building package inline-r-0.9.2 using:
/Users/jli/.stack/setup-exe-cache/x86_64-osx/Cabal-simple_mPHDZzAJ_2.2.0.1_ghc-8.4.3 --builddir=.stack-work/dist/x86_64-osx/Cabal-2.2.0.1 configure --user --package-db=clear --package-db=global --package-db=/Users/jli/.stack/snapshots/x86_64-osx/d96d1c41d18571a54eb85ebae848040464e1476263180e538beb77700203aeb1/8.4.3/pkgdb --libdir=/Users/jli/.stack/snapshots/x86_64-osx/d96d1c41d18571a54eb85ebae848040464e1476263180e538beb77700203aeb1/8.4.3/lib --bindir=/Users/jli/.stack/snapshots/x86_64-osx/d96d1c41d18571a54eb85ebae848040464e1476263180e538beb77700203aeb1/8.4.3/bin --datadir=/Users/jli/.stack/snapshots/x86_64-osx/d96d1c41d18571a54eb85ebae848040464e1476263180e538beb77700203aeb1/8.4.3/share --libexecdir=/Users/jli/.stack/snapshots/x86_64-osx/d96d1c41d18571a54eb85ebae848040464e1476263180e538beb77700203aeb1/8.4.3/libexec --sysconfdir=/Users/jli/.stack/snapshots/x86_64-osx/d96d1c41d18571a54eb85ebae848040464e1476263180e538beb77700203aeb1/8.4.3/etc --docdir=/Users/jli/.stack/snapshots/x86_64-osx/d96d1c41d18571a54eb85ebae848040464e1476263180e538beb77700203aeb1/8.4.3/doc/inline-r-0.9.2 --htmldir=/Users/jli/.stack/snapshots/x86_64-osx/d96d1c41d18571a54eb85ebae848040464e1476263180e538beb77700203aeb1/8.4.3/doc/inline-r-0.9.2 --haddockdir=/Users/jli/.stack/snapshots/x86_64-osx/d96d1c41d18571a54eb85ebae848040464e1476263180e538beb77700203aeb1/8.4.3/doc/inline-r-0.9.2 --dependency=aeson=aeson-1.4.0.0-EAbp2GiwrvTH27nXdJzV0g --dependency=base=base-4.11.1.0 --dependency=bytestring=bytestring-0.10.8.2 --dependency=containers=containers-0.5.11.0 --dependency=data-default-class=data-default-class-0.1.2.0-2kYzERBLX3wJoPfj7mwVvW --dependency=deepseq=deepseq-1.4.3.0 --dependency=exceptions=exceptions-0.10.0-DmsI5QMvE6e6QgVkMINEKb --dependency=inline-c=inline-c-0.6.1.0-FM9gF7RqOpoLWRlok3Pud0 --dependency=mtl=mtl-2.2.2 --dependency=pretty=pretty-1.1.3.6 --dependency=primitive=primitive-0.6.3.0-DaZpcxwJp2TGn8ITSgfI4C --dependency=process=process-1.6.3.0 --dependency=reflection=reflection-2.1.4-ET4Qfoy5lmWBopRK3ezJIQ --dependency=setenv=setenv-0.1.1.3-H1xmIqlPy4yIDquO6eJhBl --dependency=singletons=singletons-2.4.1-FDzlisNNwplIrNjegYYDdD --dependency=template-haskell=template-haskell-2.13.0.0 --dependency=text=text-1.2.3.0 --dependency=th-lift=th-lift-0.7.10-88ozaMeoe8eDZSlyIjheFa --dependency=th-orphans=th-orphans-0.13.6-6mvRAE1wQLBDXpoe3PtgV3 --dependency=transformers=transformers-0.5.5.0 --dependency=unix=unix-2.7.2.2 --dependency=vector=vector-0.12.0.1-GGZqQZyzchy8YFPCF67wxL --exact-configuration --ghc-option=-fhide-source-paths
Process exited with code: ExitFailure 1

-- While building package glib-0.13.8.0 using:
/private/var/folders/xh/k7b18vjx6kz051t7q29r8mfsp28xc5/T/stack75478/glib-0.13.8.0/.stack-work/dist/x86_64-osx/Cabal-2.2.0.1/setup/setup --builddir=.stack-work/dist/x86_64-osx/Cabal-2.2.0.1 configure --user --package-db=clear --package-db=global --package-db=/Users/jli/.stack/snapshots/x86_64-osx/d96d1c41d18571a54eb85ebae848040464e1476263180e538beb77700203aeb1/8.4.3/pkgdb --libdir=/Users/jli/.stack/snapshots/x86_64-osx/d96d1c41d18571a54eb85ebae848040464e1476263180e538beb77700203aeb1/8.4.3/lib --bindir=/Users/jli/.stack/snapshots/x86_64-osx/d96d1c41d18571a54eb85ebae848040464e1476263180e538beb77700203aeb1/8.4.3/bin --datadir=/Users/jli/.stack/snapshots/x86_64-osx/d96d1c41d18571a54eb85ebae848040464e1476263180e538beb77700203aeb1/8.4.3/share --libexecdir=/Users/jli/.stack/snapshots/x86_64-osx/d96d1c41d18571a54eb85ebae848040464e1476263180e538beb77700203aeb1/8.4.3/libexec --sysconfdir=/Users/jli/.stack/snapshots/x86_64-osx/d96d1c41d18571a54eb85ebae848040464e1476263180e538beb77700203aeb1/8.4.3/etc --docdir=/Users/jli/.stack/snapshots/x86_64-osx/d96d1c41d18571a54eb85ebae848040464e1476263180e538beb77700203aeb1/8.4.3/doc/glib-0.13.8.0 --htmldir=/Users/jli/.stack/snapshots/x86_64-osx/d96d1c41d18571a54eb85ebae848040464e1476263180e538beb77700203aeb1/8.4.3/doc/glib-0.13.8.0 --haddockdir=/Users/jli/.stack/snapshots/x86_64-osx/d96d1c41d18571a54eb85ebae848040464e1476263180e538beb77700203aeb1/8.4.3/doc/glib-0.13.8.0 --dependency=Cabal=Cabal-2.2.0.1 --dependency=base=base-4.11.1.0 --dependency=bytestring=bytestring-0.10.8.2 --dependency=containers=containers-0.5.11.0 --dependency=gtk2hs-buildtools=gtk2hs-buildtools-0.13.4.0-BeTuALJn73yF2BRI2iGnJc --dependency=text=text-1.2.3.0 --dependency=utf8-string=utf8-string-1.0.1.1-LRSjvSC6FZkKPp48Qszoj8 -fclosure_signals --exact-configuration --ghc-option=-fhide-source-paths
Process exited with code: ExitFailure 1
Progress 8/45

Answer 10 · 2020-04-07T23:03:37.000Z

You should use docker with the workshop in your case.

Answer 11 · 2020-04-08T13:49:44.000Z

Is it possible that you are not giving docker enough memory? -m 16g or higher? Also, did you try a small toy example of like 5 cells with 5 genes or something like that?

Answer 12 · 2020-04-08T17:01:44.000Z

Thanks. I tried as you said. sudo docker run -m 16g -it --rm -v "/Users/lijia/Downloads/:/Users/lijia/Downloads" gregoryschwartz/too-many-cells:0.2.2.0 make-tree --matrix-path test.matrix.csv --labels-file test.label.csv --draw-collection "PieRing" --output out Loading matrix [................................................................................................] 0%too-many-cells: Matrix path does not exist. CallStack (from HasCallStack): error, called at app/Main.hs:273:36 in main:Main the current path is /Users/lijia/Downloads/too-many-cells. Thanks.

…

On Apr 8, 2020, at 8:50 AM, Gregory Schwartz ***@***.***> wrote: Is it possible that you are not giving docker enough memory? -m 16g or higher? Also, did you try a small toy example of like 5 cells with 5 genes or something like that? — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#19 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AG4D5HCIYKXBJOCFZGYFFDTRLR6IRANCNFSM4MBRKXYQ>.

Answer 13 · 2020-04-08T18:54:31.000Z

You are telling docker to mount /Users/lijia/Downloads to that location in the container, but pointing to a file in / in the container. So try:

sudo docker run -m 16g -it --rm -v "/Users/lijia/Downloads/:/Users/lijia/Downloads" gregoryschwartz/too-many-cells:0.2.2.0 make-tree --matrix-path /Users/lijia/Downloads/too-many-cells/test.matrix.csv --labels-file /Users/lijia/Downloads/too-many-cells/test.label.csv --draw-collection "PieRing" --output /Users/lijia/Downloads/too-many-cells/out

Answer 14 · 2020-04-08T20:14:13.000Z

You are telling docker to mount /Users/lijia/Downloads to that location in the container, but pointing to a file in / in the container. So try:

sudo docker run -m 16g -it --rm -v "/Users/lijia/Downloads/:/Users/lijia/Downloads" gregoryschwartz/too-many-cells:0.2.2.0 make-tree --matrix-path /Users/lijia/Downloads/too-many-cells/test.matrix.csv --labels-file /Users/lijia/Downloads/too-many-cells/test.label.csv --draw-collection "PieRing" --output /Users/lijia/Downloads/too-many-cells/out

Thank you! I can run it successfully.

Answer 15 · 2020-04-08T20:40:49.000Z

For your own data you probably need to either up the memory, mount it like you did here, or both.

Answer 16 · 2020-06-03T16:36:18.000Z

Hi again Dr. Schwartz... I now ran into the same problem as @lijiacd985 did...
I followed the instructions you suggested according to the communications above but the problem remains.

Here is my command:
sudo docker run -m 16g -it --rm -v "/d/Users/zeiss/tmcdocker/:/d/Users/zeiss/tmcdocker"
gregoryschwartz/too-many-cells:0.2.2.0 make-tree
--matrix-path /d/Users/zeiss/tmcdocker/data/brain/filtered_feature_bc_matrix/
--matrix-path /d/Users/zeiss/tmcdocker/data/heart/filtered_feature_bc_matrix/
--output /d/Users/zeiss/tmcdocker/out

Error messages:
Loading matrix [....................................................] 0%
too-many-cells:
Matrix path does not exist.
CallStack (from HasCallStack):
error, called at app/Main.hs:273:36 in main:Main

Config.:
I am using the Docker Toolbox on a Windows 7 computer.
Testing with the brain+heart data set you used in your workshop.
The current path is: /d/Users/zeiss/tmcdocker/

Looking forward to your suggestions!

Regards,
Joey

Answer 17 · 2020-06-03T17:33:29.000Z

@joyeblog Is this after you got it to work with other data? If not, then check that Windows has permissions to mount in your docker Settings under Shared Drives (or the like). Also, silly question, but your Windows mount point looks like a Unix directory, not Windows, is this correct?

Answer 18 · 2020-06-03T17:52:27.000Z

@GregorySchwartz No, I have not tried my own data yet. I was thinking to replicate your workshop results first to ensure there isn't any technical errors before moving on to my sequencing dataset.

Everything is run in the Oracle Virtual Box, I gave the permission to mount on D:\Users\zeiss\tmcdocker according to the instructions from my issue #26 . Please see the attached screenshot from the Oracle Virtual Box.

Interesting question... Is the path format for Windows different from the one I had above? I tried the following command (Windows format path):
sudo docker run -m 16g -it --rm -v "D:/Users/zeiss/tmcdocker:/D:/Users/zeiss/tmcdocker"
gregoryschwartz/too-many-cells:0.2.2.0 make-tree
--matrix-path /D:/Users/zeiss/tmcdocker/data/brain/filtered_feature_bc_matrix/

And got error message:
Error response from daemon: invalid volume specification: 'D:/Users/zeiss/tmcdocker:/D:/Users/zeiss/tmcdocker'. See 'docker run --help'.

Sorry for keep bothering you with basic and silly questions.. Thanks for your time!

@joyeblog Is this after you got it to work with other data? If not, then check that Windows has permissions to mount in your docker Settings under Shared Drives (or the like). Also, silly question, but your Windows mount point looks like a Unix directory, not Windows, is this correct?

Answer 19 · 2020-06-03T18:51:10.000Z

In the volume argument, before the : is the host and after is the container. The container has a linux filesystem, so you don't want to repeat the invalid path. So you should have that first part, but change the second to whatever you want and have another volume argument pointing to the output folder on the container mapped to the host.

Answer 20 · 2020-06-03T19:11:51.000Z

Thanks again for the quick response! Honestly it sounds a bit too technical for me. Would you please use an example? Let's say,
The current path on docker-machine: /home/docker
The current path mounted on the Windows system: D:/Users/zeiss/tmcdocker
The folder I want all the output files in is also: D:/Users/zeiss/tmcdocker

In this case, what exactly should I write inside the quotation mark below?
sudo docker run -m 16g -it --rm -v "???"
gregoryschwartz/too-many-cells:0.2.2.0 make-tree
--matrix-path ./data/brain/filtered_feature_bc_matrix/
--matrix-path ./data/heart/filtered_feature_bc_matrix/
--output out

Thanks,
Joey

In the volume argument, before the : is the host and after is the container. The container has a linux filesystem, so you don't want to repeat the invalid path. So you should have that first part, but change the second to whatever you want and have another volume argument pointing to the output folder on the container mapped to the host.

Answer 21 · 2020-06-08T13:31:19.000Z

I don't know what your file system looks like, but maybe something like

-v "D:/Users/zeiss/tmcdocker:/work"
gregoryschwartz/too-many-cells:0.2.2.0 make-tree
--matrix-path /work/data/brain/filtered_feature_bc_matrix/
--matrix-path /work/data/heart/filtered_feature_bc_matrix/
--output /work/out

Answer 22 · 2020-06-19T14:22:22.000Z

Thanks a lot! I will give a try.