Run example got ReadOnlyMemoryError
Closed this issue · 2 comments
chengchingwen commented
hi, I try to run the example in this repo, but got ReadOnlyMemoryError().
I'm using julia v1.2 with MPI v0.10 and CuArrays v1.2.1
here is the error log:
root@e0d8932c3ee9:/data/Julia-MPI-CuArray# mpirun -np 2 julia 02-broadcast.jl
------------------------------------------------------------------------------
Running on 2 processes
------------------------------------------------------------------------------
ERROR: LoadError: ReadOnlyMemoryError()
Stacktrace:
[1] Bcast!(::CuArray{Float64,2}, ::Int64, ::Int64, ::MPI.Comm) at /root/.julia/packages/MPI/RzbIV/src/collective.jl:28
[2] Bcast! at /root/.julia/packages/MPI/RzbIV/src/collective.jl:35 [inlined]
[3] do_broadcast() at /data/Julia-MPI-CuArray/02-broadcast-impl.jl:30
[4] main() at /data/Julia-MPI-CuArray/02-broadcast.jl:13
[5] top-level scope at /data/Julia-MPI-CuArray/02-broadcast.jl:16
[6] include at ./boot.jl:328 [inlined]
[7] include_relative(::Module, ::String) at ./loading.jl:1094
[8] include(::Module, ::String) at ./Base.jl:31
[9] exec_options(::Base.JLOptions) at ./client.jl:295
[10] _start() at ./client.jl:464
in expression starting at /data/Julia-MPI-CuArray/02-broadcast.jl:16
kose-y commented
What version of MPI (not MPI.jl) are you using? And is it compiled with CUDA-aware MPI enabled?
It's probably because the MPI is not compiled with CUDA-aware support.
For example, OpenMPI should be built this way.
chengchingwen commented
I'm using openmpi-3.1.2. Actually, I use the docker image of uber/horovod : 0.15.2-tf1.12.0-torch1.0.0-py3.5 and install julia in it, But looks like it did not build the mpi with CUDA-aware support. I'll try to build it and test again, thanks.