'utf-8' codec can't decode byte 0xe8 in position 0: invalid continuation bytereturn collect_results_cpu(results, size, tmpdir)
xu19971109 opened this issue · 2 comments
Prerequisite
- I have searched Issues and Discussions but cannot get the expected help.
- The bug has not been fixed in the latest version(https://github.com/open-mmlab/mmengine).
Environment
The torch is from nvidia offical docker(nvcr.io/nvidia/pytorch:20.04-py3, torch_version='1.12.0a0+bd13bc6').
environment:
OrderedDict([
('sys.platform', 'linux'),
('Python', '3.8.13 | packaged by conda-forge | (default, Mar 25 2022, 06:04:10) [GCC 10.3.0]'),
('CUDA available', True),
('numpy_random_seed', 2147483648),
('GPU 0,1,2,3,4,5,6,7', 'NVIDIA GeForce RTX 3090'),
('CUDA_HOME', '/usr/local/cuda'),
('NVCC', 'Cuda compilation tools, release 11.6, V11.6.124'),
('GCC', 'gcc (Ubuntu 9.4.0-1ubuntu1~20.04.1) 9.4.0'),
('PyTorch', '1.12.0a0+bd13bc6'),
('PyTorch compiling details', 'PyTorch built with:\n
- GCC 9.4\n
- C++ Version: 201402\n
- Intel(R) Math Kernel Library Version 2019.0.5 Product Build 20190808 for Intel(R) 64 architecture applications\n
- Intel(R) MKL-DNN v2.5.2 (Git Hash N/A)\n
- OpenMP 201511 (a.k.a. OpenMP 4.5)\n
- LAPACK is enabled (usually provided by MKL)\n
- NNPACK is enabled\n
- CPU capability usage: AVX2\n
- CUDA Runtime 11.6\n
- NVCC architecture flags:
-gencode;arch=compute_52,code=sm_52;
-gencode;arch=compute_60,code=sm_60;
-gencode;arch=compute_61,code=sm_61;
-gencode;arch=compute_70,code=sm_70;
-gencode;arch=compute_75,code=sm_75;
-gencode;arch=compute_80,code=sm_80;
-gencode;arch=compute_86,code=sm_86;
-gencode;arch=compute_86,code=compute_86\n
- CuDNN 8.4\n
- Magma 2.5.2\n
- Build settings: - BLAS_INFO=mkl, BUILD_TYPE=Release, CUDA_VERSION=11.6, CUDNN_VERSION=8.4.0, CXX_COMPILER=/usr/bin/c++, CXX_FLAGS=-fno-gnu-unique -fvisibility-inlines-hidden -DUSE_PTHREADPOOL -fopenmp -DNDEBUG -DUSE_KINETO -DUSE_FBGEMM -DUSE_QNNPACK -DUSE_PYTORCH_QNNPACK -DUSE_XNNPACK -DSYMBOLICATE_MOBILE_DEBUG_HANDLE -DEDGE_PROFILER_USE_KINETO -O2 -fPIC -Wno-narrowing -Wall -Wextra -Werror=return-type -Wno-missing-field-initializers -Wno-type-limits -Wno-array-bounds -Wno-unknown-pragmas -Wno-sign-compare -Wno-unused-parameter -Wno-unused-function -Wno-unused-result -Wno-unused-local-typedefs -Wno-strict-overflow -Wno-strict-aliasing -Wno-error=deprecated-declarations -Wno-stringop-overflow -Wno-psabi -Wno-error=pedantic -Wno-error=redundant-decls -Wno-error=old-style-cast -fdiagnostics-color=always -faligned-new -Wno-unused-but-set-variable -Wno-maybe-uninitialized -fno-math-errno -fno-trapping-math -Werror=format -Werror=cast-function-type -Wno-stringop-overflow, LAPACK_INFO=mkl, PERF_WITH_AVX=1, PERF_WITH_AVX2=1, PERF_WITH_AVX512=1, TORCH_VERSION=1.12.0, USE_CUDA=ON, USE_CUDNN=ON, USE_EXCEPTION_PTR=1, USE_GFLAGS=OFF, USE_GLOG=OFF, USE_MKL=ON, USE_MKLDNN=OFF, USE_MPI=ON, USE_NCCL=ON, USE_NNPACK=ON, USE_OPENMP=ON, USE_ROCM=OFF, \n'),
('TorchVision', '0.13.0a0'),
('OpenCV', '3.4.11'),
('MMEngine', '0.7.3')])
Reproduces the problem - code sample
It comes from Training and testing on multiple GPUs and multiple machines
tmpdir = dir_tensor.numpy().tobytes().decode().rstrip()
Reproduces the problem - command or script
It showed up when val at 900/4630 iters.
The start script is:
export NCCL_SOCKET_IFNAME=xxxxxxx
export GLOO_SOCKET_IFNAME=xxxxxxx
export NCCL_DEBUG=INFO
export NCCL_IB_DISABLE=1
export WORLD_SIZE=2
export CUDA_VISIBLE_DEVICES='0,1,2,3,4,5,6,7'
python -m torch.distributed.launch \
--nnodes=2 \
--node_rank=1 \
--master_addr='ip_addr' \
--nproc_per_node=8 \
--master_port=29700 \
./tools/train.py \
configs/_freespace_/pidnet-l.py \
--resume \
--launcher pytorch ${@:3}
Reproduces the problem - error message
File "/data1/xuxin/code2/mmengine/mmengine/dist/dist.py", line 981, in collect_results_cpu
tmpdir = dir_tensor.numpy().tobytes().decode().rstrip()
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xb9 in position 0: invalid start byte
ubuntu:5119:5150 [0] NCCL INFO comm 0x7f2480009010 rank 8 nranks 16 cudaDev 0 busId 1000 - Abort COMPLETE
[E ProcessGroupNCCL.cpp:464] Some NCCL operations have failed or timed out. Due to the asynchronous nature of CUDA kernels, subsequent GPU operations might run on corrupted/incomplete data. To avoid this inconsistency, we are taking the entire
process down.
terminate called after throwing an instance of 'std::runtime_error'
what(): [Rank 8] Watchdog caught collective operation timeout: WorkNCCL(SeqNum=90544, OpType=BROADCAST, Timeout(ms)=1800000) ran for 1805629 milliseconds before timing out.
[E ProcessGroupNCCL.cpp:464] Some NCCL operations have failed or timed out. Due to the asynchronous nature of CUDA kernels, subsequent GPU operations might run on corrupted/incomplete data. To avoid this inconsistency, we are taking the entire
process down.
terminate called after throwing an instance of 'std::runtime_error'
what(): [Rank 11] Watchdog caught collective operation timeout: WorkNCCL(SeqNum=90544, OpType=BROADCAST, Timeout(ms)=1800000) ran for 1801084 milliseconds before timing out.
Additional information
- I used my own dataset, about 1.5M pictures, I trained them on mmsegmentation 1.0.0
- When I trained on Single machine and multiple GPUs, it is successful!
- Another problem was that the total iters of val remained the same after Training and testing on multiple GPUs and multiple machines, but GPUs utilization was normal for both machines
Hi, in this line, will this modification help you:
tmpdir = dir_tensor.numpy().tobytes().decode('utf-8').rstrip()
If it does, we'll enhance this part of code ASAP
It doesn't work.