terminate called after throwing an instance of 'at::Error' what(): CUDA error (3): initialization error (check_status at /pytorch/aten/src/ATen/cuda/detail/CUDAHooks.cpp:36)

Question

terminate called after throwing an instance of 'at::Error' what(): CUDA error (3): initialization error (check_status at /pytorch/aten/src/ATen/cuda/detail/CUDAHooks.cpp:36)

JinQiangWang2021 opened this issue 3 years ago · 3 comments

JinQiangWang2021 commented 3 years ago

/home/account/anaconda3/envs/RL17/bin/python /home/account/Documents/Deep_RL_Implementations/results/Cart_Pole.py
/home/account/anaconda3/envs/RL17/lib/python3.7/site-packages/gym/envs/registration.py:14: PkgResourcesDeprecationWarning: Parameters to load are deprecated. Call .resolve and .require separately.
result = entry_point.load(False)

AGENT NAME: A3C
1.1: A3C
TITLE CartPole
layer info [20, 10, [2, 1]]
layer info [20, 10, [2, 1]]
{'learning_rate': 0.005, 'linear_hidden_units': [20, 10], 'final_layer_activation': ['SOFTMAX', None], 'gradient_clipping_norm': 5.0, 'discount_rate': 0.99, 'epsilon_decay_rate_denominator': 1.0, 'normalise_rewards': True, 'exploration_worker_difference': 2.0, 'clip_rewards': False, 'Actor': {'learning_rate': 0.0003, 'linear_hidden_units': [64, 64], 'final_layer_activation': 'Softmax', 'batch_norm': False, 'tau': 0.005, 'gradient_clipping_norm': 5, 'initialiser': 'Xavier'}, 'Critic': {'learning_rate': 0.0003, 'linear_hidden_units': [64, 64], 'final_layer_activation': None, 'batch_norm': False, 'buffer_size': 1000000, 'tau': 0.005, 'gradient_clipping_norm': 5, 'initialiser': 'Xavier'}, 'min_steps_before_learning': 400, 'batch_size': 256, 'mu': 0.0, 'theta': 0.15, 'sigma': 0.25, 'action_noise_std': 0.2, 'action_noise_clipping_range': 0.5, 'update_every_n_steps': 1, 'learning_updates_per_learning_session': 1, 'automatically_tune_entropy_hyperparameter': True, 'entropy_term_weight': None, 'add_extra_noise': False, 'do_evaluation_iterations': True, 'output_activation': None, 'hidden_activations': 'relu', 'dropout': 0.0, 'initialiser': 'default', 'batch_norm': False, 'columns_of_data_to_be_embedded': [], 'embedding_dimensions': [], 'y_range': ()}
RANDOM SEED 2741982467
terminate called after throwing an instance of 'at::Error'
what(): CUDA error (3): initialization error (check_status at /pytorch/aten/src/ATen/cuda/detail/CUDAHooks.cpp:36)
frame #0: torch::autograd::Engine::thread_init(int) + 0x34 (0x7f4a94844264 in /home/jqw/anaconda3/envs/RL17/lib/python3.7/site-packages/torch/_C.cpython-37m-x86_64-linux-gnu.so)
frame #1: torch::autograd::python::PythonEngine::thread_init(int) + 0x2a (0x7f4a94873a5a in /home/jqw/anaconda3/envs/RL17/lib/python3.7/site-packages/torch/_C.cpython-37m-x86_64-linux-gnu.so)
frame #2: + 0xc92bd (0x7f4a6872a2bd in /home/jqw/anaconda3/envs/RL17/bin/../lib/libstdc++.so.6)
frame #3: + 0x76ba (0x7f4aa8e396ba in /lib/x86_64-linux-gnu/libpthread.so.0)
frame #4: clone + 0x6d (0x7f4aa8b6f51d in /lib/x86_64-linux-gnu/libc.so.6)

terminate called after throwing an instance of 'at::Error'
what(): CUDA error (3): initialization error (check_status at /pytorch/aten/src/ATen/cuda/detail/CUDAHooks.cpp:36)
frame #0: torch::autograd::Engine::thread_init(int) + 0x34 (0x7f4a94844264 in /home/jqw/anaconda3/envs/RL17/lib/python3.7/site-packages/torch/_C.cpython-37m-x86_64-linux-gnu.so)
frame #1: torch::autograd::python::PythonEngine::thread_init(int) + 0x2a (0x7f4a94873a5a in /home/jqw/anaconda3/envs/RL17/lib/python3.7/site-packages/torch/_C.cpython-37m-x86_64-linux-gnu.so)
frame #2: + 0xc92bd (0x7f4a6872a2bd in /home/jqw/anaconda3/envs/RL17/bin/../lib/libstdc++.so.6)
frame #3: + 0x76ba (0x7f4aa8e396ba in /lib/x86_64-linux-gnu/libpthread.so.0)
frame #4: clone + 0x6d (0x7f4aa8b6f51d in /lib/x86_64-linux-gnu/libc.so.6)

Answer 1 · 2021-11-08T13:56:31.000Z

I have Solved this question:

import os 

os.environ["CUDA_DEVICE_ORDER"]="PCI_BUS_ID"
os.environ["CUDA_VISIBLE_DEVICES"]=""

Answer 2 · 2021-12-15T08:29:03.000Z

nice

I have Solved this question:

import os 

os.environ["CUDA_DEVICE_ORDER"]="PCI_BUS_ID"
os.environ["CUDA_VISIBLE_DEVICES"]=""

Answer 3 · 2021-12-15T08:29:25.000Z

您好，邮件已收到。我查阅后，尽快回复您!