demo errors due to insufficient gpu ram or missing module paddle.fluid.core_avx?

Question

demo errors due to insufficient gpu ram or missing module paddle.fluid.core_avx?

KimBenjaminTang opened this issue 5 years ago · 0 comments

Hello and first of all thank you for your contribution!

I wanted to execute the provided paddle demo but am just receiving errors.
I tested it twice with the same notebook on Google Colab and Kaggle but both times it fails.

This is my notebook: https://colab.research.google.com/drive/1CkgfxGwsbEvfWxkE6iKwShKIK5lFlARu

WARNING: Do not have avx core. You may not build with AVX, but AVX is supported on local machine.
 You could build paddle WITH_AVX=ON to get better performance.
The original error is: No module named 'paddle.fluid.core_avx'
W1209 03:51:00.008404   422 init.cc:162] AVX is available, Please re-compile on local machine
name: "cspn_affinity_propagate.tmp_23"
type {
  type: LOD_TENSOR
  lod_tensor {
    tensor {
      data_type: FP32
      dims: -1
      dims: 1
      dims: 48
      dims: 64
      dims: 128
    }
    lod_level: 0
  }
}
persistable: false

W1209 03:51:00.632901   422 device_context.cc:259] Please NOTE: device: 0, CUDA Capability: 37, Driver API Version: 10.1, Runtime API Version: 9.0
W1209 03:51:00.642139   422 device_context.cc:267] device: 0, cuDNN Version: 7.6.
An exception was thrown!
 Invoke operator fill_constant error.
Python Callstacks: 
  File "/usr/local/lib/python3.6/dist-packages/paddle/fluid/framework.py", line 1844, in _prepend_op
    attrs=kwargs.get("attrs", None))
  File "/usr/local/lib/python3.6/dist-packages/paddle/fluid/initializer.py", line 189, in __call__
    stop_gradient=True)
  File "/usr/local/lib/python3.6/dist-packages/paddle/fluid/framework.py", line 1627, in create_var
    kwargs['initializer'](var, self)
  File "/usr/local/lib/python3.6/dist-packages/paddle/fluid/layer_helper_base.py", line 383, in set_variable_initializer
    initializer=initializer)
  File "/usr/local/lib/python3.6/dist-packages/paddle/fluid/layers/tensor.py", line 142, in create_global_var
    value=float(value), force_cpu=force_cpu))
  File "/usr/local/lib/python3.6/dist-packages/paddle/fluid/optimizer.py", line 226, in _create_global_learning_rate
    persistable=True)
  File "/usr/local/lib/python3.6/dist-packages/paddle/fluid/optimizer.py", line 365, in _create_optimization_pass
    self._create_global_learning_rate()
  File "/usr/local/lib/python3.6/dist-packages/paddle/fluid/optimizer.py", line 532, in apply_gradients
    optimize_ops = self._create_optimization_pass(params_grads)
  File "/usr/local/lib/python3.6/dist-packages/paddle/fluid/optimizer.py", line 562, in apply_optimize
    optimize_ops = self.apply_gradients(params_grads)
  File "/usr/local/lib/python3.6/dist-packages/paddle/fluid/optimizer.py", line 601, in minimize
    loss, startup_program=startup_program, params_grads=params_grads)
  File "/usr/local/lib/python3.6/dist-packages/paddle/fluid/dygraph/base.py", line 86, in __impl__
    return func(*args, **kwargs)
  File "/usr/local/lib/python3.6/dist-packages/paddle/fluid/wrapped_decorator.py", line 25, in __impl__
    return wrapped_func(*args, **kwargs)
  File "</usr/local/lib/python3.6/dist-packages/decorator.py:decorator-gen-20>", line 2, in minimize
  File "demo.py", line 75, in demo
    optim.minimize(output)
  File "demo.py", line 97, in <module>
    MODULE.demo()
C++ Callstacks: 
Enforce failed. Expected allocating <= available, but received allocating:10485338519 > available:1249705728.
Insufficient GPU memory to allocation. at [/paddle/paddle/fluid/platform/gpu_info.cc:293]
PaddlePaddle Call Stacks: 
0       0x7f41c0889955p void paddle::platform::EnforceNotMet::Init<std::string>(std::string, char const*, int) + 357
1       0x7f41c0889cb2p paddle::platform::EnforceNotMet::EnforceNotMet(std::string const&, char const*, int) + 82
2       0x7f41c260abe9p paddle::platform::GpuMaxChunkSize() + 617
3       0x7f41c2534064p
4       0x7f41fd309827p
5       0x7f41c253450dp paddle::memory::legacy::GetGPUBuddyAllocator(int) + 109
6       0x7f41c2534721p void* paddle::memory::legacy::Alloc<paddle::platform::CUDAPlace>(paddle::platform::CUDAPlace const&, unsigned long) + 33
7       0x7f41c2534df5p paddle::memory::allocation::NaiveBestFitAllocator::AllocateImpl(unsigned long) + 405
8       0x7f41c252f113p paddle::memory::allocation::AllocatorFacade::Alloc(boost::variant<paddle::platform::CUDAPlace, paddle::platform::CPUPlace, paddle::platform::CUDAPinnedPlace, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_> const&, unsigned long) + 227
9       0x7f41c252f3bbp paddle::memory::allocation::AllocatorFacade::AllocShared(boost::variant<paddle::platform::CUDAPlace, paddle::platform::CPUPlace, paddle::platform::CUDAPinnedPlace, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_> const&, unsigned long) + 27
10      0x7f41c2199d6cp paddle::memory::AllocShared(boost::variant<paddle::platform::CUDAPlace, paddle::platform::CPUPlace, paddle::platform::CUDAPinnedPlace, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_> const&, unsigned long) + 44
11      0x7f41c2507458p paddle::framework::Tensor::mutable_data(boost::variant<paddle::platform::CUDAPlace, paddle::platform::CPUPlace, paddle::platform::CUDAPinnedPlace, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_>, paddle::framework::proto::VarType_Type, unsigned long) + 136
12      0x7f41c0c29994p paddle::operators::FillConstantKernel<float>::Compute(paddle::framework::ExecutionContext const&) const + 500
13      0x7f41c0c2c8b0p std::_Function_handler<void (paddle::framework::ExecutionContext const&), paddle::framework::OpKernelRegistrarFunctor<paddle::platform::CUDAPlace, false, 0ul, paddle::operators::FillConstantKernel<float>, paddle::operators::FillConstantKernel<double>, paddle::operators::FillConstantKernel<long>, paddle::operators::FillConstantKernel<int>, paddle::operators::FillConstantKernel<paddle::platform::float16> >::operator()(char const*, char const*, int) const::{lambda(paddle::framework::ExecutionContext const&)#1}>::_M_invoke(std::_Any_data const&, paddle::framework::ExecutionContext const&) + 32
14      0x7f41c24b576dp paddle::framework::OperatorWithKernel::RunImpl(paddle::framework::Scope const&, boost::variant<paddle::platform::CUDAPlace, paddle::platform::CPUPlace, paddle::platform::CUDAPinnedPlace, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_> const&, paddle::framework::RuntimeContext*) const + 381
15      0x7f41c24b5dabp paddle::framework::OperatorWithKernel::RunImpl(paddle::framework::Scope const&, boost::variant<paddle::platform::CUDAPlace, paddle::platform::CPUPlace, paddle::platform::CUDAPinnedPlace, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_> const&) const + 555
16      0x7f41c24b321cp paddle::framework::OperatorBase::Run(paddle::framework::Scope const&, boost::variant<paddle::platform::CUDAPlace, paddle::platform::CPUPlace, paddle::platform::CUDAPinnedPlace, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_> const&) + 300
17      0x7f41c09f8216p paddle::framework::Executor::RunPreparedContext(paddle::framework::ExecutorPrepareContext*, paddle::framework::Scope*, bool, bool, bool) + 438
18      0x7f41c09fadc4p paddle::framework::Executor::Run(paddle::framework::ProgramDesc const&, paddle::framework::Scope*, int, bool, bool, std::vector<std::string, std::allocator<std::string> > const&, bool) + 132
19      0x7f41c087bae3p
20      0x7f41c08b9b54p
21            0x5674fcp _PyCFunction_FastCallDict + 860
22            0x50abb3p
23            0x50c5b9p _PyEval_EvalFrameDefault + 1097
24            0x508245p
25            0x50a080p
26            0x50aa7dp
27            0x50d390p _PyEval_EvalFrameDefault + 4640
28            0x508245p
29            0x50a080p
30            0x50aa7dp
31            0x50d390p _PyEval_EvalFrameDefault + 4640
32            0x508245p
33            0x50a080p
34            0x50aa7dp
35            0x50c5b9p _PyEval_EvalFrameDefault + 1097
36            0x508245p
37            0x50a080p
38            0x50aa7dp
39            0x50c5b9p _PyEval_EvalFrameDefault + 1097
40            0x508245p
41            0x50b403p PyEval_EvalCode + 35
42            0x635222p
43            0x6352d7p PyRun_FileExFlags + 151
44            0x638a8fp PyRun_SimpleFileExFlags + 383
45            0x639631p Py_Main + 1425
46            0x4b0f40p main + 224
47      0x7f41fd53ab97p __libc_start_main + 231
48            0x5b2fdap _start + 42

Traceback (most recent call last):
  File "demo.py", line 97, in <module>
    MODULE.demo()
  File "demo.py", line 79, in demo
    exe.run(fluid.default_startup_program())
  File "/usr/local/lib/python3.6/dist-packages/paddle/fluid/executor.py", line 644, in run
    raise e
  File "/usr/local/lib/python3.6/dist-packages/paddle/fluid/executor.py", line 640, in run
    use_program_cache=use_program_cache)
  File "/usr/local/lib/python3.6/dist-packages/paddle/fluid/executor.py", line 669, in _run_impl
    use_program_cache=use_program_cache)
  File "/usr/local/lib/python3.6/dist-packages/paddle/fluid/executor.py", line 766, in _run_program
    exe.run(program.desc, scope, 0, True, True, fetch_var_name)
paddle.fluid.core_noavx.EnforceNotMet: Invoke operator fill_constant error.
Python Callstacks: 
  File "/usr/local/lib/python3.6/dist-packages/paddle/fluid/framework.py", line 1844, in _prepend_op
    attrs=kwargs.get("attrs", None))
  File "/usr/local/lib/python3.6/dist-packages/paddle/fluid/initializer.py", line 189, in __call__
    stop_gradient=True)
  File "/usr/local/lib/python3.6/dist-packages/paddle/fluid/framework.py", line 1627, in create_var
    kwargs['initializer'](var, self)
  File "/usr/local/lib/python3.6/dist-packages/paddle/fluid/layer_helper_base.py", line 383, in set_variable_initializer
    initializer=initializer)
  File "/usr/local/lib/python3.6/dist-packages/paddle/fluid/layers/tensor.py", line 142, in create_global_var
    value=float(value), force_cpu=force_cpu))
  File "/usr/local/lib/python3.6/dist-packages/paddle/fluid/optimizer.py", line 226, in _create_global_learning_rate
    persistable=True)
  File "/usr/local/lib/python3.6/dist-packages/paddle/fluid/optimizer.py", line 365, in _create_optimization_pass
    self._create_global_learning_rate()
  File "/usr/local/lib/python3.6/dist-packages/paddle/fluid/optimizer.py", line 532, in apply_gradients
    optimize_ops = self._create_optimization_pass(params_grads)
  File "/usr/local/lib/python3.6/dist-packages/paddle/fluid/optimizer.py", line 562, in apply_optimize
    optimize_ops = self.apply_gradients(params_grads)
  File "/usr/local/lib/python3.6/dist-packages/paddle/fluid/optimizer.py", line 601, in minimize
    loss, startup_program=startup_program, params_grads=params_grads)
  File "/usr/local/lib/python3.6/dist-packages/paddle/fluid/dygraph/base.py", line 86, in __impl__
    return func(*args, **kwargs)
  File "/usr/local/lib/python3.6/dist-packages/paddle/fluid/wrapped_decorator.py", line 25, in __impl__
    return wrapped_func(*args, **kwargs)
  File "</usr/local/lib/python3.6/dist-packages/decorator.py:decorator-gen-20>", line 2, in minimize
  File "demo.py", line 75, in demo
    optim.minimize(output)
  File "demo.py", line 97, in <module>
    MODULE.demo()
C++ Callstacks: 
Enforce failed. Expected allocating <= available, but received allocating:10485338519 > available:1249705728.
Insufficient GPU memory to allocation. at [/paddle/paddle/fluid/platform/gpu_info.cc:293]
PaddlePaddle Call Stacks: 
0       0x7f41c0889955p void paddle::platform::EnforceNotMet::Init<std::string>(std::string, char const*, int) + 357
1       0x7f41c0889cb2p paddle::platform::EnforceNotMet::EnforceNotMet(std::string const&, char const*, int) + 82
2       0x7f41c260abe9p paddle::platform::GpuMaxChunkSize() + 617
3       0x7f41c2534064p
4       0x7f41fd309827p
5       0x7f41c253450dp paddle::memory::legacy::GetGPUBuddyAllocator(int) + 109
6       0x7f41c2534721p void* paddle::memory::legacy::Alloc<paddle::platform::CUDAPlace>(paddle::platform::CUDAPlace const&, unsigned long) + 33
7       0x7f41c2534df5p paddle::memory::allocation::NaiveBestFitAllocator::AllocateImpl(unsigned long) + 405
8       0x7f41c252f113p paddle::memory::allocation::AllocatorFacade::Alloc(boost::variant<paddle::platform::CUDAPlace, paddle::platform::CPUPlace, paddle::platform::CUDAPinnedPlace, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_> const&, unsigned long) + 227
9       0x7f41c252f3bbp paddle::memory::allocation::AllocatorFacade::AllocShared(boost::variant<paddle::platform::CUDAPlace, paddle::platform::CPUPlace, paddle::platform::CUDAPinnedPlace, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_> const&, unsigned long) + 27
10      0x7f41c2199d6cp paddle::memory::AllocShared(boost::variant<paddle::platform::CUDAPlace, paddle::platform::CPUPlace, paddle::platform::CUDAPinnedPlace, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_> const&, unsigned long) + 44
11      0x7f41c2507458p paddle::framework::Tensor::mutable_data(boost::variant<paddle::platform::CUDAPlace, paddle::platform::CPUPlace, paddle::platform::CUDAPinnedPlace, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_>, paddle::framework::proto::VarType_Type, unsigned long) + 136
12      0x7f41c0c29994p paddle::operators::FillConstantKernel<float>::Compute(paddle::framework::ExecutionContext const&) const + 500
13      0x7f41c0c2c8b0p std::_Function_handler<void (paddle::framework::ExecutionContext const&), paddle::framework::OpKernelRegistrarFunctor<paddle::platform::CUDAPlace, false, 0ul, paddle::operators::FillConstantKernel<float>, paddle::operators::FillConstantKernel<double>, paddle::operators::FillConstantKernel<long>, paddle::operators::FillConstantKernel<int>, paddle::operators::FillConstantKernel<paddle::platform::float16> >::operator()(char const*, char const*, int) const::{lambda(paddle::framework::ExecutionContext const&)#1}>::_M_invoke(std::_Any_data const&, paddle::framework::ExecutionContext const&) + 32
14      0x7f41c24b576dp paddle::framework::OperatorWithKernel::RunImpl(paddle::framework::Scope const&, boost::variant<paddle::platform::CUDAPlace, paddle::platform::CPUPlace, paddle::platform::CUDAPinnedPlace, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_> const&, paddle::framework::RuntimeContext*) const + 381
15      0x7f41c24b5dabp paddle::framework::OperatorWithKernel::RunImpl(paddle::framework::Scope const&, boost::variant<paddle::platform::CUDAPlace, paddle::platform::CPUPlace, paddle::platform::CUDAPinnedPlace, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_> const&) const + 555
16      0x7f41c24b321cp paddle::framework::OperatorBase::Run(paddle::framework::Scope const&, boost::variant<paddle::platform::CUDAPlace, paddle::platform::CPUPlace, paddle::platform::CUDAPinnedPlace, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_> const&) + 300
17      0x7f41c09f8216p paddle::framework::Executor::RunPreparedContext(paddle::framework::ExecutorPrepareContext*, paddle::framework::Scope*, bool, bool, bool) + 438
18      0x7f41c09fadc4p paddle::framework::Executor::Run(paddle::framework::ProgramDesc const&, paddle::framework::Scope*, int, bool, bool, std::vector<std::string, std::allocator<std::string> > const&, bool) + 132
19      0x7f41c087bae3p
20      0x7f41c08b9b54p
21            0x5674fcp _PyCFunction_FastCallDict + 860
22            0x50abb3p
23            0x50c5b9p _PyEval_EvalFrameDefault + 1097
24            0x508245p
25            0x50a080p
26            0x50aa7dp
27            0x50d390p _PyEval_EvalFrameDefault + 4640
28            0x508245p
29            0x50a080p
30            0x50aa7dp
31            0x50d390p _PyEval_EvalFrameDefault + 4640
32            0x508245p
33            0x50a080p
34            0x50aa7dp
35            0x50c5b9p _PyEval_EvalFrameDefault + 1097
36            0x508245p
37            0x50a080p
38            0x50aa7dp
39            0x50c5b9p _PyEval_EvalFrameDefault + 1097
40            0x508245p
41            0x50b403p PyEval_EvalCode + 35
42            0x635222p
43            0x6352d7p PyRun_FileExFlags + 151
44            0x638a8fp PyRun_SimpleFileExFlags + 383
45            0x639631p Py_Main + 1425
46            0x4b0f40p main + 224
47      0x7f41fd53ab97p __libc_start_main + 231
48            0x5b2fdap _start + 42

So what I understand from this error, is that not enough GPU memory can be allocated. But there is around 13GB ram on Google Colab with the GPU available. How much is required for this demo?
Or is there another error that leads to this?

I also noticed that in the beginning it states "The original error is: No module named 'paddle.fluid.core_avx'" but from my understanding this is a warning and not what leads to this error, right?

Thanks for your help!