scanner-research/scanner

Config GPU usage

Closed this issue · 2 comments

Hi,

I am trying to run the example face_detection app.
However, I always encounter out of memory error. Below is the error message:

F1214 03:43:10.334967  1536 cuda.h:45] GPUassert: out of memory /opt/scanner/scanner/util/memory.cpp 131
*** Check failure stack trace: ***
    @     0x7f010a8f15cd  google::LogMessage::Fail()
    @     0x7f010a8f3433  google::LogMessage::SendToLog()
  0%|                                                                                                                                                                                                     | 0/232 [00:00<?, ?it/s, jobs=1, workers=1, tasks=232]    @     0x7f010a8f115b  google::LogMessage::Flush()
    @     0x7f010a8f3e1e  google::LogMessageFatal::~LogMessageFatal()
    @     0x7f007ca29b35  cuAssert()
    @     0x7f007ca2efda  scanner::SystemAllocator::allocate()
    @     0x7f007ca2de71  scanner::new_block_buffer_sizes_()
    @     0x7f007ca2e3de  scanner::new_block_buffer_size_()
    @     0x7f007c9bfb1a  scanner::internal::PreEvaluateWorker::yield()
    @     0x7f007c984439  _ZN7scanner8internal12_GLOBAL__N_119pre_evaluate_driverERNS_5QueueISt5tupleIJSt5dequeINS0_10TaskStreamESaIS5_EENS0_13EvalWorkEntryEEEEESB_NS0_21PreEvaluateWorkerArgsE
    @     0x7f007c994934  _ZNSt6thread5_ImplISt12_Bind_simpleIFPFvRN7scanner5QueueISt5tupleIJSt5dequeINS2_8internal10TaskStreamESaIS7_EENS6_13EvalWorkEntryEEEEESD_NS6_21PreEvaluateWorkerArgsEESt17reference_wrapperISC_ESI_SE_EEE6_M_runEv
    @     0x7f00f1dcbc80  (unknown)
    @     0x7f01141ce6ba  start_thread
  0%|                                                                                                                                                                                                     | 0/232 [00:00<?, ?it/s, jobs=1, workers=1, tasks=232]    @     0x7f0113f0441d  clone
    @              (nil)  (unknown)
Aborted (core dumped)

I would like to know is there any configuration I can set whether to use GPU, and which GPU is going to be used? There are three GPU cards on the machine, and the first and second ones are fully used. I want to specify Scanner to use the third one.

Appreciate any suggestion!

fpoms commented

Hi @jacky841102,

Scanner accepts the standard methods for limiting programs to use only a subset of the GPUs on a machine.

Specifically, you can set

export CUDA_VISIBLE_DEVICES=<gpu_id>

In this case you would set <gpu_id> = 2

Thanks!