What is the purpose of this code?
Opened this issue · 8 comments
int requestedAlgoCount = CUDNN_CONVOLUTION_FWD_ALGO_COUNT;
int returnedAlgoCount = -1;
cudnnConvolutionFwdAlgoPerf_t fwd_results[2 * requestedAlgoCount];
checkCUDNN(cudnnFindConvolutionForwardAlgorithm(*handle, src_tensor_desc,
filter_desc, conv_desc, dst_tensor_desc, requestedAlgoCount,
&returnedAlgoCount, fwd_results));
fwd_algo = fwd_results[0].algo;
You can refer to cudnnFindConvolutionForwardAlgorithm() in cuDNN-API document.
cuDNN implement different algorithm for convolution. Different algorithms require different need for memory and have different performance. This function attempts all algorithms available for cudnnConvolutionForward(). It will attempt both the provided convDesc mathType and CUDNN_DEFAULT_MATH. Therefore, you can see the "fwd_results" is a 2 * requestedAlgoCount array.
This output "fwd_results" is a user-allocated array to store performance metrics sorted ascending by compute time. So I choose the first algorithm. If you want lower memory size, you can make other choices.
“fwd_algo” will later be used in cudnnGetConvolutionForwardWorkspaceSize() to allocate temporary memory size according to the algorithm you choose and the actual execution of convolution cudnnConvolutionForward() also need it.
Why are there two algorithm choices in backward propagation, cudnnConvolutionBackwardFilter and cudnnConvolutionBackwardData. What is their role? Can you answer it?
In backward propagation, cudnnConvolutionBackwardFilter is used to get the differential of filter, and cudnnConvolutionBackwardData is used to get the differential of input. These two operation is different, so they need different algorithm.
The total number of resulting algorithms can be queried through the API cudnnGetConvolutionForwardAlgorithmMaxCount(). Also, you can find it in cudnn.h
What is the process of cudnn backpropagation? There is too little information on the Internet.
cuDNN doesn’t provide api for getting differential from loss, so you should write your own code. After that, you can refer to backpropagation algorithm and use cuDNN api to compute the differential.
What does this ”cudnnConvolutionBackwardData is used to get the differential of input“ mean? like Is the derivative of p in y(p(x))?
What does this ”cudnnConvolutionBackwardData is used to get the differential of input“ mean? like Is the derivative of p in y(p(x))?
Yes. After that, you can use the derivative of p to get the derivative of parameters in p(x), so the output of cudnnConvolutionBackwardData would be passed to the backforward function of its previous operation. You can refer to main.cu.