Execute matmul op faild

Question

Execute matmul op faild

Gebixiaochen opened this issue 2 years ago · 13 comments

I tried to run the matmul op and follow the codes in function "void run_matmul_bias_gelu" in fusion_sample.cpp. My code is as below:

int main() {
  auto ha = loadBinary<float>("/workspace/features.bin");
  auto hb = loadBinary<float>("/workspace/weight.bin");
  auto hc = loadBinary<float>("/workspace/output.bin");

  float *a_ptr;
  cudaMalloc((void **)&a_ptr, ha.size() * sizeof(float));
  cudaMemcpy(a_ptr, ha.data(), ha.size() * sizeof(float),
             cudaMemcpyHostToDevice);
  float *b_ptr;
  cudaMalloc((void **)&b_ptr, hb.size() * sizeof(float));
  cudaMemcpy(b_ptr, hb.data(), hb.size() * sizeof(float),
             cudaMemcpyHostToDevice);

  float *c_ptr;
  cudaMalloc((void **)&c_ptr, hc.size() * sizeof(float));

  const int m = ha.size() / 96;
  const int n = 96;
  const int k = 96;

  int64_t stride[3];
  int64_t a_dim[3] = {1, m, k};
  int64_t b_dim[3] = {1, k, n};
  int64_t c_dim[3] = {1, m, n};
  generateStrides(a_dim, stride, 3, CUDNN_TENSOR_NCHW);
  auto aMatrixTensor =
      cudnn_frontend::TensorBuilder()
          .setDim(3, a_dim)
          .setStride(3, stride)
          .setId('a')
          .setAlignment(
              16)  // 16B alignment is needed to run a tensor core engine
          .setDataType(CUDNN_DATA_FLOAT)
          .build();

  generateStrides(b_dim, stride, 3, CUDNN_TENSOR_NCHW);
  auto bMatrixTensor = cudnn_frontend::TensorBuilder()
                           .setDim(3, b_dim)
                           .setStride(3, stride)
                           .setId('b')
                           .setAlignment(16)
                           .setDataType(CUDNN_DATA_FLOAT)
                           .build();

  generateStrides(c_dim, stride, 3, CUDNN_TENSOR_NCHW);
  auto afterMatMulTensor = cudnn_frontend::TensorBuilder()
                               .setDim(3, c_dim)
                               .setStride(3, stride)
                               .setId('A')  // after matmul
                               .setAlignment(16)
                               .setVirtual()
                               .setDataType(CUDNN_DATA_FLOAT)
                               .build();

  std::cout << aMatrixTensor.describe() << std::endl;
  std::cout << bMatrixTensor.describe() << std::endl;
  std::cout << afterMatMulTensor.describe() << std::endl;

  // Define the matmul desc
  auto matmulDesc = cudnn_frontend::MatMulDescBuilder()
                        .setComputeType(CUDNN_DATA_FLOAT)
                        .build();
  std::cout << matmulDesc.describe() << std::endl;

  // Create a matmul Node
  auto matmul_op = cudnn_frontend::OperationBuilder(
                       CUDNN_BACKEND_OPERATION_MATMUL_DESCRIPTOR)
                       .setaMatDesc(aMatrixTensor)
                       .setbMatDesc(bMatrixTensor)
                       .setcMatDesc(afterMatMulTensor)
                       .setmatmulDesc(matmulDesc)
                       .build();
  std::cout << matmul_op.describe() << std::endl;

  std::array<cudnn_frontend::Operation const *, 1> ops = {&matmul_op};
  cudnnHandle_t handle_;
  checkCudnnErr(cudnnCreate(&handle_));

  auto opGraph = cudnn_frontend::OperationGraphBuilder()
                     .setHandle(handle_)
                     .setOperationGraph(ops.size(), ops.data())
                     .build();
  auto plan =
      get_execplan_from_heuristics_else_fall_back(std::move(opGraph), handle_);

  auto workspace_size = plan.getWorkspaceSize();
  std::cout << plan.describe() << " requires workspace " << workspace_size
            << std::endl;

  void *workspace_ptr = nullptr;
  if (workspace_size > 0) {
    cudaMalloc(&workspace_ptr, (size_t)workspace_size);
  }

  void *data_ptrs[] = {a_ptr, b_ptr, c_ptr};
  int64_t uids[] = {'a', 'b', 'c'};
  auto variantPack = cudnn_frontend::VariantPackBuilder()
                         .setWorkspacePointer(workspace_ptr)
                         .setDataPointers(3, data_ptrs)
                         .setUids(3, uids)
                         .build();

  cudnnStatus_t status = cudnnBackendExecute(handle_, plan.get_raw_desc(),
                                             variantPack.get_raw_desc());

  if (workspace_size > 0) {
    (cudaFree(workspace_ptr));
  }
  checkCudnnErr(cudnnDestroy(handle_));
  cudnn_frontend::throw_if(
      [status]() { return (status != CUDNN_STATUS_SUCCESS); },
      "Plan execute error", status);
}

I run it under cuda-memcheck and got logs as below, is there anything g wrong with my codes? I use docker:nvcr.io/nvidia/pytorch:21.10-py3 and my driver version is Driver Version: 515.65.01

root@3c261e44c675:/workspace/trt_inference/bin# cuda-memcheck ./trtexc
========= CUDA-MEMCHECK
ha size is 5164704
hb size is 9216
hc size is 5164704
CUDNN_BACKEND_TENSOR_DESCRIPTOR : Datatype: CUDNN_DATA_FLOAT Id: 97 Alignment: 16 nDims 3 VectorCount: 1 vectorDimension -1 Dim [ 1,53799,96 ] Str [,5164704,96,1] isVirtual: 0 isByValue: 0
CUDNN_BACKEND_TENSOR_DESCRIPTOR : Datatype: CUDNN_DATA_FLOAT Id: 98 Alignment: 16 nDims 3 VectorCount: 1 vectorDimension -1 Dim [ 1,96,96 ] Str [,9216,96,1] isVirtual: 0 isByValue: 0
CUDNN_BACKEND_TENSOR_DESCRIPTOR : Datatype: CUDNN_DATA_FLOAT Id: 65 Alignment: 16 nDims 3 VectorCount: 1 vectorDimension -1 Dim [ 1,53799,96 ] Str [,5164704,96,1] isVirtual: 1 isByValue: 0
CUDNN_BACKEND_MATMUL_DESCRIPTOR : Math precision 0
CUDNN_BACKEND_OPERATION : OpMode: 19 X 0 Y 0 W 0 B 0 T 0 DW 0 DY 0 DX 0 C 0 A Mtrix 0x557468e919e0 B Mtrix 0x5574c343d5d0 C Mtrix 0x5574c35f37b0 P 0 MatMul 0x5574c35f39c0 Reduction 0 alphabetaType 4 Alpha: 1 1 Alpha2: 1 1 Beta: 0 0
Heuristic has 3 configurations
CUDNN_BACKEND_EXECUTION_PLAN_DESCRIPTOR : Matmul_eng0_k24=27, numeric_notes:[CUDNN_NUMERICAL_NOTE_TENSOR_CORE,] behavior_notes:[CUDNN_BEHAVIOR_NOTE_RUNTIME_COMPILATION,] workSpaceSize: 0 requires workspace 0
terminate called after throwing an instance of 'cudnn_frontend::cudnnException'
what(): Plan execute error
========= Error: process didn't terminate successfully
========= No CUDA-MEMCHECK results found

Hope to get help! Thanks.

Answer 1 · 2022-11-25T08:02:28.000Z

Thanks for posting.

May I know

what cuDNN version you are using?
What GPU you are using (nvidia-smi could help)?
Can you provide cuDNN backend logs as well ? See

-Anerudhan

Answer 2 · 2022-11-25T08:48:50.000Z

Thanks for reply !

my cuDNN version is 8.2.4
GPU is 3090
I enable info/warn/err and got the logs as be low

I! CuDNN (v8204) function cudnnBackendCreateDescriptor() called:
i! descriptorType: type=cudnnBackendDescriptorType_t; val=CUDNN_BACKEND_TENSOR_DESCRIPTOR (17);
i! Time: 2022-11-25T08:47:46.254294 (0d+0h+0m+0s since start)
i! Process=391; Thread=391; GPU=NULL; Handle=NULL; StreamId=NULL.

I! CuDNN (v8204) function cudnnBackendSetAttribute() called:
i! descriptor: type=CUDNN_BACKEND_TENSOR_DESCRIPTOR:
i! type: type=cudnnDataType_t; val=CUDNN_DATA_FLOAT (0);
i! nbDims: type=int; val=-1;
i! dimA: type=int; val=[];
i! strideA: type=int; val=[];
i! uid: type=int64_t; val=-1;
i! alignmentInBytes: type=int64_t; val=1;
i! isVirtual: type=bool; val=false;
i! attributeName: type=cudnnBackendAttributeName_t; val=CUDNN_ATTR_TENSOR_DATA_TYPE (901);
i! attributeType: type=cudnnBackendAttributeType_t; val=CUDNN_TYPE_DATA_TYPE (1);
i! elementCount: type=int64_t; val=1;
i! [0]: type=cudnnDataType_t; val=CUDNN_DATA_FLOAT (0);
i! Time: 2022-11-25T08:47:46.254362 (0d+0h+0m+0s since start)
i! Process=391; Thread=391; GPU=NULL; Handle=NULL; StreamId=NULL.

I! CuDNN (v8204) function cudnnBackendSetAttribute() called:
i! descriptor: type=CUDNN_BACKEND_TENSOR_DESCRIPTOR:
i! type: type=cudnnDataType_t; val=CUDNN_DATA_FLOAT (0);
i! nbDims: type=int; val=-1;
i! dimA: type=int; val=[];
i! strideA: type=int; val=[];
i! uid: type=int64_t; val=-1;
i! alignmentInBytes: type=int64_t; val=1;
i! isVirtual: type=bool; val=false;
i! attributeName: type=cudnnBackendAttributeName_t; val=CUDNN_ATTR_TENSOR_DIMENSIONS (902);
i! attributeType: type=cudnnBackendAttributeType_t; val=CUDNN_TYPE_INT64 (3);
i! elementCount: type=int64_t; val=3;
i! [0]: type=int64_t; val=1;
i! [1]: type=int64_t; val=53799;
i! [2]: type=int64_t; val=96;
i! Time: 2022-11-25T08:47:46.254374 (0d+0h+0m+0s since start)
i! Process=391; Thread=391; GPU=NULL; Handle=NULL; StreamId=NULL.

I! CuDNN (v8204) function cudnnBackendSetAttribute() called:
i! descriptor: type=CUDNN_BACKEND_TENSOR_DESCRIPTOR:
i! type: type=cudnnDataType_t; val=CUDNN_DATA_FLOAT (0);
i! nbDims: type=int; val=3;
i! dimA: type=int; val=[1,53799,96];
i! strideA: type=int; val=[0,0,0];
i! uid: type=int64_t; val=-1;
i! alignmentInBytes: type=int64_t; val=1;
i! isVirtual: type=bool; val=false;
i! attributeName: type=cudnnBackendAttributeName_t; val=CUDNN_ATTR_TENSOR_STRIDES (903);
i! attributeType: type=cudnnBackendAttributeType_t; val=CUDNN_TYPE_INT64 (3);
i! elementCount: type=int64_t; val=3;
i! [0]: type=int64_t; val=5164704;
i! [1]: type=int64_t; val=96;
i! [2]: type=int64_t; val=1;
i! Time: 2022-11-25T08:47:46.254394 (0d+0h+0m+0s since start)
i! Process=391; Thread=391; GPU=NULL; Handle=NULL; StreamId=NULL.

I! CuDNN (v8204) function cudnnBackendSetAttribute() called:
i! descriptor: type=CUDNN_BACKEND_TENSOR_DESCRIPTOR:
i! type: type=cudnnDataType_t; val=CUDNN_DATA_FLOAT (0);
i! nbDims: type=int; val=3;
i! dimA: type=int; val=[1,53799,96];
i! strideA: type=int; val=[5164704,96,1];
i! uid: type=int64_t; val=-1;
i! alignmentInBytes: type=int64_t; val=1;
i! isVirtual: type=bool; val=false;
i! attributeName: type=cudnnBackendAttributeName_t; val=CUDNN_ATTR_TENSOR_UNIQUE_ID (906);
i! attributeType: type=cudnnBackendAttributeType_t; val=CUDNN_TYPE_INT64 (3);
i! elementCount: type=int64_t; val=1;
i! [0]: type=int64_t; val=97;
i! Time: 2022-11-25T08:47:46.254402 (0d+0h+0m+0s since start)
i! Process=391; Thread=391; GPU=NULL; Handle=NULL; StreamId=NULL.

I! CuDNN (v8204) function cudnnBackendSetAttribute() called:
i! descriptor: type=CUDNN_BACKEND_TENSOR_DESCRIPTOR:
i! type: type=cudnnDataType_t; val=CUDNN_DATA_FLOAT (0);
i! nbDims: type=int; val=3;
i! dimA: type=int; val=[1,53799,96];
i! strideA: type=int; val=[5164704,96,1];
i! uid: type=int64_t; val=97;
i! alignmentInBytes: type=int64_t; val=1;
i! isVirtual: type=bool; val=false;
i! attributeName: type=cudnnBackendAttributeName_t; val=CUDNN_ATTR_TENSOR_BYTE_ALIGNMENT (900);
i! attributeType: type=cudnnBackendAttributeType_t; val=CUDNN_TYPE_INT64 (3);
i! elementCount: type=int64_t; val=1;
i! [0]: type=int64_t; val=16;
i! Time: 2022-11-25T08:47:46.254408 (0d+0h+0m+0s since start)
i! Process=391; Thread=391; GPU=NULL; Handle=NULL; StreamId=NULL.

I! CuDNN (v8204) function cudnnBackendFinalize() called:
i! descriptor: type=CUDNN_BACKEND_TENSOR_DESCRIPTOR:
i! type: type=cudnnDataType_t; val=CUDNN_DATA_FLOAT (0);
i! nbDims: type=int; val=3;
i! dimA: type=int; val=[1,53799,96];
i! strideA: type=int; val=[5164704,96,1];
i! uid: type=int64_t; val=97;
i! alignmentInBytes: type=int64_t; val=16;
i! isVirtual: type=bool; val=false;
i! Time: 2022-11-25T08:47:46.254425 (0d+0h+0m+0s since start)
i! Process=391; Thread=391; GPU=NULL; Handle=NULL; StreamId=NULL.

I! CuDNN (v8204) function cudnnBackendCreateDescriptor() called:
i! descriptorType: type=cudnnBackendDescriptorType_t; val=CUDNN_BACKEND_TENSOR_DESCRIPTOR (17);
i! Time: 2022-11-25T08:47:46.254443 (0d+0h+0m+0s since start)
i! Process=391; Thread=391; GPU=NULL; Handle=NULL; StreamId=NULL.

I! CuDNN (v8204) function cudnnBackendSetAttribute() called:
i! descriptor: type=CUDNN_BACKEND_TENSOR_DESCRIPTOR:
i! type: type=cudnnDataType_t; val=CUDNN_DATA_FLOAT (0);
i! nbDims: type=int; val=-1;
i! dimA: type=int; val=[];
i! strideA: type=int; val=[];
i! uid: type=int64_t; val=-1;
i! alignmentInBytes: type=int64_t; val=1;
i! isVirtual: type=bool; val=false;
i! attributeName: type=cudnnBackendAttributeName_t; val=CUDNN_ATTR_TENSOR_DATA_TYPE (901);
i! attributeType: type=cudnnBackendAttributeType_t; val=CUDNN_TYPE_DATA_TYPE (1);
i! elementCount: type=int64_t; val=1;
i! [0]: type=cudnnDataType_t; val=CUDNN_DATA_FLOAT (0);
i! Time: 2022-11-25T08:47:46.254450 (0d+0h+0m+0s since start)
i! Process=391; Thread=391; GPU=NULL; Handle=NULL; StreamId=NULL.

I! CuDNN (v8204) function cudnnBackendSetAttribute() called:
i! descriptor: type=CUDNN_BACKEND_TENSOR_DESCRIPTOR:
i! type: type=cudnnDataType_t; val=CUDNN_DATA_FLOAT (0);
i! nbDims: type=int; val=-1;
i! dimA: type=int; val=[];
i! strideA: type=int; val=[];
i! uid: type=int64_t; val=-1;
i! alignmentInBytes: type=int64_t; val=1;
i! isVirtual: type=bool; val=false;
i! attributeName: type=cudnnBackendAttributeName_t; val=CUDNN_ATTR_TENSOR_DIMENSIONS (902);
i! attributeType: type=cudnnBackendAttributeType_t; val=CUDNN_TYPE_INT64 (3);
i! elementCount: type=int64_t; val=3;
i! [0]: type=int64_t; val=1;
i! [1]: type=int64_t; val=96;
i! [2]: type=int64_t; val=96;
i! Time: 2022-11-25T08:47:46.254456 (0d+0h+0m+0s since start)
i! Process=391; Thread=391; GPU=NULL; Handle=NULL; StreamId=NULL.

I! CuDNN (v8204) function cudnnBackendSetAttribute() called:
i! descriptor: type=CUDNN_BACKEND_TENSOR_DESCRIPTOR:
i! type: type=cudnnDataType_t; val=CUDNN_DATA_FLOAT (0);
i! nbDims: type=int; val=3;
i! dimA: type=int; val=[1,96,96];
i! strideA: type=int; val=[0,0,0];
i! uid: type=int64_t; val=-1;
i! alignmentInBytes: type=int64_t; val=1;
i! isVirtual: type=bool; val=false;
i! attributeName: type=cudnnBackendAttributeName_t; val=CUDNN_ATTR_TENSOR_STRIDES (903);
i! attributeType: type=cudnnBackendAttributeType_t; val=CUDNN_TYPE_INT64 (3);
i! elementCount: type=int64_t; val=3;
i! [0]: type=int64_t; val=9216;
i! [1]: type=int64_t; val=96;
i! [2]: type=int64_t; val=1;
i! Time: 2022-11-25T08:47:46.254468 (0d+0h+0m+0s since start)
i! Process=391; Thread=391; GPU=NULL; Handle=NULL; StreamId=NULL.

I! CuDNN (v8204) function cudnnBackendSetAttribute() called:
i! descriptor: type=CUDNN_BACKEND_TENSOR_DESCRIPTOR:
i! type: type=cudnnDataType_t; val=CUDNN_DATA_FLOAT (0);
i! nbDims: type=int; val=3;
i! dimA: type=int; val=[1,96,96];
i! strideA: type=int; val=[9216,96,1];
i! uid: type=int64_t; val=-1;
i! alignmentInBytes: type=int64_t; val=1;
i! isVirtual: type=bool; val=false;
i! attributeName: type=cudnnBackendAttributeName_t; val=CUDNN_ATTR_TENSOR_UNIQUE_ID (906);
i! attributeType: type=cudnnBackendAttributeType_t; val=CUDNN_TYPE_INT64 (3);
i! elementCount: type=int64_t; val=1;
i! [0]: type=int64_t; val=98;
i! Time: 2022-11-25T08:47:46.254475 (0d+0h+0m+0s since start)
i! Process=391; Thread=391; GPU=NULL; Handle=NULL; StreamId=NULL.

I! CuDNN (v8204) function cudnnBackendSetAttribute() called:
i! descriptor: type=CUDNN_BACKEND_TENSOR_DESCRIPTOR:
i! type: type=cudnnDataType_t; val=CUDNN_DATA_FLOAT (0);
i! nbDims: type=int; val=3;
i! dimA: type=int; val=[1,96,96];
i! strideA: type=int; val=[9216,96,1];
i! uid: type=int64_t; val=98;
i! alignmentInBytes: type=int64_t; val=1;
i! isVirtual: type=bool; val=false;
i! attributeName: type=cudnnBackendAttributeName_t; val=CUDNN_ATTR_TENSOR_BYTE_ALIGNMENT (900);
i! attributeType: type=cudnnBackendAttributeType_t; val=CUDNN_TYPE_INT64 (3);
i! elementCount: type=int64_t; val=1;
i! [0]: type=int64_t; val=16;
i! Time: 2022-11-25T08:47:46.254483 (0d+0h+0m+0s since start)
i! Process=391; Thread=391; GPU=NULL; Handle=NULL; StreamId=NULL.

I! CuDNN (v8204) function cudnnBackendFinalize() called:
i! descriptor: type=CUDNN_BACKEND_TENSOR_DESCRIPTOR:
i! type: type=cudnnDataType_t; val=CUDNN_DATA_FLOAT (0);
i! nbDims: type=int; val=3;
i! dimA: type=int; val=[1,96,96];
i! strideA: type=int; val=[9216,96,1];
i! uid: type=int64_t; val=98;
i! alignmentInBytes: type=int64_t; val=16;
i! isVirtual: type=bool; val=false;
i! Time: 2022-11-25T08:47:46.254493 (0d+0h+0m+0s since start)
i! Process=391; Thread=391; GPU=NULL; Handle=NULL; StreamId=NULL.

I! CuDNN (v8204) function cudnnBackendCreateDescriptor() called:
i! descriptorType: type=cudnnBackendDescriptorType_t; val=CUDNN_BACKEND_TENSOR_DESCRIPTOR (17);
i! Time: 2022-11-25T08:47:46.254498 (0d+0h+0m+0s since start)
i! Process=391; Thread=391; GPU=NULL; Handle=NULL; StreamId=NULL.

I! CuDNN (v8204) function cudnnBackendSetAttribute() called:
i! descriptor: type=CUDNN_BACKEND_TENSOR_DESCRIPTOR:
i! type: type=cudnnDataType_t; val=CUDNN_DATA_FLOAT (0);
i! nbDims: type=int; val=-1;
i! dimA: type=int; val=[];
i! strideA: type=int; val=[];
i! uid: type=int64_t; val=-1;
i! alignmentInBytes: type=int64_t; val=1;
i! isVirtual: type=bool; val=false;
i! attributeName: type=cudnnBackendAttributeName_t; val=CUDNN_ATTR_TENSOR_DATA_TYPE (901);
i! attributeType: type=cudnnBackendAttributeType_t; val=CUDNN_TYPE_DATA_TYPE (1);
i! elementCount: type=int64_t; val=1;
i! [0]: type=cudnnDataType_t; val=CUDNN_DATA_FLOAT (0);
i! Time: 2022-11-25T08:47:46.254504 (0d+0h+0m+0s since start)
i! Process=391; Thread=391; GPU=NULL; Handle=NULL; StreamId=NULL.

I! CuDNN (v8204) function cudnnBackendSetAttribute() called:
i! descriptor: type=CUDNN_BACKEND_TENSOR_DESCRIPTOR:
i! type: type=cudnnDataType_t; val=CUDNN_DATA_FLOAT (0);
i! nbDims: type=int; val=-1;
i! dimA: type=int; val=[];
i! strideA: type=int; val=[];
i! uid: type=int64_t; val=-1;
i! alignmentInBytes: type=int64_t; val=1;
i! isVirtual: type=bool; val=false;
i! attributeName: type=cudnnBackendAttributeName_t; val=CUDNN_ATTR_TENSOR_DIMENSIONS (902);
i! attributeType: type=cudnnBackendAttributeType_t; val=CUDNN_TYPE_INT64 (3);
i! elementCount: type=int64_t; val=3;
i! [0]: type=int64_t; val=1;
i! [1]: type=int64_t; val=53799;
i! [2]: type=int64_t; val=96;
i! Time: 2022-11-25T08:47:46.254511 (0d+0h+0m+0s since start)
i! Process=391; Thread=391; GPU=NULL; Handle=NULL; StreamId=NULL.

I! CuDNN (v8204) function cudnnBackendSetAttribute() called:
i! descriptor: type=CUDNN_BACKEND_TENSOR_DESCRIPTOR:
i! type: type=cudnnDataType_t; val=CUDNN_DATA_FLOAT (0);
i! nbDims: type=int; val=3;
i! dimA: type=int; val=[1,53799,96];
i! strideA: type=int; val=[0,0,0];
i! uid: type=int64_t; val=-1;
i! alignmentInBytes: type=int64_t; val=1;
i! isVirtual: type=bool; val=false;
i! attributeName: type=cudnnBackendAttributeName_t; val=CUDNN_ATTR_TENSOR_STRIDES (903);
i! attributeType: type=cudnnBackendAttributeType_t; val=CUDNN_TYPE_INT64 (3);
i! elementCount: type=int64_t; val=3;
i! [0]: type=int64_t; val=5164704;
i! [1]: type=int64_t; val=96;
i! [2]: type=int64_t; val=1;
i! Time: 2022-11-25T08:47:46.254517 (0d+0h+0m+0s since start)
i! Process=391; Thread=391; GPU=NULL; Handle=NULL; StreamId=NULL.

I! CuDNN (v8204) function cudnnBackendSetAttribute() called:
i! descriptor: type=CUDNN_BACKEND_TENSOR_DESCRIPTOR:
i! type: type=cudnnDataType_t; val=CUDNN_DATA_FLOAT (0);
i! nbDims: type=int; val=3;
i! dimA: type=int; val=[1,53799,96];
i! strideA: type=int; val=[5164704,96,1];
i! uid: type=int64_t; val=-1;
i! alignmentInBytes: type=int64_t; val=1;
i! isVirtual: type=bool; val=false;
i! attributeName: type=cudnnBackendAttributeName_t; val=CUDNN_ATTR_TENSOR_UNIQUE_ID (906);
i! attributeType: type=cudnnBackendAttributeType_t; val=CUDNN_TYPE_INT64 (3);
i! elementCount: type=int64_t; val=1;
i! [0]: type=int64_t; val=65;
i! Time: 2022-11-25T08:47:46.254525 (0d+0h+0m+0s since start)
i! Process=391; Thread=391; GPU=NULL; Handle=NULL; StreamId=NULL.

I! CuDNN (v8204) function cudnnBackendSetAttribute() called:
i! descriptor: type=CUDNN_BACKEND_TENSOR_DESCRIPTOR:
i! type: type=cudnnDataType_t; val=CUDNN_DATA_FLOAT (0);
i! nbDims: type=int; val=3;
i! dimA: type=int; val=[1,53799,96];
i! strideA: type=int; val=[5164704,96,1];
i! uid: type=int64_t; val=65;
i! alignmentInBytes: type=int64_t; val=1;
i! isVirtual: type=bool; val=false;
i! attributeName: type=cudnnBackendAttributeName_t; val=CUDNN_ATTR_TENSOR_BYTE_ALIGNMENT (900);
i! attributeType: type=cudnnBackendAttributeType_t; val=CUDNN_TYPE_INT64 (3);
i! elementCount: type=int64_t; val=1;
i! [0]: type=int64_t; val=16;
i! Time: 2022-11-25T08:47:46.254532 (0d+0h+0m+0s since start)
i! Process=391; Thread=391; GPU=NULL; Handle=NULL; StreamId=NULL.

I! CuDNN (v8204) function cudnnBackendSetAttribute() called:
i! descriptor: type=CUDNN_BACKEND_TENSOR_DESCRIPTOR:
i! type: type=cudnnDataType_t; val=CUDNN_DATA_FLOAT (0);
i! nbDims: type=int; val=3;
i! dimA: type=int; val=[1,53799,96];
i! strideA: type=int; val=[5164704,96,1];
i! uid: type=int64_t; val=65;
i! alignmentInBytes: type=int64_t; val=16;
i! isVirtual: type=bool; val=false;
i! attributeName: type=cudnnBackendAttributeName_t; val=CUDNN_ATTR_TENSOR_IS_VIRTUAL (907);
i! attributeType: type=cudnnBackendAttributeType_t; val=CUDNN_TYPE_BOOLEAN (2);
i! elementCount: type=int64_t; val=1;
i! [0]: type=bool; val=true;
i! Time: 2022-11-25T08:47:46.254538 (0d+0h+0m+0s since start)
i! Process=391; Thread=391; GPU=NULL; Handle=NULL; StreamId=NULL.

I! CuDNN (v8204) function cudnnBackendFinalize() called:
i! descriptor: type=CUDNN_BACKEND_TENSOR_DESCRIPTOR:
i! type: type=cudnnDataType_t; val=CUDNN_DATA_FLOAT (0);
i! nbDims: type=int; val=3;
i! dimA: type=int; val=[1,53799,96];
i! strideA: type=int; val=[5164704,96,1];
i! uid: type=int64_t; val=65;
i! alignmentInBytes: type=int64_t; val=16;
i! isVirtual: type=bool; val=true;
i! Time: 2022-11-25T08:47:46.254543 (0d+0h+0m+0s since start)
i! Process=391; Thread=391; GPU=NULL; Handle=NULL; StreamId=NULL.

I! CuDNN (v8204) function cudnnBackendCreateDescriptor() called:
i! descriptorType: type=cudnnBackendDescriptorType_t; val=CUDNN_BACKEND_MATMUL_DESCRIPTOR (18);
i! Time: 2022-11-25T08:47:46.254586 (0d+0h+0m+0s since start)
i! Process=391; Thread=391; GPU=NULL; Handle=NULL; StreamId=NULL.

I! CuDNN (v8204) function cudnnBackendSetAttribute() called:
i! descriptor: type=CUDNN_BACKEND_MATMUL_DESCRIPTOR; val=NOT_IMPLEMENTED;
i! attributeName: type=cudnnBackendAttributeName_t; val=CUDNN_ATTR_MATMUL_COMP_TYPE (1500);
i! attributeType: type=cudnnBackendAttributeType_t; val=CUDNN_TYPE_DATA_TYPE (1);
i! elementCount: type=int64_t; val=1;
i! [0]: type=cudnnDataType_t; val=CUDNN_DATA_FLOAT (0);
i! Time: 2022-11-25T08:47:46.254595 (0d+0h+0m+0s since start)
i! Process=391; Thread=391; GPU=NULL; Handle=NULL; StreamId=NULL.

I! CuDNN (v8204) function cudnnBackendFinalize() called:
i! descriptor: type=CUDNN_BACKEND_MATMUL_DESCRIPTOR; val=NOT_IMPLEMENTED;
i! Time: 2022-11-25T08:47:46.254601 (0d+0h+0m+0s since start)
i! Process=391; Thread=391; GPU=NULL; Handle=NULL; StreamId=NULL.

I! CuDNN (v8204) function cudnnBackendCreateDescriptor() called:
i! descriptorType: type=cudnnBackendDescriptorType_t; val=CUDNN_BACKEND_OPERATION_MATMUL_DESCRIPTOR (19);
i! Time: 2022-11-25T08:47:46.254611 (0d+0h+0m+0s since start)
i! Process=391; Thread=391; GPU=NULL; Handle=NULL; StreamId=NULL.

I! CuDNN (v8204) function cudnnBackendSetAttribute() called:
i! descriptor: type=CUDNN_BACKEND_OPERATION_MATMUL_DESCRIPTOR; val=NOT_IMPLEMENTED;
i! attributeName: type=cudnnBackendAttributeName_t; val=CUDNN_ATTR_OPERATION_MATMUL_ADESC (1520);
i! attributeType: type=cudnnBackendAttributeType_t; val=CUDNN_TYPE_BACKEND_DESCRIPTOR (15);
i! elementCount: type=int64_t; val=1;
i! [0]: type=CUDNN_BACKEND_TENSOR_DESCRIPTOR; val=0x55d770801a90;
i! : type=bool; val=true;
i! Time: 2022-11-25T08:47:46.254618 (0d+0h+0m+0s since start)
i! Process=391; Thread=391; GPU=NULL; Handle=NULL; StreamId=NULL.

I! CuDNN (v8204) function cudnnBackendSetAttribute() called:
i! descriptor: type=CUDNN_BACKEND_OPERATION_MATMUL_DESCRIPTOR; val=NOT_IMPLEMENTED;
i! attributeName: type=cudnnBackendAttributeName_t; val=CUDNN_ATTR_OPERATION_MATMUL_BDESC (1521);
i! attributeType: type=cudnnBackendAttributeType_t; val=CUDNN_TYPE_BACKEND_DESCRIPTOR (15);
i! elementCount: type=int64_t; val=1;
i! [0]: type=CUDNN_BACKEND_TENSOR_DESCRIPTOR; val=0x55d774a718f0;
i! : type=bool; val=true;
i! Time: 2022-11-25T08:47:46.254624 (0d+0h+0m+0s since start)
i! Process=391; Thread=391; GPU=NULL; Handle=NULL; StreamId=NULL.

I! CuDNN (v8204) function cudnnBackendSetAttribute() called:
i! descriptor: type=CUDNN_BACKEND_OPERATION_MATMUL_DESCRIPTOR; val=NOT_IMPLEMENTED;
i! attributeName: type=cudnnBackendAttributeName_t; val=CUDNN_ATTR_OPERATION_MATMUL_CDESC (1522);
i! attributeType: type=cudnnBackendAttributeType_t; val=CUDNN_TYPE_BACKEND_DESCRIPTOR (15);
i! elementCount: type=int64_t; val=1;
i! [0]: type=CUDNN_BACKEND_TENSOR_DESCRIPTOR; val=0x55d774a71a10;
i! : type=bool; val=true;
i! Time: 2022-11-25T08:47:46.254629 (0d+0h+0m+0s since start)
i! Process=391; Thread=391; GPU=NULL; Handle=NULL; StreamId=NULL.

I! CuDNN (v8204) function cudnnBackendSetAttribute() called:
i! descriptor: type=CUDNN_BACKEND_OPERATION_MATMUL_DESCRIPTOR; val=NOT_IMPLEMENTED;
i! attributeName: type=cudnnBackendAttributeName_t; val=CUDNN_ATTR_OPERATION_MATMUL_DESC (1523);
i! attributeType: type=cudnnBackendAttributeType_t; val=CUDNN_TYPE_BACKEND_DESCRIPTOR (15);
i! elementCount: type=int64_t; val=1;
i! [0]: type=CUDNN_BACKEND_MATMUL_DESCRIPTOR; val=0x55d774a71b00;
i! : type=bool; val=true;
i! Time: 2022-11-25T08:47:46.254634 (0d+0h+0m+0s since start)
i! Process=391; Thread=391; GPU=NULL; Handle=NULL; StreamId=NULL.

I! CuDNN (v8204) function cudnnBackendFinalize() called:
i! descriptor: type=CUDNN_BACKEND_OPERATION_MATMUL_DESCRIPTOR; val=NOT_IMPLEMENTED;
i! Time: 2022-11-25T08:47:46.254638 (0d+0h+0m+0s since start)
i! Process=391; Thread=391; GPU=NULL; Handle=NULL; StreamId=NULL.

I! CuDNN (v8204) function cudnnCreate() called:
i! handle: location=host; addr=0x7ffdaf5d5088;
i! Time: 2022-11-25T08:47:46.254668 (0d+0h+0m+0s since start)
i! Process=391; Thread=391; GPU=NULL; Handle=NULL; StreamId=NULL.

I! CuDNN (v8204) function cudnnBackendCreateDescriptor() called:
i! descriptorType: type=cudnnBackendDescriptorType_t; val=CUDNN_BACKEND_OPERATIONGRAPH_DESCRIPTOR (15);
i! Time: 2022-11-25T08:47:46.578905 (0d+0h+0m+0s since start)
i! Process=391; Thread=391; GPU=NULL; Handle=NULL; StreamId=NULL.

I! CuDNN (v8204) function cudnnBackendSetAttribute() called:
i! descriptor: type=CUDNN_BACKEND_OPERATIONGRAPH_DESCRIPTOR:
i! attributeName: type=cudnnBackendAttributeName_t; val=CUDNN_ATTR_OPERATIONGRAPH_OPS (801);
i! attributeType: type=cudnnBackendAttributeType_t; val=CUDNN_TYPE_BACKEND_DESCRIPTOR (15);
i! elementCount: type=int64_t; val=1;
i! [0]: type=CUDNN_BACKEND_OPERATION_MATMUL_DESCRIPTOR; val=0x55d742b2b870;
i! : type=bool; val=true;
i! Time: 2022-11-25T08:47:46.578955 (0d+0h+0m+0s since start)
i! Process=391; Thread=391; GPU=NULL; Handle=NULL; StreamId=NULL.

I! CuDNN (v8204) function cudnnBackendSetAttribute() called:
i! descriptor: type=CUDNN_BACKEND_OPERATIONGRAPH_DESCRIPTOR:
i! attributeName: type=cudnnBackendAttributeName_t; val=CUDNN_ATTR_OPERATIONGRAPH_HANDLE (800);
i! attributeType: type=cudnnBackendAttributeType_t; val=CUDNN_TYPE_HANDLE (0);
i! elementCount: type=int64_t; val=1;
i! [0]: type=cudnnHandle_t; streamId=(nil) (defaultStream);
i! Time: 2022-11-25T08:47:46.578974 (0d+0h+0m+0s since start)
i! Process=391; Thread=391; GPU=NULL; Handle=NULL; StreamId=NULL.

I! CuDNN (v8204) function cudnnBackendFinalize() called:
i! descriptor: type=CUDNN_BACKEND_OPERATIONGRAPH_DESCRIPTOR:
i! Time: 2022-11-25T08:47:46.578981 (0d+0h+0m+0s since start)
i! Process=391; Thread=391; GPU=NULL; Handle=NULL; StreamId=NULL.

I! CuDNN (v8204) function cudnnBackendCreateDescriptor() called:
i! descriptorType: type=cudnnBackendDescriptorType_t; val=CUDNN_BACKEND_ENGINEHEUR_DESCRIPTOR (4);
i! Time: 2022-11-25T08:47:46.579050 (0d+0h+0m+0s since start)
i! Process=391; Thread=391; GPU=NULL; Handle=NULL; StreamId=NULL.

I! CuDNN (v8204) function cudnnBackendSetAttribute() called:
i! descriptor: type=CUDNN_BACKEND_ENGINEHEUR_DESCRIPTOR; val=NOT_IMPLEMENTED;
i! attributeName: type=cudnnBackendAttributeName_t; val=CUDNN_ATTR_ENGINEHEUR_OPERATION_GRAPH (201);
i! attributeType: type=cudnnBackendAttributeType_t; val=CUDNN_TYPE_BACKEND_DESCRIPTOR (15);
i! elementCount: type=int64_t; val=1;
i! [0]: type=CUDNN_BACKEND_OPERATIONGRAPH_DESCRIPTOR; val=0x55d780dc2120;
i! : type=bool; val=true;
i! Time: 2022-11-25T08:47:46.579061 (0d+0h+0m+0s since start)
i! Process=391; Thread=391; GPU=NULL; Handle=NULL; StreamId=NULL.

I! CuDNN (v8204) function cudnnBackendSetAttribute() called:
i! descriptor: type=CUDNN_BACKEND_ENGINEHEUR_DESCRIPTOR; val=NOT_IMPLEMENTED;
i! attributeName: type=cudnnBackendAttributeName_t; val=CUDNN_ATTR_ENGINEHEUR_MODE (200);
i! attributeType: type=cudnnBackendAttributeType_t; val=CUDNN_TYPE_HEUR_MODE (8);
i! elementCount: type=int64_t; val=1;
i! [0]: type=int; val=0;
i! Time: 2022-11-25T08:47:46.579072 (0d+0h+0m+0s since start)
i! Process=391; Thread=391; GPU=NULL; Handle=NULL; StreamId=NULL.

I! CuDNN (v8204) function cudnnBackendFinalize() called:
i! descriptor: type=CUDNN_BACKEND_ENGINEHEUR_DESCRIPTOR; val=NOT_IMPLEMENTED;
i! Time: 2022-11-25T08:47:46.579077 (0d+0h+0m+0s since start)
i! Process=391; Thread=391; GPU=NULL; Handle=NULL; StreamId=NULL.

I! CuDNN (v8204) function cudnnBackendGetAttribute() called:
i! descriptor: type=CUDNN_BACKEND_ENGINEHEUR_DESCRIPTOR; val=NOT_IMPLEMENTED;
i! attributeName: type=cudnnBackendAttributeName_t; val=CUDNN_ATTR_ENGINEHEUR_RESULTS (202);
i! attributeType: type=cudnnBackendAttributeType_t; val=CUDNN_TYPE_BACKEND_DESCRIPTOR (15);
i! requestedElementCount: type=int64_t; val=0;
i! elementCount: location=host; addr=0x7ffdaf5d4d70;
i! arrayOfElements: location=host; addr=NULL_PTR;
i! Time: 2022-11-25T08:47:46.579096 (0d+0h+0m+0s since start)
i! Process=391; Thread=391; GPU=NULL; Handle=NULL; StreamId=NULL.

I! CuDNN (v8204) function cudnnBackendGetAttribute() called:
i! descriptor: type=CUDNN_BACKEND_ENGINEHEUR_DESCRIPTOR; val=NOT_IMPLEMENTED;
i! attributeName: type=cudnnBackendAttributeName_t; val=CUDNN_ATTR_ENGINEHEUR_RESULTS (202);
i! attributeType: type=cudnnBackendAttributeType_t; val=CUDNN_TYPE_BACKEND_DESCRIPTOR (15);
i! requestedElementCount: type=int64_t; val=0;
i! elementCount: location=host; addr=0x7ffdaf5d4d70;
i! arrayOfElements: location=host; addr=NULL_PTR;
i! Time: 2022-11-25T08:47:46.579119 (0d+0h+0m+0s since start)
i! Process=391; Thread=391; GPU=NULL; Handle=NULL; StreamId=NULL.

I! CuDNN (v8204) function cudnnBackendCreateDescriptor() called:
i! descriptorType: type=cudnnBackendDescriptorType_t; val=CUDNN_BACKEND_ENGINECFG_DESCRIPTOR (3);
i! Time: 2022-11-25T08:47:46.579125 (0d+0h+0m+0s since start)
i! Process=391; Thread=391; GPU=NULL; Handle=NULL; StreamId=NULL.

I! CuDNN (v8204) function cudnnBackendCreateDescriptor() called:
i! descriptorType: type=cudnnBackendDescriptorType_t; val=CUDNN_BACKEND_ENGINECFG_DESCRIPTOR (3);
i! Time: 2022-11-25T08:47:46.579130 (0d+0h+0m+0s since start)
i! Process=391; Thread=391; GPU=NULL; Handle=NULL; StreamId=NULL.

I! CuDNN (v8204) function cudnnBackendCreateDescriptor() called:
i! descriptorType: type=cudnnBackendDescriptorType_t; val=CUDNN_BACKEND_ENGINECFG_DESCRIPTOR (3);
i! Time: 2022-11-25T08:47:46.579135 (0d+0h+0m+0s since start)
i! Process=391; Thread=391; GPU=NULL; Handle=NULL; StreamId=NULL.

I! CuDNN (v8204) function cudnnBackendGetAttribute() called:
i! descriptor: type=CUDNN_BACKEND_ENGINEHEUR_DESCRIPTOR; val=NOT_IMPLEMENTED;
i! attributeName: type=cudnnBackendAttributeName_t; val=CUDNN_ATTR_ENGINEHEUR_RESULTS (202);
i! attributeType: type=cudnnBackendAttributeType_t; val=CUDNN_TYPE_BACKEND_DESCRIPTOR (15);
i! requestedElementCount: type=int64_t; val=3;
i! elementCount: location=host; addr=0x7ffdaf5d4c88;
i! arrayOfElements: location=host; addr=0x55d780ec1cb0;
i! Time: 2022-11-25T08:47:46.579141 (0d+0h+0m+0s since start)
i! Process=391; Thread=391; GPU=NULL; Handle=NULL; StreamId=NULL.

I! CuDNN (v8204) function cudnnBackendCreateDescriptor() called:
i! descriptorType: type=cudnnBackendDescriptorType_t; val=CUDNN_BACKEND_EXECUTION_PLAN_DESCRIPTOR (5);
i! Time: 2022-11-25T08:47:46.579182 (0d+0h+0m+0s since start)
i! Process=391; Thread=391; GPU=NULL; Handle=NULL; StreamId=NULL.

I! CuDNN (v8204) function cudnnBackendSetAttribute() called:
i! descriptor: type=CUDNN_BACKEND_EXECUTION_PLAN_DESCRIPTOR:
i! engcfg: type=CUDNN_BACKEND_ENGINECFG_DESCRIPTOR:
i! engine: type=CUDNN_BACKEND_ENGINE_DESCRIPTOR:
i! opGraph: type=void*; val=NULL_PTR;
i! engine_id: type=int; val=-1;
i! knobDesc: type=cudnnBackendDescriptor_t; val=NULL_PTR;
i! numericalNotes: type=cudnnBackendDescriptor_t; val=NULL_PTR;
i! behaviorNotes: type=cudnnBackendDescriptor_t; val=NULL_PTR;
i! attributeName: type=cudnnBackendAttributeName_t; val=CUDNN_ATTR_EXECUTION_PLAN_ENGINE_CONFIG (401);
i! attributeType: type=cudnnBackendAttributeType_t; val=CUDNN_TYPE_BACKEND_DESCRIPTOR (15);
i! elementCount: type=int64_t; val=1;
i! [0]: type=CUDNN_BACKEND_ENGINECFG_DESCRIPTOR; val=0x55d77efbb970;
i! : type=bool; val=true;
i! Time: 2022-11-25T08:47:46.579193 (0d+0h+0m+0s since start)
i! Process=391; Thread=391; GPU=NULL; Handle=NULL; StreamId=NULL.

I! CuDNN (v8204) function cudnnBackendSetAttribute() called:
i! descriptor: type=CUDNN_BACKEND_EXECUTION_PLAN_DESCRIPTOR:
i! engcfg: type=CUDNN_BACKEND_ENGINECFG_DESCRIPTOR:
i! engine: type=CUDNN_BACKEND_ENGINE_DESCRIPTOR:
i! opGraph: type=CUDNN_BACKEND_OPERATIONGRAPH_DESCRIPTOR:
i! engine_id: type=int; val=0;
i! knobDesc: type=cudnnBackendDescriptor_t:
i! CUDNN_KNOB_TYPE_KERNEL_CFG: type=int; val=27;
i! numericalNotes: type=cudnnBackendDescriptor_t:
i! CUDNN_NUMERICAL_NOTE_TENSOR_CORE: type=bool; val=true;
i! behaviorNotes: type=cudnnBackendDescriptor_t:
i! CUDNN_BEHAVIOR_NOTE_RUNTIME_COMPILATION: type=bool; val=true;
i! attributeName: type=cudnnBackendAttributeName_t; val=CUDNN_ATTR_EXECUTION_PLAN_HANDLE (400);
i! attributeType: type=cudnnBackendAttributeType_t; val=CUDNN_TYPE_HANDLE (0);
i! elementCount: type=int64_t; val=1;
i! [0]: type=cudnnHandle_t; streamId=(nil) (defaultStream);
i! Time: 2022-11-25T08:47:46.579202 (0d+0h+0m+0s since start)
i! Process=391; Thread=391; GPU=NULL; Handle=NULL; StreamId=NULL.

I! CuDNN (v8204) function cudnnBackendFinalize() called:
i! descriptor: type=CUDNN_BACKEND_EXECUTION_PLAN_DESCRIPTOR:
i! engcfg: type=CUDNN_BACKEND_ENGINECFG_DESCRIPTOR:
i! engine: type=CUDNN_BACKEND_ENGINE_DESCRIPTOR:
i! opGraph: type=CUDNN_BACKEND_OPERATIONGRAPH_DESCRIPTOR:
i! engine_id: type=int; val=0;
i! knobDesc: type=cudnnBackendDescriptor_t:
i! CUDNN_KNOB_TYPE_KERNEL_CFG: type=int; val=27;
i! numericalNotes: type=cudnnBackendDescriptor_t:
i! CUDNN_NUMERICAL_NOTE_TENSOR_CORE: type=bool; val=true;
i! behaviorNotes: type=cudnnBackendDescriptor_t:
i! CUDNN_BEHAVIOR_NOTE_RUNTIME_COMPILATION: type=bool; val=true;
i! Time: 2022-11-25T08:47:46.579211 (0d+0h+0m+0s since start)
i! Process=391; Thread=391; GPU=NULL; Handle=NULL; StreamId=NULL.

I! CuDNN (v8204) function cudnnBackendCreateDescriptor() called:
i! descriptorType: type=cudnnBackendDescriptorType_t; val=CUDNN_BACKEND_ENGINE_DESCRIPTOR (2);
i! Time: 2022-11-25T08:47:48.178545 (0d+0h+0m+2s since start)
i! Process=391; Thread=391; GPU=NULL; Handle=NULL; StreamId=NULL.

I! CuDNN (v8204) function cudnnBackendGetAttribute() called:
i! descriptor: type=CUDNN_BACKEND_ENGINECFG_DESCRIPTOR:
i! engine: type=CUDNN_BACKEND_ENGINE_DESCRIPTOR:
i! opGraph: type=CUDNN_BACKEND_OPERATIONGRAPH_DESCRIPTOR:
i! engine_id: type=int; val=0;
i! knobDesc: type=cudnnBackendDescriptor_t:
i! CUDNN_KNOB_TYPE_KERNEL_CFG: type=int; val=27;
i! numericalNotes: type=cudnnBackendDescriptor_t:
i! CUDNN_NUMERICAL_NOTE_TENSOR_CORE: type=bool; val=true;
i! behaviorNotes: type=cudnnBackendDescriptor_t:
i! CUDNN_BEHAVIOR_NOTE_RUNTIME_COMPILATION: type=bool; val=true;
i! attributeName: type=cudnnBackendAttributeName_t; val=CUDNN_ATTR_ENGINECFG_ENGINE (300);
i! attributeType: type=cudnnBackendAttributeType_t; val=CUDNN_TYPE_BACKEND_DESCRIPTOR (15);
i! requestedElementCount: type=int64_t; val=1;
i! elementCount: location=host; addr=0x7ffdaf5d4c90;
i! arrayOfElements: location=host; addr=0x7ffdaf5d4c88;
i! Time: 2022-11-25T08:47:48.178596 (0d+0h+0m+2s since start)
i! Process=391; Thread=391; GPU=NULL; Handle=NULL; StreamId=NULL.

I! CuDNN (v8204) function cudnnBackendCreateDescriptor() called:
i! descriptorType: type=cudnnBackendDescriptorType_t; val=CUDNN_BACKEND_KNOB_CHOICE_DESCRIPTOR (7);
i! Time: 2022-11-25T08:47:48.178611 (0d+0h+0m+2s since start)
i! Process=391; Thread=391; GPU=NULL; Handle=NULL; StreamId=NULL.

I! CuDNN (v8204) function cudnnBackendCreateDescriptor() called:
i! descriptorType: type=cudnnBackendDescriptorType_t; val=CUDNN_BACKEND_KNOB_CHOICE_DESCRIPTOR (7);
i! Time: 2022-11-25T08:47:48.178616 (0d+0h+0m+2s since start)
i! Process=391; Thread=391; GPU=NULL; Handle=NULL; StreamId=NULL.

I! CuDNN (v8204) function cudnnBackendCreateDescriptor() called:
i! descriptorType: type=cudnnBackendDescriptorType_t; val=CUDNN_BACKEND_KNOB_CHOICE_DESCRIPTOR (7);
i! Time: 2022-11-25T08:47:48.178621 (0d+0h+0m+2s since start)
i! Process=391; Thread=391; GPU=NULL; Handle=NULL; StreamId=NULL.

I! CuDNN (v8204) function cudnnBackendCreateDescriptor() called:
i! descriptorType: type=cudnnBackendDescriptorType_t; val=CUDNN_BACKEND_KNOB_CHOICE_DESCRIPTOR (7);
i! Time: 2022-11-25T08:47:48.178625 (0d+0h+0m+2s since start)
i! Process=391; Thread=391; GPU=NULL; Handle=NULL; StreamId=NULL.

I! CuDNN (v8204) function cudnnBackendCreateDescriptor() called:
i! descriptorType: type=cudnnBackendDescriptorType_t; val=CUDNN_BACKEND_KNOB_CHOICE_DESCRIPTOR (7);
i! Time: 2022-11-25T08:47:48.178637 (0d+0h+0m+2s since start)
i! Process=391; Thread=391; GPU=NULL; Handle=NULL; StreamId=NULL.

I! CuDNN (v8204) function cudnnBackendCreateDescriptor() called:
i! descriptorType: type=cudnnBackendDescriptorType_t; val=CUDNN_BACKEND_KNOB_CHOICE_DESCRIPTOR (7);
i! Time: 2022-11-25T08:47:48.178642 (0d+0h+0m+2s since start)
i! Process=391; Thread=391; GPU=NULL; Handle=NULL; StreamId=NULL.

I! CuDNN (v8204) function cudnnBackendCreateDescriptor() called:
i! descriptorType: type=cudnnBackendDescriptorType_t; val=CUDNN_BACKEND_KNOB_CHOICE_DESCRIPTOR (7);
i! Time: 2022-11-25T08:47:48.178646 (0d+0h+0m+2s since start)
i! Process=391; Thread=391; GPU=NULL; Handle=NULL; StreamId=NULL.

I! CuDNN (v8204) function cudnnBackendCreateDescriptor() called:
i! descriptorType: type=cudnnBackendDescriptorType_t; val=CUDNN_BACKEND_KNOB_CHOICE_DESCRIPTOR (7);
i! Time: 2022-11-25T08:47:48.178650 (0d+0h+0m+2s since start)
i! Process=391; Thread=391; GPU=NULL; Handle=NULL; StreamId=NULL.

I! CuDNN (v8204) function cudnnBackendCreateDescriptor() called:
i! descriptorType: type=cudnnBackendDescriptorType_t; val=CUDNN_BACKEND_KNOB_CHOICE_DESCRIPTOR (7);
i! Time: 2022-11-25T08:47:48.178655 (0d+0h+0m+2s since start)
i! Process=391; Thread=391; GPU=NULL; Handle=NULL; StreamId=NULL.

I! CuDNN (v8204) function cudnnBackendCreateDescriptor() called:
i! descriptorType: type=cudnnBackendDescriptorType_t; val=CUDNN_BACKEND_KNOB_CHOICE_DESCRIPTOR (7);
i! Time: 2022-11-25T08:47:48.178659 (0d+0h+0m+2s since start)
i! Process=391; Thread=391; GPU=NULL; Handle=NULL; StreamId=NULL.

I! CuDNN (v8204) function cudnnBackendCreateDescriptor() called:
i! descriptorType: type=cudnnBackendDescriptorType_t; val=CUDNN_BACKEND_KNOB_CHOICE_DESCRIPTOR (7);
i! Time: 2022-11-25T08:47:48.178663 (0d+0h+0m+2s since start)
i! Process=391; Thread=391; GPU=NULL; Handle=NULL; StreamId=NULL.

I! CuDNN (v8204) function cudnnBackendCreateDescriptor() called:
i! descriptorType: type=cudnnBackendDescriptorType_t; val=CUDNN_BACKEND_KNOB_CHOICE_DESCRIPTOR (7);
i! Time: 2022-11-25T08:47:48.178669 (0d+0h+0m+2s since start)
i! Process=391; Thread=391; GPU=NULL; Handle=NULL; StreamId=NULL.

I! CuDNN (v8204) function cudnnBackendCreateDescriptor() called:
i! descriptorType: type=cudnnBackendDescriptorType_t; val=CUDNN_BACKEND_KNOB_CHOICE_DESCRIPTOR (7);
i! Time: 2022-11-25T08:47:48.178675 (0d+0h+0m+2s since start)
i! Process=391; Thread=391; GPU=NULL; Handle=NULL; StreamId=NULL.

I! CuDNN (v8204) function cudnnBackendCreateDescriptor() called:
i! descriptorType: type=cudnnBackendDescriptorType_t; val=CUDNN_BACKEND_KNOB_CHOICE_DESCRIPTOR (7);
i! Time: 2022-11-25T08:47:48.178679 (0d+0h+0m+2s since start)
i! Process=391; Thread=391; GPU=NULL; Handle=NULL; StreamId=NULL.

I! CuDNN (v8204) function cudnnBackendCreateDescriptor() called:
i! descriptorType: type=cudnnBackendDescriptorType_t; val=CUDNN_BACKEND_KNOB_CHOICE_DESCRIPTOR (7);
i! Time: 2022-11-25T08:47:48.178684 (0d+0h+0m+2s since start)
i! Process=391; Thread=391; GPU=NULL; Handle=NULL; StreamId=NULL.

I! CuDNN (v8204) function cudnnBackendCreateDescriptor() called:
i! descriptorType: type=cudnnBackendDescriptorType_t; val=CUDNN_BACKEND_KNOB_CHOICE_DESCRIPTOR (7);
i! Time: 2022-11-25T08:47:48.178688 (0d+0h+0m+2s since start)
i! Process=391; Thread=391; GPU=NULL; Handle=NULL; StreamId=NULL.

I! CuDNN (v8204) function cudnnBackendCreateDescriptor() called:
i! descriptorType: type=cudnnBackendDescriptorType_t; val=CUDNN_BACKEND_KNOB_CHOICE_DESCRIPTOR (7);
i! Time: 2022-11-25T08:47:48.178692 (0d+0h+0m+2s since start)
i! Process=391; Thread=391; GPU=NULL; Handle=NULL; StreamId=NULL.

I! CuDNN (v8204) function cudnnBackendCreateDescriptor() called:
i! descriptorType: type=cudnnBackendDescriptorType_t; val=CUDNN_BACKEND_KNOB_CHOICE_DESCRIPTOR (7);
i! Time: 2022-11-25T08:47:48.178698 (0d+0h+0m+2s since start)
i! Process=391; Thread=391; GPU=NULL; Handle=NULL; StreamId=NULL.

I! CuDNN (v8204) function cudnnBackendCreateDescriptor() called:
i! descriptorType: type=cudnnBackendDescriptorType_t; val=CUDNN_BACKEND_KNOB_CHOICE_DESCRIPTOR (7);
i! Time: 2022-11-25T08:47:48.178703 (0d+0h+0m+2s since start)
i! Process=391; Thread=391; GPU=NULL; Handle=NULL; StreamId=NULL.

I! CuDNN (v8204) function cudnnBackendCreateDescriptor() called:
i! descriptorType: type=cudnnBackendDescriptorType_t; val=CUDNN_BACKEND_KNOB_CHOICE_DESCRIPTOR (7);
i! Time: 2022-11-25T08:47:48.178707 (0d+0h+0m+2s since start)
i! Process=391; Thread=391; GPU=NULL; Handle=NULL; StreamId=NULL.

I! CuDNN (v8204) function cudnnBackendCreateDescriptor() called:
i! descriptorType: type=cudnnBackendDescriptorType_t; val=CUDNN_BACKEND_KNOB_CHOICE_DESCRIPTOR (7);
i! Time: 2022-11-25T08:47:48.178712 (0d+0h+0m+2s since start)
i! Process=391; Thread=391; GPU=NULL; Handle=NULL; StreamId=NULL.

I! CuDNN (v8204) function cudnnBackendCreateDescriptor() called:
i! descriptorType: type=cudnnBackendDescriptorType_t; val=CUDNN_BACKEND_KNOB_CHOICE_DESCRIPTOR (7);
i! Time: 2022-11-25T08:47:48.178716 (0d+0h+0m+2s since start)
i! Process=391; Thread=391; GPU=NULL; Handle=NULL; StreamId=NULL.

I! CuDNN (v8204) function cudnnBackendCreateDescriptor() called:
i! descriptorType: type=cudnnBackendDescriptorType_t; val=CUDNN_BACKEND_KNOB_CHOICE_DESCRIPTOR (7);
i! Time: 2022-11-25T08:47:48.178720 (0d+0h+0m+2s since start)
i! Process=391; Thread=391; GPU=NULL; Handle=NULL; StreamId=NULL.

I! CuDNN (v8204) function cudnnBackendCreateDescriptor() called:
i! descriptorType: type=cudnnBackendDescriptorType_t; val=CUDNN_BACKEND_KNOB_CHOICE_DESCRIPTOR (7);
i! Time: 2022-11-25T08:47:48.178725 (0d+0h+0m+2s since start)
i! Process=391; Thread=391; GPU=NULL; Handle=NULL; StreamId=NULL.

I! CuDNN (v8204) function cudnnBackendCreateDescriptor() called:
i! descriptorType: type=cudnnBackendDescriptorType_t; val=CUDNN_BACKEND_KNOB_CHOICE_DESCRIPTOR (7);
i! Time: 2022-11-25T08:47:48.178729 (0d+0h+0m+2s since start)
i! Process=391; Thread=391; GPU=NULL; Handle=NULL; StreamId=NULL.

I! CuDNN (v8204) function cudnnBackendGetAttribute() called:
i! descriptor: type=CUDNN_BACKEND_ENGINE_DESCRIPTOR:
i! opGraph: type=CUDNN_BACKEND_OPERATIONGRAPH_DESCRIPTOR:
i! engine_id: type=int; val=0;
i! knobDesc: type=cudnnBackendDescriptor_t:
i! CUDNN_KNOB_TYPE_KERNEL_CFG: type=int; val=27;
i! numericalNotes: type=cudnnBackendDescriptor_t:
i! CUDNN_NUMERICAL_NOTE_TENSOR_CORE: type=bool; val=true;
i! behaviorNotes: type=cudnnBackendDescriptor_t:
i! CUDNN_BEHAVIOR_NOTE_RUNTIME_COMPILATION: type=bool; val=true;
i! attributeName: type=cudnnBackendAttributeName_t; val=CUDNN_ATTR_ENGINE_GLOBAL_INDEX (1301);
i! attributeType: type=cudnnBackendAttributeType_t; val=CUDNN_TYPE_INT64 (3);
i! requestedElementCount: type=int64_t; val=1;
i! elementCount: location=host; addr=0x7ffdaf5d4800;
i! arrayOfElements: location=host; addr=0x7ffdaf5d4808;
i! Time: 2022-11-25T08:47:48.178736 (0d+0h+0m+2s since start)
i! Process=391; Thread=391; GPU=NULL; Handle=NULL; StreamId=NULL.

I! CuDNN (v8204) function cudnnBackendGetAttribute() called:
i! descriptor: type=CUDNN_BACKEND_ENGINECFG_DESCRIPTOR:
i! engine: type=CUDNN_BACKEND_ENGINE_DESCRIPTOR:
i! opGraph: type=CUDNN_BACKEND_OPERATIONGRAPH_DESCRIPTOR:
i! engine_id: type=int; val=0;
i! knobDesc: type=cudnnBackendDescriptor_t:
i! CUDNN_KNOB_TYPE_KERNEL_CFG: type=int; val=27;
i! numericalNotes: type=cudnnBackendDescriptor_t:
i! CUDNN_NUMERICAL_NOTE_TENSOR_CORE: type=bool; val=true;
i! behaviorNotes: type=cudnnBackendDescriptor_t:
i! CUDNN_BEHAVIOR_NOTE_RUNTIME_COMPILATION: type=bool; val=true;
i! attributeName: type=cudnnBackendAttributeName_t; val=CUDNN_ATTR_ENGINECFG_KNOB_CHOICES (302);
i! attributeType: type=cudnnBackendAttributeType_t; val=CUDNN_TYPE_BACKEND_DESCRIPTOR (15);
i! requestedElementCount: type=int64_t; val=25;
i! elementCount: location=host; addr=0x7ffdaf5d4810;
i! arrayOfElements: location=host; addr=0x7ffdaf5d4840;
i! Time: 2022-11-25T08:47:48.178745 (0d+0h+0m+2s since start)
i! Process=391; Thread=391; GPU=NULL; Handle=NULL; StreamId=NULL.

I! CuDNN (v8204) function cudnnBackendGetAttribute() called:
i! descriptor: type=CUDNN_BACKEND_KNOB_CHOICE_DESCRIPTOR; val=NOT_IMPLEMENTED;
i! attributeName: type=cudnnBackendAttributeName_t; val=CUDNN_ATTR_KNOB_CHOICE_KNOB_TYPE (600);
i! attributeType: type=cudnnBackendAttributeType_t; val=CUDNN_TYPE_KNOB_TYPE (9);
i! requestedElementCount: type=int64_t; val=1;
i! elementCount: location=host; addr=NULL_PTR;
i! arrayOfElements: location=host; addr=0x7ffdaf5d47fc;
i! Time: 2022-11-25T08:47:48.178760 (0d+0h+0m+2s since start)
i! Process=391; Thread=391; GPU=NULL; Handle=NULL; StreamId=NULL.

I! CuDNN (v8204) function cudnnBackendGetAttribute() called:
i! descriptor: type=CUDNN_BACKEND_KNOB_CHOICE_DESCRIPTOR; val=NOT_IMPLEMENTED;
i! attributeName: type=cudnnBackendAttributeName_t; val=CUDNN_ATTR_KNOB_CHOICE_KNOB_VALUE (601);
i! attributeType: type=cudnnBackendAttributeType_t; val=CUDNN_TYPE_INT64 (3);
i! requestedElementCount: type=int64_t; val=1;
i! elementCount: location=host; addr=NULL_PTR;
i! arrayOfElements: location=host; addr=0x7ffdaf5d4818;
i! Time: 2022-11-25T08:47:48.178770 (0d+0h+0m+2s since start)
i! Process=391; Thread=391; GPU=NULL; Handle=NULL; StreamId=NULL.

I! CuDNN (v8204) function cudnnBackendDestroyDescriptor() called:
i! descriptor: type=CUDNN_BACKEND_KNOB_CHOICE_DESCRIPTOR; val=NOT_IMPLEMENTED;
i! Time: 2022-11-25T08:47:48.178786 (0d+0h+0m+2s since start)
i! Process=391; Thread=391; GPU=NULL; Handle=NULL; StreamId=NULL.

I! CuDNN (v8204) function cudnnBackendDestroyDescriptor() called:
i! descriptor: type=CUDNN_BACKEND_KNOB_CHOICE_DESCRIPTOR; val=NOT_IMPLEMENTED;
i! Time: 2022-11-25T08:47:48.178791 (0d+0h+0m+2s since start)
i! Process=391; Thread=391; GPU=NULL; Handle=NULL; StreamId=NULL.

I! CuDNN (v8204) function cudnnBackendDestroyDescriptor() called:
i! descriptor: type=CUDNN_BACKEND_KNOB_CHOICE_DESCRIPTOR; val=NOT_IMPLEMENTED;
i! Time: 2022-11-25T08:47:48.178795 (0d+0h+0m+2s since start)
i! Process=391; Thread=391; GPU=NULL; Handle=NULL; StreamId=NULL.

I! CuDNN (v8204) function cudnnBackendDestroyDescriptor() called:
i! descriptor: type=CUDNN_BACKEND_KNOB_CHOICE_DESCRIPTOR; val=NOT_IMPLEMENTED;
i! Time: 2022-11-25T08:47:48.178799 (0d+0h+0m+2s since start)
i! Process=391; Thread=391; GPU=NULL; Handle=NULL; StreamId=NULL.

I! CuDNN (v8204) function cudnnBackendDestroyDescriptor() called:
i! descriptor: type=CUDNN_BACKEND_KNOB_CHOICE_DESCRIPTOR; val=NOT_IMPLEMENTED;
i! Time: 2022-11-25T08:47:48.178803 (0d+0h+0m+2s since start)
i! Process=391; Thread=391; GPU=NULL; Handle=NULL; StreamId=NULL.

I! CuDNN (v8204) function cudnnBackendDestroyDescriptor() called:
i! descriptor: type=CUDNN_BACKEND_KNOB_CHOICE_DESCRIPTOR; val=NOT_IMPLEMENTED;
i! Time: 2022-11-25T08:47:48.178807 (0d+0h+0m+2s since start)
i! Process=391; Thread=391; GPU=NULL; Handle=NULL; StreamId=NULL.

I! CuDNN (v8204) function cudnnBackendDestroyDescriptor() called:
i! descriptor: type=CUDNN_BACKEND_KNOB_CHOICE_DESCRIPTOR; val=NOT_IMPLEMENTED;
i! Time: 2022-11-25T08:47:48.178811 (0d+0h+0m+2s since start)
i! Process=391; Thread=391; GPU=NULL; Handle=NULL; StreamId=NULL.

I! CuDNN (v8204) function cudnnBackendDestroyDescriptor() called:
i! descriptor: type=CUDNN_BACKEND_KNOB_CHOICE_DESCRIPTOR; val=NOT_IMPLEMENTED;
i! Time: 2022-11-25T08:47:48.178816 (0d+0h+0m+2s since start)
i! Process=391; Thread=391; GPU=NULL; Handle=NULL; StreamId=NULL.

I! CuDNN (v8204) function cudnnBackendDestroyDescriptor() called:
i! descriptor: type=CUDNN_BACKEND_KNOB_CHOICE_DESCRIPTOR; val=NOT_IMPLEMENTED;
i! Time: 2022-11-25T08:47:48.178820 (0d+0h+0m+2s since start)
i! Process=391; Thread=391; GPU=NULL; Handle=NULL; StreamId=NULL.

I! CuDNN (v8204) function cudnnBackendDestroyDescriptor() called:
i! descriptor: type=CUDNN_BACKEND_KNOB_CHOICE_DESCRIPTOR; val=NOT_IMPLEMENTED;
i! Time: 2022-11-25T08:47:48.178825 (0d+0h+0m+2s since start)
i! Process=391; Thread=391; GPU=NULL; Handle=NULL; StreamId=NULL.

I! CuDNN (v8204) function cudnnBackendDestroyDescriptor() called:
i! descriptor: type=CUDNN_BACKEND_KNOB_CHOICE_DESCRIPTOR; val=NOT_IMPLEMENTED;
i! Time: 2022-11-25T08:47:48.178830 (0d+0h+0m+2s since start)
i! Process=391; Thread=391; GPU=NULL; Handle=NULL; StreamId=NULL.

I! CuDNN (v8204) function cudnnBackendDestroyDescriptor() called:
i! descriptor: type=CUDNN_BACKEND_KNOB_CHOICE_DESCRIPTOR; val=NOT_IMPLEMENTED;
i! Time: 2022-11-25T08:47:48.178834 (0d+0h+0m+2s since start)
i! Process=391; Thread=391; GPU=NULL; Handle=NULL; StreamId=NULL.

I! CuDNN (v8204) function cudnnBackendDestroyDescriptor() called:
i! descriptor: type=CUDNN_BACKEND_KNOB_CHOICE_DESCRIPTOR; val=NOT_IMPLEMENTED;
i! Time: 2022-11-25T08:47:48.178838 (0d+0h+0m+2s since start)
i! Process=391; Thread=391; GPU=NULL; Handle=NULL; StreamId=NULL.

I! CuDNN (v8204) function cudnnBackendDestroyDescriptor() called:
i! descriptor: type=CUDNN_BACKEND_KNOB_CHOICE_DESCRIPTOR; val=NOT_IMPLEMENTED;
i! Time: 2022-11-25T08:47:48.178842 (0d+0h+0m+2s since start)
i! Process=391; Thread=391; GPU=NULL; Handle=NULL; StreamId=NULL.

I! CuDNN (v8204) function cudnnBackendDestroyDescriptor() called:
i! descriptor: type=CUDNN_BACKEND_KNOB_CHOICE_DESCRIPTOR; val=NOT_IMPLEMENTED;
i! Time: 2022-11-25T08:47:48.178846 (0d+0h+0m+2s since start)
i! Process=391; Thread=391; GPU=NULL; Handle=NULL; StreamId=NULL.

I! CuDNN (v8204) function cudnnBackendDestroyDescriptor() called:
i! descriptor: type=CUDNN_BACKEND_KNOB_CHOICE_DESCRIPTOR; val=NOT_IMPLEMENTED;
i! Time: 2022-11-25T08:47:48.178851 (0d+0h+0m+2s since start)
i! Process=391; Thread=391; GPU=NULL; Handle=NULL; StreamId=NULL.

I! CuDNN (v8204) function cudnnBackendDestroyDescriptor() called:
i! descriptor: type=CUDNN_BACKEND_KNOB_CHOICE_DESCRIPTOR; val=NOT_IMPLEMENTED;
i! Time: 2022-11-25T08:47:48.178855 (0d+0h+0m+2s since start)
i! Process=391; Thread=391; GPU=NULL; Handle=NULL; StreamId=NULL.

I! CuDNN (v8204) function cudnnBackendDestroyDescriptor() called:
i! descriptor: type=CUDNN_BACKEND_KNOB_CHOICE_DESCRIPTOR; val=NOT_IMPLEMENTED;
i! Time: 2022-11-25T08:47:48.178859 (0d+0h+0m+2s since start)
i! Process=391; Thread=391; GPU=NULL; Handle=NULL; StreamId=NULL.

I! CuDNN (v8204) function cudnnBackendDestroyDescriptor() called:
i! descriptor: type=CUDNN_BACKEND_KNOB_CHOICE_DESCRIPTOR; val=NOT_IMPLEMENTED;
i! Time: 2022-11-25T08:47:48.178863 (0d+0h+0m+2s since start)
i! Process=391; Thread=391; GPU=NULL; Handle=NULL; StreamId=NULL.

I! CuDNN (v8204) function cudnnBackendDestroyDescriptor() called:
i! descriptor: type=CUDNN_BACKEND_KNOB_CHOICE_DESCRIPTOR; val=NOT_IMPLEMENTED;
i! Time: 2022-11-25T08:47:48.178868 (0d+0h+0m+2s since start)
i! Process=391; Thread=391; GPU=NULL; Handle=NULL; StreamId=NULL.

I! CuDNN (v8204) function cudnnBackendDestroyDescriptor() called:
i! descriptor: type=CUDNN_BACKEND_KNOB_CHOICE_DESCRIPTOR; val=NOT_IMPLEMENTED;
i! Time: 2022-11-25T08:47:48.178872 (0d+0h+0m+2s since start)
i! Process=391; Thread=391; GPU=NULL; Handle=NULL; StreamId=NULL.

I! CuDNN (v8204) function cudnnBackendDestroyDescriptor() called:
i! descriptor: type=CUDNN_BACKEND_KNOB_CHOICE_DESCRIPTOR; val=NOT_IMPLEMENTED;
i! Time: 2022-11-25T08:47:48.178876 (0d+0h+0m+2s since start)
i! Process=391; Thread=391; GPU=NULL; Handle=NULL; StreamId=NULL.

I! CuDNN (v8204) function cudnnBackendDestroyDescriptor() called:
i! descriptor: type=CUDNN_BACKEND_KNOB_CHOICE_DESCRIPTOR; val=NOT_IMPLEMENTED;
i! Time: 2022-11-25T08:47:48.178880 (0d+0h+0m+2s since start)
i! Process=391; Thread=391; GPU=NULL; Handle=NULL; StreamId=NULL.

I! CuDNN (v8204) function cudnnBackendDestroyDescriptor() called:
i! descriptor: type=CUDNN_BACKEND_KNOB_CHOICE_DESCRIPTOR; val=NOT_IMPLEMENTED;
i! Time: 2022-11-25T08:47:48.178886 (0d+0h+0m+2s since start)
i! Process=391; Thread=391; GPU=NULL; Handle=NULL; StreamId=NULL.

I! CuDNN (v8204) function cudnnBackendDestroyDescriptor() called:
i! descriptor: type=CUDNN_BACKEND_KNOB_CHOICE_DESCRIPTOR; val=NOT_IMPLEMENTED;
i! Time: 2022-11-25T08:47:48.178890 (0d+0h+0m+2s since start)
i! Process=391; Thread=391; GPU=NULL; Handle=NULL; StreamId=NULL.

I! CuDNN (v8204) function cudnnBackendGetAttribute() called:
i! descriptor: type=CUDNN_BACKEND_ENGINE_DESCRIPTOR:
i! opGraph: type=CUDNN_BACKEND_OPERATIONGRAPH_DESCRIPTOR:
i! engine_id: type=int; val=0;
i! knobDesc: type=cudnnBackendDescriptor_t:
i! CUDNN_KNOB_TYPE_KERNEL_CFG: type=int; val=27;
i! numericalNotes: type=cudnnBackendDescriptor_t:
i! CUDNN_NUMERICAL_NOTE_TENSOR_CORE: type=bool; val=true;
i! behaviorNotes: type=cudnnBackendDescriptor_t:
i! CUDNN_BEHAVIOR_NOTE_RUNTIME_COMPILATION: type=bool; val=true;
i! attributeName: type=cudnnBackendAttributeName_t; val=CUDNN_ATTR_ENGINE_NUMERICAL_NOTE (1303);
i! attributeType: type=cudnnBackendAttributeType_t; val=CUDNN_TYPE_NUMERICAL_NOTE (11);
i! requestedElementCount: type=int64_t; val=6;
i! elementCount: location=host; addr=0x7ffdaf5d4c98;
i! arrayOfElements: location=host; addr=NULL_PTR;
i! Time: 2022-11-25T08:47:48.178897 (0d+0h+0m+2s since start)
i! Process=391; Thread=391; GPU=NULL; Handle=NULL; StreamId=NULL.

I! CuDNN (v8204) function cudnnBackendGetAttribute() called:
i! descriptor: type=CUDNN_BACKEND_ENGINE_DESCRIPTOR:
i! opGraph: type=CUDNN_BACKEND_OPERATIONGRAPH_DESCRIPTOR:
i! engine_id: type=int; val=0;
i! knobDesc: type=cudnnBackendDescriptor_t:
i! CUDNN_KNOB_TYPE_KERNEL_CFG: type=int; val=27;
i! numericalNotes: type=cudnnBackendDescriptor_t:
i! CUDNN_NUMERICAL_NOTE_TENSOR_CORE: type=bool; val=true;
i! behaviorNotes: type=cudnnBackendDescriptor_t:
i! CUDNN_BEHAVIOR_NOTE_RUNTIME_COMPILATION: type=bool; val=true;
i! attributeName: type=cudnnBackendAttributeName_t; val=CUDNN_ATTR_ENGINE_NUMERICAL_NOTE (1303);
i! attributeType: type=cudnnBackendAttributeType_t; val=CUDNN_TYPE_NUMERICAL_NOTE (11);
i! requestedElementCount: type=int64_t; val=6;
i! elementCount: location=host; addr=0x7ffdaf5d4c98;
i! arrayOfElements: location=host; addr=0x55d7829f8f70;
i! Time: 2022-11-25T08:47:48.178904 (0d+0h+0m+2s since start)
i! Process=391; Thread=391; GPU=NULL; Handle=NULL; StreamId=NULL.

I! CuDNN (v8204) function cudnnBackendGetAttribute() called:
i! descriptor: type=CUDNN_BACKEND_ENGINE_DESCRIPTOR:
i! opGraph: type=CUDNN_BACKEND_OPERATIONGRAPH_DESCRIPTOR:
i! engine_id: type=int; val=0;
i! knobDesc: type=cudnnBackendDescriptor_t:
i! CUDNN_KNOB_TYPE_KERNEL_CFG: type=int; val=27;
i! numericalNotes: type=cudnnBackendDescriptor_t:
i! CUDNN_NUMERICAL_NOTE_TENSOR_CORE: type=bool; val=true;
i! behaviorNotes: type=cudnnBackendDescriptor_t:
i! CUDNN_BEHAVIOR_NOTE_RUNTIME_COMPILATION: type=bool; val=true;
i! attributeName: type=cudnnBackendAttributeName_t; val=CUDNN_ATTR_ENGINE_BEHAVIOR_NOTE (1305);
i! attributeType: type=cudnnBackendAttributeType_t; val=CUDNN_TYPE_BEHAVIOR_NOTE (19);
i! requestedElementCount: type=int64_t; val=1;
i! elementCount: location=host; addr=0x7ffdaf5d4c98;
i! arrayOfElements: location=host; addr=NULL_PTR;
i! Time: 2022-11-25T08:47:48.178911 (0d+0h+0m+2s since start)
i! Process=391; Thread=391; GPU=NULL; Handle=NULL; StreamId=NULL.

I! CuDNN (v8204) function cudnnBackendGetAttribute() called:
i! descriptor: type=CUDNN_BACKEND_ENGINE_DESCRIPTOR:
i! opGraph: type=CUDNN_BACKEND_OPERATIONGRAPH_DESCRIPTOR:
i! engine_id: type=int; val=0;
i! knobDesc: type=cudnnBackendDescriptor_t:
i! CUDNN_KNOB_TYPE_KERNEL_CFG: type=int; val=27;
i! numericalNotes: type=cudnnBackendDescriptor_t:
i! CUDNN_NUMERICAL_NOTE_TENSOR_CORE: type=bool; val=true;
i! behaviorNotes: type=cudnnBackendDescriptor_t:
i! CUDNN_BEHAVIOR_NOTE_RUNTIME_COMPILATION: type=bool; val=true;
i! attributeName: type=cudnnBackendAttributeName_t; val=CUDNN_ATTR_ENGINE_BEHAVIOR_NOTE (1305);
i! attributeType: type=cudnnBackendAttributeType_t; val=CUDNN_TYPE_BEHAVIOR_NOTE (19);
i! requestedElementCount: type=int64_t; val=1;
i! elementCount: location=host; addr=0x7ffdaf5d4c98;
i! arrayOfElements: location=host; addr=0x55d783c57ed0;
i! Time: 2022-11-25T08:47:48.178917 (0d+0h+0m+2s since start)
i! Process=391; Thread=391; GPU=NULL; Handle=NULL; StreamId=NULL.

I! CuDNN (v8204) function cudnnBackendGetAttribute() called:
i! descriptor: type=CUDNN_BACKEND_EXECUTION_PLAN_DESCRIPTOR:
i! engcfg: type=CUDNN_BACKEND_ENGINECFG_DESCRIPTOR:
i! engine: type=CUDNN_BACKEND_ENGINE_DESCRIPTOR:
i! opGraph: type=CUDNN_BACKEND_OPERATIONGRAPH_DESCRIPTOR:
i! engine_id: type=int; val=0;
i! knobDesc: type=cudnnBackendDescriptor_t:
i! CUDNN_KNOB_TYPE_KERNEL_CFG: type=int; val=27;
i! numericalNotes: type=cudnnBackendDescriptor_t:
i! CUDNN_NUMERICAL_NOTE_TENSOR_CORE: type=bool; val=true;
i! behaviorNotes: type=cudnnBackendDescriptor_t:
i! CUDNN_BEHAVIOR_NOTE_RUNTIME_COMPILATION: type=bool; val=true;
i! attributeName: type=cudnnBackendAttributeName_t; val=CUDNN_ATTR_EXECUTION_PLAN_WORKSPACE_SIZE (402);
i! attributeType: type=cudnnBackendAttributeType_t; val=CUDNN_TYPE_INT64 (3);
i! requestedElementCount: type=int64_t; val=1;
i! elementCount: location=host; addr=NULL_PTR;
i! arrayOfElements: location=host; addr=0x7ffdaf5d4f78;
i! Time: 2022-11-25T08:47:48.178926 (0d+0h+0m+2s since start)
i! Process=391; Thread=391; GPU=NULL; Handle=NULL; StreamId=NULL.

I! CuDNN (v8204) function cudnnBackendDestroyDescriptor() called:
i! descriptor: type=CUDNN_BACKEND_ENGINE_DESCRIPTOR:
i! opGraph: type=CUDNN_BACKEND_OPERATIONGRAPH_DESCRIPTOR:
i! engine_id: type=int; val=0;
i! knobDesc: type=cudnnBackendDescriptor_t:
i! CUDNN_KNOB_TYPE_KERNEL_CFG: type=int; val=27;
i! numericalNotes: type=cudnnBackendDescriptor_t:
i! CUDNN_NUMERICAL_NOTE_TENSOR_CORE: type=bool; val=true;
i! behaviorNotes: type=cudnnBackendDescriptor_t:
i! CUDNN_BEHAVIOR_NOTE_RUNTIME_COMPILATION: type=bool; val=true;
i! Time: 2022-11-25T08:47:48.178932 (0d+0h+0m+2s since start)
i! Process=391; Thread=391; GPU=NULL; Handle=NULL; StreamId=NULL.

I! CuDNN (v8204) function cudnnBackendDestroyDescriptor() called:
i! descriptor: type=CUDNN_BACKEND_ENGINECFG_DESCRIPTOR:
i! engine: type=CUDNN_BACKEND_ENGINE_DESCRIPTOR:
i! opGraph: type=CUDNN_BACKEND_OPERATIONGRAPH_DESCRIPTOR:
i! engine_id: type=int; val=0;
i! knobDesc: type=cudnnBackendDescriptor_t:
i! CUDNN_KNOB_TYPE_KERNEL_CFG: type=int; val=11;
i! numericalNotes: type=cudnnBackendDescriptor_t:
i! CUDNN_NUMERICAL_NOTE_TENSOR_CORE: type=bool; val=true;
i! behaviorNotes: type=cudnnBackendDescriptor_t:
i! CUDNN_BEHAVIOR_NOTE_RUNTIME_COMPILATION: type=bool; val=true;
i! Time: 2022-11-25T08:47:48.178939 (0d+0h+0m+2s since start)
i! Process=391; Thread=391; GPU=NULL; Handle=NULL; StreamId=NULL.

I! CuDNN (v8204) function cudnnBackendDestroyDescriptor() called:
i! descriptor: type=CUDNN_BACKEND_ENGINECFG_DESCRIPTOR:
i! engine: type=CUDNN_BACKEND_ENGINE_DESCRIPTOR:
i! opGraph: type=CUDNN_BACKEND_OPERATIONGRAPH_DESCRIPTOR:
i! engine_id: type=int; val=0;
i! knobDesc: type=cudnnBackendDescriptor_t:
i! CUDNN_KNOB_TYPE_KERNEL_CFG: type=int; val=10;
i! numericalNotes: type=cudnnBackendDescriptor_t:
i! CUDNN_NUMERICAL_NOTE_TENSOR_CORE: type=bool; val=true;
i! behaviorNotes: type=cudnnBackendDescriptor_t:
i! CUDNN_BEHAVIOR_NOTE_RUNTIME_COMPILATION: type=bool; val=true;
i! Time: 2022-11-25T08:47:48.178945 (0d+0h+0m+2s since start)
i! Process=391; Thread=391; GPU=NULL; Handle=NULL; StreamId=NULL.

I! CuDNN (v8204) function cudnnBackendDestroyDescriptor() called:
i! descriptor: type=CUDNN_BACKEND_ENGINEHEUR_DESCRIPTOR; val=NOT_IMPLEMENTED;
i! Time: 2022-11-25T08:47:48.178952 (0d+0h+0m+2s since start)
i! Process=391; Thread=391; GPU=NULL; Handle=NULL; StreamId=NULL.

I! CuDNN (v8204) function cudnnBackendCreateDescriptor() called:
i! descriptorType: type=cudnnBackendDescriptorType_t; val=CUDNN_BACKEND_VARIANT_PACK_DESCRIPTOR (16);
i! Time: 2022-11-25T08:47:48.178969 (0d+0h+0m+2s since start)
i! Process=391; Thread=391; GPU=NULL; Handle=NULL; StreamId=NULL.

I! CuDNN (v8204) function cudnnBackendSetAttribute() called:
i! descriptor: type=CUDNN_BACKEND_VARIANT_PACK_DESCRIPTOR:
i! attributeName: type=cudnnBackendAttributeName_t; val=CUDNN_ATTR_VARIANT_PACK_DATA_POINTERS (1001);
i! attributeType: type=cudnnBackendAttributeType_t; val=CUDNN_TYPE_VOID_PTR (6);
i! elementCount: type=int64_t; val=3;
i! [0]: location=dev; addr=0x55d782b88500;
i! [1]: location=dev; addr=0x55d782b88500;
i! [2]: location=dev; addr=0x55d782b88500;
i! Time: 2022-11-25T08:47:48.178977 (0d+0h+0m+2s since start)
i! Process=391; Thread=391; GPU=NULL; Handle=NULL; StreamId=NULL.

I! CuDNN (v8204) function cudnnBackendSetAttribute() called:
i! descriptor: type=CUDNN_BACKEND_VARIANT_PACK_DESCRIPTOR:
i! Uid=0: location=dev; addr=0x7f2bfc200000;
i! Uid=0: location=dev; addr=0x7f2bfd600000;
i! Uid=0: location=dev; addr=0x7f2c00800000;
i! attributeName: type=cudnnBackendAttributeName_t; val=CUDNN_ATTR_VARIANT_PACK_UNIQUE_IDS (1000);
i! attributeType: type=cudnnBackendAttributeType_t; val=CUDNN_TYPE_INT64 (3);
i! elementCount: type=int64_t; val=3;
i! [0]: type=int64_t; val=97;
i! [1]: type=int64_t; val=98;
i! [2]: type=int64_t; val=99;
i! Time: 2022-11-25T08:47:48.178984 (0d+0h+0m+2s since start)
i! Process=391; Thread=391; GPU=NULL; Handle=NULL; StreamId=NULL.

I! CuDNN (v8204) function cudnnBackendSetAttribute() called:
i! descriptor: type=CUDNN_BACKEND_VARIANT_PACK_DESCRIPTOR:
i! Uid=97: location=dev; addr=0x7f2bfc200000;
i! Uid=98: location=dev; addr=0x7f2bfd600000;
i! Uid=99: location=dev; addr=0x7f2c00800000;
i! attributeName: type=cudnnBackendAttributeName_t; val=CUDNN_ATTR_VARIANT_PACK_WORKSPACE (1003);
i! attributeType: type=cudnnBackendAttributeType_t; val=CUDNN_TYPE_VOID_PTR (6);
i! elementCount: type=int64_t; val=1;
i! [0]: location=dev; addr=0x7ffdaf5d52d0;
i! Time: 2022-11-25T08:47:48.178990 (0d+0h+0m+2s since start)
i! Process=391; Thread=391; GPU=NULL; Handle=NULL; StreamId=NULL.

I! CuDNN (v8204) function cudnnBackendFinalize() called:
i! descriptor: type=CUDNN_BACKEND_VARIANT_PACK_DESCRIPTOR:
i! Uid=97: location=dev; addr=0x7f2bfc200000;
i! Uid=98: location=dev; addr=0x7f2bfd600000;
i! Uid=99: location=dev; addr=0x7f2c00800000;
i! Time: 2022-11-25T08:47:48.178995 (0d+0h+0m+2s since start)
i! Process=391; Thread=391; GPU=NULL; Handle=NULL; StreamId=NULL.

I! CuDNN (v8204) function cudnnBackendExecute() called:
i! handle: type=cudnnHandle_t; streamId=(nil) (defaultStream);
i! executionPlan: type=CUDNN_BACKEND_EXECUTION_PLAN_DESCRIPTOR:
i! engcfg: type=CUDNN_BACKEND_ENGINECFG_DESCRIPTOR:
i! engine: type=CUDNN_BACKEND_ENGINE_DESCRIPTOR:
i! opGraph: type=CUDNN_BACKEND_OPERATIONGRAPH_DESCRIPTOR:
i! engine_id: type=int; val=0;
i! knobDesc: type=cudnnBackendDescriptor_t:
i! CUDNN_KNOB_TYPE_KERNEL_CFG: type=int; val=27;
i! numericalNotes: type=cudnnBackendDescriptor_t:
i! CUDNN_NUMERICAL_NOTE_TENSOR_CORE: type=bool; val=true;
i! behaviorNotes: type=cudnnBackendDescriptor_t:
i! CUDNN_BEHAVIOR_NOTE_RUNTIME_COMPILATION: type=bool; val=true;
i! variantPack: type=CUDNN_BACKEND_VARIANT_PACK_DESCRIPTOR:
i! Uid=97: location=dev; addr=0x7f2bfc200000;
i! Uid=98: location=dev; addr=0x7f2bfd600000;
i! Uid=99: location=dev; addr=0x7f2c00800000;
i! Time: 2022-11-25T08:47:48.179006 (0d+0h+0m+2s since start)
i! Process=391; Thread=391; GPU=0; Handle=0x55d774a71d40; StreamId=(nil) (defaultStream).

I! CuDNN (v8204) function cudnnBackendExecuteInternal() called:
i! enable_tf32: type=bool; val=true;
i! operation: type=internalType; val=GENERIC_FUSION_PATTERN;
i! engine_id: type=int; val=0;
i! knobDesc: type=cudnnBackendDescriptor_t:
i! CUDNN_KNOB_TYPE_KERNEL_CFG: type=int; val=27;
i! numericalNotes: type=cudnnBackendDescriptor_t:
i! CUDNN_NUMERICAL_NOTE_TENSOR_CORE: type=bool; val=true;
i! behaviorNotes: type=cudnnBackendDescriptor_t:
i! CUDNN_BEHAVIOR_NOTE_RUNTIME_COMPILATION: type=bool; val=true;
i! Time: 2022-11-25T08:47:48.179015 (0d+0h+0m+2s since start)
i! Process=391; Thread=391; GPU=0; Handle=0x55d774a71d40; StreamId=(nil) (defaultStream).

I! CuDNN (v8204) function cudnnDestroy() called:
i! Time: 2022-11-25T08:47:48.179025 (0d+0h+0m+2s since start)
i! Process=391; Thread=391; GPU=NULL; Handle=NULL; StreamId=NULL.

Answer 3 · 2022-11-26T02:01:14.000Z

Is it possible for you try on the latest cuDNN 8.6.0?

https://developer.nvidia.com/rdp/cudnn-download

Thanks

Answer 4 · 2022-11-28T02:09:22.000Z

is it caused by the version of cudnn? I use the docker released by NVIDIA : https://docs.nvidia.com/deeplearning/frameworks/pytorch-release-notes/rel_21-10.html#rel_21-10
it's a bit trouble for me to update cudnn to 8.6.0 because I need also to update the driver version up to 520. may you please try testing on the same docker as I used above?

Answer 5 · 2022-11-28T05:49:31.000Z

Why not use 22.10 container ? 21.10 container seems a year old?

…

On Sun, Nov 27, 2022, 6:09 PM Gebixiaochen ***@***.***> wrote: is it caused by the version of cudnn? I use the docker released by NVIDIA : https://docs.nvidia.com/deeplearning/frameworks/pytorch-release-notes/rel_21-10.html#rel_21-10 it's a bit trouble for me to update cudnn to 8.6.0 because I need also to update the driver version up to 520. may you please try testing on the same docker as I used above? — Reply to this email directly, view it on GitHub <#39 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAG73USVPKLFO67OUYVGNF3WKQH5ZANCNFSM6AAAAAASLBDTFE> . You are receiving this because you commented.Message ID: ***@***.***>

Answer 6 · 2022-11-28T05:58:52.000Z

As I said, 22.10 needs to update the driver version higher than 520.
"Release 22.10 is based on [CUDA 11.8.0], which requires [NVIDIA Driver] release 520 or later."

Answer 7 · 2023-02-16T11:40:25.000Z

As I said, 22.10 needs to update the driver version higher than 520. "Release 22.10 is based on [CUDA 11.8.0], which requires [NVIDIA Driver] release 520 or later."

Have you solved this problem? I got the same error...

Answer 8 · 2023-02-17T07:32:31.000Z

Hi @mazhaojia123,

Thanks for checking out.

cuDNN does not go back and fix in previous updates. Rather the fixes are in future cuDNN versions.

You can update the cuDNN version within the container through debian installation described here
https://docs.nvidia.com/deeplearning/cudnn/install-guide/index.html#package-manager-ubuntu-install

export OS=ubuntu2004
export cudnn_version=8.8.0.121
export cuda_version=cuda11.8

wget https://developer.download.nvidia.com/compute/cuda/repos/${OS}/x86_64/cuda-${OS}.pin 

sudo mv cuda-${OS}.pin /etc/apt/preferences.d/cuda-repository-pin-600
sudo apt-key adv --fetch-keys https://developer.download.nvidia.com/compute/cuda/repos/${OS}/x86_64/3bf863cc.pub
sudo add-apt-repository "deb https://developer.download.nvidia.com/compute/cuda/repos/${OS}/x86_64/ /"
sudo apt-get update

sudo apt-get install libcudnn8=${cudnn_version}-1+${cuda_version}
sudo apt-get install libcudnn8-dev=${cudnn_version}-1+${cuda_version}

cudnn built with 11.8 is backward compatible with the existing cuda version (11.4) in the container.

Answer 9 · 2023-02-18T02:13:23.000Z

@Anerudhan, Thanks.

Answer 10 · 2023-02-18T09:25:07.000Z

@Anerudhan
I am confused about how to write a matmul using cudnn.

I write a matmul with the frontend api but I got the message: “Fusion with float inputs is only supported on Ampere or later”. I want to run these codes on the v100 GPU. Is there a way to solve the problem?
I also tried the backend api, but I cannot find any code samples about how to use the backend API to write a matmul-like op. I felt that it is difficult to put descriptors together according to the documents. So if the question 1 cannot be solved, is there any tutorial I can find to learn to use the backend api to write a matmul?

Generally, all I want is writing a matmul with cudnn on v100. But I have problems on both the frontend and backend api. So I am looking forward to some help.

Thanks!!!

Answer 11 · 2023-02-18T14:42:41.000Z

Hi @mazhaojia123

Thanks for trying this out.

cudnn_frontend is a C++ layer on top of backend API. Backend API is too verbose and will be difficult to write the Matmul on.
Since you are are getting NOT_SUPPORTED error. This will persist even if you use the backend API.

Coming back to your question on Matmul on V100.
On V100 we only support the Matmul if the tensor data type is Half precision.

input :- half
output:- half
compute type :- float

Matmul with float datatype is only supported from Ampere architecture and above.

A sample with half input is shown here:
https://github.com/NVIDIA/cudnn-frontend/blob/main/samples/test_list.cpp#L933

But this requires a change in https://github.com/NVIDIA/cudnn-frontend/blob/main/samples/fusion_sample.cpp#L1698
where you need to remove the if condition

Let me know if you have further questions.

Answer 12 · 2023-02-19T08:24:14.000Z

@Anerudhan Thank you ! The information is helpful !

Answer 13 · 2023-11-02T06:29:34.000Z

Closing this issue.

Please re-open if you have any further questions