CUDA Graph open fail

[tuanzhangCS]

System information

• OS Platform and Distribution (e.g., Linux Ubuntu 20.04): Linux
• DeepRec version or commit id: be62ec3
• Python version: 3.8
• Bazel version (if compiling from source): 0.26.1
• GCC/Compiler version (if compiling from source): gcc 9.4.0
• CUDA/cuDNN version: cuda11.6/cuDNN8

Describe the current behavior
My usage is:

SessionOptions options;
options.config.mutable_gpu_options()->set_cuda_graph_enable_jit(true);
auto status = tensorflow::LoadSavedModel(options, tensorflow::RunOptions(), model_path, {"serve"}, &bundle);
for (int i = 0; i < loops; ++i) {
   status = bundle.session->Run(inputs, out_names, {}, &outputs);
}

But I can't observe cudaGraph used in nsight system.

Describe the expected behavior
Some sub graph can run by cuda Graph.

Are there any examples of using cuda Graph?

[tuanzhangCS]

I found that because I didn't compile DeepRec with XLA.
After I fix it, there will be cudaGraphLaunch in the graph when running.
But sometimes the results are random error, why?

[candyzone]

session run error or random results?

[tuanzhangCS]

session run error or random results?

Random results.