xiangwang1223/neural_graph_collaborative_filtering

sparse_tensor_dense_matmul cause ResourceExhaustedError

gumanchang opened this issue · 3 comments

gumanchang commented

Hi, I meet this error use my own dataset(about 2 million users), gpu is NVIDIA® Tesla® V100 32GB
code: temp_embed.append(tf.sparse_tensor_dense_matmul(A_fold_hat[f], ego_embeddings))
ResourceExhaustedError (see above for traceback): OOM when allocating tensor with shape[2231226,64] and type float on /job:localhost/replica:0/task:0/device:GPU:0 by allocator GPU_0_bfc
So, something wrong ?
Thank you :)

xiangwang1223 commented

Hi, Thanks for your interest. The error should be caused by the large scale of users. That is, in NGCF, we need to create an adjacency (or Laplacian) matrix, further converted into a sparse tensor in tf, whose size is equal to (#users+#items)*(#users+#items).

One possible solution might be to set the hyperparameter 'n_fold' a larger value (say, 10000), but I cannot guarantee that.

Thanks.

gumanchang commented

Hi, I try you suggestion, but I got the opposite conclusion. When 'n_fold' = 1 it works well, but 'n_fold' = 10000 it too slow to run.
I know some GCN implementations do not have this 'n_fold' hyperparameter.
So, can I remove this hyperparameter without losing precision?

xiangwang1223 commented

Hi, yes, you can remove the hyperparameter directly. I set this parameter in case of the memory error.