MemoryCopyError with large dataset

[Elrondsen]

Hello,

I am trying to use the kmcuda package for kmeans clustering in R.
I have succesfully installed the packages on our AWS p2.xlarge instance using the manual here on GitHub and succesfully ran the examples.

I am trying to validate Prado's algorithm for finding the best clustering with unknown number of clusters and hence I run the kmcuda function inside two for loops - one for number of clusters and second for different initializations.
The tested dataset is 1000 datapoints of 1000 dimensions. The algorithm runs fine until the 909 clusters when it fails. I have also tried to run PCA first (which reduced the dimensions to 6), but the algorithm crashed while computing 911 clusters (if I read the logs well).

This is the log of the last run before the crash together with the R error log.
`arguments: 1 0x7ffedca05558 0.000 0.10 0 1000 1000 911 334 0 0 2 0x4d2ae020 0x49f10a30 0x44ff5080 (nil)
reassignments threshold: 0
yinyang groups: 91
GPU #0 memory: used 90112000 bytes (0.8%), free 11906842624 bytes, total 11996954624 bytes
GPU #0 has 49152 bytes of shared memory per block
transposing the samples...
transpose <<<(32, 32), (32, 8)>>> 1000, 1000
performing kmeans++...
done
running Lloyd until reassignments drop below 110
plans: [(0, 1000)]
planc: [(0, 911)]
iteration 1: 1000 reassignments
iteration 2: 3 reassignments
transposing the samples...
transpose <<<(32, 29), (32, 8)>>> 911, 1000
performing kmeans++...
step 1cudaMemcpyAsync( host_dists + offset, (*dists)[devi].get(), length * sizeof(float), cudaMemcpyDeviceToHost)
/home/ubuntu/kmcuda/src/kmeans.cu:814 -> invalid argument

kmeans_cuda_plus_plus failed
kmeans_init_centroids() failed for yinyang groups: invalid argument
kmeans_cuda_yy failed: no error

Error in kmeans_cuda(dist, j, tolerance = 1e-04, seed = i + 333, verbosity = 2, :
kmeans_cuda error 5 MemoryCopyError `

The kmeans_cuda function is just a wrapper for the external call: kmeans_cuda <- function(samples,clusters,tolerance,seed,verbosity,average_distance){ .External("kmeans_cuda", samples = samples, clusters = clusters, tolerance = tolerance, seed = seed, verbosity = verbosity, average_distance = average_distance) }

Thanks for your answer beforehand.

[vmarkovtsev]

Can you please try with yinyang optimization disabled: yinyang_t=0.

Anyway, if you can send me the data to reproduce the error to vadim at sourced dot tech I will be able to debug. Seems like a problem with kmeans++ initialization, most probably it cannot choose more clusters for some reason.

[Elrondsen]

Thanks for your superquick answer.
I have tried your suggestion to disable yinyang and the algorithm worked.

Nevertheless I have just sent you simplified code and data to test and debug. Thanks for the results beforehand.