JaneliaSciComp/JRCLUST

splitting high firing rate cells can take a very long time

Closed this issue · 7 comments

agbondy commented

On the order of ten minutes if using the "hierarchical clustering" option. Looking at the computer's resources, the GPU is idle and the CPU is only 10% active, suggesting it may be possible to dramatically improve performance of whatever the expensive underlying operation is?

aliddell commented

@agbondy sorry for the delay on this. I have a user locally who started seeing this or something similar right around the time you did. Some questions:

  1. What model CPU/how much memory do you have?
  2. What size clusters are you looking at?
  3. Are you seeing this with all clusters?
  4. If you set a breakpoint here:
    a) what size is sampledSpikes?
    b) if you step through into pca (which is where I'm presuming this is happening, since it's happening there for us) and into its local functions, is it locking up at the call to svd?
  5. Another question: do you see this when splitting using either k-means or k-medoids?

We're seeing the svd taking unreasonably long for smallish (~1000x250) matrices here.

aliddell commented

Would you mind pulling the latest version and seeing if this problem still presents itself?

agbondy commented

Sorry for the super delayed reply. Was writing a grant. I'll pull the latest version and get back to you.

agbondy commented

OK, updated to the latest code and it's still happening. It's happening in the hierarchical clustering function itself (matlab's linkage.m function) at line 257: " Z = linkagemex(Y',method,{'euc'},memEff); ". The size of Y is 140449 by 5, where 140449 is the number of events in the cluster. For this cluster, that line alone takes several minutes to run.

I'm running Matlab R2018a on a Dell Precision Workstation with an Intel Xeon Gold 6136 CPU (12 cores at 3GHz), 384GB of DDR4 RAM and an NVIDIA Quadro P5000 GPU with 16GB of RAM.

aliddell commented

@agbondy okay, so this isn't related to that SVD issue. For a random 140449x5 matrix of doubles I get a clustering in 287 seconds on a Core i7-8700K (@3.7 GHz), so that kind of timing is probably typical with hierarchical clustering at that scale. You mentioned tens of minutes though. What other bottlenecks are you seeing? When I profile, the vast majority (99%) of time in doAutoSplit is spent in clusterdata. Would you be willing to profile for me?

As an aside, what about using k-means? It should be faster in general, would that work for you?

agbondy commented

I am fairly confident now that the only significant bottleneck is the hierarchical clustering itself. The "tens of minutes" case may have been a cluster with even more events. Our sessions can last up to five hours long, so there are cases where the matrix could be even bigger than the example above. K-means is a perfectly suitable alternative in those cases.

aliddell commented

Okay, I'm going to close this.