kwikteam/phy-contrib

Memory Error in phy

Closed this issue · 8 comments

JoanaPNeto commented

Hey,
I tried to open with phy some data that has been sorted with Kilosort.
The raw data file has ~32GB where we have 719 channels.
Kilosort took around 1h30 but after when I tried to open in phy:

[phy] F:\DataKilosort\1440chprobe\719ch>phy template-gui params.py
21:26:54 [E] cli:37 An error has occurred (MemoryError):

Did you ever get this type of error?
I attached the phy.log.
Thank you,
Joana

phy.pdf

rossant commented

Do you know how many templates you have? Also, how much RAM do you have?

JoanaPNeto commented

64GB of RAM and 2880 templates (~4 times the number of channels).

On Thu, Sep 29, 2016 at 12:29 PM, Cyrille Rossant notifications@github.com
wrote:

Do you know how many templates you have? Also, how much RAM do you have?

—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub
#66 (comment),
or mute the thread
https://github.com/notifications/unsubscribe-auth/AQJuWUJ4iLBrTUFbwXserbHvGgQQM19Dks5qu6EfgaJpZM4KJy1f
.

rossant commented

OK, the problem is that templates are stored as dense and not sparse arrays at the moment (that will change later). In your case it seems to crash when templates are substracted from the raw data in the trace view.

There are several workarounds until sparse arrays are implemented.

  • Edit ~/.phy/TemplateGUI/state.json and edit the interval of TraceView to be something like [0, 0.010] -- there will be less spikes to display and it might not crash
  • Provide an option to disable this subtraction (but not implemented yet)

Otherwise could you send me a subset of your data (less than 1 second) so that I can debug properly?

yger commented

I'm also pushing for a sparse representation of the templates, as we are
doing in SpyKING CIRCUS. I love the phy GUI, but for us this is also not
able (yet) to load datasets with thousands of channels and/or templates. If
there is anything we could do to help, maybe at some point we could discuss
about that. We would be happy to try to contribute to the project!

Best

Pierre

Le jeu. 29 sept. 2016 13:45, Cyrille Rossant notifications@github.com a
écrit :

OK, the problem is that templates are stored as dense and not sparse
arrays at the moment (that will change later). In your case it seems to
crash when templates are substracted from the raw data in the trace view.

There are several workarounds until sparse arrays are implemented.

  • Edit ~/.phy/TemplateGUI/state.json and edit the interval of
    TraceView to be something like [0, 0.010] -- there will be less spikes
    to display and it might not crash
  • Provide an option to disable this subtraction (but not implemented
    yet)

Otherwise could you send me a subset of your data (less than 1 second) so
that I can debug properly?

—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub
#66 (comment),
or mute the thread
https://github.com/notifications/unsubscribe-auth/ABmE_x1e5s7aElaFMveAjxgM3JMEKSbwks5qu6TWgaJpZM4KJy1f
.

JoanaPNeto commented

Thank you.

Inside state.json (shown bellow) where should I add that interval?

{
  "ClusterView": {
    "quality": "n_spikes",
    "similarity": "closest_clusters"
  },
  "CorrelogramView0": {
    "bin_size": 0.001,
    "excerpt_size": 10000,
    "n_excerpts": 100,
    "uniform_normalization": true,
    "window_size": 0.05
  },
  "FeatureView0": {
    "scaling": 2.0736
  },
  "TraceView0": {
    "origin": null
  },
  "WaveformView0": {
    "do_zoom_on_channels": true,
    "overlap": false
  },
  "geometry_state": {
    "geometry": {
      "__qbytearray__":
"QWRuUXl3QUJBQUFBQUFBM0FBQUFEZ0FBQnNvQUFBUDRBQUFBUVFBQUFEUUFBQWJBQUFBRDdnQUFBQUFBQUE9PQ=="
    },
    "state": {
      "__qbytearray__":
"QUFBQS93QUFBQUQ5QUFBQUFRQUFBQUVBQUFhQUFBQURqUHdDQUFBQUFmd0FBQUFaQUFBRGpBQUFBRmNBLy8vLy9BRUFBQUFEL0FBQUFBQUFBQWFBQUFBQXRBRC8vLy84QWdBQUFBTDhBQUFBR1FBQUE0d0FBQUJYQVAvLy8vd0JBQUFBQS93QUFBQUFBQUFCU1FBQUFEZ0EvLy8vL0FJQUFBQUQrd0FBQUJZQVF3QnNBSFVBY3dCMEFHVUFjZ0JXQUdrQVpRQjNBUUFBQUJrQUFBRU1BQUFBR1FELy8vLzdBQUFBSEFCVEFHa0FiUUJwQUd3QVlRQnlBR2tBZEFCNUFGWUFhUUJsQUhjQkFBQUJLd0FBQVlRQUFBQVpBUC8vLy9zQUFBQWdBRU1BYndCeUFISUFaUUJzQUc4QVp3QnlBR0VBYlFCV0FHa0FaUUIzQURBQkFBQUN0UUFBQVBBQUFBQVpBUC8vLy93QUFBRlBBQUFETUFBQUFEZ0EvLy8vL0FJQUFBQUMrd0FBQUJnQVJnQmxBR0VBZEFCMUFISUFaUUJXQUdrQVpRQjNBREFCQUFBQUdRQUFBaGdBQUFBWkFQLy8vL3NBQUFBVUFGUUFjZ0JoQUdNQVpRQldBR2tBWlFCM0FEQUJBQUFDTndBQUFXNEFBQUFaQVAvLy8vc0FBQUFhQUZjQVlRQjJBR1VBWmdCdkFISUFiUUJXQUdrQVpRQjNBREFCQUFBRWhRQUFBZnNBQUFBNEFQLy8vL3NBQUFBZ0FFTUFid0J5QUhJQVpRQnNBRzhBWndCeUFHRUFiUUJXQUdrQVpRQjNBREVCQUFBQ3B3QUFBUDRBQUFBQUFBQUFBUHdBQUFGTUFBQUROZ0FBQUFBQS8vLy8vQUlBQUFBQyt3QUFBQmdBUmdCbEFHRUFkQUIxQUhJQVpRQldBR2tBWlFCM0FERUJBQUFBR1FBQUFvUUFBQUFBQUFBQUFQc0FBQUFVQUZRQWNnQmhBR01BWlFCV0FHa0FaUUIzQURFQkFBQUNvd0FBQVFJQUFBQUFBQUFBQVBzQUFBQWFBRmNBWVFCMkFHVUFaZ0J2QUhJQWJRQldBR2tBWlFCM0FERUJBQUFFaUFBQUFmZ0FBQUFBQUFBQUFBQUFBQUFBQUFPTUFBQUFCQUFBQUFRQUFBQUlBQUFBQ1B3QUFBQUE="
    }
  }
}
rossant commented

Replace

  "TraceView0": {
    "origin": null
  },

by

  "TraceView0": {
    "origin": null,
    "interval": [0, 0.010]
  },
ajuavinett commented

Hello -- getting this same memory issue now on a 32GB computer with a ~15 GB (11 min) 384-ch (1024 templates) Neuropixels probe recording. The GUI loads up just fine on a 64GB computer, and only takes ~8 GB of RAM, but when I load it up on another computer, it stalls and sucks up all 32GB of RAM.

EDIT (solved): Deleted the state.json file found in Users/*/.phy/TemplateGUI. Note: The .phy/TemplateGUI is by default in the user directory (this is not in each individual .phy folder). I think the previous time I had the TemplateGUI open it was on some bizarre setting that was really difficult to display with a larger dataset. Now opens without a problem.

Seems like a good note in general: you can reset your GUI viewing settings by deleting the state.json file.

rossant commented

Thank you @ajuavinett for the information