openNSx: NSx files with large MetaTags.Timestamp field require excessive memory for time adjustment

Question

openNSx: NSx files with large MetaTags.Timestamp field require excessive memory for time adjustment

jmw182 opened this issue 6 years ago · 9 comments

I have NSx files where the value of NSx.MetaTags.Timestamp is large. I think this is due to a bug in the 256-channel version of Central running on the wNSP prototype (Central Suite v7.0.0 Build 2, Central v7.0 Build 16, nPlay Firmware v7.0.2 Build 1 for WNSP256), in which the timestamp does not reset to 0 when a recording is initiated,

OpenNSx adds long sections of 0-padding to these files (because of the section beginning on line 823). My workaround is to change line 833 to skip this adjustment if the Timestamp is > 10 seconds. I do not reset the Timestamp value either so that the NEV file can be manually aligned if needed. I can submit a pull request with my fix.

Answer 1 · 2020-07-21T00:07:59.000Z

v7.1 added the nozeropad option which can disable this time adjustment. The branch in my pull request will now automatically apply this option if the recording start time (NSx.MetaTags.Timestamp) is greater than 10 seconds, and will print a warning.

Answer 2 · 2020-07-21T16:41:06.000Z

Jeff, this is by design, actually. It's not a bug. It's always been this way. In the past, the timestamp would reset before start of recording, but with the 64-bit timestamp this won't happen again. BUT... in openNSx you have the option to skip this by using the flag "nozeropad". It's implemented in the latest version of NPMK (7.1.1.0: June 11, 2020).

Answer 3 · 2020-07-21T17:03:07.000Z

Thanks @kianabc. I agree that the new "nozeropad" flag provides me with the mechanism to avoid this problem, but the default behavior is still a bit confusing. I'm assuming the reason for zero-padding is to help align the NEV and NSx files when the NSx start timestamp is a small number slightly greater than 0. But for the larger 64-bit timestamps, the zero padding matrix can be much larger than the actual data matrix, and can require much more memory than what is available in RAM. Should the default behavior in this case be to use the "nozeropad" option without requiring the user to specify it?

A counterpoint I can think of is that users might not be aware of the MetaTags.Timestamp value and might not know how to align the NSx and NEV files. Would it be reasonable for openNSx to also generate a timestamp vector to make this alignment easier?

Answer 4 · 2020-07-21T19:33:28.000Z

OK, I think I'm going to add a warning to the user and make them aware of the issue and let them choose. In general it's not possible (or ideal) to change the default behavior of the code or it'll break things for everyone. The one thing I'd change immediately is .Data.Electrode to .Data.Channel. Long story as to why it is the way it is, but I wish I could change it.

Thanks for your feedback.

Kian

Answer 5 · 2020-07-21T19:34:09.000Z

Can you elaborate on your timestamp vector idea? I just read that part of your message.

Answer 6 · 2020-07-21T20:05:17.000Z

Thanks. I understand needing to maintain previous behavior to not break everyone's analyses, so a warning would help.

For the timestamp vector, I mean adding a field to the NSx structure with a 1 x num_samples vector (or cell array of vectors if Data is also an array) with the timestamp value of each column of the Data matrix. This would be equivalent to the timestamp vectors in the NEV structures (e.g. NEV.Data.Spikes.TimeStamp) and should make aligning data from the two files easier.

I'm not sure if these timestamp values are already stored for each sample in the binary NSx format, but they could also be generated in matlab from the values that already exist.

For simple raw NS6 files (where Data is not a cell array), I think this would just be NSx.TimeStamp = (0:size(NSx.Data,2)-1) + NSx.MetaTags.Timestamp(1);

For files with reduced lower sampling rates, I think it could be generated with NSx.TimeStamp =(0:NSx.MetaTags.TimeRes/NSx.MetaTags.SamplingFreq:size(NSx.Data,2)-1) + NSx.MetaTags.Timestamp(1);

And if Data is a cell array, a vector could be generated for each chunk of data by looping over the cells and MetaTags.Timestamp values.

Answer 7 · 2020-08-13T19:31:44.000Z

Hi Jeff,

I made this change. I also ask the user if they want to see the warning again and if not, it won't ask them again. It also refers them to a knowledge base article that describes the situation very nicely (I think). Once I test it (waiting for 3.0 files) I'll upload it.

Thanks for your feedback. You'll get a notification for a new version if your fork is using NPMKverChecker.

Best,
Kian

Answer 8 · 2020-08-13T19:34:04.000Z

Thanks Kian!

Answer 9 · 2020-09-11T22:30:20.000Z

I'm going to upload this and create a new released. Tested it and it looks to be working fine. Thanks again.