SpikeGLX abruptly stops recording

Question

SpikeGLX abruptly stops recording

Closed this issue 7 years ago · 7 comments

Hello, I've had this issue a couple times now where SpikeGLX stops recording, without any error messages. When I start a new acquisition, everything is fine. Thanks in advance for your help.

Answer 1 · 2017-11-30T19:55:53.000Z

Hello Ashley, Eager to help. I need much more context info about settings and what you see. - What version of the software are you using? - Has anything frozen, or have the voltages just flatlined? - How long is a run before it shows the problem and is it a very repeatable time or variable? - What’s the longest you ran successfully with this probe? - What option probe is it? - Does this happen only with this probe or any probe? - Is it just the graphs window that stops drawing or does the data acquisition stop or does file writing stop? - When this happens, does the acquisition clock in the upper left keep running? - If you are writing a file, does the file clock in the upper right keep running? - If writing, do the status data in the bottom of the console window show file writing activity? - Do files have the expected span of data, even though the display isn’t changing? - Sometimes warning messages are reported in the content area of the console window even though a dialog box doesn’t appear on the screen. - What are the settings in the graphs window? A screen shot is helpful. In the picture I don’t see the vertical time cursor which is puzzling. That may be because the graph time (secs) is very short but I can’t tell. - I can’t tell if the traces have gone to zero or are saturated. I need to know if the time or space averaging settings are enabled. Also you can read voltages from the graph with the mouse and tell me what the voltages are. - If also running nidq at the same time, does that also appear to stop? Let’s start with those. Bill From: Ashley Juavinett [mailto:notifications@github.com] Sent: Thursday, November 30, 2017 2:10 PM To: billkarsh/SpikeGLX <SpikeGLX@noreply.github.com> Cc: Subscribed <subscribed@noreply.github.com> Subject: [billkarsh/SpikeGLX] SpikeGLX abruptly stops recording (#2) Hello, I've had this issue a couple times now where SpikeGLX stops recording, without any error messages. When I start a new acquisition, everything is fine. Thanks in advance for your help. [spikeglx fail]<https://user-images.githubusercontent.com/7157126/33449843-03314e72-d5d8-11e7-8377-523c2799e586.PNG> — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub<#2>, or mute the thread<https://github.com/notifications/unsubscribe-auth/AGxB0BJSo1b9hDX_g89bigvy9q3qKSPtks5s7v2FgaJpZM4QxDYr>.

Answer 2 · 2017-11-30T20:12:37.000Z

Hi Bill, thanks for the quick reply (and for your great software). I've answered your questions below.

What version of the software are you using? v20170814 Imec v4.3
Has anything frozen, or have the voltages just flatlined? everything else seems to be fine
How long is a run before it shows the problem and is it a very repeatable time or variable? 30 minutes on this instance, not sure about other instances
What’s the longest you ran successfully with this probe? 45 minutes
What option probe is it? option 4
Does this happen only with this probe or any probe? this has happened before but unfortunately i didn't make a note of it and may have deleted the experiment, but so far i've only done experiments with opt 1 or 4 probes, more like it was an opt 4
Is it just the graphs window that stops drawing or does the data acquisition stop or does file writing stop? the sync channel was still displaying and recording, it seems. when i open the file in spike viewer, it looks as i showed.
When this happens, does the acquisition clock in the upper left keep running? yes, acquisition clock was still going.
If you are writing a file, does the file clock in the upper right keep running? yep, still running.
If writing, do the status data in the bottom of the console window show file writing activity? i didn't know to look for this, so i'm not sure.
Do files have the expected span of data, even though the display isn’t changing? they're about the right size, if that's what you're asking!
Sometimes warning messages are reported in the content area of the console window even though a dialog box doesn’t appear on the screen. there were no messages in the console window (simply: acquisition started, completed, acquisition stopped)
What are the settings in the graphs window? A screen shot is helpful. In the picture I don’t see the vertical time cursor which is puzzling. That may be because the graph time (secs) is very short but I can’t tell. i've attached a full screenshot from the file viewer, unfortunately i didn't take a screenshot during acquisition
I can’t tell if the traces have gone to zero or are saturated. I need to know if the time or space averaging settings are enabled. Also you can read voltages from the graph with the mouse and tell me what the voltages are. voltages go to zero
If also running nidq at the same time, does that also appear to stop? not running nidq

Answer 3 · 2017-11-30T21:02:08.000Z

Hi Ashley, More questions: - What mode of file writing “triggering” are you using? TTL, timed, Immediate… - Are you connecting signals to the external imec sync pins and saving the sync channel? It’s critical to distinguish whether you have a problem with the hardware, where it’s amplifiers have stopped working, or the probe is fine but the software is misbehaving. Moreover, the software might stop working in three really different ways. (1) In a catastrophic failure, the software is no longer fetching data from the hardware, but a running clock, which is driven by the data fetching, shows that’s not it. (2) The next worst thing is the data are being acquired but not written. If the files are the expected size (according to the trigger mode’s specification for when to stop writing) than we don’t think that. Also the file clock is running. Also you see “Completed” messages in the console which are written to confirm that the trigger has completed wring a file. (3) It’s possible that the data are flowing from hardware through to the files but the displays have hung up. If the graphs are still sweeping (the cursor still sweeping across) that’s alright. Again, the sweep cursor isn’t shown for spans less that about 1 second because it’s hard on the eyes. So it appears from this that the hardware has an issue and the software is alright. It sounds like you have signals connected to the imec 16-pin sync connector. If so, we can check if the graphs are sweeping and showing this channel, and whether it is showing expected signal levels in the recorded files. This is useful because the sync signals don’t depend on the same ADCs and amplifiers as the neural channels. So again, if we are getting healthy uninterrupted recording and display of the sync channel(s) then we can conclude the software is alright. If the sync channel is working but the neural channels are not then there’s a problem in the probe, headstage or flex cable. Because the voltages go to zero, it looks like the probe isn’t getting power. I would first suspect a bad connection somewhere in the chain from the BSC card back to the probe. This is not unknown, but is a more rare failure mode. More commonly the ADCs saturate. If I’m missing something that is still implicating the software, correct me on that. I honestly don’t understand why just restarting a run would correct this, unless in between trials you’ve also wiggled or touched the hardware too. There’s a lot we don’t understand about the system in detail because it is proprietary and we don’t have schematics. Bill From: Ashley Juavinett [mailto:notifications@github.com] Sent: Thursday, November 30, 2017 3:13 PM To: billkarsh/SpikeGLX <SpikeGLX@noreply.github.com> Cc: Karsh, Bill <karshb@janelia.hhmi.org>; Comment <comment@noreply.github.com> Subject: Re: [billkarsh/SpikeGLX] SpikeGLX abruptly stops recording (#2) Hi Bill, thanks for the quick reply (and for your great software). I've answered your questions below. * What version of the software are you using? v20170814 Imec v4.3 * Has anything frozen, or have the voltages just flatlined? everything else seems to be fine * How long is a run before it shows the problem and is it a very repeatable time or variable? 30 minutes on this instance, not sure about other instances * What’s the longest you ran successfully with this probe? 45 minutes * What option probe is it? option 4 * Does this happen only with this probe or any probe? this has happened before but unfortunately i didn't make a note of it and may have deleted the experiment, but so far i've only done experiments with opt 1 or 4 probes, more like it was an opt 4 * Is it just the graphs window that stops drawing or does the data acquisition stop or does file writing stop? the sync channel was still displaying and recording, it seems. when i open the file in spike viewer, it looks as i showed. * When this happens, does the acquisition clock in the upper left keep running? yes, acquisition clock was still going. * If you are writing a file, does the file clock in the upper right keep running? yep, still running. * If writing, do the status data in the bottom of the console window show file writing activity? i didn't know to look for this, so i'm not sure. * Do files have the expected span of data, even though the display isn’t changing? they're about the right size, if that's what you're asking! * Sometimes warning messages are reported in the content area of the console window even though a dialog box doesn’t appear on the screen. there were no messages in the console window (simply: acquisition started, completed, acquisition stopped) * What are the settings in the graphs window? A screen shot is helpful. In the picture I don’t see the vertical time cursor which is puzzling. That may be because the graph time (secs) is very short but I can’t tell. i've attached a full screenshot from the file viewer, unfortunately i didn't take a screenshot during acquisition * I can’t tell if the traces have gone to zero or are saturated. I need to know if the time or space averaging settings are enabled. Also you can read voltages from the graph with the mouse and tell me what the voltages are. voltages go to zero * If also running nidq at the same time, does that also appear to stop? not running nidq [spikeglx fail 2]<https://user-images.githubusercontent.com/7157126/33452376-0b796756-d5e0-11e7-9c52-a4efa7dd9f66.PNG> — You are receiving this because you commented. Reply to this email directly, view it on GitHub<#2 (comment)>, or mute the thread<https://github.com/notifications/unsubscribe-auth/AGxB0Enl63rSDl1F1HbFpSwtfjuAgM1Bks5s7ww2gaJpZM4QxDYr>.

Answer 4 · 2017-12-01T16:58:20.000Z

Hi Bill, Immediate triggering, and yes, connecting to external sync pins on base station.

More importantly, it turns out I might have been wrong about it working if I restart acquisition (this was true when this happened before, but not now). I just plugged the mouse in and the probe is correctly recognized but I am getting an IMEC setExtRef(1) error 5. Seems you may be correct about there being a hardware problem...

Answer 5 · 2017-12-01T17:13:07.000Z

Hi Ashley, SetExtRef(1) error 5 is telling me your probe is trouble. This is a classic symptom of a bad shank. With a hairline crack the connection of one or more traces can come and go depending upon how much force is applied. Soon the break will be complete and the probe will be dead. You are using a phase 3A probe, that is, an early prototype. The problem of overly delicate shanks was recognized in these but the redesign of the shank-base curvature will not roll out until phase 3B, the first commercial designs. That can’t help you now, but likely it is not your fault. The best of lab hands have suffered this. Sorry. Bill From: Ashley Juavinett [mailto:notifications@github.com] Sent: Friday, December 01, 2017 11:58 AM To: billkarsh/SpikeGLX <SpikeGLX@noreply.github.com> Cc: Karsh, Bill <karshb@janelia.hhmi.org>; Comment <comment@noreply.github.com> Subject: Re: [billkarsh/SpikeGLX] SpikeGLX abruptly stops recording (#2) Hi Bill, Immediate triggering, and yes, connecting to external sync pins on base station. More importantly, it turns out I might have been wrong about it working if I restart acquisition (this was true when this happened before, but not now). I just plugged the mouse in and the probe is correctly recognized but I am getting an IMEC setExtRef(1) error 5. Seems you may be correct about there being a hardware problem... — You are receiving this because you commented. Reply to this email directly, view it on GitHub<#2 (comment)>, or mute the thread<https://github.com/notifications/unsubscribe-auth/AGxB0MMC4-0WX3utvhhcJs0LWMVQ2wiWks5s8DAtgaJpZM4QxDYr>.

Answer 6 · 2017-12-07T03:14:32.000Z

Hi Ashley, Do you mind if this issue is closed? Bill From: Ashley Juavinett [mailto:notifications@github.com] Sent: Friday, December 01, 2017 11:58 AM To: billkarsh/SpikeGLX <SpikeGLX@noreply.github.com> Cc: Karsh, Bill <karshb@janelia.hhmi.org>; Comment <comment@noreply.github.com> Subject: Re: [billkarsh/SpikeGLX] SpikeGLX abruptly stops recording (#2) Hi Bill, Immediate triggering, and yes, connecting to external sync pins on base station. More importantly, it turns out I might have been wrong about it working if I restart acquisition (this was true when this happened before, but not now). I just plugged the mouse in and the probe is correctly recognized but I am getting an IMEC setExtRef(1) error 5. Seems you may be correct about there being a hardware problem... — You are receiving this because you commented. Reply to this email directly, view it on GitHub<#2 (comment)>, or mute the thread<https://github.com/notifications/unsubscribe-auth/AGxB0MMC4-0WX3utvhhcJs0LWMVQ2wiWks5s8DAtgaJpZM4QxDYr>.

Answer 7 · 2017-12-07T15:56:01.000Z

Yep, that’s fine! Thanks for all of your help.

…

On Dec 6, 2017, 10:14 PM -0500, billkarsh ***@***.***>, wrote: Hi Ashley, Do you mind if this issue is closed? Bill From: Ashley Juavinett ***@***.*** Sent: Friday, December 01, 2017 11:58 AM To: billkarsh/SpikeGLX ***@***.***> Cc: Karsh, Bill ***@***.***>; Comment ***@***.***> Subject: Re: [billkarsh/SpikeGLX] SpikeGLX abruptly stops recording (#2) Hi Bill, Immediate triggering, and yes, connecting to external sync pins on base station. More importantly, it turns out I might have been wrong about it working if I restart acquisition (this was true when this happened before, but not now). I just plugged the mouse in and the probe is correctly recognized but I am getting an IMEC setExtRef(1) error 5. Seems you may be correct about there being a hardware problem... — You are receiving this because you commented. Reply to this email directly, view it on GitHub<#2 (comment)>, or mute the thread<https://github.com/notifications/unsubscribe-auth/AGxB0MMC4-0WX3utvhhcJs0LWMVQ2wiWks5s8DAtgaJpZM4QxDYr>. — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub, or mute the thread.