SpikeInterface/probeinterface

incorrect wiring with io.read_spikeglx ?

Closed this issue · 29 comments

Hi,

Sorry if I'm getting confused again with probe wiring, but probeinterface.io.read_spikeglx(metapath) doesn't seem to reflect user channel order properly.

Here is what I get with

rec_prb = rec.set_probe(read_spikeglx(metapath))

sw.plot_timeseries(
    rec_prb,
    segment_index=0,
    order_channel_by_depth=True,
    time_range=(0, 0.05)
)

image

It is the same plot with order_channel_by_depth=False

and here is prb.to_dataframe()
image

is this line correct?

Thanks

Hi.
For spikeglx, channel_ids and contact_ids are in the same order because organise in the same the same.
There is no complex "wiring" there contrary to neuronexus+intan+openephys for instance.

I guess that in your config channel are already order by depth naturaly. So odering for display do not play a role.

Hi,
Thanks for your response.
Sorry, I don't understand. Initializing the probe with this method gives different results than setting directly x and y coordinates for each channel.

Here's what the probe, contact locations recording look like with read_spikeglx(metapath). It says channels span the first 3000um although this probe was configured such that channels actually span 7600um:
image
image
image

So far, the way we have been obtaining x and y coordinates for each channel was with this "SGLXMetatoCoords" module (slightly modified from Jennifer Colonel's to be able to return x and y coords. This function is meant to output directly to Kilosort channel_map format.

Now if we set the contact positions of the probe directly from the channel xy coordinates:
image
Now the probe looks correct (checkerBox pattern):
image
And the recording orders by depth properly.
image

Hi Tom.
This is really really annoying!!
This would means that we have a strong bug in the way we are reading the channel location from the meta file.

I have to admit that I do not have too much knowledge about that

@MarineChap @jenniferColonell:
could you coordinate to compare how we are reading the meta file between the 2 repos:
https://github.com/jenniferColonell/ecephys_spike_sorting/blob/master/ecephys_spike_sorting/common/SGLXMetaToCoords.py
https://github.com/SpikeInterface/probeinterface/blob/main/probeinterface/io.py#L939

My intuition is that there is a mismatch between the channel_id and the contact_id for the formula of the x/y position.

@TomBugnon : could you send us your meta file so we could include it in our tests suite ?

Sorry if I'm a bearer of bad news here.

Also, now I am noticing: in plot_timeseries() the y axis seems to be inverted relative to how phy shows depths (in phy: 0 = tip of probe. Here it looks like 0=furthest from tip). In this recording most spikes are in the thalamus, towards the tip of the electrode. This is what the recording looks like in Phy (with channel_map obtained as I said previously).
image

image

@TomBugnon

We noticed this before and the plot_timeseries is fixed in the latest SpikeInterface version (0.96.0). Can you try that out?

Yep looks correct with 0.96.0, thanks
image

@samuelgarcia
Here are two meta files, one with the longcolumn pattern and the other with the checkerboard pattern (latter used in the examples above)
https://uwmadison.box.com/s/afo1h104392dnd2md561e3bxsq622p6t

@jenniferColonell : no the new version only solve the flip top/down of the image of time series.

The channel map diffrerences are unsolved.

@TomBugnon : could your try to get the actual master and then to comment line from 806 to 829 in io.py.
And tell us if it fix the problem ?

@samuelgarcia Identical result with line 806 to 829 commented out in io.py.

Here's what the MetatoCoord function returns for the metadata file doppio_checkerbox_xxx.meta I shared above in case you need to test it directly:

chan_id: [ 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17
18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35
36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53
54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71
72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89
90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107
108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125
126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143
144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161
162 163 164 165 166 167 168 169 170 171 172 173 174 175 176 177 178 179
180 181 182 183 184 185 186 187 188 189 190 191 192 193 194 195 196 197
198 199 200 201 202 203 204 205 206 207 208 209 210 211 212 213 214 215
216 217 218 219 220 221 222 223 224 225 226 227 228 229 230 231 232 233
234 235 236 237 238 239 240 241 242 243 244 245 246 247 248 249 250 251
252 253 254 255 256 257 258 259 260 261 262 263 264 265 266 267 268 269
270 271 272 273 274 275 276 277 278 279 280 281 282 283 284 285 286 287
288 289 290 291 292 293 294 295 296 297 298 299 300 301 302 303 304 305
306 307 308 309 310 311 312 313 314 315 316 317 318 319 320 321 322 323
324 325 326 327 328 329 330 331 332 333 334 335 336 337 338 339 340 341
342 343 344 345 346 347 348 349 350 351 352 353 354 355 356 357 358 359
360 361 362 363 364 365 366 367 368 369 370 371 372 373 374 375 376 377
378 379 380 381 382 383]
chan_x: [27. 59. 11. 43. 27. 59. 11. 43. 27. 59. 11. 43. 27. 59. 11. 43. 27. 59.
11. 43. 27. 59. 11. 43. 27. 59. 11. 43. 27. 59. 11. 43. 27. 59. 11. 43.
27. 59. 11. 43. 27. 59. 11. 43. 27. 59. 11. 43. 27. 59. 11. 43. 27. 59.
11. 43. 27. 59. 11. 43. 27. 59. 11. 43. 27. 59. 11. 43. 27. 59. 11. 43.
27. 59. 11. 43. 27. 59. 11. 43. 27. 59. 11. 43. 27. 59. 11. 43. 27. 59.
11. 43. 27. 59. 11. 43. 27. 59. 11. 43. 27. 59. 11. 43. 27. 59. 11. 43.
27. 59. 11. 43. 27. 59. 11. 43. 27. 59. 11. 43. 27. 59. 11. 43. 27. 59.
11. 43. 27. 59. 11. 43. 27. 59. 11. 43. 27. 59. 11. 43. 27. 59. 11. 43.
27. 59. 11. 43. 27. 59. 11. 43. 27. 59. 11. 43. 27. 59. 11. 43. 27. 59.
11. 43. 27. 59. 11. 43. 27. 59. 11. 43. 27. 59. 11. 43. 27. 59. 11. 43.
27. 59. 11. 43. 27. 59. 11. 43. 27. 59. 11. 43. 27. 59. 11. 43. 27. 59.
11. 43. 27. 59. 11. 43. 27. 59. 11. 43. 27. 59. 11. 43. 27. 59. 11. 43.
27. 59. 11. 43. 27. 59. 11. 43. 27. 59. 11. 43. 27. 59. 11. 43. 27. 59.
11. 43. 27. 59. 11. 43. 27. 59. 11. 43. 27. 59. 11. 43. 27. 59. 11. 43.
27. 59. 11. 43. 27. 59. 11. 43. 27. 59. 11. 43. 27. 59. 11. 43. 27. 59.
11. 43. 27. 59. 11. 43. 27. 59. 11. 43. 27. 59. 11. 43. 27. 59. 11. 43.
27. 59. 11. 43. 27. 59. 11. 43. 27. 59. 11. 43. 27. 59. 11. 43. 27. 59.
11. 43. 27. 59. 11. 43. 27. 59. 11. 43. 27. 59. 11. 43. 27. 59. 11. 43.
27. 59. 11. 43. 27. 59. 11. 43. 27. 59. 11. 43. 27. 59. 11. 43. 27. 59.
11. 43. 27. 59. 11. 43. 27. 59. 11. 43. 27. 59. 11. 43. 27. 59. 11. 43.
27. 59. 11. 43. 27. 59. 11. 43. 27. 59. 11. 43. 27. 59. 11. 43. 27. 59.
11. 43. 27. 59. 11. 43.]
chan_y: [ 0. 3840. 3860. 20. 40. 3880. 3900. 60. 80. 3920. 3940. 100.
120. 3960. 3980. 140. 160. 4000. 4020. 180. 200. 4040. 4060. 220.
240. 4080. 4100. 260. 280. 4120. 4140. 300. 320. 4160. 4180. 340.
360. 4200. 4220. 380. 400. 4240. 4260. 420. 440. 4280. 4300. 460.
480. 4320. 4340. 500. 520. 4360. 4380. 540. 560. 4400. 4420. 580.
600. 4440. 4460. 620. 640. 4480. 4500. 660. 680. 4520. 4540. 700.
720. 4560. 4580. 740. 760. 4600. 4620. 780. 800. 4640. 4660. 820.
840. 4680. 4700. 860. 880. 4720. 4740. 900. 920. 4760. 4780. 940.
960. 4800. 4820. 980. 1000. 4840. 4860. 1020. 1040. 4880. 4900. 1060.
1080. 4920. 4940. 1100. 1120. 4960. 4980. 1140. 1160. 5000. 5020. 1180.
1200. 5040. 5060. 1220. 1240. 5080. 5100. 1260. 1280. 5120. 5140. 1300.
1320. 5160. 5180. 1340. 1360. 5200. 5220. 1380. 1400. 5240. 5260. 1420.
1440. 5280. 5300. 1460. 1480. 5320. 5340. 1500. 1520. 5360. 5380. 1540.
1560. 5400. 5420. 1580. 1600. 5440. 5460. 1620. 1640. 5480. 5500. 1660.
1680. 5520. 5540. 1700. 1720. 5560. 5580. 1740. 1760. 5600. 5620. 1780.
1800. 5640. 5660. 1820. 1840. 5680. 5700. 1860. 1880. 5720. 5740. 1900.
1920. 5760. 5780. 1940. 1960. 5800. 5820. 1980. 2000. 5840. 5860. 2020.
2040. 5880. 5900. 2060. 2080. 5920. 5940. 2100. 2120. 5960. 5980. 2140.
2160. 6000. 6020. 2180. 2200. 6040. 6060. 2220. 2240. 6080. 6100. 2260.
2280. 6120. 6140. 2300. 2320. 6160. 6180. 2340. 2360. 6200. 6220. 2380.
2400. 6240. 6260. 2420. 2440. 6280. 6300. 2460. 2480. 6320. 6340. 2500.
2520. 6360. 6380. 2540. 2560. 6400. 6420. 2580. 2600. 6440. 6460. 2620.
2640. 6480. 6500. 2660. 2680. 6520. 6540. 2700. 2720. 6560. 6580. 2740.
2760. 6600. 6620. 2780. 2800. 6640. 6660. 2820. 2840. 6680. 6700. 2860.
2880. 6720. 6740. 2900. 2920. 6760. 6780. 2940. 2960. 6800. 6820. 2980.
3000. 6840. 6860. 3020. 3040. 6880. 6900. 3060. 3080. 6920. 6940. 3100.
3120. 6960. 6980. 3140. 3160. 7000. 7020. 3180. 3200. 7040. 7060. 3220.
3240. 7080. 7100. 3260. 3280. 7120. 7140. 3300. 3320. 7160. 7180. 3340.
3360. 7200. 7220. 3380. 3400. 7240. 7260. 3420. 3440. 7280. 7300. 3460.
3480. 7320. 7340. 3500. 3520. 7360. 7380. 3540. 3560. 7400. 7420. 3580.
3600. 7440. 7460. 3620. 3640. 7480. 7500. 3660. 3680. 7520. 7540. 3700.
3720. 7560. 7580. 3740. 3760. 7600. 7620. 3780. 3800. 7640. 7660. 3820.]

thank you @TomBugnon

This is really helpful, we'll do more tests on our side and hopefully fix this by the end of the week!

@TomBugnon can you test this? #142

@alejoe91 Looks good with #142 !

The probe map is correct, the recording shows properly.
Channel X and Y coordinates are identical except for a translation along X-axis (MetatoCoord takes probe edge as origin, read_sglx takes lower-left channel I believe)
image

Thank you!

Hi all. In case it is helpful, I have an ImecMap class that uses Jennifer's code to parse the Neuropixel 1.0 Imec Readout Table (imro) and Imec Channel Map (cmp) directly from a .bin file's metadata (or other sources) and, and puts together a handy table that keeps straight the user order, channel order, electrode site, channel id, x and y coordinates, etc. We use funky probe layouts all the time, and so far haven't encountered any issues. Link. Implementation.

Awesome! I guess we only have test data with standard configs so we never spotted this. I'll double check that everything is the same also for NP2 and include your files for testing, then we'll merge and make a new release. Sorry about the hassle!

Thanks @jenniferColonell

Do you happen to have some test meta files for now probes, also with non-standard configurations?

Thanks @jenniferColonell

I don't see the attachments though. Could you send them over on slack?

THanks all for the help and feeback.
Alessio find the bug.
@jenniferColonell : do you allow us to push publically (in the repo) some of your files for running CI tests ?

@jenniferColonell we would use the meta files you provided for internal testing

Fix released in 0.2.14 and release notes here: https://probeinterface.readthedocs.io/en/main/releases/0.2.14.html

Note that the bug was introduced on Sept. 1st 2022, so not too long ago.

Thank you all for reporting it, providing test data, and support!