AllenInstitute/ipfx

Large number of cells missing long square features

Closed this issue · 7 comments

tmchartrand commented

I've discovered that quite a few recent cells have null records for long squares in their stored IPFX output (json output as queried from LIMS). I can't figure out why this is missing, since when I rerun them myself it succeeds, and no errors are recorded in the logs I can see.
An example
cell id: 839897427
output json: /allen/programs/celltypes/production/humancelltypes/prod259/Ephys_Roi_Result_839897299/EPHYS_FEATURE_EXTRACTION_V3_QUEUE_839897299_output.json
nwb file: /allen/programs/celltypes/production/humancelltypes/prod259/Ephys_Roi_Result_839897299/nwb2_H19.06.351.11.16.01.01.nwb

This might be a small issue, or it might be a big issue.
For that reason, we will timebox the investigation of this issue to 1/2 day. Any longer than that, then we need to plan how to address the issue, and schedule that in another sprint.

This appears to be related to the stimulus_ontology.json file that is generated by LIMS. The next research segment needs to investigate why the LIMS-generated stimulus_ontology.json files result in null "Long Square" and "Cell" features.

tmchartrand commented

To emphasize the magnitude of this issue, in a test set of ~200 recent inhibitory cells (newer unreleased data), about half seem to show this issue. For whatever reason it seems more common in this set than recently released excitatory patch-seq cells, but that may just be because they were re-processed more recently.

sgratiy commented

@tmchartrand Could you please link a couple of ephys_roi_result from the recent inhibitory cell pool that show the issue?

tmchartrand commented

@sgratiy that info is in the issue - or is there something specific missing that you need?

sgratiy commented

@tmchartrand I understood your comment about "test set of ~200 recent inhibitory cells" as if it was an additional set of data that exhibited this issue and though it would be good to have an additional example to work with.

tmchartrand commented
sgratiy commented

It appears that LIMS generated stimulus_ontology is a likely reasons for this issue:
when ontology was generated in python we were getting:
[
"code",
"EXTPGGAEND",
"EXTPGGAEND141203"
]
no with LIMS generated ontlogy we get this
[
"code",
"141203",
"EXTPGGAEND141203"
]
for all differences just
diff stimulus_ontology.json 1040487838_stimulus_ontology.json
in the folder for the cell listed above.

i.e., the stimulus code tag is not generate correctly by the
http://stash.corp.alleninstitute.org/projects/TECH/repos/lims/browse/app/strategies/ephys_nwb_stimulus_summary_v3_strategy.rb#33

tmchartrand commented

@wbwakeman I believe this was fixed a while back, closing now.