Generate transformed SWC files for Q2 2021 release

[wbwakeman]

LIMS query to identify the neuron reconstruction records for Lein U01 grant Q2 data push to BICCN archives. 80 reconstructions expected

SELECT nr.id AS neuron_reconstruction_id
FROM specimens sp JOIN ephys_roi_results er On er.id=sp.ephys_roi_result_id AND er.workflow_state = 'manual_passed' 
JOIN neuron_reconstructions nr ON nr.specimen_id=sp.id AND nr.superseded = 'f' AND nr.manual = 't'
JOIN well_known_files wkf ON wkf.attachable_id=nr.id AND wkf.well_known_file_type_id = (SELECT id FROM well_known_file_types WHERE name = '3DNeuronReconstruction') 
JOIN donors d ON d.id=sp.donor_id 
WHERE 
er.published_at = '3021-06-30' AND wkf.published_at = '3021-06-30'
--er.published_at IS NOT NULL AND wkf.published_at IS NOT NULL --uncomment to get all published reconstructions ever
ORDER BY 1;

related to #127

Tasks:

• Use Neuron morphology pipeline to create transformed versions of the reconstruction files
• Run migrate_outputs lambda
• Run latest_reconstructions.py script
• Make the files available to Wayne

[wbwakeman]

All 80 reconstructions were enqueued in the LIMS MORPHOLOGY_DATA_MOVE_QUEUE so that they transferred to AWS and started the neuron_morphology pipeline. All step functions succeeded.

Just need someone to run the script to download the raw, marker, and transformed files to a location on the Institute network file system.
https://github.com/AllenInstitute/neuron_morphology/blob/dev/pipeline/user_scripts/latest_reconstructions.py

List of reconstruction ids:

703291074, 706963251, 706967287, 707413935, 716169563, 716209783, 716225032, 716875029,
719136830, 726188074, 726244401, 726247594, 728173876, 732824188, 736899203, 756102088,
763921748, 763932191, 813020030, 813022657, 813023248, 814527800, 817736876, 825574207, 
825585295, 831475407, 832092482, 833836334, 834738392, 835415218, 839030479, 842183290,
843365843, 844974575, 845844384, 846279841, 850984385, 855939444, 865979208, 875564478, 
885591706, 885753571, 887228205, 887268286, 887321326, 889978944, 896083918, 912809707,
916693867, 936032661, 937884922, 972768276, 976055685, 978746665, 978748457, 984088173, 
984088846, 1005683664,1006972172,1007069791,1021802313,1021955768,1023088289,1023408873,
1032336566,1033837384,1040828028,1051374182,1052309685,1052812364,1055622591,1056097279,
1061228167,1063354704,1068941230,1082715183,1100989311,1100989552,1102400248,1102595665

[wbwakeman]

python latest_reconstructions.py nm-stage-deployment-out-runs /allen/aibs/technology/waynew/morph/20210630_swc/download/ --reconstruction_ids 1018012596 703291074 706963251 706967287

This command seems to work but there are not entries for the new reconstructions in the list above.

Probably need to run the migrate_outputs lambda:
https://github.com/AllenInstitute/neuron_morphology/blob/dev/pipeline/commands/lambdas/migrate_outputs.py

[gnayuy]

The all runs results are available at nm-prod-deployment-working-bucket, so I download all runs results including 80 neurons from there to /allen/programs/celltypes/workgroups/mouse_full_morphology/yangy/swc_q2_2021.

The nm-prod might be a little bit behind the dev branch for release, I'll look into it.