/np_codeocean

Tools for uploading and interacting with Mindscope Neuropixels experiments on Code Ocean

Primary LanguagePython

np_codeocean

Tools for uploading and interacting with Mindscope Neuropixels experiments on Code Ocean

Requires running as admin on Windows in order to create remote-to-remote symlinks on the Isilon.

  • upload CLI tool is provided, which uses the np_session interface to find and upload raw data for one ecephys session:

    pip install np_codeocean
upload <session-id>

    where session-id is a valid input to np_session.Session():

    • a lims ID (1333741475)
    • a workgroups foldername (DRPilot_366122_20230101)
    • a path to a session folder ( \\allen\programs\mindscope\workgroups\np-exp\1333741475_719667_20240227)

  • a folder of symlinks pointing to the raw data is created, with a new structure suitable for the KS2.5 sorting pipeline on Code Ocean

  • the symlink folder, plus metadata, are entered into a csv file, which is submitted to http://aind-data-transfer-service, which in turn runs the aind-data-transfer tool on the HPC, which follows the symlinks to the original data, median-subtracts/scales/compresses ephys data, then uploads with the AWS CLI tool

  • all compression/zipping acts on copies in temporary folders: the original raw data is not altered in anyway