Concatenating/merging locs hdf5 files.

[ajasja]

• Picasso version: 0.6.7
• Python version: 3.10
• Operating System: Win11

Description

Due to the limitations of TIFF, we have 3 split TIFF files (each ~4GB). We have picked localisations in each file separately. I thought it would be easy to merge, but it took me longer than I want to admit, so I'm sharing the code. (I hope I did not miss a built in merge function)

What I Did

The trick is that the joined array has to be a np.**rec**.array and that the frame numbers have to be updated.

from picasso import io
import numpy as np

locs1, info1 = io.load_locs(r'X:\Sodelavci\Federico\TRIF_DNA_paint\2024-01-19__Gatta_ruler_40R\C1-20240119-105728_WT_r_locs.hdf5')
locs2, info2 = io.load_locs(r'X:\Sodelavci\Federico\TRIF_DNA_paint\2024-01-19__Gatta_ruler_40R\C1-20240119-105728_WT_2_r_locs.hdf5')
locs2.frame = locs2.frame + locs1[-1].frame #add last frame number
locs3, info3 = io.load_locs(r'X:\Sodelavci\Federico\TRIF_DNA_paint\2024-01-19__Gatta_ruler_40R\C1-20240119-105728_WT_3_r_locs.hdf5')
locs3.frame = locs3.frame + locs2[-1].frame #add last frame number


concat_info = info1.copy()
concat_info[0]['Frames'] = int(concat_locs[-1].frame+1)

concat_locs = np.rec.array(np.concatenate((locs1,locs2,locs3)), dtype=locs1.dtype)
io.save_locs(r'X:\Sodelavci\Federico\TRIF_DNA_paint\2024-01-19__Gatta_ruler_40R\C1-20240119-105728_WT_concat_locs_by_python.hdf5',concat_locs, concat_info)

[rafalkowalewski1]

Hi

Thank you for sharing the code. You are right, Picasso does not use the most common data structure for handling localizations. However, np.rec.arrays are quite fast and efficient.

On the other hand, I have heard lately that pandas speed has been improved in the new releases, so I will give that a try at some point - far more people are familiar with pandas :)

Bests,
Rafal

[ajasja]

Maybe you could add a util function (append_localisations([loc1, loc2, loc3]) somewhere to picasso utils?

[rafalkowalewski1]

I can add something of the sort in picasso.lib with the next release

Bests,
Rafal

[ajasja]

Thanks @rafalkowalewski1! Is append_to_rec the one?

Best,
Ajasja

[rafalkowalewski1]

Hi Ajasja,

I am sorry, I have not added a new function yet. append_to_rec is an old function that does more or less what you'd need, however, it will not automatically adjust the frame index (which you'd need to adjust manually for now).

Bests,
Rafal

[ajasja]

No problem at all -- should we repen the issue?