LSSTDESC/healsparse

Are there file serialization formats for HTM, Q3C that would be worth make Sparse versions for?

Opened this issue · 0 comments

wmwv commented

Pardon, this is a bit off topic, but are there file serialization formats for storing data with HTM or Q3C indexing with some fast-access pattern? I'm familiar with them in databases but have not been able to find a serialized file format implementation aimed at performance.

E.g., sure, you can save them as a column in a Parquet file and load that into an dataframe with the trixel/pixel number as an integer, but that doesn't take advantage of the structure of the numbering scheme when doing things like nearest neighbor or region lookups.

The relevance here is would a HealSparse equivalent be worth developing for those?