How to cross databases?
Closed this issue · 2 comments
Hi all,
are there something already implemented to join and/or cross the .LEMONdB files?
Context:
I have the same field, several hundreds of images, and I needed to divide by blocks of ~100-200 images in order to execute lemon diffphot within a reasonable time interval. Then I have several .LEMONdB archives.
First:
I want to joint them in a unique dB to visualize it using lemon juicer. Is it possible?
Second:
Could it exist misidentifications between the coordinates? What is the error that we can expect between two star coordinates in the worst scenario? Can we flag that ones that exist in one or several dB but not in other/s?
Third:
Each star at each "block" will have different reference stars on which lemon diffphot compute the light-curve. Can we evaluate the differences and obtain a cosistent light curve that join both/all the separate data for a single star, or even for all the stars in the field?
Fourth:
Is it possible to check (print, ...) the reference stars used in a database?
My point is to know exactly what is done nowadays to deal with the .LEMONdB? What are the files(.py) that I need to check? mining.py?
First:
I want to joint them in a unique dB to visualize it using lemon juicer. Is it possible?
That's currently not possible, and it would be rather difficult (if not outright impossible) to achieve: each star may have a different set of reference stars in each of the diffphot runs, and therefore the deltas of the light curve, with respect to the artificial constant star, would be different. We would not only have to match the same star in the two .LEMONdB files (this is the easy part, because we have the astrometric coordinates), but also figure out a robust way of chaining the two light curves. It's a non-trivial amount of work for an arguably limited use case.
I have the same field, several hundreds of images, and I needed to divide by blocks of ~100-200 images in order to execute lemon diffphot within a reasonable time interval. Then I have several .LEMONdB archives.
I feel your pain :D The algorithm used by diffphot is O(n^2), so the solution in our case was to throw more CPUs and memory at the problem. We've managed to reduce ~6,000 images over a weekend using a machine with 16 CPU cores and 64 GiB of memory.
Second:
Could it exist misidentifications between the coordinates? What is the error that we can expect between two star coordinates in the worst scenario? Can we flag that ones that exist in one or several dB but not in other/s?
The precision is what Astrometry.net can give us. The astrometry command is just a high-level wrapper around it, that we implemented to make things easier. There's no way to identify stars across multiple LEMONdB files, so you would have to do that yourself.
Third:
Each star at each "block" will have different reference stars on which lemon diffphot compute the light-curve. Can we evaluate the differences and obtain a cosistent light curve that join both/all the separate data for a single star, or even for all the stars in the field?
Again — there's no implemented way to do that, so you would have to implement it yourself, sorry.
Fourth:
Is it possible to check (print, ...) the reference stars used in a database?
Please see the answer below.
My point is to know exactly what is done nowadays to deal with the .LEMONdB? What are the files(.py) that I need to check? mining.py?
There's no documented API beyond the documentation of the classes in database.py. You could use this script, or this one, as two examples and starting points to implement what you need. We definitely should document this property at some point, because you're not neither the first nor the last person who is going to need to do something similar.
Thanks for contributing to this issue. As it has been 180 days since the last activity, we'll be automatically closing the issue soon. This is often because the request was already solved in some way and it just wasn't updated or it's just no longer relevant. If that's not the case, please respond here within the next 30 days to keep it open. You can read more here: https://github.com/probot/stale#is-closing-stale-issues-really-a-good-idea.