Z = Z_RR?
abrodze opened this issue · 9 comments
The column descriptions for 'Z' and 'Z_RR' are very similar and potentially confusing why the two values are not numerically identical. Z is post QSO afterburner, correct? This makes Z = Z_RR only for galaxy and star spectype, if this is true. I suggest Z column definition be changed to "final redshift estimate", as it could equal Z_RR or a redrock redshift after QN refinement
@abrodze, what specific files are affected by this?
Z_RR would be pulled from redrock*.fits files in the redux directory and Z is present in the QSO catalogs (LSS, lya, etc) (@echaussidon would know more specifics than me). Both Z_RR and Z are present in all versions of QSO catalogs
This might be a case where a single description in column_descriptions.csv isn't sufficient.
- Redrock files use "Z"
- qso_qn renames that to "Z_RR" (and downgrades to float32), and then computes a new "Z_NEW"
- I think the QSO catalogs then use "Z_NEW" renamed back to "Z"
i.e. it appears that the "Z" in the QSO catalogs is the updated Redrock fit using the QuasarNet prior (aka "Z_NEW"), which is not the same thing as the "Z" in the original redrock output (pre-QN). While this is somewhat messy, I am sympathetic to the QSO catalogs wanting the simple name "Z" to be the best recommended redshift to use.
The consequence is that we need to be careful with the data model descriptions and not just blindly use update_column_descriptions --force ... which would could overwrite with the wrong description. Since that is fragile (never trust a human to remember to do the right thing!), we could add special cases to update_column_descriptions with a list of column names to never override.
Indeed, the LSS catalogs for QSO use the QSO catalog 'Z' as 'Z' (and consistent with the described behavior, rename the previously loaded redrock 'Z' as 'Z_RR').
I agree the simplest solution is probably to have special cases that don't get descriptions over-written.
Another one is 'ZWARN'? How do we handle the fact that the zmtl files add extra bits? (For the LSS catalogs, we rename the zmtl 'ZWARN' to 'ZWARN_MTL' and we have both.)
I think we still need to check the overall correctness of the data model in light of this question.
@ashleyjross, @aureliocarnero, I notice that Z_RR came up in PR #165. Is the work you are doing on that PR sufficient to close this issue?
It is basically the same issue, though I'm not sure that we fully solved the potential confusion. We should be sure to document as well as possible in the column descriptions.
OK, thanx. I think "document as well as possible" should be sufficient to close this issue as part of #165.
Closed by #166.