Add warnings to files that contain (TARGET|PLATE|FIBER)_(RA|DEC) or (PLATE_|)REF_EPOCH.

[weaverba137]

Add a warning note to files containing (TARGET|PLATE|FIBER)_(RA|DEC) or (PLATE_|)REF_EPOCH

• these are after fiberassign, so a warning note should be added:
  • doc/DESI_TARGET/fiberassign
  • doc/DESI_ROOT/survey/fiberassign
  • doc/DESI_SPECTRO_REDUX/SPECPROD
  • doc/DESI_ROOT/vac/RELEASE/lss
• These are before fiberassign, so no warning note required:
  • doc/DESI_TARGET/{TARG_DIR,SCND_DIR}
  • doc/DESI_SURVEYOPS
• It is OK to ignore simulations, which aren't in EDR:
  • doc/DESI_SPECTRO_SIM/PIXPROD/NIGHT/EXPID/simspec-EXPID.rst

@araichoor, please add commits to branch edr-cleanup.

[weaverba137]

@araichoor, note that we have merged the edr-cleanup branch, so please create a new branch if you are working on this.

[araichoor]

for info, I was thinking to add those two sentences, when relevant:

"For the SURVEY=cmx m33 tile (TILEID=80615) tile and all the SURVEY=sv1 tiles (except TILEID=80971-80976, the dc3r2 ones), proper-motion correction was applied at the fiberassign design step; thus the following columns can have different values than in the desitarget products: TARGET_RA, TARGET_DEC, REF_EPOCH, PLATE_RA, PLATE_DEC, and PLATE_REF_EPOCH. "

"For targets with a non-zero proper motion, FIBER_RA and FIBER_DEC refer to the position at the reference epoch (but note that the proper-motion correction has been applied at the time of the observation, it is just not recorded in FIBER_RA and FIBER_DEC)."

which would looks like this:

One subtle remark:
Technically, I think the (RA, DEC, REF_EPOCH) from the MTL ledgers come from the desitarget catalogs.
However, for sv1 (and cmxm33), where we did apply proper-motion correction in the fiberassign step, we did not have yet MTL ledgers.
So I think it s not worth it to mention MTL ledgers in the first sentence.

I have the following questions:

• an item I don t address here is that, for tiles designed after sv1 (even today in the main), where do not apply proper-motion correction, for targets with no proper-motion REF_EPOCH=0 is updated to 2015.5; I don t know if that s worth it to mention;
• in the first sentence, shall I adapt the list of columns mentioned at the end w.r.t. they are present or not in the file in question (for instance, PLATE_REF_EPOCH is not present in redrock)? Or is it ok to have this "superset" of columns mentioned in each file where some of those appear (that will avoid possible mistakes in the editing of each file)?
• for the second sentence, I ve not mentioned what column should be used for the reference epoch; technically, it should be PLATE_REF_EPOCH, as this is what PlateMaker uses, but as this column is e.g. not present in the redrock file, I m not sure if I shall mention it; note that, so far, and probably for the rest of the survey, REF_EPOCH=PLATE_REF_EPOCH (and same for RA, DEC), so REF_EPOCH would work; but I m not sure we d want to go such in details there...
• when I mention "the desitarget catalogs" or the "fiberassign step", I don t know if I should/could add some links to other places (to the desidatamodel documentation? to the github codes?).

Any suggestion/comment is welcome!

[weaverba137]

@sbailey, could you take a look at the notes above? I'm not enough of an expert to judge the level of detail that is needed here.

[sbailey]

@araichoor

• an item I don t address here is that, for tiles designed after sv1 (even today in the main), where do not apply proper-motion correction, for targets with no proper-motion REF_EPOCH=0 is updated to 2015.5; I don t know if that s worth it to mention;

I don't think we need to mention that since for PMRA=PMDEC=0 any REF_EPOCH is equivalent.

• in the first sentence, shall I adapt the list of columns mentioned at the end w.r.t. they are present or not in the file in question (for instance, PLATE_REF_EPOCH is not present in redrock)? Or is it ok to have this "superset" of columns mentioned in each file where some of those appear (that will avoid possible mistakes in the editing of each file)?

I think it is ok to list the superset of columns, even if you are adding the comment to a file that doesn't have a particular column.

• for the second sentence, I ve not mentioned what column should be used for the reference epoch; technically, it should be PLATE_REF_EPOCH, as this is what PlateMaker uses, but as this column is e.g. not present in the redrock file, I m not sure if I shall mention it; note that, so far, and probably for the rest of the survey, REF_EPOCH=PLATE_REF_EPOCH (and same for RA, DEC), so REF_EPOCH would work; but I m not sure we d want to go such in details there...

I don't think you need to mentione PLATE_REF_EPOCH since it is identical to REF_EPOCH other than not being propagated.

• when I mention "the desitarget catalogs" or the "fiberassign step", I don t know if I should/could add some links to other places (to the desidatamodel documentation? to the github codes?).

You could link to DESI_TARGET/TARG_DIR/DR/VERSION/targets/PHASE/RESOLVE/OBSCON/PHASEtargets-OBSCON-RESOLVE-hp-HP and DESI_TARGET/fiberassign/tiles/TILES_VERSION/TILEXX/fiberassign-TILEID but I think it would also be fine if you don't.

i.e. I like your text and think it would be fine to post as-is, or optionally with those links. Thanks!

[araichoor]

awesome, thanks for the feedback.
I ve tested to add the links, and it works fine.
so I think I ll proceed with using this formulation (same as before, with links):

"For the SURVEY=cmx m33 tile (TILEID=80615) tile and all the SURVEY=sv1 tiles (except TILEID=80971-80976, the dc3r2 ones), proper-motion correction was applied at the :doc:fiberassign </DESI_TARGET/fiberassign/tiles/TILES_VERSION/TILEXX/fiberassign-TILEID> design step; thus the following columns can have different values than in the :doc:desitarget products </DESI_TARGET/TARG_DIR/DR/VERSION/targets/PHASE/RESOLVE/OBSCON/PHASEtargets-OBSCON-RESOLVE-hp-HP>: TARGET_RA, TARGET_DEC, REF_EPOCH, PLATE_RA, PLATE_DEC, and PLATE_REF_EPOCH. "

"For targets with a non-zero proper motion, FIBER_RA and FIBER_DEC refer to the position at the reference epoch (but note that the proper-motion correction has been applied at the time of the observation, it is just not recorded in FIBER_RA and FIBER_DEC)."

looks like this:

[araichoor]

for the record, starting from the list of files containing either REF_EPOCH or FIBER_{RA,DEC} that @weaverba137 generated, I think I ll proceed as follows:

I ll ignore these ones:

# desitarget, before fiberassign
    doc/DESI_TARGET/* (except the fiberassign-TILEID.rst)
# mtl, with (ra, dec) from desitarget, ie before fiberassign
    doc/DESI_SURVEYOPS/mtl/*
# sky targets, so no PM here; and no FIBER_{RA,DEC}
    doc/DESI_ROOT/survey/fiberassign/SURVEY/TILEXX/TILEID-sky.rst
# sims
    doc/DESI_SPECTRO_SIM/PIXPROD/NIGHT/EXPID/simspec-EXPID.rst
# (dummy FIBERMAP)
    doc/DESI_SPECTRO_REDUX/SPECPROD/calibnight/NIGHT/fiberflatnight-CAMERA-NIGHT.rst 
    doc/DESI_SPECTRO_REDUX/SPECPROD/exposures/NIGHT/EXPID/fiberflat-CAMERA-EXPID.rst
    doc/DESI_SPECTRO_REDUX/SPECPROD/exposures/NIGHT/EXPID/fiberflatexp-CAMERA-EXPID.rst
# randoms
    doc/DESI_ROOT/vac/RELEASE/lss/VERSION/potential_assignments/random/RANN/fba/fba.rst
# lss data (sv3, so no pmcorr in fiberassign) + no FIBER_{RA,DEC}
    doc/DESI_ROOT/vac/RELEASE/lss/VERSION/inputs_wspec/data/dark_emlin_catalog.rst
    doc/DESI_ROOT/vac/RELEASE/lss/VERSION/inputs_wspec/data/datcomb_tarspecwdup.rst
# should be copies from the MTL ledgers, so "desitarget-like"
    doc/DESI_ROOT/vac/RELEASE/lss/VERSION/altmtl/UnivUNUM/bright/sv3mtl-bright-hp-HPX.rst
    doc/DESI_ROOT/vac/RELEASE/lss/VERSION/altmtl/UnivUNUM/dark/sv3mtl-dark-hp-HPX.rst
# obselete file
    doc/DESI_SPECTRO_DATA/NIGHT/EXPID/fibermap-EXPID.rst

I ll update only the pm-correction in fiberassign warning for these ones (no FIBER_{RA,DEC}):

    doc/DESI_TARGET/fiberassign/tiles/TILES_VERSION/TILEXX/fiberassign-TILEID.rst
    doc/DESI_ROOT/survey/fiberassign/SURVEY/TILEXX/TILEID-targ.rst
    doc/DESI_ROOT/survey/fiberassign/SURVEY/TILEXX/TILEID-scnd.rst
    doc/DESI_ROOT/survey/fiberassign/SURVEY/TILEXX/TILEID-too.rst
    doc/DESI_SPECTRO_DATA/NIGHT/EXPID/coordinates-EXPID.rst
#
    doc/DESI_SPECTRO_REDUX/SPECPROD/exposures/NIGHT/EXPID/exposure-qa-EXPID.rst
#
    doc/DESI_SPECTRO_REDUX/SPECPROD/tiles/GROUPTYPE/TILEID/GROUPID/emline-SPECTROGRAPH-TILEID-GROUPID.rst
    doc/DESI_SPECTRO_REDUX/SPECPROD/tiles/GROUPTYPE/TILEID/GROUPID/tile-qa-TILEID-GROUPID.rst

I ll update only the FIBER_{RA,DEC} warning for this one (RA, DEC only there for guide stars, but no TARGETID nor REF_EPOCH, so just ignoring):

    doc/DESI_SPECTRO_DATA/NIGHT/EXPID/pm-EXPID.rst

and I ll update both warnings for these ones:

    doc/DESI_SPECTRO_REDUX/SPECPROD/exposures/NIGHT/EXPID/cframe-CAMERA-EXPID.rst
    doc/DESI_SPECTRO_REDUX/SPECPROD/exposures/NIGHT/EXPID/fluxcalib-CAMERA-EXPID.rst
    doc/DESI_SPECTRO_REDUX/SPECPROD/exposures/NIGHT/EXPID/stdstars-SPECTROGRAPH-EXPID.rst
    doc/DESI_SPECTRO_REDUX/SPECPROD/exposures/NIGHT/EXPID/sframe-CAMERA-EXPID.rst
    doc/DESI_SPECTRO_REDUX/SPECPROD/exposures/NIGHT/EXPID/frame-CAMERA-EXPID.rst
    doc/DESI_SPECTRO_REDUX/SPECPROD/zcatalog/zpix-SURVEY-PROGRAM.rst
    doc/DESI_SPECTRO_REDUX/SPECPROD/zcatalog/ztile-SURVEY-PROGRAM-GROUPTYPE.rst
    doc/DESI_SPECTRO_REDUX/SPECPROD/preproc/NIGHT/EXPID/preproc-CAMERA-EXPID.rst
    doc/DESI_SPECTRO_REDUX/SPECPROD/preproc/NIGHT/EXPID/fibermap-EXPID.rst
    doc/DESI_SPECTRO_REDUX/SPECPROD/tiles/GROUPTYPE/TILEID/GROUPID/spectra-SPECTROGRAPH-TILEID-GROUPID.rst
    doc/DESI_SPECTRO_REDUX/SPECPROD/tiles/GROUPTYPE/TILEID/GROUPID/redrock-SPECTROGRAPH-TILEID-GROUPID.rst
    doc/DESI_SPECTRO_REDUX/SPECPROD/tiles/GROUPTYPE/TILEID/GROUPID/coadd-SPECTROGRAPH-TILEID-GROUPID.rst
    doc/DESI_SPECTRO_REDUX/SPECPROD/healpix/SURVEY/PROGRAM/PIXGROUP/PIXNUM/redrock-SURVEY-PROGRAM-PIXNUM.rst
    doc/DESI_SPECTRO_REDUX/SPECPROD/healpix/SURVEY/PROGRAM/PIXGROUP/PIXNUM/spectra-SURVEY-PROGRAM-PIXNUM.rst
    doc/DESI_SPECTRO_REDUX/SPECPROD/healpix/SURVEY/PROGRAM/PIXGROUP/PIXNUM/coadd-SURVEY-PROGRAM-PIXNUM.rst

[araichoor]

addressed in #177; closing it.