################################################################################################### Python and package vesrions: A) Python: Python 3.5.2 :: Anaconda custom (x86_64) B) Python packages - NumPy: v1.12.1 - Astropy: v1.1.1 - SciPy: v1.2.0 - Matplotlib: v1.5.1 - Numba: v0.26.0 C) Extreme deconvolution: Package written by Jo Bovy and available via https://github.com/jobovy/extreme-deconvolution I downloaded the master branch zip file (May 16, 2017) and installed it on local laptop, following the instruction in INSTALL_PYTNON. URL: https://github.com/jobovy/extreme-deconvolution.git ################################################################################################### Reproducibility The following series of scripts can be used to reproduce the results/figures/ tables given in the paper. A) Download scripts - xd-elg-download.sh: Describes where and how certain relevant files can be obtained from the web. "wget <url>" commands are provided for convenience so the user can just type "source xd-elg-download.sh" to download the desired files. Download all the files before using "reproducibility" section below. - DR3-DEEP2f**-tractor-download.sh: Use these to download Tractor files as described in produce-DECaLS-DR3-Tractor-DEEP2f234. B) Scripts for preparing and combining DEEP2 and DECaLS DR3 data: - produce-deep2-f234-photo-redz-oii.py: Combine extended DEEP2 pcat catalogs, impose badflag==0 and window function masks, append colummns from redshift catalogs, DEEP2 color-selection catalogs, and selection probability catalogs. For each remaining DEEP2 object, compute class number. - produce-DECaLS-DR3-Tractor-DEEP2f234.py: Show how DR3-DEEP2f**-tractor-download.sh scripts were generated using DECaLS DR3 bircks file. Combine all relevant Tractor files and mask objects that do not have positive flux invariance and brickprimary !=0. Append Tycho-2 stellar mask column and impose DEEP2 window functions. - produce-DECaLS-DR3-DEEP2-f234.py: Load processed DEEP2 and DECaLS catalogs. Apply Tycho-2 stellar masks to DECaLS objects impose flux positivity condition. Cross-match the two catalogs within respective field. After accounting for astrometric differences (in median), append select information from DEEP2 to DECaLS catalogs. Save the whole DECaLS catalog with an additional column indicating whether the object was found in DEEP2 or not. Lastly, estimate and save the area of intersection of the two survey based on point Monte Carlo method. Note 1: These scripts should be ran sequentially. C) Scripts for training obtaining XD selection model parameters - fit-dNdm-glim24.py: Load the intersection set data, set basic parameters for fitting, and fit power law or broken power law to number density distribution in g-magnitude. - fit-GMM-glim24.py: Load the intersection set data, set basic parameters for fitting, and fit Gaussian Mixture Models with varying number of componets to the gr vs. rz color-color density distribution. D) Scripts for figures and tables for the paper - plot-classes.py: Make gr vs. rz plots of various classes whose definitions are given in the paper. - generate-performance-references.py: Apply the FDR selection to the DEEP2 intersection set data. Show dNdz plot as well as the breakdown of classes in LaTex format. - apply-XD-fiducial-DEEP2.py: XD method applied with a fiducial set of parameters. Show the breakdown of classes and the corresponding dNdz plot, breakdown based on projections, plot boundary images, and create a movie. Compare dNdz of fiducial and FDR. - apply-XD-fNoZ50-DEEP2.py: The same as apply-XD-fiducial-DEEP2.py except using fNoZ = 0.5 instead of fNoZ = 0.25. Comparison to the fiducial case is shown through stills and a movie as well as in dNdz and dNdm plots. - apply-XD-N_total3000-DEEP2.py: The same as apply-XD-fiducial-DEEP2.py except using N_total = 3000 instead of 2400. Comparison to the fiducial case is shown through stills and a movie as well as in dNdz and dNdm plots. - apply-XD-zlim22p9-DEEP2.py: The same as apply-XD-fiducial-DEEP2.py except using zlim = 22.9 instead of 22.4. Comparison to the fiducial case is shown through stills and a movie as well as in dNdz and dNdm plots. - apply-XD-mag-var.py: Compute observed target densities based on the fiducial set of parameters but define selection boundary using different depths (e.g., glim = 24.3 instead of glim = 23.8.). Show results for changing limiting depths by pm 0.25 and pm 0.5 for each band. ################################################################################################### Example script for the user: The user can use the script "XD-user.py" to 1) make projections based on the selection algorithm using the parameters provided (XD-parameters) and 2) apply the selection algorithm on DEEP2 dataset. Note that since DEEP2 catalog may not be representative of the rest of DESI imaging catalogs, the selection on data should be treated as a sanity check rather than a rigorous demonstration of the correctness of the algorithm. ################################################################################################### Data repository: The following data can be found on ... The files A) through C) must be downloaded before running the scripts in the "Reproducbility" section. A) tycho2.fits Tycho-2 stellar catalog used for stellar masking. URL: C) deep2-f**-redz-oii.fits DEEP2 redshift catalogs that contain OII 3727 and target weight information and produced by John Moustakas. URL: D) color-selection.txt This file contains information about which DEEP2 photometric objects were selected for spectroscopic information in Field 2, 3, and 4. E) XD-parameters A directory containing GMM and dNdm parameters for all relevant classes for making XD-projections can be found here. They are the results obtained from running the scripts above. URL: F) DECaLS-DR3-DEEP2f**-glim24.fits The final data products of running the scripts above. Required for perfomring validation. URL: ################################################################################################### Utilities A) xd_elg_utils.py: This module contains the majority of custom functions used in the scripts above. B) table_utils.py: Thie module contains routines used to create latex tables. C) confidence_contours.py and confidence_level_height_estimation.py: Routines used in XD GMM fit plots. D) XD_selection_module.py: Include routines used for generating XD selection on DECaLS targets and making projections. Also include utitlies to plot XD boundaries and movies.