/pipeline

The Python data reduction pipeline for WiFeS

Primary LanguagePython

pipeline

The Python data reduction pipeline for WiFeS

July 2021: How is this code different from the main pyWiFeS repository?

  • Update to python3 and bug fixes (Adam fixed lots of them)

  • We should keep the old structure, and only add a set of scripts to tackle multinight observations: -- reduction_scripts still contains reduce_blue_data.py, reduce_red_data.py, save_blue_metadata.py and save_red_metadata.py just as before. We added multinight folder with our multinight scripts.

  • There is one config.py file now for both blue and red. Filter (band: blue/red) is set in the command line.

  • How to run: python reduce_stellar.py config.py obsdate band, e.g. python reduce_stellar.py config.py 20181128 r for red band

  • reduction steps for the original pywifes are on http://www.mso.anu.edu.au/pywifes/doku.php?id=documentation

Known problems

  • Is there still a problem with pickle files?

For Marusa's scripts to work, don't forget to add these lines (or modifies) in the .bashrc file:

export PYTHONPATH="/priv/mulga1/marusa/reduction_wifes/pipeline/src/:$PYTHONPATH" export PYWIFES_DIR=/priv/mulga1/marusa/reduction_wifes/pipeline/reference_data/ export PYTHONPATH="/priv/mulga1/marusa/reduction_wifes/pipeline/tools-master/:$PYTHONPATH" export PYTHONPATH="/priv/mulga1/marusa/reduction_wifes/pipeline/reduction_scripts/:$PYTHONPATH" export PYTHONPATH="/priv/mulga1/marusa/reduction_wifes/pipeline/reduction_scripts/extra_tools/:$PYTHONPATH"

How to run PyWiFeS reduction (stellar)

  • Update list of calibration files if using: utils/find_all_calib_files.py.
  • Edit reduction_scripts/configBlue.py and reduction_scripts/configRed.py. It is best to rename these files so they are unique and don't get overwritten when you do git pull.
  • Generate a metadata file in reduction_scripts: python generate_metadata_script_stellar.py configRed.py 20190321 If you need to specify the nights with calibration files, use python generate_metadata_script_stellar.py configRed.py 20190321 -b 2019018 -f 20190323
  • Check metadata file.
  • Run python reduce_marusa.py configBlue.py 20190321 in reduction_scripts.
  • Convert to ascii: utils/pywifes_utils.extract_stellar_spectra_ascii('/data/mash/marusa/2m3reduced/wifes/', '20190321')
  • Make a pdf with spectra: plumage/plotting.plot_nightly_spectra('/data/mash/marusa/2m3reduced/wifes/', '20190321', plot_output_path='/data/mash/marusa/2m3reduced/wifes/')
  • Delete auxiliary reduction files (because there is a lot of data): python utils/clean.py /data/mash/marusa/2m3reduced/wifes/20190321/reduced_b/ and same for reduced_r.

Modified pipeline to reduce young stars

Scripts are in pipeline/reduction_scripts/

What's changed in this version:

  • This version is adjusted for reductions of many nights. In principle you only need to provide a path to the folder for each night, the rest of params in the files stay constant.
  • generate metadata: a separate script that finds adjacent arcs for each science exposure. Output is not a pickle but python file that is imported into reduction script.
  • config file: everything you need to set. You don't have to set anything in generate metadata or reduction script anymore.

Prepare metadata

python generate_metadata_script_marusa.py configRed.py /priv/mulga1/marusa/2m3data/wifes/20190314 A new folder is created: /priv/mulga1/marusa/2m3reduced/wifes/20190114/reduced_r_ys and ys_metadata_WiFeSRed.py is stored there. Review this file to check if everything is OK.

Rename metadata file

Reduce each night

python reduce_red_data_python3.py configRed.py /priv/mulga1/marusa/2m3data/wifes/20190314 How does this script know what metadata file to read?

Convert to ascii

convert_fits_to_ascii.py

Radial velocity determination and correction

TODO

List of all files with objnames, airmass, instrument etc.

pipeline/marusa/files.fits, generated with pipeline/marusa/list_of_all_stars_observed_from_fitsfiles.py - it would be great if we could add 'Program' column to mark objects, tellurics, specphot etc. but would require more work.

TODO

  • files.fits: Marusa: Add a column Program (young stars), Adam (tellurics, spphot, standard, TESS), the rest is 'Other/...'
  • files.fits: Marusa: A column Useful: crossmatch against a separate text file with bad data
  • Adam: generate metadata: Automatically crossmatch against the list of tellurics. And add Bohlin 2013 HST flux standards to pywifes list of standards.
  • Blue
  • Simplify:
    • Copy all necessary files into pipeline
    • Make it more simple and robust.
  • What happens if there are 2 or more stars in the slit?
  • Diagnostics: It does happen that imagetype is wrong. E.g. Arcs are sometimes not arcs and then image reduction keeps crashing. Make a script to make a simple convert of all fits files in the folder into jpgs sorted into different folders based on their type. This would allow a quick look to check if all images are what they say they are.

TODO2

  • find_all_calib_files: things are hardcoded there

  • make sure all paths are in an untracked file, so when you git pull it doesn't overwrite your settings

  • put back the option to specify object name for the objects you want to reduce

  • enable individual reduction, meaning that if there are many frames of the same object that night and you don't want to co-add them but reduce separately. Manage arcs in metadata: they should go together with only one science frame.

  • Don't hardcode things. Put everything in config files!!!

  • object_list: doesn't work right now!

  • generate_metadata: It only assigns one arc to RZ Mic and many arcs to other science targets: /priv/mulga2/nstanden/20181018/...

  • generate metadata: import pu: remove this

  • include an option to chose a way of how arcs are matched with science: add coordinates

  • Propose missing calibration files: it's BROKEN!!