Scripts to help with the reduction of MUSE observations

The following scripts automate the most tedious tasks when reducing data with the MUSE pipeline. We assume that the data has been downloaded from the ESO archive with the processed calibrations. All data that are downloaded for one object are assumed to be stored in a single folders, which are conveniently named after the objects. The scripts provided here are then assumed to live in the folder that contain these object folders.

If you find these scripts useful for your research, please acknowledge their use in a suitable manner (e.g., with a footnote containing an URL to this repository) in your publications.

Here I only provide some basic information regarding the reduction procedures and how the scripts are used for the reduction. A detailed description of those processes can be found in:

Weilbacher et al. 2020, A&A 641, A28 – https://doi.org/10.1051/0004-6361/202037855
The MUSE Manual (ESO-261650) – https://www.eso.org/sci/facilities/paranal/instruments/muse/doc.html
The MUSE Data Reduction Manual (Urrutia, Streicher, Weilbacher, and Sandin) – https://data.aip.de/projects/musepipeline.html

Requirements

bash (version used here: v4.2.46; should work with any version)
python3 (version used here: 3.10.6)
astropy (version used here: 5.1)

Basic reduction (muse_scibasic)

The basic science reduction removes instrumental signatures from the data (bias, flat-field, dark, and wavelength calibration). While the required master calibration data for this purposes could be created by us with the pipeline as well, we here use the master calibration data already provided by ESO.

Generation of sof-files for muse_scibasic

For each object several exposures are taken, sometimes spread over different nights. We have to associate the correct master calibration data for each exposure, namely those for which the calibration data was taken closest in time to the actual object exposure. This is especially of relevance for the illumination correction frames. The python script gen_sofs_for_scibasic.py below takes care of this.

Usage example:

   for dirname in ./J0519+0007/*_basic_*                          
   do
	 if [ -d ${dirname} ]; then
	     echo $dirname    
	     python ./gen_sofs_for_scipost.py ./J0519+0007/ $dirname/
	 fi
   done

The output of this script are sof-files for each science exposure in the object folder.

Running muse_scibasic for all science exposures

The script esorex_scibasic.sh calls esorex with the muse_scibasic recipe for each of the sof-files created in the previous step. For 8 object exposures this takes about 30 minutes on my MUSE reduction machine. We use the recipe parameters --nifu=-1, which parallelises the operation over all detectors, --merge=true, which combines individual files into single files (where possible), and --resample=true, which creates wavelength calibrated resampled fits images for each detector to allow for a quick visual check of the result.

Example usage:

esorex_scibasic.sh ./J0057-0941/ # (replace J0057-0941 with the name of the object)

Post-processing with muse_scipost, muse_exp_align, and muse_exp_combine

Equipped with the output from the basic reduction, we can now perform the post-processing that ultimately leads to science ready data-cuboids (modulo sky-subtraction \& other possible artefacts, TBD).

Generation of sof-files for muse_scipost

The bash script gen_sofs_for_scipost.sh collates the output from muse_scibasic and associates the correct calibration files for each exposure to create the sof-files for muse_scipost. The required calibration files are the standard star exposure, telluric star exposure, LSF profile determination, and astrometric solution. We use the files downloaded as processed master-calibrations from the ESO archive (use the calSelector service accordingly when downloading MUSE raw-data from the ESO archive).

As with muse_scibasic, we use calibration data that is closest in time to the actual science exposure. These associations are created by the python-script gen_sofs_for_scipost.py , which is called by gen_sofs_for_scipost.sh. Normally, no direct interaction by the user with gen_sofs_for_scipost.py is needed.

Usage example:

./gen_sofs_for_scipost.sh ./J0057-0941/

Running muse_scipost

With the sof-files from the previous step we can now run muse_scipost on the output of muse_scibasic for each object. This is done with the following script esorex_scipost.sh. We save the individual cubes and pixeltables for each object exposure, as well as the output from the sky subtraction routine (which we want to modify below). All this is done with the script esorex_scipost.sh

Usage example

./esorex_scipost.sh ./J0057-0941/

Exposure alignment with muse_exp_align

The initial solution of the astrometry that is applied to each individual exposure is anchored on the catalogued position of the guide star and the known offset between the guide star and the centre of the MUSE field of view. Since the positional accuracy of the guide star catalogue (GSC2.3) is not good enough, we have to properly align the individual exposures prior to any attempt of combining them into a exposure stacked datacube. This is achieved with the pipeline task muse_exp_align.

The script gen_sof_and_esorex_exp_align.sh automates the process of finding all the required input files for the sof-file and then running the task. The list of files for the sof-file is displayed to the user prior to the execution of the muse_exp_align recipe.

Normally this script should be called via ./gen_sof_and_esorex_exp_align.sh ./obj_name/, but sometimes the default threshold for finding objects on which the alignment is triangulated needs to be adjusted to find more objects. In this case pass the different threshold value as follows to the script: ./gen_sof_and_esorex_exp_align.sh ./obj_name/ --threshold=THRESH (replace THRESH with the desired threshold value).

Inspect the resulting PREVIEW_FOV_0001.fits file to check that the computed final alignment of all exposures relative to each other is good (i.e. no mirror images of stars, no elongated stars, etc.).

Usage example:

./gen_sof_and_esorex_exp_align.sh ./obj_name/  # ... or
./gen_sof_and_esorex_exp_align.sh ./obj_name/ --threshold=THRESH # to adjust exp_align threshold

The resulting output files from muse_exp_align can be found in the directory ./obj_name_reduced/.

Exposure combination with muse_exp_combine

After all above steps have been completed, the final step is to resample all pixtables onto a common grid (aka “datacube”). This is achieved by the pipeline recipe muse_exp_combine, and with the script ./gen_sof_and_esorex_exp_combine.sh we automate the creation of the sof-files and the subsequent call of the muse_exp_combine recipe. For an object consisting of 8 science exposures this procedure takes about 90 minutes on my MUSE reduction machine.

Usage example

./gen_sof_and_esorex_combine.sh ./obj_name/

The combined output is than found in the folder ./obj_name_reduced/

Summary

Task	Scripts	Wall Time
`muse_scibasic`	`./gen_sofs_for_scibasic.py ./obj_name/`
	`./esorex_scibasic.sh ./obj_name/`	30 min
`muse_scipost`	`./gen_sofs_for_scipost.sh ./obj_name/`
	`./esorex_scipost.sh ./obj_name/`	60 min
`muse_exp_align`	`./gen_sof_and_esorex_exp_align.sh ./obj_name/`	5 min
	note: one can adjust the threshold (see above)
`muse_exp_combine`	`./gen_sof_and_esorex_exp_combine.sh ./obj_name/`	90 min

Running times are what typically got for the MUSE reduction server at Leiden Observatory for objects with 8 exposures.

Second pass of muse_scipost (resampling to output grid, use –autocalib=deepfield)

Generate .sof files for 2nd run

bash script: gen_sofs_for_scipost2.sh, usage gen_sofs_for_scipost2.sh ./object_dir/
- append OFFSET_LIST and OUTPUT_WCS from first run to scipost sof files
- notes
  - this requires that the initial reduction has been inspected visually to ensure that the offsets determined by muse_exp_align are reasonable
- store these in ${object_dir}/*_basic_*/*_scipost_2.sof

Run scipost with .sof files for 2nd run

bash script esorex_scipost2.sh
- usage: esorex_scipost2.sh $directory
- usage example: esorex_scipost2.sh ./J1044+0353/
- uses --autcalib=deepfield as default
- out files: ${directory}/*basic*/*scipost_2/DATACUBE_FINAL.fits
- links files of individual cubes to ${object_dir}_reduced directory (filename glob-pattern *_scipost_2_*)

Combine cubes with mpdaf

python script mpdaf_scipost_run_combine.py
- usage mpdaf_scipost_run_combine.py ${object_dir}_reduced
- usage example mpdaf_scipost_run_combine.py -d J1044+0353_reduced
- out files: ${object_dir}_reduced/{DATACUBE,EXPMAP,STATPIX}_2_mpaf_cmbd.fits
- for cubes with few exposures, better to use MAD statistics, i.e. use the script with option --mad

Generate band images and append to combined cube

python script gen_band_images.py
- usage: gen_band_images.py ${cube_file}
- example usage: gen_band_images.py ./J1044+0353_reduced/DATACUBE_scipost_2_mpdaf_cmbd.fits
- creates the following bands: “Johnson_V”, “Cousins_R”, “Cousins_I”, “SDSS_g”, “SDSS_r”, “SDSS_i”, “SDSS_z”, “PS1_g”, “PS1_r”, “PS1_i”, “PS1_z”
- band images are appended to HDUList of input FITS file

Generate continuum image & sky-mask

python script gen_cont_image.py → creates continuum image; usage information gen_cont_image.py -h.
- example usage: gen_cont_image.py -i ./J1044+0353_reduced/DATACUBE_scipost_2_mpdaf_cmbd.fits -l 1150 1400 1830 1990
note: requires working LSDCat installation
python script gen_sky_mask.py → creates sky mask from continuum image; usage information gen_sky_mask.py -h
- example usage: gen_sky_mask.py -i cont_im_DATACUBE_scipost_2_mpdaf_cmbd.fits (default parameter should be fine in most cases)

Generate sof-files for 3rd scipost run

script: gen_sofs_for_scipost3.sh, usage gen_sofs_for_scipost3.sh ./object_dir/
- appends SKY_MASK from gen_cont_image.py & gen_sky_mask.py to second run scipost sof files
- stores these in ${object_dir}/*_basic_*/*_scipost_3.sof
example usage: gen_sofs_for_scipost3.sh ./J1044+0353/

Run scipost with .sof files for 3rd run

script esorex_scipost3.sh
- uses --autcalib=deepfield as default, and --save=cube,individual,skymodel
- links files of individual cubes to ${object_dir}_reduced directory (filename glob-pattern *_scipost_3_*)

Combine cubes with mpdaf

License

/*

---------------------------------------------------------------------------------

“THE BEER-WARE LICENSE”:

<edmund.herenz@iucaa.in> wrote this file. As long as you retain this notice you

can do whatever you want with this stuff. If we meet some day, and you think

this stuff is worth it, you can buy me a beer in return. E.C. Herenz (2023,2024)

----------------------------------------------------------------------------------

*/

Knusper/muse_reduction_helpers

Scripts to help with the reduction of MUSE observations

Requirements

Basic reduction (muse_scibasic)

Generation of sof-files for muse_scibasic

Running muse_scibasic for all science exposures

Post-processing with muse_scipost, muse_exp_align, and muse_exp_combine

Generation of sof-files for muse_scipost

Running muse_scipost

Exposure alignment with muse_exp_align

Exposure combination with muse_exp_combine

Summary

Second pass of muse_scipost (resampling to output grid, use –autocalib=deepfield)

Generate .sof files for 2nd run

Run scipost with .sof files for 2nd run

Combine cubes with mpdaf

Generate band images and append to combined cube

Generate continuum image & sky-mask

Generate sof-files for 3rd scipost run

Run scipost with .sof files for 3rd run

Combine cubes with mpdaf

License