oxkat

"Ceci n'est pas une pipe[line]"
The Treachery of Images, René Magritte, 1929

What is this?

A set of Python scripts with the aim of (semi-)automatically processing MeerKAT data.
At the core is a set of functions that generate calls to various pieces of radio astronomy software, a semi-modular bunch of CASA scripts for performing reference calibration, and a fairly sizeable list of default parameters. The defaults at present cater for full-band Stokes-I continuum imaging, including the correction of direction-dependent effects.
Job script generation and dependency chains are automatically handled when running on the ilifu/IDIA cluster, UKZN's hippo cluster, or the CHPC's Lengau cluster.
Setup scripts glue the above components together into a processing recipe. The default procedure is broken down into stages, after each of which it is advisable to pause and examine the state of the process before continuing.
The intention is that the bar to entry is low. If you have stock Python then nothing else needs installing apart from Singularity, which is available on both the ilifu/IDIA and CHPC clusters. All the underlying radio astronomy packages are containerised. The Singularity layer can also be disabled for running installations on your own machine, either directly, or inside a Python virtual environment.
If you publish results that have made use of oxkat then please cite the ACSL entry, and (more importantly) the underlying packages used.
Please file bugs, suggestions, questions, etc. as issues.

Quick start

Once you have the container(s) in place then log into your machine or cluster, e.g.:
```
$ ssh ianh@slurm.ilifu.ac.za
```
Navigate to a working area / scratch space:
```
$ cd /scratch/users/ianh/XMM12
```

Clone the root contents of this repo into it:

$ git clone https://github.com/IanHeywood/oxkat.git .

Make a symlink to your MeerKAT Measurement Set (or place it in the working folder, it will not be modified at all):
```
$ ln -s /idia/projects/mightee/1538856059/1538856059_sdp_l0.full_1284.full_pol.ms .
```
The first step is to run a script that gathers some required information about the observation:
```
$ python setups/0_GET_INFO.py idia
$ ./submit_info_job.sh
```
Once this is complete you can generate and submit the jobs required for the reference calibration (1GC):
```
$ python setups/1GC.py idia
$ ./submit_1GC_jobs.sh
```
If something goes wrong you can kill the running and queued jobs on a cluster with:
```
$ source SCRIPTS/kill_1GC_jobs.sh
```
Once all the jobs have completed then you can examine the products, and move on to the setup for the next steps in the same fashion.

Please see the setups README for more details about the general workflow. Most of the settings can be tuned via the config.py file.

Containers

There is a dedicated Singularity container (oxkat-0.41.sif) available that contains all the necessary packages and dependencies. This is available in the general container repository on the ilifu/IDIA cluster, and the default settings should pick it up automatically when that cluster is being used. For other systems the container will have to be downloaded (or copied over). The container can be downloaded here.

String patterns for package-specific containers are specified in the config.py file. The scripts will search for containers that match these patterns in the container paths, so it's simple to swap a particular package out for a different version as long as you have it containerised.

Software package roll-call

Package	Stage	Purpose	Reference
`astropy`	1GC, 3GC	Coordinates, time standards, FITS file manipulation	Astropy Collaboration, 2013, Astropy Collaboration, 2018
`CASA`	1GC	Averaging, splitting, cross calibration, DI self-calibration, flagging	McMullin et al., 2007
`CubiCal`	2GC, 3GC	DI / DD self-calibration	Kenyon et al., 2018
`DDFacet`	3GC	Imaging with direction-dependent corrections	Tasse et al., 2018
`killMS`	3GC	DD self-calibration	Tasse, 2014; Smirnov & Tasse, 2014
`owlcat`	2GC, 3GC	FITS file manipulation	-
`ragavi`	1GC, 2GC	Plotting gain solutions	-
`shadeMS`	1GC	Plotting visibilities	Smirnov et al., 2022
`Singularity`	1GC, FLAG, 2GC, 3GC	Containerisation	Kurtzer, Sochat & Bauer, 2017
`tricolour`	FLAG	Flagging	Hugo et al., 2022
`wsclean`	FLAG, 2GC, 3GC	Imaging, model prediction	Offringa et al., 2014

Thank you for visiting.

joesbright/oxkat

oxkat

What is this?

Quick start

Containers

Software package roll-call