# ASKAP SDP HAS Moved! The **ASKAP SDP project**, which is astronomy data reduction software **ASKAPsoft** and other things, is now being hosted at the CSIRO Bitbucket repository which is publicly accessible. The version of ASKAPsoft in this github repo is old and not supported in any way. Go to the CSIRO Bitbucket to obtain the latest version. Information and links (including Digital Object Identifiers (DOI) for referencing) to various versions can be found if you [search the CSIRO Data Access Portal (DAP)](https://data.csiro.au/dap/search?q=ASKAPsoft). The DAP will also have contact details for queries and issues. We have moved from github to that system on CSIRO recommendation. This github repo was setup before CSIRO obtained their Bitbucket services. --------- ASKAPsoft Science Data Processor ================================ These notes provide just a brief outline for building the ASKAPsoft Science Data Processor codebase. More detailed instructions exist on the ASKAP Computing Wiki. Software Requirements ===================== ASKAPsoft is designed to be built and executed in a standard Unix/Linux environment. The current official supported environments are: * Debian 7.x (Wheezy) Linux * Mac OSX 10.9.x (Mavericks) * Cray Linux Environment (XC30 platform) Core dependencies must be fulfilled by the platform. These include, but are not limited to, a C/C++/Fortran compiler, Make, Python 2.7, Java 7 and MPI. More specific dependencies are downloaded by the ASKAPsoft build system and are installed within the ASKAPsoft development tree. Specific to the Debian platform, after a standard installation of Debian Wheezy (7.x) the following packages will need to be installed with apt-get: * g++ * gfortran * openjdk-7-jdk * python-dev * flex * bison * openmpi-bin * libopenmpi-dev * libfreetype6-dev * libpng12-dev Checkout & Bootstrap ==================== The first step of building ASKAPsoft involved checking out a copy of the code from the Subversion repository and then running the bootstrap script. Note that during the bootstrap and build process a number of tarballs containing third party software will be fetched. If Direct HTTP access is not available on your build host then please refer the the section below titled "Archive Repository". The boostrap script build and configures the developer tools necessary for building the rest of the ASKAPsoft codebase: svn co https://svn.atnf.csiro.au/askapsdp/trunk ASKAPsoft cd ASKAPsoft /usr/bin/python2.7 bootstrap.py -n The "-n" option to bootstrap says not to use "svn update" to update the workspace prior to doing the bootstrap. Building ======== Before building ASKAPsoft, certain environment variable need to be set. This is accomplished by sourcing the initaskap.sh script, which will exist in the top level directory after the bootstrap procedure described above has been completed. Then the recursive builder (rbuild) can be used to build the entire ASKAPsoft tree: cd ASKAPsoft . initaskap.sh rbuild -S -a The "-S" option to rbuild says not to do an "svn update" prior to building. To build a release tarball for a specific package, or set of packages the release target option must be set. For example to build the core data processing applications (assuming initaskap.sh has already been sourced): cd $ASKAP_ROOT cd Code/Systems/cpapps rbuild -S -t release This will create a filenamed release-<svnrev>.tgz Archive Repository ================== ASKAPsoft depends on a number of 3rd party packages. Some of these must be supplied by the operating platform. For example a compiler such as GCC and an MPI implementation must be present. Other software is downloaded during the ASKAPsoft build process, and as such HTTP access is required. If your build host relies on a HTTP proxy to access the internet then the http_proxy & https_proxy environment variables must be set in your environment. Here is an example, where "webproxy.myorg.com" is the hostname for the proxy server that is running on port 8080 export http_proxy=http://webproxy.myorg.com:8080 export https_proxy=$http_proxy Alternatively, if access to download these files is not available at all on the build host a local repository can be set up. In this case the bootstrap.ini must be configured (prior to executing the bootstrap below) to refer to the local cache via a "file" URL. For example: remote_archive = file:///home/joe/askapsoft_dependencies Documentation ============= Included within the ASKAPsoft tree is both API documentation and user documentation. The user documentation can be built like so: cd $ASKAP_ROOT/Code/Doc/user/current rbuild -n -t doc The HTML documentation tree will be created at the following location: $ASKAP_ROOT/Code/Doc/user/current/doc/_build/html Rather than a single unified API documentation tree, each package hosts its own documentation. A package can generally be identified by the fact it has a build.py in its package root directory. The following example shows how to build the "Synthesis" package API documentation: cd $ASKAP_ROOT/Code/Components/Synthesis/synthesis/current rbuild -n -t doc The HTML documentation tree will be created at the following location: $ASKAP_ROOT/Code/Components/Synthesis/synthesis/current/html Release Build ============= While the ASKAPsoft applications can be executed directly out of a development tree, a deployment consisting of the binaries and libraries is support. A release tarball can be created for each package separately if desired, however super-packages are available as a convenient grouping. For example the "cpapps" package provides the calibration, imaging & source-finding software. It can be built like so: cd $ASKAP_ROOT/Code/Systems/cpapps rbuild -n -t release This will create a file release-1234.tgz (where 1234 is the Subversion release number) which contains bin & lib directories. This tarball can be extracted in a convenient location, such as /opt or /usr/local, and PATH/LD_LIBRARY_PATH environment variables can then be set to include the bin/lib directories respectively. In the typical HPC environment, the ASKAPsoft "cpapps" package would be deployed as an [Environment Module]. [Environment Module]:http://modules.sourceforge.net/ rbuild ====== The 'rbuild' command is the main build command for developers. It has the ability of updating from the subversion repository and also recursively resolve, update and build dependencies. To get help for 'rbuild' simply type ------------------------------------------------------------------------------ %% rbuild -h Usage: rbuild [options] [<package_path>] This is the main ASKAPsoft build command for developers. It can handle dependencies, subversion updates and changes to the build system. There are two types of build targets: recursive [depends, install, stage, release, signature] and non-recursive [bclean, clean, doc, format, pylint, functest, test, deploy]. The non-recursive targets only apply to the current package, while recursive targets are applies to all dependencies of the current package. The recursive behaviour may be overridden with appropriate flag. The default <package_path> is the current directory. Options: -h, --help show this help message and exit -a, --autobuild mode where dependencies/packages are computed once. Additionally turns on no-update and no-recursion flags i.e. -n -R -f, --force force building of packages ignoring NO_BUILD files -q, --quiet do not show all builder output [default=True] -v, --verbose show all builder output -n, --no-update no svn updates, rebuild of myself or Tools rebuild. Equivalent to "-S -M -T" -N, --no-recursive-update no svn updates, rebuild of myself, Tools rebuild or recursion. Equivalent to "-S -M -T -R -v" or "python build.py TARGET" -p EXTRAOPTS, --pass-options=EXTRAOPTS pass on package specific build options, e.g. "nompi=1" or specific functional tests e.g "-t functest -p functests/mytest.py" -t TARGET, --target=TARGET select TARGET from: depends, install, stage, release, signature, bclean, clean, doc, format, pylint, functest, test, deploy [default=install] --release-name=RELEASE_NAME the name of the staging directory and the release tarball --deploy-targets=DEPLOY_TARGETS the deployement targets to execute. Select any DEPLOY_TARGETS from: authenticate,deploy_local,deploy_remote,verify [defaul t=authenticate,deploy_local,deploy_remote,verify] Advanced Options: Caution: Use these options at your own risk. -D, --debug do not run the actual package building command -M, --no-build-myself do not rebuild myself (rbuild) -P, --no-parallel do not do parallel builds of packages -R, --no-recursion do not apply target recursively to dependencies -S, --no-svn-update do not perform subversion update -T, --no-tools do not rebuild Tools -U, --update-only Ignore any target options and just do svn update -V, --no-virtualenv do not include virtualenv in a release -X, --no-exit-on-error continue building ignoring any individual package build failures ------------------------------------------------------------------------------ rbuild system ============= The rbuild system is the infrastructure underlying the rbuild command. At the package level it can be run via the rbuild script or calling the build.py files directly. In this case there is no recursive building. For example: python build.py [-q|-x] <target> It provides the following command-line options -q suppress messages to stdout and just print errors -x exit recursivebuild when an error is encountered and targets as for rbuild. Troubleshooting =============== * Make sure that you don't have a ~/.pydistutils.cfg because this can conflict with virtualenv.