Dependencies
Closed this issue · 10 comments
besides the common library, we will also need to consider the executables, line file parameters, XS files, etc. needed to run LBLRTM:
- LBLRTM
- AER Line File (whole line file only, not by-molecule because of size considerations)
- XS Files and database
- LNFL
Currently, these guys exist as SVN repositories, but we should be able to easily and cleanly convert them to Git repos.
Karen gave the go-ahead for this, but it will be low on the priority list (gotta get ABSCO_tables.py done first).
this is higher on the priority list now -- Karen thinks James wants everything that is needed in version control
LBL and LNFL have their own repos now:
https://github.com/pernak18/LNFL
https://github.com/pernak18/LBLRTM
they have also been added as submodules to the ABSCO repo. now i will work on an automated build script
the AER line file (which includes XS stuff in item 3) is a bit more difficult because it is so large (700 MB) and exceeds the file size limit for GitHub. I have IT looking into Git large file storage (https://git-lfs.github.com/). another option is the external AER Git server.
model builds are now handled automatically in https://github.com/pernak18/ABSCO/blob/master/build_models.py (see baf2434 as well)
this is designed to work for the gfortran, ifort, and pgf90 compilers (i've briefly tested all three). it also replaces the designated paths for the LBL and LNFL executables and line parameter file in the .ini file the user will write.
i keep on hitting roadblocks with the AER line file because the hypothetical repository would be so large. in addition to the 700 MB whole line file, there are the single-molecule line files, line coupling file, and cross section database, all of which add up to over a GB:
% du -hs $RELEASE_LINES/aer_v_3.6/
1.9G /nas/project/rc_static/models/aer_line_parameters/AER_line_files/aer_v_3.6/
GitHub institutes a 100 MB limit on repositories, so we've had to rethink our approach. i've already discussed Git LFS -- that's been installed, but i'm still running into the same errors as i was without LFS, possibly because i have to pay to use it (https://help.github.com/articles/about-storage-and-bandwidth-usage/ -- i get 1 GB of storage and bandwidth per month for free, perhaps i've exceeded my bandwidth for september?).
another option is committing a tarball (g-zipped) into version control, but that, too, is too large:
% ls -lh $RELEASE_LINES/aer_v_3.6.tar.gz
-rw-r--r--. 1 rpernak blue 412M Aug 2 2017 /nas/project/rc_static/models/aer_line_parameters/AER_line_files/aer_v_3.6.tar.gz
for now, i'm exploring the usage of AER's external Gitlab server, which hopefully has no such limitations. i've inquired with IT about this.
the line parameter database (LPD) has been uploaded to AER's external Gitlab server and added to this repo's submodule list so that a --recursive clone will pull all dependencies for the ABSCO library (see d35baa5). note that the LPD, LNFL, and LBLRTM submodules are in a detached HEAD state -- i.e., they were linked at a specific point in time and any modifications to them have not been applied for the ABSCO project.
a caveat is that any user that wants to use this library will need to be added to the LPD repo access list. if this is problematic, we can store the LPD somewhere (James suggests https://zenodo.org/). for now, while only a few people are using this software, we should be OK.
all dependencies have been addressed
When trying to update from the external Gitlab server I found I needed to manually modify the username in the .gitmodules and .git/config files to have the username match my own instead of the default of rpernak
thanks James. Karen and i just discovered this today (this morning). apologies. i'll work on a fix
On my dev machine im using Git 2.17.1