/haskell-stack-mirror-script

Bash script that downloads a full set of available stack and hackage resources ready to be mirrored offline.

Primary LanguageShellGNU General Public License v3.0GPL-3.0

This script downloads full stack infrastracture and all hackage packages.

In order to establish the offline mirror you need a http server to serve downloaded files.

Usage

Run mirror.sh with the following options or none accepting defaults.

Options (by argument number):

  1. Custom mirror directory (default: "mirror" in current path).
  2. Custom address of http server (default: "http://localhost:3000").

After successful run it provides:

  • mirror directory that is ready to be served by a http server (like nginx, apache, mighttpd2, lighttpd, etc)
  • generated "config.yaml" needed to be placed on clients here: "~/.stack/config.yaml"

Once you set up web server use stack without internet connection!

The script also performs:

  • checking downloaded files integrity
  • skipping of downloaded checked files on consequent runs

Requirements

  • bash + core-utils (tee, cut, test, sort, comm, printf, etc...)
  • grep
  • sed
  • git
  • wget
  • tar
  • gzip
  • sha1sum

Issues

  • Currently stack does not correcly support setup-info: field in "config.yaml". Related issue: #1. This means that in order to run stack setup you have to supply a path to "mirror/stack-setup-mirror.yaml" manually with --setup-info-yaml option. So if running on server side it can be:
stack setup --setup-info-yaml /srv/http/haskell-stack-mirror/stack-setup-mirror.yaml

On client side:

stack setup --setup-info-yaml http://custom-server-address:12345/srv/http/haskell-stack-mirror/stack-setup-mirror.yaml

  • Bash script can be error-prone with it's imperative nature and intended to be as a temporary solution until a good Haskell program will be developed.

  • stack setup yaml is parsed in a pretty tricky way as well as request fields in urls. Mirror directory does not retain original url path structure. So in theory collisions may happen.

  • Only Unix-like systems are supported directly. Running bash script in other environments may require special treatment. Sorry I don't have a suitable working non-Unix system and not so confident to adjust script to them without testing.

  • The list of required programs to be installed prior to running the script is not short. Though these are all pretty standard on any Unix-like OS.

  • As of 2017-08-03 there is one broken package on Hackage: hermes-1.3.4.3, that has missing files on the server, thus causing some warning messages to show. It's ok.

Tested configuration

Tested on

Arch Linux x86_64 (2017-03-03) with nginx.

nginx configuration example (/etc/nginx/nginx.conf) assuming that you put mirror directory into "/srv/http/haskell-stack-mirror" ```nginx events { worker_connections 1024; } http { include mime.types; default_type application/octet-stream; sendfile on; server { listen 3000; server_name haskell-stack-mirror; location / { root /srv/http/haskell-stack-mirror; } } } ```

NixOS 17.09

Resulted "config.yaml" shall be extended with NixOS specific features, for example:

nix:
  enable: true
  packages: []

Performance and disk-space usage

As of 2017-08-03 a fully downloaded mirror directory uses 25 GiB of space. Approximate time it takes to verify all files integrity ~15 minutes on a 2.2Ghz CPU. In case there are no new files it takes less than a minute to check for updates.

TODO

Investigate the possibility of an uncomplicated way to mirror remote Hackage documentation (and maybe other).

Inspiration and thanks

This project is inspired by offline-stack project and this google groups thread. Special thanks to Neil Mitchell !

Warning!

Please, do not overuse the downloading procedure from scratch. Excessive downloads will unreasonably increase load on servers.