EarthCubeGeochron/Sparrow

Installation issues on RHEL

Opened this issue · 0 comments

This issue documents some recent issues with running Sparrow on RHEL systems that are common at some universities. Many of these issues should be mentioned on a troubleshooting help page.

We successfully solved a compilation issue for the CLI (caused by Github actions) with #246 and #254, allowing us to install the Sparrow executable again.

On trying to run Sparrow, we get a mostly successful build, with a no space left on device error near the end:

------
 > [frontend 2/5] COPY packages/sheet/package.json /app/packages/sheet/:
------
failed to solve with frontend dockerfile.v0: failed to build LLB: failed to load cache: failed to register layer: open /var/lib/docker/image/devicemapper/layerdb/tmp/write-set-324648261/diff: no space left on device
ERROR: Service 'frontend' failed to build : Build failed
ERROR: Service 'frontend' needs to be built, but --no-build was passed.
One or more containers did not build successfully, aborting.

The database/Python server application are running but the frontend is not.

  • Digging into this (looking at disk space with df -h) we learn that the /var folder (where Docker stores its files) is using almost all of its allotted space. This is a common problem with "hardened" servers that parcel the disk into space-limited volumes: /var is give relatively little space on the server relative to other functional groups, although Docker stores all internal files there. On this system, it gets 10GB (fairly standard I think).
  • docker system prune and docker system prune --volumes (after backing up the database out of caution!) reclaim relatively little space. This is sensible since Sparrow is basically the only application that has been run on this server; there aren't unused resources to reclaim.
  • One solution would be to just expand /var but this requires dealing with LVM which is really obnoxious.
  • Sparrow takes up a fair bit of space, but not 10 GB! Looking at the /var/lib/docker directory in more detail, it appears that most space is used in the devicemapper directory.
  • docker info reports that the devicemapper driver is deprecated, and asks that a --storage-opt be configured:
WARNING: the devicemapper storage-driver is deprecated, and will be removed in a future release.
WARNING: devicemapper: usage of loopback devices is strongly discouraged for production use.
         Use `--storage-opt dm.thinpooldev` to specify a custom block storage device.

I think this may be a RHEL-specific storage bug that needs to be solved at the Docker level. Ubuntu systems use overlay2 by default. It feels above my pay grade, especially since I basically never use RHEL systems. Any insight from RHEL users would be appreciated.