/squashmount

Init and management script for mounting rewritable squashfs-compressed data

Primary LanguagePerl

squashmount

(C) Martin Väth (martin at mvath.de)

This project is under the BSD license 2.0 (“3-clause BSD license”). SPDX-License-Identifier: BSD-3-Clause

Init and management script for mounting rewritable squashfs-compressed data

This is the successor of the squash_dir project

It is actually a ground-up rewrite of that project in perl with a highly improved control interface.

One of its aims is to be init system independent. For openrc and systemd ready-to-use init/service-files are provided; I gladly add init-files for other init systems if I receive (tested) patches.

squashmount works with systemd in “standard” setups, but due to conceptional bugs of systemd in some setups (see section “Installation”), systemd is no longer officially supported.

What is this project?

This is squashmount, a generic initscript, user interface and management tool for keeping directories compressed by squashfs but simultaneously keep them writable using some of (depending on the configuration and what is available):

Since squashmount v15.0.0 there is also a choice between

The idea is that, as a rule, on shutdown the data is recompressed (and the temporary modified data removed). This approach is originally due to synss' script from http://forums.gentoo.org/viewtopic-t-465367-highlight-.html In that forum thread you can also ask for help about this project.

For some mount points different rules than “resquash and delete on umount” might be desired (see the examples section below), and moreover, it might be necessary to override these rules temporarily. For such things a powerful user interface is provided.

This project can be useful for any linux distribution. Historically, the main motivations was to keep the Gentoo main repository compressed. This is still one of the most striking examples:

If the Gentoo main repository (/usr/portage or /var/db/repos/gentoo) is compressed with squashmount (without DISTDIR which you should store somewhere else when using this script), the required disk space is only about 50-100 MB (depending on your compression method), instead of 200-400 MB (or much higher, the actual space requirement depending essentially on the filesystem, e.g. how inodes are used). Usually, also the access is much faster.

It is possible to combine squashmount with portage's sync-type = squashdelta to mount the Gentoo repository writable.

Screenshot with a typical usage

Demo screenshot

Requirements

The script requires of course that squashfs support is activated in the kernel (and supports the COMPRESSION method), that the mksquashfs tool is available, and also that some of the above mentioned unionfs-type tools is available and supported by the kernel.

Warning

Since v17.0.0/v10.0.0 , squashmount defaults to the COMPRESSION method zstd/lz4. This method is available only in linux-4.14/3.19 or higher or in squashfuse-0.1.101_alpha20170917/0.1.100_alpha20140523 or higher. So take care to either use a sufficiently new kernel/squashfuse version or to change the default!

Moreover, you need a decently new version of perl5 together with some of its standard modules (which might need to be installed separately if your perl5 version should be too old). Decently new perl versions should have the TERM::ANSIColor module; you need this if you want to see nicely colored output.

It is also strongly recommended to install the File::Which module (although there are some fallbacks if it is not available).

If you want that the hard status line is set, also the title script from https://github.com/vaeth/runtitle (version >=2.3) is required in your path.

Installation

If you are a Gentoo user, you can just emerge squashmount from the mv overlay.

Otherwise you just have to copy bin/squashmount into /usr/bin/squashmount or any other directory of your $PATH. For zsh completion support also copy zsh/_squashmount into a directory of your zsh's $fpath.

It is strongly recommended to put

  • alias squashmount='noglob squashmount'

into your ~/.zshrc or /etc/zsh/zshrc or /etc/zshrc, so that things like

  • squashmount start *

will work in your zsh as intended without the need to quote *. (I assume that you do not use any poor shell instead of zsh.) ;)

If you use revdep-rebuild from Gentoo or similar distributions, and if you use the default naming scheme, it is recommended to copy the content of etc/revdep-rebuild into /etc/revdep-rebuild to cancel duplicate or obsolete paths search of revdep-rebuild.

For openrc support copy openrc/init.d/squashmount to /etc/init.d/squashmount and activate it in the usual way. For systemd-support copy systemd/system/squashmount.service to your systemd unit folder (pkg-config --variable=systemdsystemunitdir systemd, usually /lib/systemd/system or /usr/lib/systemd/system) and activate it in the usual way (or e.g. copy into /etc/systemd/system)

If you use systemd be sure to compile the mount binary with systemd support (this should be the case in most distributions providing systemd; in Gentoo this means to enable USE=systemd for the util-linux package. If you compile util-linux manually, make sure to pass --with-systemd to ./configure). In this case, systemd will probably work for you in “standard” setups. With systemd-219 (or newer?) and some unusual setups like --make-shared on some partitions, it can happen nevertheless that mount appears to work, but actually nothing is mounted if systemd is in use. This is related with the fact that systemd tries to control all mounts instead of letting the kernel do it alone. Of course, this breaks tools like squashmount completely. Bug systemd upstream about such problems, but not me: I am not planning to add hacks to fix the breakage introduced by some ill-conceived systemd concepts.

For systemd, you should set an appropriate timeout: There is no general rule how long compression can take “maximally”, so the timeout is set to infinity, by default. It is strongly recommended to set this to a realistic value for you system and setting by giving a (generous) upper estimate for your needs by copying the file etc/system/squashmount.service.d/timeout.conf to /etc/systemd/squashmount/service.d/timeout.conf and editing appropriately. If you copied the main script not to /usr/bin/squashmount, you should put into the same directory a file with appropriate modified paths. For instance, if you copied the main script to /sbin/squashmount then create /etc/systemd/system/squashmount.service.d/exec.conf with the content

[Service]
ExecStart=
ExecStart=/sbin/squashmount start
ExecStop=
ExecStop=/sbin/squashmount -f --lsof=0 stop

Also copy tmpfiles.d/squashmount.conf to /usr/lib/tmpfiles.d, although this is not absolutely necessary (squashmount will create the corresponding directories anyway). If you use an init-system which does not mount /run as a ramdisk, you should cleanup /run/squashmount on every fresh start before calling squashmount start. Depending on your init-system, a way to achieve this might be to change the first letter in the crucial line in /usr/lib/tmpfiles.d/squashmount.conf from d to D! and to make sure that the processing of /usr/lib/tmpfiles.d takes place before calling squashmount start. (Since accidental cleaning can have very inconvenient consequences, and currently only systemd supports the D! syntax, d is the default.) See section Emergency Case what to do if /run/squashmount is removed accidentally anyway. If you use find_cruft, you might want to copy the content of lib/find_cruft to /usr/lib/find_cruft or /etc and adapt it to your needs.

If you plan to use portage's sync-type = squashdelta, you might want to copy the content of etc/portage/repo.postsync.d to /etc/portage/repo.postsync.d Note that the hook-file in this directory treats the mount point “gentoo” specially! See the example configuration in etc/squashmount.pl how to setup an appropriate mount point “gentoo” for this setting.

In all cases you have to copy lib/squashmount.pl to /etc/squashmount.pl and adapt it to your need! This is an essential point of squashmount, and it is impossible to use squashmount without setting up the configuration. You can optionally also copy lib/squashmount.pl to /lib/squashmount.pl or /usr/lib/squashmount.pl to provide a system-wide example config. Alternatively, you can also modify that file to use it as a fallback if /etc/squashmount.pl is not readable.

Some Examples

Essentially, the init-system (or you) has to call

  • squashmount start

on start and

  • squashmount -f --lsof=0 stop

on shutdown (at a time when the local filesystems are already/still mounted). The provided installation files for systemd and openrc do just this.

This will cause all configured mount points to be mounted/umounted correspondingly. When umounting, by default the modified data is recompressed into the squash-files (but this can be customized).

The configuration of the mount points happens in the file /etc/squashmount.pl This is a perl file, so you can use perl code in this file to source other files at your discretion.

The provided example configuration file etc/squashmount.pl is rather realistic if you are a Gentoo user: It provides the following mount points

  • (a) guest: A guest user's home directory /home/guest
  • (b) tex: The installed files from texlive /usr/share/texmf-dist
  • (c1) portage: The Gentoo repository /usr/portage
  • (c2) gentoo: The Gentoo repository when using sync-type = squashdelta
  • (d) db: The Gentoo database of installed packages /var/db

Further mount points are in the example config-file but not listed here.

For all the mount points it is reasonable to use squashmount with them for different reasons:

  • (a) The guest-user should be able to modify data in /home/guest, but its changes should usually be forgotten. (Sometimes you will not want to forget these changes, e.g. when you want to update the “default” home directory which the user sees; see below how to do this).

  • (b) The tex directory is huge, and it saves considerable space to keep it compressed on disk.

  • (c1) Keeping the Gentoo repository compressed does not only save an enormous amount of disk space but actually also speeds up portage considerable, because the disk access is faster on a single (squashed) file. Moreover, after changes with eix --sync you might want to compare the new files with previous versions in the squashed file which you can still access when you use squashmount.

  • (c2) When you prefer to use portage's sync-type = squashdelta, you already have the advantage of a compressed portage tree. However, you might want to use squashmount to make this portage tree writable (e.g. in order to temporarily fix a broken Manifest file locally).

  • (d) The db directory is short but its data is very sensible and the number files is huge: Keeping it in a compressed file gives a lot of disk space and speedup, and it makes sense to keep a compressed backup of the last mounted version.

In these examples, additional features of squashmount are used in the example configuration:

  1. For (a), it is obviously necessary to use a different treatment: Normally, the squash-file should not be generated, and the temporary data should be removed. A similar remark holds for (c2).

  2. For (d), squashmount will keep a backup even of the squash-file for the db directory.

  3. squashing of (c1) and (d) should happen only when a certain threshold of changes to the data is reached. The modified data will survive the reboot even if it is not resquashed, but it takes more diskspace of course, and there is no readonly version of the corresponding files.

  4. No resquash of the tex directory when only certain files (like the automatically generated ls-R file) were updated: In fact, if the only changes made to the directoy are in these files (it is optionally also checked that their content is not changed), the directory will be cleared when umounting/remounting/rebooting.

You can also call squashmount at runtime to resquash or clean certain directories manually or to change states for the above “default” actions on future umounts. For instance, if you changed the skeleton of the guest user in (a) you can call

  • squashmount --no-kill remount guest

to force immediate regeneration of the squashed file from the directory. Moreover, you can call squashmount to temporarily change the state of mounted directories. For example, if you want to change temporarily that the guest user's data is saved on remount, use

  • squashmount --no-kill set guest

A changed state will remain active until reset, restart, or stop is executed. Another example: If a lot of data would be resquashed at the next umount, but you want to reboot urgently, just call

  • squashmount --no-squash set

before rebooting. Conversely, if you want to squash the portage mount point despite its threshold is not reached (e.g. because you plan to make an experimental change which you plan to undo later), you can call

  • squashmount --squash restart portage

If afterwards you made your experimental change and want to undo it, call

  • squashmount --kill restart portage

If your changes take a longer time and you want to make sure that you do not forget to call the above command, you can temporarily change the state:

  • squashmount --kill set portage

Normally, this setting will survive a remount. In order to reset to the original setting after remounting, use option -r with remount:

  • squashmount -r remount portage

This is the same as calling

  • squashmount remount portage
squashmount reset portage

The above examples should perhaps be enough to give you an impression how to use squashmount. To get an exact description of the user interface and of the config file format just execute:

  • squashmount man

Emergency Case

If you accidentally removed or corrupted /run/squashmount, e.g. due to a bug in squashmount itself or in its configuration or if the init-system was misconfigured and removed /run/squashmount after calling squashmount start, you should try to umount unconditionally. You can instruct squashmount to do this with

  • squashmount -fI umount

or (if you do not want resquashing in such a situation):

  • squashmount -nfI umount

This will work reliably unless you used temporary directories in your setup. It will even work with temporary directories if their name is still stored in /run/squashmount, i.e. if the information there was not lost completely.

A Word of None-Warning

It is in general rather safe to squash a directory, even a rather vital one: Even if e.g. you boot from a kernel which has no support for some of overlay|overlayfs|aufs|unionfs-fuse|unionfs |funionfs to make the directory writable, squashmount will mount it at least as read-only (using mount --bind if necessary). Moreover, if everything goes wrong you can still use the unsquashfs tool to unpack the directory manually. Probably the only danger in packing “strange” directories are special files, hard links (this information will usually get lost), or special devices which are perhaps not supported by the used tools.

Modules and Mounting

Since version 3.0, unless configured otherwise, squashmount will attempt to modprobe the modules

  • squashfs
  • aufs
  • fuse
  • overlay
  • overlayfs
  • unionfs

when they are required; optionally/alternatively, also the corresponding kernel option can be checked in /sys/module or /proc/config.gz; also the existence of required binaries can be checked before actually the mounting is attempted. It depends on your setting whether this is done and/or whether in case of failure the corresponding tool is skipped tacitly without attempting to mount.

If no tool mounts successfully, it is attempted to use mount --bind to get the directory at least readonly on the expected place, so even in this bad situation (which probably only happens if you boot from an experimental kernel or a brand new kernel without corresponding support) you can still access the directory read-only. Hence, also rather vital directories can be compressed as long as it is not vital to write to them (and as long as the relevant programs for mounting etc. are not contained within these directories, of course).

User Permissions

If you rely only on the tool squashfuse and either unionfs-fuse or funionfs (which are all based on the fuse userspace file system), then squashmount can also be used with user permissions. To support selecting only these tools and appropriate files and dirs from the calling user, the option --user is supported since squashmount v15.0.0. For details, call squashmount man and look for the option --user.

Recent squashfs-tools

It is recommended to use a new version than 4.3 of squashfs-tools which has the -quiet option. Cf. the description of --mksquash-verbose and the $mksquash_verbose variable in ./squashmount man for details. As a Gentoo user you can install such a recent version from the mv overlay.

However, this recommendation is only an eye-candy: squashmount will work without any problems also with an unpatched version of squashfs-tools; just the display when squashing will not be so nice.