This Dracut module (dracut-sshd) integrates the OpenSSH sshd into the initramfs. It allows for remote unlocking of a fully encrypted root filesystem and remote access to the Dracut emergency shell (i.e. early userspace).
It's compatible with systems that use Dracut as initramfs manager and systemd as init system, such as Fedora, CentOS/RHEL (version 7 or greater) and SUSE. Gentoo is also to known to work with dracut-sshd, as long as it's configured with systemd and Dracut.
2018, Georg Sauthoff mail@gms.tf, GPLv3+
- Example: Open Encrypted Root Filesystem
- Example: Emergency Shell
- Install
- Space Overhead
- Host Keys
- Timeout
- Network
- Hardware Alternatives
- FAQ
- Related Work
- Tested Environments
After booting a Fedora system with encrypted root filesystem (i.e. a filesystem on a LUKS volume to be opened with cryptsetup) the Dracut initramfs blocks at the password prompt. With dracut-sshd enabled remote unlocking is then as simple as:
$ ssh headless.example.org
-sh-4.4# systemd-tty-ask-password-agent
Please enter passphrase for disk luks-123-cafe! *********
Please enter passphrase for disk luks-124-cafe! *********
-sh-4.4# Connection to 203.0.113.23 closed by remote host.
Connection to 203.0.113.23 closed.
That means under normal circumstances the completion of all password prompts automatically resumes the boot process.
The command systemd-tty-ask-password-agent --list prints an overview
over all pending password prompts.
The start of the Dracut emergency shell can be
requested via adding rd.break to the kernel command line, but
it also happens when Dracut is unable to mount the root
filesystem or other grave issues. In such cases the emergency
shell blocks the boot process. Without remote access the machine
is quite dead then.
Example session:
$ ssh headless.example.org
-sh-4.4# export TERM=vt220
-sh-4.4# export SYSTEMD=FRMXK
-sh-4.4# export LC_ALL=C
-sh-4.4# less /run/initramfs/rdsosreport.txt
-sh-4.4# journalctl -e
-sh-4.4# systemctl status
-sh-4.4# systemctl list-jobs
After fixing potential issues the emergency shell can be terminated to resume the boot:
switch_root:/root# systemctl stop dracut-emergency.service
switch_root:/root# Connection to 203.0.113.23 closed by remote host.
Connection to 203.0.113.23 closed.
Alternatively, one can send a signal to the emergency service, e.g.
with systemctl kill ... or systemctl kill --signal=... ....
Copy the 46sshd subdirectory to the Dracut module directory:
# cp -ri 46sshd /usr/lib/dracut/modules.d
Alternatively, you can install the latest stable version from the dracut-sshd copr repository.
Either way, once present under /usr/lib/dracut/modules.d it's
enabled, by default.
With an sshd that lacks systemd support (e.g. under Gentoo), one has to adjust the systemd service file:
# echo 'Skip this sed on Fedora/RHEL/CentOS/Debian/Ubuntu/...!'
# sed -e 's/^Type=notify/Type=simple/' \
-e 's@^\(ExecStart=/usr/sbin/sshd\) -D@\1 -e -D@' \
-i \
/usr/lib/dracut/modules.d/46sshd/sshd.service
Dracut-sshd includes the first available ssh authorized keys file of the following list into the initramfs:
- /root/.ssh/dracut_authorized_keys
- /etc/dracut-sshd/authorized_keys
- /root/.ssh/authorized_keys
Note that on some distributions such as Fedora
Silverblue your only option is to create a keys file
under /etc/dracut-sshd as /root isn't accessible during
dracut runtime.
Of course, our initramfs image needs network support. The simplest way to achieve this is to include networkd. To install the networkd dracut module:
# dnf install -y dracut-network
When installing from copr, dracut-network is automatically
installed as dependency.
Create a non-NetworkManager network config, e.g. via Networkd:
$ cat /etc/systemd/network/20-wired.network
[Match]
Name=e*
[Network]
DHCP=ipv4
Adjust the Name=, if necessary.
Note that the dracut networkd module doesn't include the system's network configuration files by default and note that the module isn't enabled, by default, either. Thus, you have to configure Dracut for networkd (cf. the install_items and add_dracutmodules directives). Example:
# cat /etc/dracut.conf.d/90-networkd.conf
install_items+=" /etc/systemd/network/20-wired.network "
add_dracutmodules+=" systemd-networkd "
Alternatively, early boot network connectivity can be configured by other means (i.e. kernel parameters, see below). However, the author of this README strongly recommends to use Networkd instead of NetworkManager on servers and server-like systems.
If the above example is sufficient you just need to copy the
example configuration files from the example/ subdirectory:
# cp example/20-wired.network /etc/systemd/network
# cp example/90-networkd.conf /etc/dracut.conf.d
Finally regenerate the initramfs:
# dracut -f -v
Note that Ubuntu's dracut defaults to an initramfs filename that is incompatible with Ubuntu's grub default initrd settings ... m( Thus, on Ubuntu one has to explicitly specify the initramfs filename like this:
# dracut -f -v /boot/initrd.img-$(uname -r)
Verify that this sshd module is included. Either via inspecting the verbose
output or via lsinitrd, e.g.:
# lsinitrd | grep 'authorized\|bin/sshd\|network/20'
-rw-r--r-- 1 root root 119 Jul 17 15:08 etc/systemd/network/20-wired.network
-rw------- 1 root root 99 Jul 17 17:04 root/.ssh/authorized_keys
-rwxr-xr-x 1 root root 876328 Jul 17 17:04 usr/sbin/sshd
Finally, reboot.
The space overhead of the Dracut sshd module is negligible:
enabled modules initramfs size
--------------------------------------
vanilla -network -ifcfg 16 MiB
+systemd-networkd 17 MiB
+systemd-networkd +sshd 19 MiB
+network +ifcfg 21 MiB
+network +ifcfg +sshd 21 MiB
+network +ifcfg +sshd 22 MiB
+systemd-networkd
(all numbers from a Fedora 28 system, measuring the compressed initramfs size)
Technically, the systemd-networkd Dracut module is
sufficient for establishing network connectivity. It even
includes the ip command. Since the network Dracut module is
included by default (under CentOS 7/Fedora 27/28) via the
ifcfg
Dracut module, it may make sense to explicitly exclude it when
building the initramfs on a system where networkd is available,
e.g. via
dracut -f -v --omit ifcfg
as this saves a few megabytes.
Since the initramfs is actually loaded into a tmpfs that is freed during switch-root it doesn't really pay off to safe a few mega-/kilobytes in the initramfs. A few KiBs could be saved via switching from OpenSSH's sshd to something like Dropbear, but such an alternative sshd server is likely less well audited for security issues and supports less features (e.g. ssh-ed25519 public key authentication was only added as late as 2020, and, as of 2021, there are some interoperability issues and ed25519-sk keys aren't supported).
Last but not least, in times where even embedded systems feature hundreds of megabytes RAM, temporarily occupying a few extra KiBs/MiBs before switch root has no dramatic impact.
By default, this module includes the system's
/etc/ssh/ssh_host_*_key private host keys into the
initramfs. Note that this doesn't decrease the security
in comparison with a system whose root filesystem is unencrypted:
- the generated initramfs image under /boot is only readable by the root user
- if an attacker is able to access the /boot/initramfs file (e.g. by booting the machine from a Live stick) then she is also able to access all host keys on a unencrypted root filesystem
That said, if /etc/ssh/dracut_ssh_host_*_key{,.pub}
files are present then those are included, instead.
As always, it depends on your threat model, whether it makes sense to use an extra host key for the initramfs or not. Using an extra key may complicate the life of an attacker who is able to read out the initramfs content but is unable to change it and thus the attacker has to wait for the next SSH connection to the initramfs before being able to perform a MITM attack. On the other hand, when the attacker is able to change to initramfs image then an extra key doesn't provide more security than using the system's host key as the attacker can intercept the entered password, anyway.
If your primary threat model is an attacker who gets access to decommissioned but still readable hard-disks, then the system's host key in the initramfs image provides no value to the attacker given that the root filesystem is fully encrypted (and that the host key isn't reused in the replacement system).
With recent Fedora versions (e.g. Fedora 28) a cryptsetup
password prompt shouldn't timeout. If it does then it's a
regression (cf. Bug 868421). Even if it times out
and Dracut drops into the emergency shell then remotely
connecting to it should still work with this module. In such
situations systemd-tty-ask-password-agent should
still work. See also Section 'Example: Emergency Shell' on how
to resume the boot process then.
A simple way to trigger the timeout is to enter the wrong
password 3 times when unlocking a LUKS volume. Under Fedora 28,
the timeout is then 2 minutes or so long, i.e. the emergency
shell is then started after 2 minutes, by default, even without
explicitly adding rd.shell to the kernel command line. One can
recover from such a situation with e.g.:
# systemctl restart 'systemd-cryptsetup@*'
Another example for the emergency shell getting started is that
a device that is necessary for mounting the root filesystem
simply isn't attached - or the UUIDs specified on the kernel
command line don't match. After inspecting the situation with
systemctl status ..., journalctl -e, etc. one can
regenerate some config and restart the appropriate services in a
similar fashion.
An alternative to the networkd configuration is to configure network via additional Dracut command line parameters.
This requires the activation of the network dracut module, e.g.:
# cat /etc/dracut.conf.d/90-network.conf
add_dracutmodules+=" network "
On systems without networkd (e.g. CentOS 7/RHEL 8) this is the only way to enable network connectivity in early userspace. For example, the following parameters enable DHCP on all network interfaces in early userspace:
rd.neednet=1 ip=dhcp
They need to be appended to GRUB_CMDLINE_LINUX= in
/etc/default/grub and to be effective the Grub config then
needs to be regenerated:
# grub2-editenv - unset kernelopts
# grub2-mkconfig -o /etc/grub2.cfg
# grub2-mkconfig -o /etc/grub2-efi.cfg
Note that on distributions like CentOS 7/Fedora 27/28 there is
also the old-school ifcfg network scripts system under
/etc/sysconfig/network-scripts that can be used instead of
NetworkManager. It can be launched via the auto-generated
network service that calls the old sysv init.d script. However,
the network Dracut module doesn't include neither this service
nor the network-scripts configuration (it includes some of the
scripts but the Dracut modules auto-generate the configuration
during early userspace boot based on the kernel
command line/detected hardware). With CentOS 7/Fedora 27/28 the
default network configuration (in late userspace) uses
NetworkManager which only uses the ifcfg-* files under
/etc/sysconfig/network-scripts.
The grub2-editenv call is only necessary on systems (such as
RHEL 8) where the kernel parameters are stored in /etc/grubenv
instead of in each menu entry (either in the main grub2.cfg or
under /boot/loader/entries if the system follows the boot
loader specification (bls)).
A Baseboard Management Controller (BMC) or some kind of remote KVM device can help with early boot issues, however:
- not all remote machines even have a BMC
- the BMC often is quite tedious to use and buggy
- the BMC often contains low quality proprietary software that is never updated and likely contains many security issues
- in some hosting environments a KVM must be manually attached and is charged at an hourly rate. That means you end up paying the remote hands for attaching the KVM, plus possibly an extra charge if you need it outside business hours and the hourly rate.
Thus, as a general rule, one wants to avoid a BMC/KVM as much as possible.
-
How to make the early boot sshd listen on a non-standard port?
A: If you really want to do that you can provide a
/etc/sysconfig/dracut-sshdthat definesSSHD_OPTS(see also). -
Why does sshd hangs during early-boot when running dracut-sshd inside a virtual machine (VM)?
A: Most likely the VM guest is short of entropy and thus sshd blocks during startup (without logging a warning) for an indefinite amount of time. Possible up to the systemd service restart timeout. Directing some of the VM host's entropy into the VM guest fixes this issue (cf. these comments for examples of how to do this).
-
Why do I get
Permission denied (publickey)although the same authorized key works after the system is booted?A: This can be caused by a root account that is locked with
!instead of*. In that case it's enough to change the lock method (or set a password) and regenerate the initramfs. Background: On some systems Dracut also includes/etc/shadowwhich is then used by sshd. In early userspace, there is no PAM, thus sshd uses built-in code for shadow handling. In contrast to usual PAM configuration (which is used by late userspace sshd, by default), sshd itself differentiates between*and!as invalid password field tokens. Meaning that only*allows public key authentication while!blocks any login (see also). -
How do I make it work on Ubuntu 20.04?
A: There are some pitfalls on Ubuntu. Firstly, dracut isn't installed by default (fix:
apt install dracut-core dracut-network). Secondly, dracut isn't a first class citizen on Ubuntu (i.e. it's only included in the universe repository, not in the main repository). As a result, the default dracut initramfs filename doesn't match what Ubuntu uses in its Grub configuration. Thus, you have to explicitly specify the right one (i.e./boot/initrd.img-$(uname -r)) in thedracutandlsinitrdcommands. -
How do I debug dracut-sshd issues in the early boot environment?
A: You start by dropping into the dracut emergency shell and looking at the journal and status of the involved services. For example, via
systemctl status sshd.service,journalctl -u sshdetc. You drop into the emergency shell by addingrd.break(and possiblyrd.shell) to kernel parameter command-line. Of course, you need some kind of console access when doing such debugging. Using a virtual machine usually is sufficient to reproduce issues which simplifies things.
There is the unmaintained (since 2019 or earlier) dracut-crypt-ssh module which aimed to provide SSH access for remotely unlocking an encrypted LUKS volume. Main differences to dracut-sshd:
- uses Dropbear instead of OpenSSH sshd (cf. the Space Overhead Section for the implications)
- doesn't use systemd for starting/stopping the Dropbear daemon
- generates a new set of host keys, by default
- listens on a non-standard port for ssh, by default
- arguably more complex than dracut-sshd - certainly more lines of code and some options
- comes with an unlock command that is superfluous in the
presence of
systemd-tty-ask-password-agent- and it's kind of dangerous to use, e.g. when the password prompt times out the password is echoed to the console
In 2017, a dracut-crypt-ssh pull request added support for optionally using OpenSSH's sshd instead of Dropbear, without changing the other differences. It was closed without being merged in 2021.
There are also some other dracut modules that use Dropbear:
mk-fg/dracut-crypt-sshd which was marked
deprecated in 2016 in favour of the above dracut-crypt-ssh. It
uses Dropbear and some console hacks instead of
systemd-tty-ask-password-agent.
mdcurtis/dracut-earlyssh is a fork
mk-fg/dracut-crypt-sshd. The main difference is that it also
suppports RHEL 6 (which features a quite different version of
dracut). xenoson/dracut-earlyssh is a fork of
mdcurtis/dracut-earlyssh. It has RHEL 6 support removed and some
questionable helpers removed. It creates a systemd unit file for
Dropbear although it still explicitly starts/stops it via hook
files instead of making use of the systemd dependency features.
The ArchWiki dm-crypt page lists two initramfs hooks for remote access. Both don't use Dracut nor systemd, though. Also, they use Dropbear and Tinyssh as ssh daemon.
Clevis, an automatic decryption framework, has some LUKS unlocking and Dracut support. Looking at its documentation, when it comes to automatic LUKS unlocking, the LUKS passphrase is stored encrypted in the LUKS header. Clevis then decrypts it using an external service/hardware (e.g. a Tang server or a TPM module).
Similar to Clevis, Mandos also implements a framework for unattended LUKS unlocking. Unlike Clevis, it primarily targets Debian and doesn't support TPM. That means for unlocking the Mandos client fetches the asymmetrically encrypted LUKS password from a Mandos server.
With version 248 (i.e. available since early 2021 or so), systemd integrated some automatic LUKS2 volume unlocking features. Similar to Clevis it supports TPM2 modules. In addition, it also supports smart cards and FIDO2/hmac-secret devices. At least some of those FIDO2 devices seem to support non-interactive HMAC computation and thus allow to auto-unlock LUKS volumes as long as the enrolled FIDO2 device is connected.
If your threat model goes beyond what is described in the Host Keys Section, you have to look into authenticated boot and disk encryption.
Although enterprise motherboard and server vendors often integrate unpleasant BMCs (cf. the Hardware Alternatives Section), a hardware solution for remote access to early boot doesn't have to be awful. For example, there is the open and DIY Pi-KVM project which looks quite promising.
Related Fedora ticket: Bug 524727 - Dracut + encrypted root + networking (2009)
- Fedora Silverblue 33
- Fedora 27 to 37
- CentOS 7, 8
- CentOS Stream 9 (by a contributor)
- RHEL 8 beta 1
- Rocky Linux 8.8, 9 (by a contributor)
- Gentoo (by a contributor)
- SUSE (by a contributor)
- Arch (by a contributor)
- Ubuntu 20.04 LTS
- Debian 12 (by a contributor)