This Dracut module (dracut-sshd) integrates the OpenSSH sshd into the initramfs. It allows for remote unlocking of a fully encrypted root filesystem and remote access to the Dracut emergency shell (i.e. early userspace).
2018, Georg Sauthoff [email protected], GPLv3+
After booting a Fedora system with encrypted root filesystem (i.e. a filesystem on a LUKS volume to be opened with cryptsetup) the Dracut initramfs blocks at the password prompt. With dracut-sshd enabled remote unlocking is then as simple as:
$ ssh headless.example.org
-sh-4.4# systemd-tty-ask-password-agent
Please enter passphrase for disk luks-123-cafe! *********
Please enter passphrase for disk luks-124-cafe! *********
-sh-4.4# Connection to 203.0.113.23 closed by remote host.
Connection to 203.0.113.23 closed.
That means under normal circumstances the completion of all password prompts automatically resumes the boot process.
The command systemd-tty-ask-password-agent --list prints an overview
over all pending password prompts.
The start of the Dracut emergency shell can be
requested via adding rd.break to the kernel command line, but
it also happens when Dracut is unable to mount the root
filesystem or other grave issues. In such cases the emergency
shell blocks the boot process. Without remote access the machine
is quite dead then.
Example session:
$ ssh headless.example.org
-sh-4.4# export TERM=vt220
-sh-4.4# export SYSTEMD=FRMXK
-sh-4.4# export LC_ALL=C
-sh-4.4# less /run/initramfs/rdsosreport.txt
-sh-4.4# journalctl -e
-sh-4.4# systemctl status
-sh-4.4# systemctl list-jobs
After fixing potential issues the emergency shell can be terminated to resume the boot:
switch_root:/root# systemctl stop dracut-emergency.service
switch_root:/root# Connection to 203.0.113.23 closed by remote host.
Connection to 203.0.113.23 closed.
Alternatively, one can send a signal to the emergency service, e.g.
with systemctl kill ... or systemctl kill --signal=... ....
Copy the 46sshd subdirectory to the Dracut module directory:
# cp -ri 46sshd /usr/lib/dracut/modules.d
With a sshd that lacks systemd support (e.g. under Gentoo), one has to adjust the systemd service file:
# echo 'Skip this sed on Fedora/RHEL/CentOS/Debian/Ubuntu/...!'
# sed -e 's/^Type=notify/Type=simple/' \
-e 's@^\(ExecStart=/usr/sbin/sshd\) -D@\1 -e -D@' \
-i \
/usr/lib/dracut/modules.d/46sshd/sshd.service
It's enabled, by default - unless the Dracut network module is missing. Thus:
# dnf install -y dracut-network
(this package also contains the systemd-networkd
Dracut module)
Make sure that /root/.ssh/authorized_keys contains the right
keys, as it's included in the initramfs:
# cat /root/.ssh/authorized_keys
Create a non-NetworkManager network config, e.g. via Networkd:
$ cat /etc/systemd/network/20-wired.network
[Match]
Name=en*
[Network]
DHCP=ipv4
Adjust the Name=. Even if the system doesn't have networkd
enabled (as it - say - uses NetworkManager), the sshd dracut
module unconditionally includes the networkd config files for
establishing network connectivity. However, the author of this
README strongly recommends to use Networkd instead of NetworkManager
on servers and server-like systems.
Finally regenerate the initramfs:
# dracut -f -v
Verify that this sshd module is included. Either via inspecting the verbose
output or via lsinitrd. Reboot.
The space overhead of the Dracut sshd module is negligible:
enabled modules initramfs size
--------------------------------------
vanilla -network -ifcfg 16 MiB
+systemd-networkd 17 MiB
+systemd-networkd +sshd 19 MiB
+network +ifcfg 21 MiB
+network +ifcfg +sshd 21 MiB
+network +ifcfg +sshd 22 MiB
+systemd-networkd
(all numbers from a Fedora 28 system, measuring the compressed initramfs size)
Technically, the systemd-networkd Dracut module is
sufficient for establishing network connectivity. It even
includes the ip command. Since the network Dracut module is
included by default (under CentOS 7/Fedora 27/28) via the ifcfg
Dracut module, it may make sense to explicitly exclude it when
building the initramfs on a system where networkd is available,
e.g. via
dracut -f -v --omit ifcfg
as this saves a few megabytes.
Since the initramfs is actually loaded into a tmpfs that is freed during switch-root it doesn't really pay off to safe a few mega-/kilobytes in the initramfs. A few KiBs could be saved via switching from OpenSSH's sshd to something like Dropbear, but such an alternative sshd server is likely less well audited for security issues and supports less features (e.g. as of 2018 Dropbear doesn't support public authentication with ssh-ed25519 keys).
Last but not least, in times where even embedded systems feature hundreds of megabytes RAM, temporarily occupying a few extra KiBs/MiBs before switch root has no dramatic impact.
By default, this module includes the system's
/etc/ssh/ssh_host_ed25519_key private host key into the
initramfs. Note that this doesn't decrease the security
in comparison with a system whose root filesystem is unencrypted:
- the generated initramfs image under /boot is only readable by the root user
- if an attacker is able to access the /boot/initramfs file (e.g. by booting the machine from a Live stick) then she is also able to access all host keys on a unencrypted root filesystem
That said, if the /etc/ssh/dracut_ssh_host_ed25519_key{,.pub}
files are present then those are included, instead.
As always, it depends on your threat model, whether it makes sense to use an extra host key for the initramfs or not. Using an extra key may complicate the life of an attacker who is able to read out the initramfs content but is unable to change it and thus the attacker has to wait for the next SSH connection to the initramfs before being able to perform a MITM attack. On the other hand, when the attacker is able to change to initramfs image then an extra key doesn't provide more security than using the system's host key as the attacker can intercept the entered password, anyway.
If your primary threat model is an attacker who gets access to decommissioned but still readable hard-disks, then the system's host key in the initramfs image provides no value to the attacker given that the root filesystem is fully encrypted (and that the host key isn't reused in the replacement system).
With recent Fedora versions (e.g. Fedora 28) a cryptsetup
password prompt shouldn't timeout. If it does then it's a
regression (cf. Bug 868421). Even if it times out
and Dracut drops into the emergency shell then remotely
connecting to it should still work with this module. In such
situations systemd-tty-ask-password-agent should
still work. See also Section 'Example: Emergency Shell' on how
to resume the boot process then.
A simple way to trigger the timeout is to enter the wrong
password 3 times when unlocking a LUKS volume. Under Fedora 28,
the timeout is then 2 minutes or so long, i.e. the emergency
shell is then started after 2 minutes, by default, even without
explicitly adding rd.shell to the kernel command line. One can
recover from such a situation with e.g.:
# systemctl restart 'systemd-cryptsetup@*'
Another example for the emergency shell getting started is that
a device that is necessary for mounting the root filesystem
simply isn't attached - or the UUIDs specified on the kernel
command line don't match. After inspecting the situation with
systemctl status ..., journalctl -e, etc. one can
regenerate some config and restart the appropriate services in a
similar fashion.
An alternative to the networkd configuration is to configure network via additional Dracut command line parameters.
On systems without networkd (e.g. CentOS 7) this is the only way to enable network connectivity in early userspace. For example, the following parameters enable DHCP on all network interfaces in early userspace:
rd.neednet=1 ip=dhcp
They need to be appended to GRUB_CMDLINE_LINUX= in
/etc/sysconfig/grub and to be effective the Grub config then
needs to be regenerated:
# grub2-mkconfig -o /etc/grub2.cfg
# grub2-mkconfig -o /etc/grub2-efi.cfg
Note that on distributions like CentOS 7/Fedora 27/28 there is
also the old-school [ifcfg][ifcg] network scripts system under
/etc/sysconfig/network-scripts that can be used instead of
NetworkManager. It can be launched via the auto-generated
network service that calls the old sysv init.d script. However,
the network Dracut module doesn't include neither this service
nor the network-scripts configuration (it includes some of the
scripts but the Dracut modules auto-generate the configuration
during early userspace boot based on the kernel
command line/detected hardware). With CentOS 7/Fedora 27/28 the
default network configuration (in late userspace) uses
NetworkManager which only uses the ifcfg-* files under
/etc/sysconfig/network-scripts.
A Baseboard Management Controller (BMC) or some kind of remote KVM device can help with early boot issues, however:
- not all remote machines even have a BMC
- the BMC often is quite tedious to use and buggy
- the BMC often contains low quality proprietary software that is never updated and likely contains many security issues
- in some hosting environments a KVM must be manually attached and is charged at an hourly rate. That means you end up paying the remote hands for attaching the KVM, plus possibly an extra charge if you need it outside business hours and the hourly rate.
Thus, as a general rule, one wants to avoid a BMC/KVM as much as possible.
There is dracut-crypt-ssh module which aims to provide SSH access for remotely unlocking an encrypted LUKS volume. Main differences to dracut-sshd:
- uses Dropbear instead of OpenSSH sshd (cf. the Space Overhead Section for the implications)
- doesn't use systemd for starting/stopping the Dropbear daemon
- generates a new set of host keys, by default
- listens on a non-standard port for ssh, by default
- arguably more complex than dracut-sshd - certainly more lines of code and some options
- comes with an unlock command that is superfluous in the
presence of
systemd-tty-ask-password-agent- and it's kind of dangerous to use, e.g. when the password prompt times out the password is echoed to the console
A dracut-crypt-ssh pull request (open as of 2018) for optionally using OpenSSH's sshd instead of Dropbear. Main differences to dracut-sshd:
- doesn't use systemd for starting/stopping the sshd daemon
- generates a new set of host keys, by default
- listens on a non-standard port for ssh, by default
- arguably more complex than dracut-sshd - certainly more lines of code and some options
- unlock command still present
- pull-request evolved via additional commits without cleanup rebases
There is mk-fg/dracut-crypt-sshd which was marked
deprecated in 2016 in favour of the above dracut-crypt-ssh. It
uses Dropbear and some console hacks instead of
systemd-tty-ask-password-agent.
mdcurtis/dracut-earlyssh is a fork
mk-fg/dracut-crypt-sshd. The main difference is that it also
suppports RHEL 6 (which features a quite different version of
dracut). xenoson/dracut-earlyssh is a fork of
mdcurtis/dracut-earlyssh. It has RHEL 6 support removed and some
questionable helpers removed. It creates a systemd unit file for
Dropbear although it still explicitly starts/stops it via hook
files instead of making use of the systemd dependency features.
Clevis, an automatic decryption framework, has some LUKS unlocking and Dracut support. Looking at its documentation, when it comes to automatic LUKS unlocking, the LUKS passphrase is stored encrypted in the LUKS header. Clevis then decrypts it using an external service/hardware (e.g. a TPM module).
The ArchWiki dm-crypt page lists two initramfs hooks for remote access. Both don't use Dracut nor systemd, though. Also, they use Dropbear and Tinyssh as ssh daemon.
Related ticket: Bug 524727 - Dracut + encrypted root + networking (2009)
- Fedora 28
- Fedora 27
- CentOS 7