Full disk encryption with Proxmox and remote decryption through SSH

Introduction

This article shows how to setup encrypted Proxmox hypervisor with remote decryption through SSH. It's useful if you want to protect your data and all virtual machines in case of physical theft of your homelab server. Keep in mind that in order to decrypt the system (for example after system reboot) you need to log in through SSH and enter decryption password.

OS used: Debian 12
Software used: Proxmox 8, KeePassXC 2.7.4

Setup overview

In order to have encrypted Proxmox you need to install encrypted Debian first and after that install Proxmox.

Instructions are for following setup:

• computer with 1 disk and 1 network connection
• disk will be partitioned with one partition for system and other for data
• data partition will be using ZFS filesysetm
• Debian installation using Debian netinst.iso from USB
• connected keyboard and monitor during installation and initial configuration, after that access will be through SSH

Debian installation

Here are the steps to install Debian using its installer.

Locales

Choose your language, location, locales and keymap (keyboard language).

Network

Use DHCP for network. We will set static IP after installation. It is recommended to have static IP for hypervisor in order do connect to it when device with DHCP server fails.

Host name

• set hostname for your server
• set domain if you use one in your network environment

User creation

Set root password and user password. Passwords can be simple and easy to type as they will be changed later to much harder after enabling SSH access.

Disk partitioning

Choose Manual partitioning and create following partitions:

• /efi - size: 100MB - filesystem: EFI System Partition
• /boot - size: 1GB - filesystem: ext4
• / - size: 32GB - filesystem: physical volume for encryption with default option aes256-xts-plain64
  • after creating encrypted partition choose Configure encrypted volumes
  • select Create encrypted volumes and select just created encrypted partition
  • select Finish
  • set password for partition. Simple password can initially be used, as it will be changed later to much harder when there is SSH access.
  • set mountpoint to / for just created Encrypted volume (it is shown on top of disk list) - filesystem: ext4
• select Finish partitioning and write changes to disk

Info

• data partition will be created later after installing Proxmox, because than you will be able to format it as ZFS filesystem. Standard Debian doesn't include ZFS packages.
• Don't create swap partition, later we will create swapfile which is a more flexible solution as you can change swapfile size without worrying about partition layout.

Packages to install

During packages installation choose only:

• SSH server
• standard system utilities

Initial Debian configuration

After installation login as root using keyboard and external monitor and perform following actions.

Install base packages

Install vim text editor if you use it.

# apt install vim

Set static IP

• Set static IP by changing /etc/network/interfaces file.

# vim /etc/network/interfaces

/etc/network/interfaces

# The primary network interface
allow-hotplug eno1
iface eno1 inet static
        address 192.168.1.5/24  # Enter IP address of your host
        gateway 192.168.1.1  # Enter IP address of your host gateway

• Add this static IP to /etc/hosts file by changing line with 127.0.1.1 address. Debian installer creates 127.0.1.1 address for systems without a permanent IP as a workaround for some software as explained here in Debian docs.

# vim /etc/hosts

/etc/hosts

127.0.0.1       localhost
192.168.1.5     hostname.domainname hostname  # Change IP from 127.0.1.1 to your host static IP set in /etc/network/interfaces

• Reboot computer after changing files.

Creation of SSH keys

Create SSH keys and set remote access using instructions from this post: SSH keys with KeePassXC.

Recommended packages

With remote SSH access to the server, it's easy to perform additional configuration. I recommend installing following packages:

# apt install htop tree lm-sensors

• htop - interactive system monitor
• tree - file and directory listing in tree view
• lm-sensors - hardware health monitoring, run by using command sensors which display information about CPU temperature, fan speed etc.

Red color prompt for root

To set red color prompt for root just backup file /root/.bashrc and overwrite original with /etc/skel/.bashrc:

# cp /root/.bashrc /root/.bashrc_backup && \
  cp /etc/skel/.bashrc /root/

then change:

• PS1='${debian_chroot:+($debian_chroot)}\[\033[01;32m\]\u@\h\[\033[00m\]:\[\033[01;34m\]\w\[\033[00m\]\$ '

to:

• PS1='${debian_chroot:+($debian_chroot)}\[\033[01;31m\]\u@\h\[\033[00m\]:\[\033[01;34m\]\w\[\033[00m\]\$ '

(change is from 32m to 31m - [\033[01;32m\] to [\033[01;31m\])

# vim /root/.bashrc

/root/.bashrc

(...)
PS1='${debian_chroot:+($debian_chroot)}\[\033[01;31m\]\u@\h\[\033[00m\]:\[\033[01;34m\]\w\[\033[00m\]\$ '
(...)

Dropbear installation for disk decryption through SSH

In order to decrypt disk through SSH during system startup you'll need to use dropbear. It is a lightweight SSH server that can start and run with initial kernel before disk decryption. Install dropbear with following command:

# apt install dropbear-initramfs

Creating dropbear SSH keys

You create SSH keys for dropbear in similar way you created SSH keys for this remote host. Once generated, also add them to KeePassXC.

• Create keys on your computer:

$ ssh-keygen -a 100 -t ed25519 -f ~/.ssh/id_ed25519_myname_dropbear -C "$(whoami)@$(hostname)@$(date +'%Y-%m-%d')"
(enter password for private key)

$ chmod 600 ~/.ssh/id_ed25519_myname_dropbear.pub

• Copy public key to remote host:

$ scp ~/.ssh/id_ed25519_myname_dropbear.pub hostname:id_ed25519_myname_dropbear.pub

hostname - hostname that was set in ~/.ssh/config when creating SSH keys for remote host

• Login to remote host, append public key to dropbear authorized_keys file, after that remove public key:

# cat /home/username/id_ed25519_myname_dropbear.pub >> /etc/dropbear/initramfs/authorized_keys
# rm /home/username/id_ed25519_myname_dropbear.pub

• Add dropbear keys to KeePassXC, delete dropbear private key from disk, re-open KeePassXC database for key to be loaded by ssh-agent.

Hardening dropbear

Harden the dropbear server by adding following parameters to line DROPBEAR_OPTIONS=:

# vim /etc/dropbear/initramfs/dropbear.conf

/etc/dropbear/initramfs/dropbear.conf

DROPBEAR_OPTIONS="-I 60 -j -k -s"

• -I 60 - disconnect SSH session after 60 seconds with no activity (idle timeout)
• -j - disables local ssh port forwarding
• -k - disables remote ssh port forwarding
• -s - disables password logins (only key login is available)

Set dropbear IP address

Dropbear should have different IP address than host. Otherwise there can be a collision of SSH keys on the same IP address.
Change dropbear IP address by adding IP=... line below DEVICE=:

# vim /etc/initramfs-tools/initramfs.conf

/etc/initramfs-tools/initramfs.conf

DEVICE=
IP=192.168.1.4:::255.255.255.0::eno1:off

• eno1 - name of you network device that can be shown using ip a command and is configured in /etc/network/interfaces
• IP=<host-ip>::<gateway-ip>:<netmask>:<hostname>:<device>:<autoconf>:<dns-server-0-ip>:<dns-server-1-ip>:<ntp0-ip> taken from Linux kernel documentation

Update initramfs

For the changes in dropbear configuration to take effect, you must update initramfs:

# update-initramfs -u

Create dropbear SSH config

On your computer add configuration to SSH config file. Use your own names, below is example config. User should be root:

$ vim ~/.ssh/config

~/.ssh/config

(...)
Host hostname-dropbear
HostName ip-address or hostname.domainname
User root
IdentitiesOnly yes
IdentityFile ~/.ssh/id_ed25519_myname_dropbear.pub

Test dropbear

• Reboot remote host
• Login to remote host using command:

$ ssh hostname-dropbear

• Run command to decrypt disk:

# cryptroot-unlock

• Enter disk password

Changing passwords to more secure

Now you have SSH access to encrypted remote host, so it's time to change initial passwords to more secure ones. Create entries in KeePassXC and generate passwords. After that change passwords:

• Login to remote host

$ ssh hostname

• Change user password (you need to be logged as user) with command:

$ passwd

• Change root password (you need to be logged as root - login as user and then change to root with su -) with command:

# passwd

• Change disk password:

# cryptsetup luksChangeKey /dev/sdXY

• /dev/sdXY - XY is letter and number of encrypted disk, can be checked using lsblk command

Creating swapfile

Now lets create swapfile for scenarios when your server may run out of RAM.

# dd if=/dev/zero of=/swapfile bs=1GiB count=2 && \
  chmod -v 0600 /swapfile && \
  mkswap /swapfile

Above command creates and enables swapfile with size 2GB (bs=1GB * count=2).

Add swapfile to fstab:

# vim /etc/fstab

/etc/fstab

/swapfile   swap    swap    defaults    0   0

Reboot system. After reboot verify if swapfile is working.

# swapon --show
# free -h

If you want less aggressive use of swapfile you can set lower swappiness value (default is 60):

# bash -c "echo 'vm.swappiness = 1' > /etc/sysctl.d/swappiness.conf"

Reboot the system afterwards.

Proxmox installation

Source

Following instructions are based on official Proxmox documentation.

Setting Proxmox repository

Add Proxmox repository to Debian:

# echo "deb [arch=amd64] http://download.proxmox.com/debian/pve bookworm pve-no-subscription" > /etc/apt/sources.list.d/pve-install-repo.list ; \
  wget https://enterprise.proxmox.com/debian/proxmox-release-bookworm.gpg -O /etc/apt/trusted.gpg.d/proxmox-release-bookworm.gpg

and verify:

# sha512sum /etc/apt/trusted.gpg.d/proxmox-release-bookworm.gpg
7da6fe34168adc6e479327ba517796d4702fa2f8b4f0a9833f5ea6e6b48f6507a6da403a274fe201595edc86a84463d50383d07f64bdde2e3658108db7d6dc87 /etc/apt/trusted.gpg.d/proxmox-release-bookworm.gpg

Installing Proxmox

Update system and install Proxmox kernel. After that reboot system.

# apt update && apt full-upgrade && \
  apt install proxmox-default-kernel
# reboot

Install Proxmox with additional packages such as postfix for mail notifications.

# apt install proxmox-ve postfix mailutils postfix-pcre open-iscsi

Note

During postfix configuration choose:

• General mail configuration type: Satellite system
• System mail name: hostname.domainname (enter your server hostname and domainname)
• rest of options leave default
• actual configuration will be done later in configuration file

Remove old Debian kernel and os-prober package (os-prober package scans all partitions of your host to create dual-boot GRUB entries but this can lead to problems with partitions that are assigned to virtual machines). After that reboot system.

# apt remove linux-image-amd64 'linux-image-6.1*' && \
  update-grub && \
  apt remove os-prober
# reboot

Postfix configuration

For postfix to send emails you need to configure it with your email account. Following example works for Gmail.

• Create file with password for application generated on Google account:

Google Account > Security > 2-Step Verification > App passwords > generate password

# vim /etc/postfix/sasl_passwd

/etc/postfix/sasl_passwd

smtp.gmail.com [email protected]:yourpassword

[email protected] - your Gmail address which you will be using for sending emails from Proxmox
yourpassword - password for application generated on Google account

• Put your password to postfix database and harden password file permissions:

# postmap hash:/etc/postfix/sasl_passwd && \
  chmod 600 /etc/postfix/sasl_passwd*

• Edit file which changes email From field (otherwise email will be From: root) and input your host name. This way if you receive email you will know which host has sent it:

# vim /etc/postfix/smtp_header_checks

/etc/postfix/smtp_header_checks

/^From:.*/ REPLACE From: yourhostname <[email protected]>

yourhostname - your Proxmox host name
[email protected] - this name doesn't matter, when postfix send email this name will be replaced with previously configured in /etc/postfix/sasl_passwd

• Put edited file to postfix database:

# postmap hash:/etc/postfix/smtp_header_checks

• Edit main configuration file to complete postfix setup and append following lines at the end of the file:

# vim /etc/postfix/main.cf

/etc/postfix/main.cf

(...)
# google mail configuration
relayhost = smtp.gmail.com:587
smtp_use_tls = yes
smtp_sasl_auth_enable = yes
smtp_sasl_security_options =
smtp_sasl_password_maps = hash:/etc/postfix/sasl_passwd
smtp_tls_CAfile = /etc/ssl/certs/Entrust_Root_Certification_Authority.pem
smtp_tls_session_cache_database = btree:/var/lib/postfix/smtp_tls_session_cache
smtp_tls_session_cache_timeout = 3600s
smtp_header_checks = pcre:/etc/postfix/smtp_header_checks

• Restart postfix:

# systemctl restart postfix

• Send test email:

# echo "test message" | mail -s "test subject" [email protected]

[email protected] - here enter email to which you want to send test message

Tip

If you didn't receive email check your sending email address in /etc/postfix/sasl_passwd file and also be sure that there is no space or other character after password. If there will be need to correct data in /etc/postfix/sasl_passwd remember to run postmap hash:... command after that change.

View logs using journalctl -e command to diagnose any issues that may have occurred.
-e - show last 1000 entries from systemd journal

Proxmox configuration

Log in to Proxmox

Log in to Proxmox using your web browser and enter address:

https://AAA.BBB.CCC.DDD:8006

AAA.BBB.CCC.DDD - your Proxmox host IP address

First time you need to log in as root with option PAM Authentication.

Setting up firewall

Proxmox has hidden anti lockout rules for ports 22 (SSH), 8006 (Proxmox Web GUI), 3128 (SPICE Proxy) so you don't need to open them but I prefer to set rules for them in case something in the future changes.

Enter Datacenter > Firewall and add following basic rules to block all input traffic except Proxmox web interface, SSH, ping:

Type	Action	Macro	Interface	Protocol	Source	S.Port	Destination	D.Port	Log level	Comment
in	ACCEPT			tcp				8006	nolog	Proxmox Web UI
in	ACCEPT			tcp				22	nolog	SSH
in	ACCEPT			icmp					nolog	ICMP (eg. PING)

• Type - direction of network traffic
• Action - action to perform by firewall rule
• Interface - network interface which is used on Proxmox host, leave empty for all interfaces
• Protocol - set protocol to which firewall rule applies
• Source - if you set 192.168.1.0/24 then only address from subnet 192.168.1.0/24 will be able to connect to Proxmox (for example you will be able to connect when your address is 192.168.1.6/24)
• D.Port (Destination Port) - set port to which firewall rule applies
• Comment - your comment for firewall rule

IMPORTANT: Firewall isn't enabled by default so the rules will apply after enabling firewall. Check the rules to be sure you won't cut off yourself from device and enable firewall in:

• Datacenter > Firewall > Options > Firewall > set Yes

Tip

If you block access to Proxmox with firewall you can disable firewall using local access to Proxmox host with keyboard by editing file /etc/pve/firewall/cluster.fw. Change enable: 1 option to enable: 0.

Note

If you would like to block some ports or traffic to host remember that Proxmox firewall priorities for hosts are as follows:
Custom node rules > Custom datacenter rules > Anti lockout rules > Default incoming/outgoing/forward policies

Package repositories

Disable enterprise repository if you don't have active enterprise subscription and plan on using only free no-subscirption repository:

• Datacenter > your Proxmox host > Updates > Repositories > select pve-enterprise > click button Disable

Network configuration for virtual machines

Create network bridge for use in virtual machines (only network bridges can be chosen for virtual machines and not physical network cards). If you have several network cards in Proxmox server it is good practice to use one for managing Proxmox and other for virtual machines and separate those networks for example with VLANs created on your home router/firewall device. That way if virtual machine is compromised it can't scan/access Proxmox hypervisor.

To create network bridge:

• Datacenter > your Proxmox host > System > Network > click button Create > choose Linux Bridge:
  • IPv4/CIDR, Gateway (IPv4), IPv6/CIDR, Gateway (IPv6) - if you plan on using your existing DHCP server (for example your home router) don't enter any IP address. Virtal machines will get addresses from DHCP.
  • VLAN aware - check this option if you plan on having some of virtual machines using this network bridge in different VLANs and those VLANs are available on this network interface (when creating VM you will be able to set which VLAN ID to use).
  • Bridge ports - enter network interface which is used on Proxmox host for virtual machines.
  • Comment - your comment for this network bridge

Tip

If you will have error when creating bridge iface xxx - ip address can't be set on interface if bridged in vmbr0 (500) then you need to delete IP address (column CIDR) and gateway (column Gateway) from this interface and add this data to this new bridge. After changes remember to click Apply Configuration.

Set email address for notifications

You need to enter your email address to receive notifications from Proxmox.

• Datacenter > Notifications > select mail-to-root in Notification Targets > click Modify > Additional Recipients(s) enter here your email address
• After that click Test to send test email.

Encrypted ZFS partition

Create encrypted ZFS partition

• Create new partition from free space on disk:

# cfdisk

Select Free space > New > press Enter to accept partition size > select Write to write changes to disk > select Quit to exit from cfdisk

• Show your disk partitions:

# lsblk -f

-f - show filesystems on disk partitions

• Encrypt newly created partition

# cryptsetup -y -v luksFormat /dev/sdXY

sdXY - enter partition name where X is disk letter and Y is partition number
-v, --verbose - show verbose information
-y, --verify-passphrase - enter password, then enter it again to verify it

• Create ZFS filesystem

# cryptsetup luksOpen /dev/sdXY crypt-data
# zpool create -o ashift=13 datapool crypt-data ; \
  zfs set xattr=sa datapool ; \
  zfs set compression=zstd datapool ; \
  zfs set atime=off datapool ; \
  zfs set recordsize=32K datapool

crypt-data - your name for decrypted partition
datapool - your ZFS zpool name

• Check if your ZFS zpool was created correctly and your attributes were set accordingly:

# zpool list
# zfs get all

Tip

Great resources on the ZFS filesystem and its other attributes can be found here:

• Ars Technica - ZFS 101—Understanding ZFS storage and performance
• JRS Systems: the blog - ZFS tuning cheat sheet

Automatic decryption of ZFS partition on system start

• Create folder containg file with your password in root folder with permission to access only by root:

# mkdir -v -m 700 /root/luks

-v - verbose information from mkdir command
-m 700 - set permission 700 for folder

# vim /root/luks/crypt-data

/root/luks/crypt-data

enter you password in this file

# chmod 600 /root/luks/crypt-data
# perl -pi -e 'chomp if eof' /root/luks/crypt-data

perl -pi -e 'chomp if eof' <keyfile> - removes possible end of line symbol \n if it was added when you were writing password in file, otherwise it can be treated as part of the password

• Add decryption command to crypttab file:

# vim /etc/crypttab

/etc/crypttab

(...)
crypt-data /dev/disk/by-id/<your_disk_name>-partX /root/luks/crypt-data luks

crypt-data - your name for decrypted partition
/dev/disk/by-id/<your_disk_name>-partX - you can check your disk name with partition by running command:ls -la /dev/disk/by-id/
/root/... - path to file with passphrase
luks - encryption type

Note

You can use UUID instead of /dev/disk/by-id/. UUID can be found by running command: ls -la /dev/disk/by-uuid/.

• Restart your Proxmox machine and check if ZFS zpool is present:

# zpool list

Add ZFS partition to Proxmox

Now you need to add newly created ZFS partition to Proxmox.

Add ZFS zvol for virtual machines

• Datacenter > Storage > click button Add > ZFS:
  • ID - enter name for this storage (for example: vm)
  • ZFS Pool - select your ZFS zpool
  • Content - Disk image, Container
  • Block Size - 64k - recommended value for zvol (ZFS block device for virtual machines)

Add ZFS dataset for backups and ISO images

• Run command:

# zfs create datapool/backups-iso -o recordsize=1M

datapool - ZFS zpool name
backups-iso - enter name for your dataset
-o recordsize - ZFS dataset record size, for large files like backups and ISO images 1M can be a good choice

• Add ZFS dataset in Proxmox:
  • Datacenter > Storage > click button Add > Directory
    • ID - enter name for this storage (for example: backups-iso)
    • Directory - enter created dataset path /datapool/backups-iso
    • Content - ISO image, Container template, VZDump backup file

Add ZFS dataset for snippets

• Run command:

# zfs create datapool/snippets -o recordsize=32K

datapool - ZFS zpool name
snippets - enter name for your dataset
-o recordsize - ZFS dataset record size

• Add ZFS dataset in Proxmox:
  • Datacenter > Storage > click button Add > Directory
    • ID - enter name for this storage (for example: snippets)
    • Directory - enter created dataset path /datapool/snippets
    • Content - Snippets

Finish

The only thing left now is to create virtual machines, but that is a topic for another blog post.