Fast initialization of Debian VM using Ansible

Introduction

Virtual machine deployment in Proxmox can be achieved faster by creating virtual machine template with desired settings and then cloning this machine. This new virtual machine requires some changes after cloning so to speed-up this process Ansible can be used. Example how to do this will be shown in this article.

Software used: Proxmox 8, Debian 12 cloud image, Ansible 2.14.3

Creating Debian template machine

In this article we will use Debian cloud image named nocloud to quickly create virtual machine template. This image has root user without password and most of necessary packages. After creating this virtual machine there will be minimal work to do to tune it to your preferences.

Get Debian cloud image

Create ZFS dataset on Proxmox for downloading cloud images.

# zfs create datapool/cloud-images -o recordsize=1M

datapool - your ZFS zpool name
cloud-images - enter name for your dataset
-o recordsize - ZFS dataset record size, for large files like backups and ISO images 1M can be a good choice

Download latest cloud image in raw format from Debian Official Cloud Images web page (for Debian 12 it will be in bookworm path).

In this instruction nocloud image will be used.

wget -P /path/to/save/ https://URL/image.raw

# wget -P /datapool/cloud-images/ https://cloud.debian.org/images/cloud/bookworm/<date-time>/debian-12-nocloud-amd64-<date-time>.raw

-P - folder where the file will be downloaded

Note

Debian web page has following cloud image types:

generic: Should run in any environment using cloud-init, for e.g. OpenStack, DigitalOcean and also on bare metal.
genericcloud: Similar to generic. Should run in any virtualised environment. Is smaller than generic by excluding drivers for physical hardware.
nocloud: Mostly useful for testing the build process itself. Doesn't have cloud-init installed, but instead allows root login without a password.
azure: Optimized for the Microsoft Azure environment
ec2: Optimized for the Amazon EC2

Create virtual machine

Create new virtual machine in Proxmox without assigning a virtual disk.

Click button Create VM:
- Tab General:
  - Node - select your Proxmox host
  - VM ID - enter ID for your virtual machine (in Proxmox it is a number)
  - Name - enter name for your virtual machine, it will be your machine hostname
- Tab OS:
  - select Do not use any media
  - in other fields leave default values
- Tab System:
  - Qemu Agent - check this field
  - in other fields leave default values
- Tab Disks:
  - delete default disk scsci0
- Tab CPU:
  - Cores - select number of cores you want to assign to virtual machine
  - Type - select host if you want maximum performance
  - in other fields leave default values
- Tab Memory:
  - Memory (MiB) - enter amount of RAM you want to assign to virtual machine
- Tab Network:
  - Bridge - select network bridge for use by this virtual machine
  - VLAN Tag - enter VLAN number if you separate virtual machines from other parts of your network using VLANs, if you don't use VLANs leave this field empty
  - in other fields leave default values

Import Debian cloud image to Proxmox virtual machine

Import downloaded cloud image file using command:

qm disk import <vmID> <image_file> <Proxmox_storage>

# qm disk import 100 /datapool/cloud-images/debian-12-nocloud-amd64-xxx.raw local-zfs

100 - created virtual machine ID in Proxomx
local-zfs - storage in Proxmox to which import downloaded cloud image as virtual machine disk

Next go to created virtual machine:

Datacenter > your Proxmox host > virtual machine > Hardware > select Unused Disk 0 > click Edit > check Discard > click Add
Datacenter > your Proxmox host > virtual machine > Options > select Boot Order > click Edit > check scsci0 and drag and drop it as first disk to boot

Discard - discard unused blocks on mounted filesystem. This can save space on host when you delete data from virtual machine, because image can then be automatically shrunk.

You can also remove CD/DVD drive as it won't be needed by this machine.

Datacenter > your Proxmox host > virtual machine > Hardware > select CD/DVD drive > click Remove

Configuration of Debian virtual machine

Plan for initial configuration

Plan for initial configuration is as follows:

Create new user and give it sudo permission
Install SSH server and check IP address of virtual machine
Clean root bash history
Reboot system and login as user

After above actions you will have SSH access to machine so it will be easy to perform next actions by copying and pasting commands from this article to terminal:

Login as user using SSH
Make color prompt for user
Disable root account, this system will be for operation with sudo only
Set timezone
Set grub timeout to 0 seconds
Install qemu-guest-agent, cloud-guest-utils and other packages
Resize disk
Create swapfile
Create firewall rules
Enable Fail2Ban for SSH (optional)
Configure unattended-upgrades
Clean user bash history

Start your virtual machine in Proxmox. If everything was configured correctly you should see login screen in virtual machine Console:

Datacenter > your Proxmox host > virtual machine > Console

Enter root as login and you will be logged in to the system.

Create new user and give it sudo permission

Create new user, for purpose of this tutorial we will name user debian. You will be asked to set password for this user. Password can be temporary for template virtual machine and you will set final password after cloning:

adduser <username>

# adduser debian

Adding user `debian' ...
Adding new group `debian' (1000) ...
Adding new user `debian' (1000) with group `debian (1000)' ...
New password: <here you will be prompted for new password>
Retype new password:
passwd: password updated successfully
Changing the user information for debian
Enter the new value, or press ENTER for the default
    Full Name []: 
    Room Number []: 
    Work Phone []: 
    Home Phone []: 
    Other []: 
Is the information correct? [Y/n] y
Adding new user `debian` to supplemental / extra groups `users` ...
Adding user `debian` to group `users` ...

Add user to administrative groups:

usermod -a -G <group> <user>

# usermod -a -G adm,sudo debian

-a - append user to supplementary groups, works only with -G option
-G - list of supplementary groups

Install SSH server and check IP address of virtual machine

Install SSH package:

# apt update && apt upgrade
# apt install openssh-server

Check IP address of virtual machine:

# ip a

Clean root bash history

Clean root bash history and after that reboot your system:

# history -c && history -w
# reboot

-c - clear history list in current shell session
-w - write current history list to the history file, overwriting the history file's contents

Login as user using SSH from your computer:

ssh -o PubkeyAuthentication=no [email protected] or username@ip-address

$ ssh -o PubkeyAuthentication=no debian@<vm-ip-address>

-o PubkeyAuthentication=no - use password login (SSH can otherwise try using other keys for this host and after 5 tries it will fail because the key isn't set yet)

Make color prompt for user

To make color prompt for user edit .bashrc file in user home folder and uncomment #force_color_prompt line:

$ vim ~/.bashrc

~/.bashrc

(...)
(uncomment following line)
force_color_prompt=yes
(...)

Logout and login again as user and now you have color prompt.

Disable root account

As you have user with sudo permissions you don't need root account. To disable root account run command:

$ sudo passwd -l root

Set timezone

You can check timezone by using command:

$ timedatectl

Debian nocloud image defaults to UTC timezone. To list available timezones use:

$ timedatectl list-timezones

In order to change timezone use command:

$ sudo timedatectl set-timezone America/Los_Angeles
or if you want to choose timezone from available options
$ sudo dpkg-reconfigure tzdata

Set grub timeout to 0 seconds

At boot time of virtual machine grub is showed and there is 5s timer before system starts. If you want to disable it change GRUB_TIMEOUT to 0 in /etc/default/grub.d/15_timeout.cfg file and after that run command update-grub:

$ sudo vim /etc/default/grub.d/15_timeout.cfg

/etc/default/grub.d/15_timeout.cfg

GRUB_TIMEOUT=0

$ sudo update-grub

Install packages

Install packages for easier management of virtual machine in Proxmox:

$ sudo apt install qemu-guest-agent cloud-guest-utils

qemu-guest-agent - is a helper daemon that is used to exchange information between the host and guest. More information can be found in QEMU documentation. Installing this package allows to see IP address in Proxmox in virtual machine's Summary tab.
cloud-guest-utils - contains growpart command which makes very easy to resize partition after disk resize in Proxmox.

I also like to have following tools on my virtual machines:

$ sudo apt install tree htop

tree - list files and folders in tree structure
htop - process viewer

After above actions clean apt:

$ sudo apt clean && sudo apt autoremove

apt clean - cleans apt packages cache /var/cache/apt/archives/
apt autoremove - removes orphaned packages which are not longer needed

Resize disk

If you would like to install additional packages or use swapfile you will need to resize disk because by default nocloud image is 2GB in size and after previous actions there isn't much space left.

Stop your virutal machine.
In Proxmox resize new disk:
- Datacenter > your Proxmox host > virtual machine > Hardware > select Hard Disk (scsi0) > click Disk Action > select Resize > Size Increment (GiB) - enter number in gigabytes by how much to increase the disk
Start your virtual machine and check partition sizes and file system free space with commands:

$ lsblk ; \
  df -h

Commands for growing partition and resizing ext4 filesystem in virutal machine are:

$ sudo growpart /dev/sda 1 && \
  sudo resize2fs /dev/sda1

Check if your root partition resized correctly:

$ lsblk ; \
  df -h

Create swapfile

Debian cloud image has no swapfile. It is left up to the user to create swapfile with appropriate size. Swapfile is useful if virtual machine runs out of free RAM. In such situation instead of applications and services crashing system will use swapfile. I typically create small swapfile of size 512MB and when virtual machine is starting to use swapfile I allocate more RAM to virtual machine.

Actions to create swapfile are as follows and can be written down as one chain of commands:

Create empty file in / folder of size 512MB (1MiB * 512) named swapfile
Set 600 permissions for this file
Make swap filesystem in created swapfile
Add swapfile mouting at the beggining of system boot by adding line /swapfile swap swap defaults 0 0 to /etc/fstab
Set swappiness level to minimize system usage of swapfile

$ sudo dd if=/dev/zero of=/swapfile bs=1MiB count=512 && \
  sudo chmod -v 0600 /swapfile && \
  sudo mkswap /swapfile && \
  echo /swapfile swap swap defaults 0 0 | sudo tee -a /etc/fstab && \
  sudo bash -c "echo 'vm.swappiness = 1' > /etc/sysctl.d/swappiness.conf"

Reboot virtual machine:

$ sudo reboot

Check if swapfile is preset:

$ sudo swapon --show

Create firewall rules

Install firewall to drop unwanted traffic. Since Debian 10 nftables is default firewall. Of course you can still use iptables or ufw but in this tutorial nftables will be used.

You can find more info under these links:

ArchLinux Wiki - nftables
nftables.org Wiki - Quick reference-nftables in 10 minutes
nftables.org Wiki
Examples available after installing nftables are in folder /usr/share/doc/nftables/examples/

Install nftables package:

$ sudo apt install nftables

Enable firewall:

$ sudo systemctl enable nftables.service

In order to set firewall rules edit file /etc/nftables.conf and replace its content with code provided below. For this template virtual machine following firewall rules will be set:

Accept localhost traffic
Drop invalid connections
Accept traffic originated from this virtual machine
Accept ICMP (for PING responses)
Accept traffic at port 22 (SSH)
Allow ICMPv6 packets
Drop any other traffic

$ sudo vim /etc/nftables.conf

/etc/nftables.conf

#!/usr/sbin/nft -f

flush ruleset

table inet filter {
        chain input {
                type filter hook input priority filter; policy drop;

                # Accept any localhost traffic
                iif lo accept

                # Drop and count invalid packets
                ct state invalid counter drop

                # Accept traffic originated from us
                ct state { established,related } accept

                # Accept ICMP PING for IPv4
                icmp type echo-request accept

                # Accept traffic on ports
                tcp dport { 22 } ct state new accept

                # ICMPv6 packets which must not be dropped, see https://tools.ietf.org/html/rfc4890#section-4.4.1
                meta nfproto ipv6 icmpv6 type { destination-unreachable, packet-too-big, time-exceeded, parameter-problem,
                                                echo-reply, echo-request,
                                                nd-router-solicit, nd-router-advert, nd-neighbor-solicit, nd-neighbor-advert,
                                                148, 149 } accept
                ip6 saddr fe80::/10 icmpv6 type { 130, 131, 132, 143, 151, 152, 153 } accept

                # Drop and count any other traffic
                counter drop
        }

        chain forward {
                type filter hook forward priority filter; policy drop;

                # Drop and count everything forwarded to us. We do not forward. That is routers job.
                counter
        }

        chain output {
                type filter hook output priority filter; policy accept;

                # Accept every outbound connection
        }
}

Reload rules after changing /etc/nftables.conf with command:

$ sudo systemctl restart nftables.service

In order to see firewall ruleset and counters statistics use command:

$ sudo nft list ruleset

Enable Fail2Ban for SSH

Optional action to increase security is to enable fail2ban. Tutorial how to do it can be found in this post Fail2Ban simple config for SSH.

Configure unattended-upgrades

Debian nocloud image by default has package unattended-upgrades. This package performs automatic updates and upgrades every day. If you want to configure automatic reboot (for example after kernel update) and specify time of day when the reboot is performed edit file /etc/apt/apt.conf.d/50unattended-upgrades.

Lines that you should uncomment and change:

$ sudo vim /etc/apt/apt.conf.d/50unattended-upgrades

/etc/apt/apt.conf.d/50unattended-upgrades

(...)
Unattended-Upgrade::Automatic-Reboot "true";
(...)
Unattended-Upgrade::Automatic-Reboot-WithUsers "true";
(...)
Unattended-Upgrade::Automatic-Reboot-Time "02:00";
(...)

Clean user bash history

Clean user bash history:

$ history -c && history -w

Poweroff your system and it is ready for cloning:

$ sudo poweroff

Cloning Debian VM

Plan for cloned virtual machine

When cloning virtual machine following actions should be taken:

In Proxmox:
- Resize disk in Proxmox
In virtual machine:
- Resize partition and filesystem in virtual machine
- Regenerate SSH host keys
- Regenerate machine-id
- Change hostname
- Add SSH public key for remote access
- Disable SSH access with password and only allow SSH key login

All of above actions in virtual machine will be automated with Ansible script.

Post configuration setting:

In virtual machine:
- Reboot system
- Change user password

Clone virtual machine

Clone virtual machine using following action in Proxmox:

Datacenter > your Proxmox host > press right mouse button on virtual machine > select Clone
- Name - set new name for cloned virtual machine
- in other fields leave default values

Resize disk

Debian nocloud image has 2GB disk with EXT4 filesystem. If you need more space you can resize disk but after that you will need to grow partition and resize filesystem. This will be handled by Ansible script.

Datacenter > your Proxmox host > virtual machine > Hardware > select Hard Disk (scsi0) > click Disk Action > select Resize > Size Increment (GiB) - input value in GB by which increase disk size

Regenerate SSH host keys

Cloned virtual machine has the same SSH host keys as the base virtual machine. Host keys are used for verification of SSH connection from client side (key fingerprint). If the same host SSH keys are used on another server you can't distinguish these servers by SSH connection.

Regenerate machine-id

When cloning virtual machine machine-id must be recreated as explained in Debian Wiki MachineId. machine-id is used in Debian to obtain IP address form DHCP server, so cloning virtual machine without changing machine-id can result in two machines on the network receiving the same IP address from DHCP server.

Change hostname

Hostname also needs to be manually changed to preferred one. Hostname needs to be changed in following files:

/etc/hosts
/etc/hostname

Add SSH key for remote access

In order to connect to cloned virtual machine using SSH keys you need to:

Generate SSH keys
Add entry to SSH config file on computer from which you connect to virtual machine

Instruction how to do it is shown in post SSH keys with KeePassXC - Creation of SSH keys.

After enabling SSH connection using keys it is time to harden SSH access by disabling user password login and root login. Instruction how it's done is shown here SSH keys with KeePassXC - Basic hardening of remote host SSH server.

Automate actions with Ansible

Ansible is an open-source automation tool that simplifies provisioning and configuration of systems and applications across multiple servers.
In order to use Ansible you need to install it on computer from which you administer virtual machine:

$ sudo apt install ansible

Then you need to create two files to put code for ansible:

sh file with bash script from witch you will invoke ansible script
yml (YAML) file with ansible script

As example create ansible folder in your home directory with following structure and files:

~/ansible/
  ├── init_vm.sh
  └── /playbooks/
       └── init_vm.yml

In init_vm.sh change hostname to target hostname of your cloned virtual machine.

$ vim ~/ansible/init_vm.sh

~/ansible/init_vm.sh

#!/bin/bash

#--- change following values ---
new_hostname=vm  # enter new hostname
id_address=192.168.1.10  # enter ip address of your cloned virtual machine
default_username=debian  # this is user of your template machine
#-------------------------------

ansible-playbook -i "${id_address}," ./playbooks/init_${new_hostname}.yml --user ${default_username} --ask-pass --ssh-common-args='-o PubkeyAuthentication=no -o StrictHostKeyChecking=no' --ask-become-pass

# -i, --inventory - specify host to connect to
# --user - connect as name of this user
# --ask-pass - ask for connection password
# --ssh-common-args='-o PubkeyAuthentication=no' - use password for connecting using SSH
# --ssh-common-args='-o StrictHostKeyChecking=no' - bypass host SSH fingerprint verification
# --ask-become-pass - ask for privilege escalation password (sudo)

In init_vm.yml change values in vars: section.

$ vim ~/ansible/playbooks/init_vm.yml

~/ansible/playbooks/init_vm.yml

---
- name: Initialize VM after cloning from template
  hosts: all
  become: true
  become_method: sudo
  vars:
    #--- change following values ---
    hostname_new: hostname  # new hostname
    domain_name: domainname  # new domainname
    default_username: debian  # this is user of your template machine
    public_key_path: /home/yourusername/.ssh/id_ed25519_myname.pub  # enter path to public key on computer from which you connect to virtual machine
    #-------------------------------

  tasks:

    - name: Specify SSH host key files to delete
      ansible.builtin.find:
        paths: /etc/ssh
        patterns: "ssh_host_*"
        recurse: true
        file_type: file
      register: collected_files

    - name: Delete SSH host key files
      ansible.builtin.file:
        path: "{{ item.path }}"
        state: absent
      with_items: "{{ collected_files.files }}"

    - name: Regenerate SSH host key files
      ansible.builtin.command: ssh-keygen -A
      register: ssh_host_keys

    - name: Grow partition to resized disk
      ansible.builtin.command: growpart /dev/sda 1
      register: growpart
      failed_when: growpart.rc == 2  # failed when return code equals 2
      # growpart return codes: 0 = partition grow success; 1 = partition could not be grown because lack of space; 2 = error

    - name: Remove machine-id files
      ansible.builtin.file:
        path: "{{ item }}"
        state: absent
      with_items:
        - /etc/machine-id
        - /var/lib/dbus/machine-id

    - name: Generate /etc/machine-id
      ansible.builtin.command: dbus-uuidgen --ensure=/etc/machine-id

    - name: Recreate symbolic link to /var/lib/dbus/machine-id
      ansible.builtin.file:
        src: /etc/machine-id
        dest: /var/lib/dbus/machine-id
        owner: root
        group: root
        state: link

    - name: Resize EXT4 filesystem to new partition size
      ansible.builtin.command: /usr/sbin/resize2fs /dev/sda1
      register: resize_fs
      changed_when: resize_fs.rc == 0

    - name: Change hostname
      ansible.builtin.hostname:
        name: "{{ hostname_new }}"
        use: "debian"  # use Debian OS strategy to update the hostname

    - name: Make sure an entry in /etc/hosts exists
      ansible.builtin.lineinfile:
        path: /etc/hosts
        regexp: "^127.0.1.1"
        line: "127.0.1.1 {{ hostname_new }}.{{ domain_name }} {{ hostname_new }}"
        state: present

    - name: Copy SSH public key to host
      ansible.posix.authorized_key:
        user: "{{ default_username }}"
        state: present
        key: "{{ lookup('file', public_key_path) }}"

    - name: Create empty SSH hardening config file with appropriate permissions
      ansible.builtin.file:
        path: /etc/ssh/sshd_config.d/no-root_no-pass.conf
        state: touch
        mode: '0644'

    - name: Add SSH hardening rules to config file
      ansible.builtin.copy:
        content: |
          PermitRootLogin no
          PasswordAuthentication no
        dest: /etc/ssh/sshd_config.d/no-root_no-pass.conf

    - name: Print return information from tasks
      ansible.builtin.debug:
        msg:
          - "{{ ssh_host_keys.stdout_lines }}"
          - "{{ growpart.stdout_lines }}"
          - "{{ resize_fs.stdout_lines }}"

Now run followng command to invoke ansible script:

$ sh ~/ansible/init_vm.sh
SSH password:
BECOME password[defaults to SSH password]:

Running that command you will be asked for virtual machine user password to establish SSH connection and virtual machine user password for sudo privileges.

Reboot system

After running Ansible script it is time to reboot your virtual machine for all changes to take effect.

Change user password

Last part is to change user password to more secure. Login as user using SSH and invoke command:

$ passwd

Fast initialization of Debian VM using Ansible

Introduction

Creating Debian template machine

Get Debian cloud image

Create virtual machine

Import Debian cloud image to Proxmox virtual machine

Configuration of Debian virtual machine

Plan for initial configuration

Login

Create new user and give it sudo permission

Install SSH server and check IP address of virtual machine

Clean root bash history

Login as user using SSH

Make color prompt for user

Disable root account

Set timezone

Set grub timeout to 0 seconds

Install packages

Resize disk

Create swapfile

Create firewall rules

Enable Fail2Ban for SSH

Configure unattended-upgrades

Clean user bash history

Cloning Debian VM

Plan for cloned virtual machine

Clone virtual machine

Resize disk

Regenerate SSH host keys

Regenerate machine-id

Change hostname

Add SSH key for remote access

Disable SSH access with password and only allow SSH key login

Automate actions with Ansible

Reboot system

Change user password