Self-hosted GitLab runner for CI/CD pipelines

Introduction

In this post I will continue explaining how to get more functionality out of GitLab. One way to automate homelab is to use CI/CD pipelines, and GitLab provides good tools for that. To utilize these CI/CD pipelines we need a GitLab Runner - a program that can be run on a server or even your own computer. However for security reasons, it's best to create a new virtual machine specifically for this task. This post will go into detail how to prepare VM with GitLab Runner.

OS used: Debian 12
Software used: GitLab Community Edition 17.11.1

Initial requirements

Working GitLab instance. To create GitLab instance you can use instruction from post Installing GitLab as a self-hosted git repository.

GitLab Runner VM requirements

In the beginning create new virtual machine, for example using this tutorial Fast initialization of Debian VM using Ansible.
I suggest allocating following resources to this new virtual machine: 20GB disk size, 2 CPU, and 2GB RAM. Increase these resources if pipelines run slowly or when disk space becomes low.

GitLab Runner installation

GitLab Runner needs following components to run properly:

Executor - execution environment for commands run in pipelines (can be Docker, Kubernetes, VirtualBox etc.)
GitLab Runner binary - actual program that communicates with GitLab instance and executes configured pipelines

After installing those components you need to:

Register GitLab Runner with your GitLab instance

Docker executor installation

GitLab Runner needs an executor for running pipelines. List of all executors can be found here: Selecting the executor.

We will use Docker executor. To install Docker run following command (taken from the Official Docker Docs how to Install Docker Engine on Debian):

$ sudo apt-get update && \
  sudo apt-get install ca-certificates curl && \
  sudo install -m 0755 -d /etc/apt/keyrings && \
  sudo curl -fsSL https://download.docker.com/linux/debian/gpg -o /etc/apt/keyrings/docker.asc && \
  sudo chmod a+r /etc/apt/keyrings/docker.asc && \
  echo \
    "deb [arch=$(dpkg --print-architecture) signed-by=/etc/apt/keyrings/docker.asc] https://download.docker.com/linux/debian \
    $(. /etc/os-release && echo "$VERSION_CODENAME") stable" | \
    sudo tee /etc/apt/sources.list.d/docker.list > /dev/null && \
  sudo apt-get update && \
  sudo apt-get install docker-ce docker-ce-cli containerd.io docker-buildx-plugin docker-compose-plugin

Docker and host firewall

Currently on Linux, Docker manage firewall rules using iptables and dynamically creates rules that are needed at the system start. Debian by default uses nftables instead of iptables, but have compatibility layer enabled, so iptablescommands are still valid and Docker works correctly.

Docker creates its own firewall rules (such as forwarding traffic between containers), so it's better to disable your custom firewall rules in the system, and instead enable firewall for the whole virtual machine in Proxmox.

Tip

If you want your Docker instance protected with firewall enable firewall for virtual machine in Proxmox. Then open following ports using firewall:

TCP 22 - SSH access to virtual machine
TCP 443 - HTTPS for optional applications run in containers

Runner installation

Runner can be installed for self-hosted purposes on a bare Linux system, in Docker or in Kubernetes. Here I will show how to install it on a Linux system using steps from official documentation Install GitLab Runner using the official GitLab repositories.

Add GitLab repository:

$ curl -L "https://packages.gitlab.com/install/repositories/runner/gitlab-runner/script.deb.sh" | sudo bash

Install GitLab Runner:

$ sudo apt install gitlab-runner

Runner registration

After installing GitLab Runner you need to register it with your GitLab instance:

Login to GitLab as Admin user to register GitLab Runner for all repositories
Click Admin (left down corner) > CI/CD > Runners > click Create instance runner
- Select Run untagged jobs - allow this Runner to run all jobs if you don't plan to use tags
- Enter description in Runner description - for example host name of your VM
- Click Create runner
- Register runner > Operating systems > select Linux
- Perform actions mentioned in Steps on page Register runner:
  - Step 1 - copy and paste command to your VM to register Runner, run this command with sudo to register Runner in system-mode (otherwise it will register in user-mode and user-mode requires you to manually start builds processing).
  - Step 2 - command from Step 1 will prompt you for the following data (if you press Enter, default value shown below the prompt will be used):
    - Enter the GitLab instance URL (for example, https://gitlab.com/) - URL address of your GitLab instance
    - Enter a name for the runner. This is stored only in the local config.toml file - name for the Runner
    - Enter an executor: docker, docker-windows, docker-autoscaler, instance, shell, virtualbox, docker+machine, kubernetes, custom, ssh, parallels - write docker when prompted for choosing executor
    - Enter the default Docker image (for example, ruby:2.7) - default docker image that will be used when you write GitLab CI/CD and don't specify image to use - I entered alpine:latest
  - Step 3 - verify that the Runner is available to pick up jobs by running command sudo gitlab-runner run
  - Click View runners - you will see screen with a list of registered Runners - if everything went ok your Runner should have status Online

In a future post I will show how to create a simple CI/CD pipline in GitLab using GitLab Runner.

Automatic upgrades

If you are using unattended-upgrades for system upgrades you can enable automatic upgrades for Docker and GitLab Runner by editing file /etc/apt/apt.conf.d/50unattended-upgrades:

$ sudo vim /etc/apt/apt.conf.d/50unattended-upgrades

/etc/apt/apt.conf.d/50unattended-upgrades

(...)
Unattended-Upgrade::Origins-Pattern {
        "origin=Debian,codename=${distro_codename}-updates";
        "origin=Debian,codename=${distro_codename},label=Debian";
        "origin=Debian,codename=${distro_codename},label=Debian-Security";
        "origin=Debian,codename=${distro_codename}-security,label=Debian-Security";

        // --- Add these lines ---
        // Docker repository
        "origin=Docker,suite=${distro_codename},component=stable";
        // GitLab Runner repository
        "origin=packages.gitlab.com/runner/gitlab-runner";
        // -----------------------

        (...)
};
(...)

Test if config is correct by running command:

$ sudo unattended-upgrades --dry-run --debug

If you have following line in the command output, then something is wrong with the configuration (probably a typo):

Marking not allowed <apt_pkg.PackageFile object: filename:'/var/lib/apt/lists/download.docker.com_linux_debian_dists_bookworm_stable_binary-amd64_Packages'  a=bookworm,c=stable,v=,o=Docker,l=Docker CE arch='amd64' site='download.docker.com' IndexType='Debian Package Index' Size=230910 ID:12>

Pruning Docker images

When you run your pipelines, Docker pulls images to the VM’s disk, creates containers, creates volumes, and also stores data in build cache for multi-stage builds.

Check space used by Docker with following command:

$ sudo docker system df

To reclaim space you should run following command (script installed with GitLab Runner):

$ sudo clear-docker-cache

To automate this process you should run above command with cron regularly (e.g. once a week).

Install cron (Debian cloud image not always has cron package):

$ sudo apt install cron

Configure cron job:

$ sudo crontab -u root -e

(...)
# Run command at 05:00 every Sunday
0 5 * * 0 clear-docker-cache

If the above command doesn't prune all the Docker resources you would like (because clear-docker-cache only removes resources tagged by GitLab Runner), you should run command:

$ sudo docker system prune --volumes -a -f

--volumes - prune anonymous volumes
-a - remove all unused images not just dangling ones
-f - do not prompt for confirmation

Troubleshooting

If you have problems with GitLab Runner you can view logs of GitLab Runner:

$ sudo journalctl --unit=gitlab-runner.service -n 100 --no-pager

--unit - show log entries for specific systemd unit
-n - show the most recent events and limit the number of lines
--no-pager - disable paging of output entries