Skip to content

Simple CI/CD pipeline in GitLab using GitLab Runner

Introduction

This blog post is continuation of previous one: Self-hosted GitLab runner for CI/CD pipelines. Here I will show how to create a simple deployment using GitLab and GitLab Runner.

OS used: Debian 12
Software used: GitLab Community Edition 17.11.1, Ansible 2.14.3

Source

Introduction

This example will use following elements:

  • Your computer - lets call it dev-machine
  • Host with webserver - lets call it webserver
  • Host with GitLab - lets call it gitlab

Plan for deploying with GitLab CI/CD:

  • Create folder with simple webpage using index.html file on dev-machine
  • Create new Git repository using this folder
  • Provide access for GitLab CI/CD to webserver using SSH:
    • Create SSH keys (public and private) to access webserver
    • Create new user on webserver and add SSH public key to its authorized keys
  • Deploy index.html using GitLab CI/CD to webserver:
    • Add private SSH key to GitLab variable
    • Write simple gitlab-ci.yml to deploy webpage to webserver

I will assume that you have virtual machine with webserver to which you can deploy HTML files, and I won't cover its creation in this post. The path where HTML files will be kept on the webserver will be /var/www/html/.

Create simple webpage

Create simple webpage on your computer. Create folder /simple-webpage/ in your home folder and add index.html file in /simple-webpage/html/ subfolder.

$ mkdir ~/simple-webpage/html && \
  vim ~/simple-webpage/html/index.html
~/simple-webpage/html/index.html
<!DOCTYPE html>
<html lang="en">

<head>
    <meta charset="UTF-8">
    <meta name="viewport" content="width=device-width, initial-scale=1.0">
    <title>Simple WebPage</title>
</head>

<body>
    <section>
        <p>
            Lorem ipsum dolor sit, amet consectetur adipisicing elit. Deleniti aliquam similique cupiditate voluptatibus
            delectus exercitationem, eum magni iure porro quasi consectetur, quaerat dolor esse odio perferendis, ipsa
            harum quibusdam repudiandae?
        </p>
    </section>
</body>

</html>

Create Git repository using existing folder

I will assume that you have configured access to your GitLab and can push/pull data from it using SSH.
Use following commands to add existing folder to GitLab:

$ cd /path/to/folder/ && \
  git init && \
  git add . && \
  git commit -m "Initial commit" && \
  git push --set-upstream git@<gitlab-domainname>:<username>/<reponame>.git main && \
  git remote add origin git@<gitlab-domainname>:<username>/<reponame>.git && \
  git push --set-upstream origin main
  • /path/to/folder/ - folder from which you want to create a new repository, in this example it will be ~/simple-webpage/
  • <gitlab-domainname> - domain name of your GitLab instance
  • <username> - user with access to GitLab, on whose account repository will be created
  • <reponame> - name of repository
  • git push --set-upstream origin main - this line was needed for me so VS Code could start tracking changes on the remote repository (it is similar when you select Publish Branch option in VS Code)

Create SSH keys to access webserver host

Now create SSH keys (public and private) that you will need to allow GitLab CI/CD access to the folder on webserver host. Use following command:

$ key_name=mykey && \
  ssh-keygen -t ed25519 -f ~/.ssh/id_ed25519_${key_name} -C "gitlab-deploy@$(date +'%Y-%m-%d')" -N "" && \
  chmod 600 ~/.ssh/id_ed25519_${key_name}.pub
  • key_name=mykey - enter a name for your key instead of mykey
  • -t - key algorithm type
  • -f - location and name of keys to be generated (private and public keys will be generated)
  • -C - comment for key, in above command it will be gitlab-deploy@<current-date>
  • -N "" - don't prompt to set passphrase for ssh private key because this key will be used with automated GitLab CI/CD pipeline

Create new user on webserver host

As a good security practice let's create a new user on the webserver for deployment purposes. This user will only be able to access folder where you put index.html. To make this action more reusable and suited for infrastructure as a code approach let's use Ansible.

Create folder ansible:

$ mkdir ~/ansible

and there create following empty files:

~/ansible
├── ansible.cfg
├── change_owner_dir.yml
├── create_user.yml
└── inventory.yml

ansible.cfg
ansible.cfg file is read when using ansible-playbook command. In this configuration file we set to use Ansible inventory file from the same folder as ansible.cfg and always ask to escalate privileges when running ansible-playbook <playbook_name.yml>. Privilege escalation is needed because all actions that will be performed on the server, like creating user, need to be run with sudo. Otherwise you would always need to run:
ansible-playbook <playbook_name.yml> --ask-become-pass instead of ansible-playbook <playbook_name.yml>.

~/ansible/ansible.cfg
[defaults]
inventory = ./inventory.yml

[privilege_escalation]
become_ask_pass = True

inventory.yml
In this file define access values to your webserver.

~/ansible/inventory.yml
servers:
  hosts:
    vm:
      ansible_host: webserver_domain_name  # enter domain name or ip address of your webserver
      ansible_user: user_name  # enter user with administration privileges (sudo command) on webserver
      ansible_ssh_private_key_file: ~/.ssh/private_key_name  # enter SSH private key name that you use to access webserver if you use KeePassXC to store private key (like in one of my previous tutorials); the private key will be provided by ssh-agent when you unlock KeePassXC database

create_user.yml
Ansible playbook to create user. Optionally change value in vars to change user to be created.

~/ansible/create_user.yml
- name: Create user with home folder and add SSH authorized key to log in as this user using SSH
  hosts: all
  become: true
  become_method: sudo

  vars:
    username: gitlab-deploy # enter user to create

  vars_prompt:
    - name: ssh_public_key
      prompt: Input SSH public key
      private: false # if true hide input when entered
      unsafe: true # option to be able to use special characters such as '{' and '%' and not create templating errors

  tasks:
    - name: Show host name and user to create
      ansible.builtin.debug:
        msg:
          - "Host: {{ ansible_facts['fqdn'] }}"
          - "User to create: {{ username }}"

    - name: Create user with home folder
      ansible.builtin.user:
        name: "{{ username }}"
        create_home: true
        shell: /bin/bash # because Debian by default create user with /bin/sh shell
        password_lock: true

    - name: Harden permissions for user home folder
      ansible.builtin.file:
        path: "/home/{{ username }}"
        state: directory
        mode: "0700"

    - name: Add SSH authorized key (public key)
      ansible.posix.authorized_key:
        user: "{{ username }}"
        state: present
        key: "{{ ssh_public_key }}"

change_owner_dir.yml
Ansible playbook to change permissions and group of folder. Change values in vars to set folder path and group name.

~/ansible/change_owner_dir.yml
- name: Change permissions and group of folder
  hosts: all
  become: true
  become_method: sudo

  vars:
    folder_path: "/var/html" # enter path on webserver where you will put sample webpage
    group: gitlab-deploy # enter group to set ownership to

  tasks:
    - name: Show host name
      ansible.builtin.debug:
        msg:
          - "Host: {{ ansible_facts['fqdn'] }}"

    - name: Change permissions and group of folder
      ansible.builtin.file:
        path: "{{ folder_path }}"
        state: directory
        group: "{{ group }}"
        mode: "0770"

Now run the playbooks that you created:

  • create_user.yml 

    $ ansible-playbook create_user.yml
    You will be asked to input SSH public key - copy content of SSH public key file that you created in step Create SSH keys to access webserver host.

  • change_owner_dir.yml 

    $ ansible-playbook change_owner_dir.yml

Check on your webserver if the new user was created and permissions to folder where you will put webpage were set correctly. Remember that if this folder is a subfolder check if parent folder also allows access to newly created gitlab-user. If not you will need to change change_owner_dir.yml to your needs and directory structure. Adding ansible parameter recurse for ansible.builtin.file can help here - it will recursively set the specified attributes on folder and subfolders.

Add SSH private key to GitLab variable

Enter your newly created repository in GitLab web UI. There go to the following option to set GitLab variable:

  • GitLab web UI > Projects > Personal > <name_of_your_repository>:
    • Settings > CI/CD > Variables > click Add variable:
      • Type - File (variable will hold path to temporary file with value from Value field)
      • Visibility - Visible (unfortunately Masked can't be used because GitLab shows error when Value of variable has whitespace characters, and SSH private key has them)
      • Flags - Protect variable - uncheck if you want to use this variable on unprotected branches (by default main branch in your repository is protected, if you will only commit to main than it's ok, but if you will commit to other branch then uncheck this option or make other branch protected). I unchecked this option because I developed .gitlab-ci.yml file in cicd branch.
      • Description - enter description to later know what this variable is used for, in this example you can enter: SSH private key for user gitlab-deploy
      • Key - enter name for your variable, in this example SSH_PRIVATE_KEY (must be one line, with no spaces, using only letters, numbers, or _)
      • Value - copy content of SSH private key file that you created in step Create SSH keys to access webserver host. Important: paste private key including a new line at the end.
      • Rest of options change to your liking or leave defaults

When you will use this variable in GitLab CI/CD job, a temporary file containing the private key will be created on the runner for each CI/CD job and its path will be stored in the $SSH_PRIVATE_KEY environment variable.

Add connection variables to GitLab

Also set new variables with information needed for connecting using SSH, such as:

  • REMOTE_USER
  • REMOTE_HOST
  • REMOTE_PATH

Use following values:

  • GitLab web UI > Projects > Personal > <name_of_your_repository>:

    • Settings > CI/CD > Variables > click Add variable:

      • Type - Variable
      • Visibility - Visible
      • Description - User used for deployment at webserver
      • Key - REMOTE_USER
      • Value - gitlab-deploy
      • Rest of options change to your liking or leave defaults

    • Settings > CI/CD > Variables > click Add variable:

      • Type - Variable
      • Visibility - Visible
      • Description - Webserver domain name
      • Key - REMOTE_HOST
      • Value - <address-of-your-webserver>
      • Rest of options change to your liking or leave defaults

    • Settings > CI/CD > Variables > click Add variable:

      • Type - Variable
      • Visibility - Visible
      • Description - Path where webpage will be deployed at webserver
      • Key - REMOTE_PATH
      • Value - <path-on-your-webserver-where-to-put-webpage>
      • Rest of options change to your liking or leave defaults

Simple gitlab-ci.yml to check if SSH access is working

It's time to test previous settings. Create .gitlab-ci.yml file in your webpage repository. This simple CI/CD job will connect to webserver and list gitlab-deploy user home folder.

$ vim ~/simple-webpage/.gitlab-ci.yml
~/simple-webpage/.gitlab-ci.yml
stages:
  - deploy

workflow:
  name: "${CI_COMMIT_MESSAGE}, author ${CI_COMMIT_AUTHOR}"

list-home-folder:
  stage: deploy
  image: alpine:latest
  before_script:
    - apk update --no-cache
    - apk add --no-cache openssh-client
    - eval $(ssh-agent -s)
    - chmod 400 "${SSH_PRIVATE_KEY}"
    - ssh-add "${SSH_PRIVATE_KEY}"
    - mkdir -p ~/.ssh
    - chmod 700 ~/.ssh
  script:
    - ssh -o StrictHostKeyChecking=no ${REMOTE_USER}@${REMOTE_HOST} "ls -la ~/"

After creating this file push it to your GitLab. When a .gitlab-ci.yml file is pushed to the repository, GitLab will automatically detect it and start a CI/CD pipeline. Go to Build > Pipelines in your GitLab project to see the pipeline’s status. If everything was ok it should be marked as Passed.

Delete SSH keys from disk

When you know that connection is working you can delete SSH public and private key files that you created in step Create SSH keys to access webserver host from your disk.

gitlab-ci.yml to deploy webpage

To deploy webpage to webserver you need a pipeline that copy files from GitLab repository to webserver. To do it at every commit use following code in .gitlab-ci.yml:

$ vim ~/simple-webpage/.gitlab-ci.yml
~/simple-webpage/.gitlab-ci.yml
stages:
  - deploy

workflow:
  name: "${CI_COMMIT_MESSAGE}, author ${CI_COMMIT_AUTHOR}"

copy-files:
  stage: deploy
  image: alpine:latest
  before_script:
    - apk update --no-cache
    - apk add --no-cache openssh-client
    - eval $(ssh-agent -s)
    - chmod 400 "${SSH_PRIVATE_KEY}"
    - ssh-add "${SSH_PRIVATE_KEY}"
    - mkdir -p ~/.ssh
    - chmod 700 ~/.ssh
  script:
    - scp -r ./simple-webpage/html/* ${REMOTE_USER}@${REMOTE_HOST}:${REMOTE_PATH}
    - echo "List files after copying:"
    - ssh -o StrictHostKeyChecking=no ${REMOTE_USER}@${REMOTE_HOST} "ls -la ${REMOTE_PATH}"

  • ./simple-webpage/html/* - copy all files from /html/ folder in your repository to remote host

After modifying this file push it to your GitLab. When index.html file will be pushed to the repository, GitLab will automatically detect new push and start a CI/CD pipeline. Go to Build > Pipelines in your GitLab project to see the pipeline’s status. If everything was ok it should be marked as Passed. See log of job copy-files and you should see list of copied files at the end because of command ls -la ${REMOTE_PATH}.