Traefik reverse proxy with Podman

Introduction

If you are running many containers with Podman and want to have HTTPS connection to every one it can be tedious to manually obtain SSL certificate for every service. A simpler solution is to run a reverse proxy in front of your containers and configure it to handle the certificates. Traefik is a popular choice for a reverse proxy. In this post I will show how to set up a configuration where Traefik runs in a container and acts as reverse proxy to other containers running on the same machine, and all managed with Podman.

OS used: Debian 12
Software used: Podman 4.3.1, Traefik 3.2.3

Source

• Traefik official documentation
• Great Traefik tutorial by Christian Lempa - Simple HTTPs for Docker! // Traefik Tutorial (updated)

Information about reverse proxy

It’s important to remember that when using a reverse proxy to handle SSL certificates, communication is encrypted only from external clients to the reverse proxy. Communication from reverse proxy to containers behind it is unencrypted. However, because these containers reside on the same machine, the risk of malicious activity is very low. More thought must be put into a configuration where the reverse proxy is on one machine and forwards communication to other machines. In such scenario, you should create separate networks for connecting those machines.

Prerequisites

Create a new virtual machine for example using this tutorial Fast initialization of Debian VM using Ansible.

If you're unfamiliar with Podman, please refer to one of my previous posts for an introduction - Installing Podman for managing containers and running container with dynamic DNS client for Cloudflare. In this article I won't get into details but will simply provide Podman installation instructions.

Podman installation

• Install Podman package:

$ sudo apt install podman

• Run a command to ensure that a user session for user running Podman is spawned at boot and remains active even after logouts:

$ sudo loginctl enable-linger <username>

<username> - enter username of user using Podman

• Reboot the system for above command to take effect.

Allow containers use port 443

In Linux regular users can't open ports below 1024. Because you will run Podman container that listens on port 443, you need to run following command:

$ sudo bash -c "echo 'net.ipv4.ip_unprivileged_port_start=443' > /etc/sysctl.d/podman_unprivileged_port_start.conf"

Reboot the system afterwards.

Firewall configuration

Open port 443 on firewall.

Using iptables

Use following command to add rule to iptables:

$ sudo iptables -A INPUT -p tcp --dport 443 -j ACCEPT && \
  sudo iptables -L -v && \
  sudo netfilter-persistent save

Using nftables

Add following line to your input chain in nftables config file:

$ sudo vim /etc/nftables.conf

/etc/nftables.conf

table inet filter {
        chain input {
            (...)
            tcp dport { 443 } ct state new accept
            (...)
}

Reload rules after changing /etc/nftables.conf with command:

$ sudo systemctl restart nftables.service

In order to see firewall ruleset and counters statistics use command:

$ sudo nft list ruleset

Cloudflare API Token

To obtain Let's Encrypt certificate you need Cloudflare API Token in order for Traefik to be able to change TXT field of your DNS record. Log in to Cloudflare and perform following actions:

• Click Avatar in right top corner > My Profile > API Tokens > click Create Token > Create Custom Token:
• Token name - enter name for your Token
• Permissions - create entry and set following values in fields:
  • Zone | DNS | Edit
• Zone Resources - set following
  • Include | Specific zone | chose your zone
  • Example: Include | Specific zone | example.com
• Client IP Address Filtering - if you have dynamic Public IP you can leave it empty
• TTL - leave empty for Token to be valid forever
• Click Continue to summary > click Create Token
• Copy Token - Token will be shown only once, if you loose it, regenerate new Token (or delete it and generate new one)
• Save Token to Podman secret (we use read command to prevent showing sensitive data in Bash history):

$ read -p "Enter Token: " token && \
  echo -n  ${token} | podman secret create traefik-cloudflare -

echo -n - do not output the trailing newline, otherwise you would have \n at the end of token

Dash - at the end of podman secret create <secret-name> - is important. It means that this command will read data from stdin. As a result you will have Podman secret named traefik-cloudflare with your Cloudflare API Token.

Podman configuration for Traefik

Traefik needs to alter other containers for their proper reverse proxy configuration. To modify other containers, a container needs access to the podman.socket.

• Enable podman.socket functionality with command:

  $ systemctl --user enable --now podman.socket
  

• Then create new containers network, lets call it reverse-proxy:

  $ podman network create reverse-proxy
  

• Also create Podman volume for storing obtained certificates:

  $ podman volume create traefik-acme
  

Note

Data stored using volumes is located in following path:

/home/<username>/.local/share/containers/storage/volumes/<volume-name>/_data

Run Traefik

Warning

During first run use Let's Encrypt staging server to obtain test certificate. Let's Encrypt rate limit how many request for a certificate you can send to theirs production server, so if you make some mistake in the configuration you will be locked for some time (couple of days or even a week).

Use following line for the staging server:

--certificatesresolvers.cloudflare.acme.caserver="https://acme-staging-v02.api.letsencrypt.org/directory"

Use following line for the production server:

--certificatesresolvers.cloudflare.acme.caserver="https://acme-v02.api.letsencrypt.org/directory"

• Create a script file containing the command to run Traefik container. Put this script, for example, in the folder traefik, that you will create in your home folder. The command is long, so placing it in a script will make it easier to manage and modify.

$ mkdir ~/traefik && \
  vim ~/traefik/traefik-container.sh

~/traefik/traefik-container.sh

#!/bin/bash

podman run -d \
        --name traefik \
        --volume /run/user/${UID}/podman/podman.sock:/var/run/docker.sock \
        --volume traefik-acme:/etc/traefik/acme \
        --secret=traefik-cloudflare,type=env,target=CF_DNS_API_TOKEN \
        --net reverse-proxy \
        -p 443:443 \
        docker.io/traefik:3.2.2 \
        --providers.docker=true \
        --providers.docker.exposedbydefault=false \
        --entrypoints.websecure.address=:443 \
        --certificatesresolvers.cloudflare.acme.storage=/etc/traefik/acme/acme.json \
        --certificatesresolvers.cloudflare.acme.dnschallenge.provider=cloudflare \
        --certificatesresolvers.cloudflare.acme.caserver="https://acme-v02.api.letsencrypt.org/directory" \
        --certificatesresolvers.cloudflare.acme.email=[email protected] \
        --certificatesresolvers.cloudflare.acme.dnschallenge.resolvers=1.1.1.1:53,1.0.0.1:53

Explanation of podman run arguments:

-d - run container in detached mode (detach from it after starting)
--volume /run/user/${UID}/podman/podman.sock:/var/run/docker.sock - mount podman socket from host inside container using UID of user running Podman
--volume traefik-acme:/etc/traefik/acme - mount persistent volume to keep obtained certificate between restarts of container
--secret=traefik,type=env,target=CF_DNS_API_TOKEN - use secret created earlier and mount it as CF_DNS_API_TOKEN environment variable inside container (environment variable name is taken from Traefik documentation)
--net <network> - use specific containers network
-p <host-port>:<container-port> - publish container port to be accessible from outside
docker.io/traefik:3.2.2 - image version from Docker Hub that is used for the container

Explanation of Traefik configuration commands. Documentation can be found here: Traefik Static Configuration: CLI:

--providers.docker=true - set Docker/Podman backend as provider and use podman.socket
--providers.consulcatalog.exposedbydefault=false - don't expose containers to Traefik by default, you will need to specify which container needs to be exposed when running them
--entrypoints.<name>.address - adding entry point that we name websecure and assigning port number for it
--certificatesresolvers.cloudflare.acme.storage - path were Traefik will create file with obtained certificate inside container
--certificatesresolvers.<name>.acme.dnschallenge.provider - use cloudflare for DNS-01 challange to obtain certificate from Let's Encrypt
--certificatesresolvers.<name>.acme.caserver - here you can change Let' Encrypt address to staging, see warning message below
--certificatesresolvers.<name>.acme.email - email used in your Cloudflare account
--certificatesresolvers.<name>.acme.dnschallenge.resolvers - use Cloudflare DNS (1.1.1.1, 1.0.0.1) to resolve domain name to prevent using your local DNS server

• Start you script:

  $ bash ~/traefik/traefik-container.sh
  

• Check if the container was created successfully:

  $ popdman ps
  

Tip

If you encounter issues with your Traefik configuration, you can run Traefik with more verbose logs using argument --log.level="DEBUG".

Command to run such configuration:

$ podman run -d \
     --name traefik \
     (...) \
     docker.io/traefik:3.2.2 \
     (...) \
     --log.level="DEBUG"

Now you can view logs with commands:

$ podman logs traefik
$ podman logs -f traefik

-f - follow logs (show new incoming log entries)

Run test container

We will run test Nginx container to check if options were set correctly.

Create local DNS entry

Enter domain name and IP address of machine with Traefik container in your local DNS server (to properly resolve this address by machines in your network), for example:
DNS A record: nginx-test.exampledomain.com | IPv4 address: 192.168.1.22.

Create test container

Run container with command:

$ podman run -d \
         --name nginx-test \
         --net reverse-proxy \
         --label traefik.enable=true \
         --label traefik.http.routers.nginx-https.tls=true \
         --label traefik.http.routers.nginx-https.tls.certresolver=cloudflare \
         --label traefik.http.routers.nginx-https.rule="Host(\`nginx-test.exampledomain.com\`)" \
         --label traefik.http.routers.nginx-https.entrypoints=websecure \
         docker.io/nginx:1.27.3

Explanation of podman run arguments:

-d - run container in detached mode (detach from it after starting)
--net <network> - use specific containers network
--label traefik.enable=true - enable exposing container to Traefik
--label traefik.http.routers.<name>.tls=true - enable tls
--label traefik.http.routers.<name>.tls.certresolver - specify provider used for obtaining certificate, in our case it's cloudflare
--label traefik.http.routers.<name>.rule=Host - specify which domain name use for certificate, this label uses backtick ` symbols, so use an escape symbol to use them \`
--label traefik.http.routers.<name>.entrypoints - define which entry point Traefik uses for this container, here we use websecure, that was defined during the creation of Traefik container
docker.io/nginx:1.27.3 - image version from Docker Hub that is used for the container

Wait a moment for Traefik to acquire certificate (or watch logs). Then you should be able to visit nginx default site entering the following address in your browser: https://nginx-test.exampledomain.com. Verify the certificate, if it's Let's Encrypt, then your setup is correct.

Tip

Traefik recognizes ports exposed by containers and forwards them. If somehow you can't connect to a container behind Traefik, use this label in the container command, to specify which port to forward to Traefik:

--label traefik.http.services.<name>.loadbalancer.server.port=80

Running Traefik container with every system boot

Once you checked that everything is working, you can configure Traefik container to run with every system boot.

• First create folder for your user to store systemd files (because Podman is run as user):

  $ mkdir -p ~/.config/systemd/user
  

• Create script file with command to generate systemd service and save it to file:

  $ vim ~/traefik/traefik-container-service.sh
  

  ~/traefik/traefik-container-service.sh

  #!/bin/bash
  
  # Generate systemd service
  # podman generate systemd --new --name CONTAINER_NAME > /path/to/file.service
  podman generate systemd --new --name traefik > ~/.config/systemd/user/traefik.service
  
  # Reload user's systemd unit list and enable newly generated service
  systemctl --user daemon-reload
  systemctl --user enable traefik.service
  systemctl --user start traefik.service
  

• Start you script:

  $ bash ~/traefik/traefik-container-service.sh
  

• Check if service is running:

  $ systemctl --user status traefik.service
  

Update Traefik

To update Traefik you need to perform following steps.

• Stop traefik.service (this will also stop and remove container):

  $ systemctl --user stop traefik.service
  

• Change version of Traefik in ~/traefik/traefik-container.sh file.

• Run script ~/traefik/traefik-container.sh.
• Run script ~/traefik/traefik-container-service.sh.
• If the new version is working correctly, you can remove old images with command:

  $ podman image prune -a