Skip to content

Podman

Run local AI with llama.cpp using Podman Quadlets

Introduction

In the last post I showed how to run Ollama container using Podman Quadlets. Today I will demonstrate how to create llama.cpp container for local AI with the same approach. Llama.cpp is an engine used to run local AI models saved in GGUF files. While Ollama uses llama.cpp under the hood, it adds extra layers of functionality that can make running models slightly slower than using llama.cpp directly. Running Ollama was easier some time ago, but that is no longer the case, which I will show in this post.

OS used: Debian 13
Software used: Podman 5.4.2, Nvidia Container Toolkit 1.19.1, llama.cpp b9680, Open WebUI 0.9.6

Podman Quadlets

Introduction

In one of my previous posts about Podman I wrote that in the newer version of Podman (> 4.4), quadlets are recommended instead of podman-compose (both as an alternative to docker-compose). Now with Debian 13 we have an appropriate version of Podman to play with quadlets. Following current homelab trends, I will show how to install local AI using Podman and Quadlets, specifically setting up Ollama and Open WebUI. I want to keep this post short, so I will focus on installation on bare metal desktop computer. In such setup you don't need to configure PCI Passthrough which you need with virtual machines.

OS used: Debian 13
Software used: Podman 5.4.2, Nvidia Container Toolkit 1.19.1, Ollama 0.30.8, Open WebUI 0.9.6

Self hosted Vaultwarden password manager with Podman

Introduction

In the last blog post I described how to install Bitwarden with Docker. There is a lighter alternative to Bitwarden server called Vaultwarden and it's compatible with official Bitwarden client. Other differences between those two can be easily found online, so I won't go into details here. To make things more interesting, this instruction will show how to install Vaultwarden with Podman.

OS used: Debian 12
Software used: Podman 4.3.1, Vaultwarden 1.32.7

Traefik reverse proxy with Podman

Introduction

If you are running many containers with Podman and want to have HTTPS connection to every one it can be tedious to manually obtain SSL certificate for every service. A simpler solution is to run a reverse proxy in front of your containers and configure it to handle the certificates. Traefik is a popular choice for a reverse proxy. In this post I will show how to set up a configuration where Traefik runs in a container and acts as reverse proxy to other containers running on the same machine, and all managed with Podman.

OS used: Debian 12
Software used: Podman 4.3.1, Traefik 3.2.3

Installing Podman for managing containers and running container with dynamic DNS client for Cloudflare

Introduction

Podman is a Docker alternative with distinction that it is a daemonless container engine and containers created with Podman can be run by regular users without root privileges. This enhances security because if a process escapes container, it won't run on the host with root privileges but with user privileges running the container. If it's so secure, why is Docker still popular? The answer is that the daemonless architecture of Podman makes it more complex to manage, especially set up containers to run with every system start. Also there is less documentation available for Podman compared to Docker, so this post will try to summarize how to create functioning Podman host.

OS used: Debian 12
Software used: Podman 4.3.1