Run local AI with llama.cpp using Podman Quadlets

Introduction

In the last post I showed how to run Ollama container using Podman Quadlets. Today I will demonstrate how to create llama.cpp container for local AI with the same approach. Llama.cpp is an engine used to run local AI models saved in GGUF files. While Ollama uses llama.cpp under the hood, it adds extra layers of functionality that can make running models slightly slower than using llama.cpp directly. Running Ollama was easier some time ago, but that is no longer the case, which I will show in this post.

OS used: Debian 13
Software used: Podman 5.4.2, Nvidia Container Toolkit 1.19.1, llama.cpp b9680, Open WebUI 0.9.6

Source

• Official Podman 5.4.2 Quadlets Documentation
• Official llama.cpp Docker Documentation

Differences between llama.cpp and Ollama

Advantages:

• llama.cpp is slightly faster than Ollama.

Disadvantages:

• In llama.cpp, changing model parameters (such as context size) requires restarting the process with the new settings.
• In Ollama, you can modify model parameters dynamically via API calls (e.g., using curl or Open WebUI) without needing to restart the service.

Install Podman

Install podman:

$ sudo apt install podman

If you will be using this setup on bare metal server with GPU, then following command should be used to ensure that a user session is spawned at boot and kept active even after logouts from GUI or tty sessions:

$ sudo loginctl enable-linger <username>

• <username> - enter username of user using Podman

Reboot system for above command to take effect.

Install GPU drivers

Now you should install GPU drivers (if you haven't already). I have Nvidia card, so I will show how to install Nvidia drivers.

Nvidia drivers from Debian repository

Install following packages:

$ sudo apt install linux-headers-$(dpkg --print-architecture) \
  nvidia-kernel-dkms \
  nvidia-driver \
  firmware-misc-nonfree \
  libnvoptix1

Official Nvidia drivers

If you prefer newer drivers than those from Debian repositories, you can download drivers from Nvidia webpage. Save them to your ~/Downloads/ directory.

To install proprietary Nvidia drivers you need to block nouveau kernel driver by creating a file in /etc/modprobe.d/ directory:

$ sudo vim /etc/modprobe.d/nvidia-disable-nouveau.conf

/etc/modprobe.d/nvidia-disable-nouveau.conf

blacklist nouveau
options nouveau modeset=0

After that update initramfs:

$ sudo update-initramfs -u

Reboot system.

After reboot install packages needed by Nvidia drivers installer:

$ sudo apt install build-essential \
  linux-headers-$(dpkg --print-architecture) \
  dkms

Now run Nvidia drivers installer:

$ sudo bash ~/Downloads/NVIDIA-Linux-x86_64-{version}.run

Note

INFO 1:
Nvidia drivers temporarily extract data to /tmp/ directory. In Debian 13 this folder is kept in memory and it's max size depends on available RAM. For 8GB RAM and more the drivers extracts with no problems.

INFO 2:
During installation you can see following message: WARNING: Unable to determine the path to install the libglvnd EGL vendor library config files. Check that you have pkg-config and the libglvnd development libraries installed, or specify a path with --glvnd-egl-config-path.. You can ingore it. Drivers work fine if you continue.

INFO 3:
During installation there can also be a message that xserver is being used. On Wayland you can ignore it. On X11 go to the terminal Ctrl+Alt+F1 and type following command:

• For Gnome

  $ sudo systemctl stop gdm
  

• For KDE

  $ sudo systemctl stop sddm
  

After that continue installation in terminal.

Install Nvidia Container Toolkit

For containers to have access to Nvidia GPU you need to install the Nvidia Container Toolkit:

$ curl -fsSL https://nvidia.github.io/libnvidia-container/gpgkey | sudo gpg --dearmor -o /usr/share/keyrings/nvidia-container-toolkit-keyring.gpg \
  && curl -s -L https://nvidia.github.io/libnvidia-container/stable/deb/nvidia-container-toolkit.list | \
  sed 's#deb https://#deb [signed-by=/usr/share/keyrings/nvidia-container-toolkit-keyring.gpg] https://#g' | \
  sudo tee /etc/apt/sources.list.d/nvidia-container-toolkit.list && \
  sudo apt-get update && \
  sudo apt-get install -y nvidia-container-toolkit

Check the list of generated CDI (Container Device Interface) devices:

$ nvidia-ctk cdi list

nvidia.com/gpu=0
nvidia.com/gpu=GPU-...
nvidia.com/gpu=all

If you don't see Nvidia GPU in the line nvidia.com/gpu=, manually generate CDI:

$ sudo systemctl restart nvidia-cdi-refresh.service

Reboot system and check if CDI are still present:

$ nvidia-ctk cdi list

Tip

If CDI devices are absent (I had this problem on Debian 13 with Nvidia drivers installed from Debian repository), use this command:

$ sudo nvidia-ctk cdi generate --output=/etc/cdi/nvidia.yaml

IMPORTANT: In this case generate CDI after every Nvidia driver update.

Verify that containers have access to GPU:

$ podman run --rm --device nvidia.com/gpu=all \
  docker.io/nvidia/cuda:11.0.3-base-ubuntu20.04 nvidia-smi

Download AI models

Llama.cpp run AI models from GGUF files. You can download models in such format from Hugging Face.

Create directory where you will download AI models. For this tutorial I will create directory in user home:

$ mkdir -p ~/ai-models

Create a separate directory for files of every model. This allows automatic loading of mmproj files, which provide multimodal capabilities (e.g., vision) for specific models.

Directory structure example:

~/ai-models/
 │
 │  # multimodal model (e.g. with vision capabilities)
 ├─ /gemma-4-E2B-it-Q8_0/
 │    ├─ gemma-4-E2B-it-Q8_0.gguf
 │    └─ mmproj-gemma-4-E2B-it-Q8_0.gguf  # file name must start with "mmproj"
 │
 │  # single file model
 ├─ /Qwen3-Coder-30B-A3B-Instruct-Q4_K_M/
 │    └─ Qwen3-Coder-30B-A3B-Instruct-Q4_K_M.gguf
 │

Next download AI model from Hugging Face for testing, for example gemma4:e2b from Google.

Create Podman Quadlets

Now you can start to create Podman Quadlets.

Quadlets files for user containers should be created in $HOME/.config/containers/systemd/ directory. Create this directory:

$ mkdir -p ~/.config/containers/systemd

Containers network

$ vim ~/.config/containers/systemd/llama-cpp.network

~/.config/containers/systemd/llama-cpp.network

[Network]
NetworkName=llama-cpp

llama.cpp container

With Nvidia support

$ vim ~/.config/containers/systemd/llama-cpp.container

~/.config/containers/systemd/llama-cpp.container

[Container]
ContainerName=llama-cpp
# CUDA 12 support
Image=ghcr.io/ggml-org/llama.cpp:server-cuda
AutoUpdate=registry
# %h expands to the home directory of the user
Volume=%h/ai-models:/models
Timezone=Europe/Warsaw
Network=llama-cpp.network
PublishPort=8080:8080
AddDevice=nvidia.com/gpu=all
Exec=--models-dir /models --models-max 1 --ctx-size 4096 --host 0.0.0.0 --port 8080 --n-gpu-layers auto

[Service]
Restart=on-failure

[Install]
# Start by default on boot
# If you don’t want an unit to start at boot, remove the [Install] section
WantedBy=multi-user.target default.target

Info

You can check meaning of llama.cpp server parameters (Exec=) in this documentation: LLaMA.cpp HTTP Server.

Open WebUI container

Llama.cpp has its own simple WebUI, but we will also install Open WebUI for more advanced functionality.

$ vim ~/.config/containers/systemd/open-webui.container

~/.config/containers/systemd/open-webui.container

[Container]
ContainerName=open-webui
Image=ghcr.io/open-webui/open-webui:main
AutoUpdate=registry
Volume=open-webui:/app/backend/data
Timezone=Europe/Warsaw
Environment=WEBUI_AUTH=False
Network=llama-cpp.network
PublishPort=3000:8080

[Service]
Restart=on-failure

[Install]
# Start by default on boot
# If you don’t want an unit to start at boot, remove the [Install] section
WantedBy=multi-user.target default.target

Apply Podman Quadlets

Reload systemd daemon:

$ systemctl --user daemon-reload

Start Quadlets:

$ systemctl --user start llama-cpp.service && \
  systemctl --user start open-webui.service

Auto update containers

Because we used latest images from containers registry and AutoUpdate option in .container files, we can auto update containers with following command:

$ podman auto-update

Delete unused images

You can delete old unused images that will be left after AutoUpdate by using command:

$ podman image prune -a

Start Open WebUI

Enter following address in your browser to use:

• Open WebUI - http://localhost:3000
• llama.cpp WebUI - http://localhost:8080

Since this installation is intended for local desktop use, we have not configured HTTPS or user authentication in Open WebUI. Protect access to Open WebUI and llama.cpp WebUI by blocking ports 3000 and 8080 in your desktop firewall.

Configure Open WebUI to connect to llama.cpp:

• Go to Admin Panel > Settings > Connections > enable OpenAI API
  • Click Add Connection (plus icon):
  • Set the following:
    • Connection Type: External
    • URL: http://llama-cpp:8080/v1
    • API Key: none (leave blank)
    • Provider: pick llama.cpp (this unlocks the loaded-model indicator and the eject button in the model selector)