Run local AI with llama.cpp using Podman Quadlets
Introduction
In the last post I showed how to run Ollama container using Podman Quadlets. Today I will demonstrate how to create llama.cpp container for local AI with the same approach. Llama.cpp is an engine used to run local AI models saved in GGUF files. While Ollama uses llama.cpp under the hood, it adds extra layers of functionality that can make running models slightly slower than using llama.cpp directly. Running Ollama was easier some time ago, but that is no longer the case, which I will show in this post.
OS used: Debian 13
Software used: Podman 5.4.2, Nvidia Container Toolkit 1.19.1, llama.cpp b9680, Open WebUI 0.9.6