Run AI Agents with Raspberry PI and Docker similarly to Containers

Last Updated on 12th July 2025 by peppe8o

This tutorial will show you how to run Artificial Intelligence (AI) agents on a Raspberry PI computer board with Docker. This means that you will be able to run AI programs with a single command line, just as you would with a standard Docker container.

This is an interesting option and alternative to run Ollama in Raspberry PI or use Hugging Face AI Models on Raspberry PI via Python, as the Docker approach saves resources and gives you a better and simplified deployment model.

Docker Models

At the date of this tutorial, Docker has just launched the Docker Models to run AI agents. It’s still in beta version, and some issues need to be fixed, but some agents can already run, and this will allow you to test them.

It’s a new feature for Docker, and it currently collects a very few AI models (only for text generation) that you can see in the Docker AI Hub. Moreover, Docker integrates the ability to use the Hugging Face marketplace as models (only for those models of Hugging Face supporting GGUF).

I bet the number of these models will increase soon, as well as I bet there will be more model types (for example, for image management). That’s because they allow for the integration of AI in your project with a very simplified process, as people who use Docker for container images can confirm.

To run the Docker models, you will need to use a “special” container: the docker/model-runner. It will enable your system to run the AI models. It is important to note that the Docker Model command could not work in older docker-engine versions. So, if you have an old version (commasd-> docker version), I suggest you update it (I tested it with 28.3.2).

Running AI models on a Raspberry PI will require a bit of CPU and RAM. For this reason, even if you can test it with older Raspberry PI computer models (only those supporting the 64-bit OS), you will probably need at least a Raspberry PI 4 Model B with 4GB of RAM or a newer model.

What We Need

As usual, I suggest adding from now to your favourite e-commerce shopping cart all the needed hardware, so that at the end you will be able to evaluate overall costs and decide if to continue with the project or remove them from the shopping cart. So, hardware will be only:

• Raspberry PI Computer Board (including proper power supply or using a smartphone micro USB charger with at least 3A)
• high-speed micro SD card (at least 32 GB, at least class 10). AI Models can use a lot of storage, so the bigger the storage, the more AI models you will be able to save in it

Step-by-Step Procedure

Prepare the Raspberry PI Operating System

Start preparing your OS. Please note that we must use a 64-bit OS to get the model-runner working.

You can install Raspberry PI OS Lite (to get a faster, headless OS) or you can install Raspberry PI OS Desktop, in this case working from its internal terminal. Please find the differences between the 2 OS versions in my Raspberry PI OS Lite vs Desktop article.

For low RAM Raspberry PI computer models, the Lite OS is strongly suggested. A few more optimisations can be performed with the following links:

• Reduce the GPU to 16MB (only for computer boards up to Raspberry PI 4 and only with Raspberry PI OS Lite) -> Manage Raspberry PI GPU Memory Split
• Change Swap File to 2000MB -> Set Raspberry PI Swap Memory

Make sure that your OS is updated. From the terminal:

sudo apt update -y && sudo apt full-upgrade -y

Finally, install Docker with the following tutorial: Beginner’s Guide to Install and Use Docker with Raspberry PI.

Pulling an AI Model

The first step is choosing a model and downloading (pulling) it. According to the Docker Model docs, you can get models from the Docker AI Hub or from the Hugging Face space.

Pulling from the Docker AI Hub

Please go to the Docker AI Hub and choose a model to use. Here you will find several models. Please select the “ai/gemma3” (also available from this link). Please scroll down the page until you find the following table:

It is important to look at this as Raspberry PI is a low-capability board, and using resource-intensive models will result in very slow responses from the AI agent. A very rough way of choosing a model that’s going to perform decently on our Raspberry PI is by looking at the size column of the model: the lower the size, the better it should work (even if it will give less precise responses).

I suggest you start with models having a size lower than 1GB, and then test other models to fine-tune your needs.

For example, let’s use the gemma3 variant using 1B of parameters and with a Q4_K_M quantization, which has a size of 760 MB. You can pull it with the following command:

docker model pull ai/gemma3:1B-Q4_K_M

Please wait for the download to end. When you get back the prompt, and at any time, you can list the models saved in your local storage with the following command:

docker model ls

Which should give you a result similar to the following:

pi@raspberrypi:~ $ docker model ls
MODEL NAME           PARAMETERS  QUANTIZATION    ARCHITECTURE  MODEL ID      CREATED       SIZE
ai/gemma3:1B-Q4_K_M  999.89 M    IQ2_XXS/Q4_K_M  gemma3        dbe29fa01698  3 months ago  762.49 MiB

Pulling from the Hugging Face

Pulling an image from the Hugging Face models hub should be simple. I write “should” as at the time of this tutorial, it didn’t work for me (I suppose there are some issues as Docker Model is still in beta release).

The trick is filtering the models from the Hugging Face hub by selecting only those supporting GGUF. You can also use this Hugging Face models link with GGUF filter to see all of those having the GGUF in the model name, which is more approximate but still a valid filter.

Once you have identified the complete model name, you should be able to pull the model by prepending the “hf.co” prefix to the model name. But even the command shown in the official Docker Model docs:

docker model pull hf.co/bartowski/Llama-3.2-1B-Instruct-GGUF

gives me an error. So, at the moment, it needs bug fixes.

Running AI on Raspberry PI with Docker

It’s time to run our AI program on the Raspberry PI with Docker. The command is as simple as creating and running containers.

You have 2 run options. You can run a single interaction with your AI agent by appending the text you want it to elaborate on, inside brackets:

docker model run ai/gemma3:1B-Q4_K_M "Write in 50 words what is a Raspberry PI"

It will answer the request similarly to the following:

pi@raspberrypi:~ $ docker model run ai/gemma3:1B-Q4_K_M "Write in 50 words what is a Raspberry PI"
A Raspberry Pi is a tiny, affordable, and versatile computer built on a low-cost microcontroller. It’s popular for projects, learning coding, and creating DIY electronics, offering a fun and accessible way to explore technology.

At the end of the answer, you will get the prompt back, and you will be able to ask a new question to your AI agent.

The second way to run the AI agent on Raspberry PI with Docker is in interactive mode. In this case, you don’t provide the question in the docker model run command, and it will start a chat with you. Here you can write your questions:

pi@raspberrypi:~ $ docker model run ai/gemma3:1B-Q4_K_M
Interactive chat mode started. Type '/bye' to exit.
> Write in 50 words what is a Raspberry PI
A Raspberry Pi is a small, affordable, and versatile computer built around a Raspberry Pi chip. It's popular for hobbyists, makers, and developers due to its low cost, powerful processing capabilities, and open-source nature.

At any time, you can close the chat by using the “/bye” command.

Cleaning the Raspberry PI Storage from AI Models

As said, if you have a little storage space and you pull several AI models, you will probably fill all your storage. You can check at any time the overall space from your Raspberry PI with the df -H command.

If you need to free it, you can list all the AI models downloaded with the command:

docker model ls

At this point, you can remove unused models with the docker model rm command with the complete model name, as shown in the following:

docker model rm ai/gemma3:1B-Q4_K_M

This will give you back the disk space occupied by the model.

Docker Model Docs

The complete and official docs for the Docker Model can be found at https://docs.docker.com/reference/cli/docker/model/.

Final Thoughts

My personal opinion of Docker Models is that it’s going to change the approach the ICT world has with AI. The way Docker simplifies service management applied to AI agent deployment will help professionals as well as hobbyists to use AI programs.

At the date of this post, there are still some limitations due to the beta release.

For example, it’s hard to use Hugging Face models, which would drastically increase the available models: testing different models, sometimes I got a simple server error, while sometimes I got errors caused by the missing authentication token from the Docker model pull command. There’s still work to do to make it usable for a production-ready system.

Moreover, the interactive mode doesn’t have memory of the previous questions in the same session. This is another point to fix to make the chat agents work as desired.

I hope that the bugs from this beta version will be fixed soon, so that I will be able to test more model types and give you back more project ideas.

What’s next

If you are interested in more Raspberry PI projects (both with Lite and Desktop OS), take a look at my Raspberry PI Tutorials.

Enjoy!