TL;DR

# Install Ollama on Raspberry Pi (ARM64)
curl -fsSL https://ollama.com/install.sh | sh

# Pull a model that actually works on Pi
ollama pull qwen2.5:0.5b
ollama pull phi3:mini

# Test it
ollama run qwen2.5:0.5b "Write a Python function to read a CSV file"

# Check memory usage
ollama ps
free -h

Raspberry Pi 5 with 8 GB RAM can run models up to 3B parameters at usable speeds. Stick to 0.5B-1.5B models for interactive use. Anything above 7B is not practical.

Why Run LLMs on a Raspberry Pi

Running a language model on a Raspberry Pi is not about competing with a GPU server. It is about specific use cases where a small, always-on, low-power device makes sense:

• Home assistant integration – a local chatbot for Home Assistant without cloud API calls
• Offline document processing – summarize, classify, or extract data from files without internet
• Edge AI – process sensor data or logs locally before sending results upstream
• Learning and experimentation – understand LLM inference constraints on constrained hardware
• Privacy – keep everything on a device you physically control

The Pi 5 is the first Raspberry Pi that makes this remotely practical. Earlier models lack the memory bandwidth and CPU performance for even small LLMs.

Hardware Requirements

Recommended minimum: Raspberry Pi 5 with 8 GB RAM. The Pi 5’s Cortex-A76 cores are roughly 2x faster per clock than the Pi 4’s Cortex-A72, and the improved memory bandwidth makes a real difference for inference workloads.

Additional recommendations:

• Storage: Use an NVMe SSD via the Pi 5’s PCIe slot (or a USB 3.0 SSD). Model loading from microSD is painfully slow.
• Cooling: Active cooling is essential. LLM inference pushes the CPU hard and sustained thermal throttling destroys performance. Use the official Pi 5 active cooler or a third-party heatsink with fan.
• Power supply: Use the official 27W USB-C power supply. Underpowered supplies cause throttling.

Installing Ollama

Ollama provides an official ARM64 build. Installation is the same as on x86:

curl -fsSL https://ollama.com/install.sh | sh

This installs Ollama and sets up a systemd service. Verify:

systemctl status ollama
ollama --version

Ollama runs entirely on CPU on the Pi – there is no GPU acceleration available. All inference happens on the ARM cores.

Which Models Actually Work

This is the critical section. Most models that work well on desktop GPUs are far too large for a Pi. Here are the models that produce usable results on a Pi 5 with 8 GB RAM:

Performance Benchmarks: Raspberry Pi 5 (8 GB)

All benchmarks measured with default Q4_K_M quantization, active cooling, no other significant processes running.

Practical cutoff: Models above 3B parameters drop below 5 tokens/sec, which makes interactive chat frustrating. For batch processing where latency does not matter, 3B-4B models are workable. Anything 7B or above will technically load (with swap) but is not practical.

Recommended Models by Use Case

Home assistant / chatbot: Qwen2.5 1.5B – best quality you can get at interactive speeds.

Code generation (simple): Qwen2.5-Coder 1.5B – surprisingly capable for small functions and scripts.

Text classification / extraction: Qwen2.5 0.5B – fast enough for processing many documents in sequence.

Summarization: Llama 3.2 1B or Qwen2.5 1.5B – both handle short document summarization.

Pull the recommended models:

ollama pull qwen2.5:1.5b
ollama pull qwen2.5:0.5b
ollama pull llama3.2:1b

Memory Management

On a Pi with 8 GB, memory is the primary constraint. Ollama loads the entire model into RAM, and the OS and other services need memory too.

Check available memory before loading a model:

free -h

Expect roughly 6.5-7 GB available on a clean Pi 5 (8 GB) with a minimal OS installation. Subtract the model’s RAM footprint from that.

Ollama memory settings:

# Set how long Ollama keeps a model in memory after the last request
# Default is 5 minutes. On Pi, set shorter to free RAM sooner.
sudo systemctl edit ollama

Add:

[Service]
Environment="OLLAMA_KEEP_ALIVE=2m"
Environment="OLLAMA_MAX_LOADED_MODELS=1"

sudo systemctl daemon-reload
sudo systemctl restart ollama

OLLAMA_MAX_LOADED_MODELS=1 prevents Ollama from trying to load multiple models simultaneously, which would immediately exhaust memory.

Swap Configuration

Configure swap as a safety net, but do not rely on it for model inference – swapping during inference makes performance unusable.

# Increase swap to 2 GB (default is often 100 MB or 1 GB)
sudo dphys-swapfile swapoff
sudo sed -i 's/CONF_SWAPSIZE=.*/CONF_SWAPSIZE=2048/' /etc/dphys-swapfile
sudo dphys-swapfile setup
sudo dphys-swapfile swapon

# If using zram instead (better for Pi):
sudo apt install zram-tools
echo -e "ALGO=lz4\nPERCENT=50" | sudo tee /etc/default/zramswap
sudo systemctl restart zramswap

zram compresses swap in memory, which is faster than swapping to storage. On a Pi with limited I/O bandwidth, zram is the better option.

Monitor during inference:

# Watch memory in real time while running a query
watch -n 1 'free -h && echo "---" && ollama ps'

Overclocking Considerations

The Pi 5 can be overclocked to improve inference speed. This is a meaningful optimization since LLM inference on CPU is compute-bound.

Edit /boot/firmware/config.txt:

# Moderate overclock (stable on most Pi 5 units with active cooling)
arm_freq=2800
gpu_freq=910

# Aggressive overclock (requires good cooling, test stability)
# arm_freq=3000
# gpu_freq=1000

Reboot and verify:

sudo reboot
# After reboot:
vcgencmd measure_clock arm
vcgencmd get_throttled
# 0x0 means no throttling

Caution: Overclocking voids the warranty and can cause instability. Always use active cooling and monitor temperatures:

vcgencmd measure_temp
# Target: stay below 80C under sustained load

A 2.4 GHz to 2.8 GHz overclock typically yields 10-15% faster inference. Going to 3.0 GHz can add another 5-7% but increases power draw and heat significantly.

Practical Use Cases

Home Assistant Integration

Run Ollama on a Pi 5 and connect it to Home Assistant’s conversation agent:

# Home Assistant configuration.yaml
conversation:
  intents:
    - intent: OllamaQuery
      sentences:
        - "ask the ai {query}"

rest_command:
  ollama_query:
    url: "http://localhost:11434/api/generate"
    method: POST
    content_type: "application/json"
    payload: '{"model": "qwen2.5:1.5b", "prompt": "{{ query }}", "stream": false}'

Offline Log Analysis

Process system logs without sending them off-device:

#!/bin/bash
# Summarize recent auth failures
LOGS=$(journalctl -u sshd --since "1 hour ago" --no-pager | tail -50)
ollama run qwen2.5:1.5b "Summarize these SSH log entries. List unique source IPs and whether this looks like a brute force attempt: $LOGS"

Edge Data Processing

For IoT or sensor applications, use the Ollama API programmatically:

import requests

def classify_reading(sensor_data: dict) -> str:
    prompt = f"Classify this sensor reading as normal, warning, or critical: {sensor_data}"
    response = requests.post(
        "http://localhost:11434/api/generate",
        json={"model": "qwen2.5:0.5b", "prompt": prompt, "stream": False}
    )
    return response.json()["response"]

Limitations and Honest Assessment

Running LLMs on a Raspberry Pi is constrained. Be realistic about what to expect:

• Quality: Sub-2B models make mistakes that 7B+ models do not. Expect factual errors, incomplete code, and weaker reasoning.
• Speed: Even the fastest models on Pi produce text slower than a person reads. Long responses take 30-60 seconds.
• Context window: Small models typically have 2K-4K token context windows. They cannot process long documents.
• No GPU acceleration: Everything runs on CPU. There is no path to GPU inference on Pi hardware.
• Concurrent users: One user at a time. The Pi cannot handle multiple simultaneous inference requests.

For anything beyond simple, short-context tasks, a proper GPU-equipped machine running Ollama will deliver a fundamentally better experience. The Pi is for specific edge and offline use cases where the constraints are acceptable.