TL;DR

Deploying Ollama on Kubernetes transforms local AI inference into a production-grade service with horizontal scaling, persistent model storage, and service mesh integration. This guide covers container orchestration patterns specifically for LLM workloads running on self-hosted infrastructure.

The core deployment uses StatefulSets rather than Deployments to maintain stable network identities and persistent volume claims for model storage. Each Ollama pod serves the REST API on port 11434 and requires GPU node affinity when using NVIDIA runtime. Configure OLLAMA_HOST=0.0.0.0:11434 to bind the service to all interfaces within the pod network, and set OLLAMA_MODELS=/models pointing to your PersistentVolume mount path.

Key architectural decisions include using PersistentVolumeClaims to store downloaded models separately from ephemeral container storage, implementing horizontal pod autoscaling based on CPU and memory metrics (since Ollama lacks native Prometheus endpoints), and configuring service mesh sidecars for request routing and circuit breaking. GPU scheduling requires node labels and resource limits matching your NVIDIA device plugin configuration.

Production considerations involve setting OLLAMA_ORIGINS to restrict CORS access, implementing readiness probes that query the API health endpoint, and using init containers to pre-pull models like llama3.2:3b before pod startup. Network policies should restrict ingress to your application namespace while allowing egress to ollama.com for model downloads.

Caution: Always validate AI-generated Kubernetes manifests in a staging cluster before applying to production. LLM-suggested resource limits may not match your actual GPU memory constraints, and incorrect volume configurations can cause model download failures or data loss during pod restarts.

This approach works for teams running private AI infrastructure on bare-metal Kubernetes, Rancher clusters, or managed Kubernetes with GPU node pools, eliminating cloud API dependencies while maintaining enterprise deployment patterns.

Why Run Ollama on Kubernetes

Running Ollama on Kubernetes transforms a single-node LLM server into a production-grade AI infrastructure platform. While Ollama works perfectly on a single Linux machine, Kubernetes orchestration becomes essential when you need to serve multiple teams, handle variable workloads, or maintain high availability for AI-powered applications.

Kubernetes namespaces let you run separate Ollama instances for different projects without interference. A development team can test llama3.2 models in one namespace while production services use mistral in another. Resource quotas prevent any single deployment from consuming all available GPU memory, which matters when running multiple 7B or 13B parameter models simultaneously.

Dynamic Scaling for Inference Workloads

Container orchestration enables horizontal pod autoscaling based on CPU or custom metrics. When your Open WebUI frontend experiences traffic spikes, Kubernetes can spin up additional Ollama pods to distribute the load. The REST API on port 11434 works seamlessly behind a Kubernetes Service, allowing multiple backend pods to serve requests through a single endpoint.

Persistent Model Storage

PersistentVolumeClaims solve the model download problem elegantly. Instead of each pod pulling multi-gigabyte GGUF files from ollama.com, you mount a shared volume at the OLLAMA_MODELS path. The first pod downloads codellama:13b once, and subsequent pods access the cached model immediately. This approach reduces startup time from minutes to seconds.

Integration with Service Mesh

Tools like Istio or Linkerd provide request tracing, automatic retries, and circuit breaking for Ollama endpoints. When an inference request times out, the service mesh can route to a different pod automatically. This reliability layer becomes critical when Ollama backs customer-facing features rather than internal experiments.

Caution: Always validate Kubernetes manifests in a staging environment before applying to production clusters. AI workloads consume substantial resources, and misconfigured resource limits can destabilize entire nodes.

Architecture Overview: Ollama in K8s

Deploying Ollama on Kubernetes requires understanding how the stateless API layer interacts with stateful model storage. Unlike traditional microservices, Ollama pods need persistent volumes for model files and GPU access for inference workloads.

The typical architecture consists of three layers. First, Ollama pods run as a StatefulSet rather than a Deployment because each pod needs stable network identity and persistent model storage. Second, a headless Service exposes port 11434 for internal cluster communication. Third, persistent volumes store GGUF model files that can reach multiple gigabytes per model.

apiVersion: v1
kind: Service
metadata:
  name: ollama-headless
spec:
  clusterIP: None
  selector:
    app: ollama
  ports:
  - port: 11434
    targetPort: 11434

Storage Strategy

Model files live in persistent volumes mounted at the path specified by OLLAMA_MODELS. A common pattern uses a shared ReadWriteMany volume for model storage with separate ReadWriteOnce volumes for each pod’s runtime data. This prevents redundant model downloads across replicas while maintaining pod isolation.

GPU Scheduling

Kubernetes node selectors and taints ensure Ollama pods land on GPU-equipped nodes. The OLLAMA_NUM_GPU environment variable controls GPU allocation per pod. Set resource limits matching your GPU count to prevent scheduling conflicts.

resources:
  limits:
    nvidia.com/gpu: 1
env:
- name: OLLAMA_NUM_GPU
  value: "1"

Network Considerations

Set OLLAMA_ORIGINS to allow cross-origin requests from web interfaces like Open WebUI running in separate pods. The OLLAMA_HOST variable typically stays at default 0.0.0.0:11434 for cluster-internal access.

Caution: Always validate resource limits and storage configurations in a staging namespace before production deployment. Misconfigured GPU requests can block pod scheduling across your entire cluster.

Container Image Preparation and Registry Setup

Building a production-ready Ollama container requires careful attention to model persistence and image optimization. The official Ollama container image runs on port 11434 and expects models to be stored in /root/.ollama, which must be backed by persistent storage in Kubernetes environments.

Start with the official Ollama image from Docker Hub. Create a custom Dockerfile that pre-loads specific models during the build phase to reduce startup time:

FROM ollama/ollama:latest

ENV OLLAMA_HOST=0.0.0.0:11434
ENV OLLAMA_MODELS=/models

RUN mkdir -p /models

COPY preload-models.sh /usr/local/bin/
RUN chmod +x /usr/local/bin/preload-models.sh

EXPOSE 11434

The preload script pulls models during image build rather than at runtime:

#!/bin/bash
ollama serve &
OLLAMA_PID=$!
sleep 5

ollama pull llama3.2:3b
ollama pull mistral:7b

kill $OLLAMA_PID

Registry Configuration

Push your custom image to a private registry accessible from your Kubernetes cluster. For Harbor or GitLab registries, authenticate using image pull secrets:

kubectl create secret docker-registry ollama-registry \
  --docker-server=registry.example.com \
  --docker-username=robot-account \
  --docker-password=token-value \
  --namespace=ai-workloads

Caution: AI-generated Dockerfiles may suggest invalid OLLAMA environment variables. Always verify against official documentation before deploying. The OLLAMA_NUM_GPU variable controls GPU allocation, not OLLAMA_NUM_GPU which does not exist.

Tag images with model versions and build dates for rollback capability. A typical tag structure looks like registry.example.com/ollama:llama3.2-20260315 to track which models are embedded in each image version.

StatefulSet Configuration and Storage Management

StatefulSets provide stable network identities and persistent storage for Ollama pods, essential for maintaining model caches across restarts. Unlike Deployments, StatefulSets guarantee ordered pod creation and stable hostnames, critical when multiple Ollama instances share a distributed model storage layer.

Configure a volumeClaimTemplate to provision storage for each Ollama replica. Models downloaded via ollama pull persist across pod restarts:

apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: ollama
spec:
  serviceName: ollama
  replicas: 3
  selector:
    matchLabels:
      app: ollama
  volumeClaimTemplates:
  - metadata:
      name: ollama-models
    spec:
      accessModes: ["ReadWriteOnce"]
      storageClassName: fast-ssd
      resources:
        requests:
          storage: 100Gi
  template:
    spec:
      containers:
      - name: ollama
        image: ollama/ollama:latest
        ports:
        - containerPort: 11434
        env:
        - name: OLLAMA_MODELS
          value: /models
        - name: OLLAMA_NUM_GPU
          value: "1"
        volumeMounts:
        - name: ollama-models
          mountPath: /models

Storage Class Selection

Choose storage classes based on model size and access patterns. NVMe-backed storage classes significantly reduce model load times for large models like llama3.1:70b. Network-attached storage works for smaller models but introduces latency during inference.

Caution: Verify storage class IOPS limits before deploying production workloads. AI-generated Kubernetes manifests may specify storage sizes without considering actual model requirements – a 70B parameter model requires 40GB minimum.

Headless Service Configuration

Create a headless service for StatefulSet pod discovery:

apiVersion: v1
kind: Service
metadata:
  name: ollama
spec:
  clusterIP: None
  selector:
    app: ollama
  ports:
  - port: 11434
    targetPort: 11434

This enables DNS-based service discovery at ollama-0.ollama.default.svc.cluster.local for direct pod addressing in multi-replica scenarios.

Service Exposure and Load Balancing

Exposing Ollama services within a Kubernetes cluster requires careful consideration of access patterns and traffic distribution. The default port 11434 must be accessible to client applications while maintaining security boundaries.

For internal cluster access, create a ClusterIP service that routes traffic to Ollama pods:

apiVersion: v1
kind: Service
metadata:
  name: ollama-service
  namespace: ai-workloads
spec:
  type: ClusterIP
  selector:
    app: ollama
  ports:
  - port: 11434
    targetPort: 11434
    protocol: TCP
    name: http

This configuration allows other pods to reach Ollama at http://ollama-service.ai-workloads.svc.cluster.local:11434. Applications like Open WebUI or custom inference clients can use this DNS name for model requests.

Load Balancing Across Replicas

When running multiple Ollama replicas, Kubernetes distributes requests using round-robin by default. For stateless inference workloads, this works well. However, model loading creates temporary CPU spikes, so consider session affinity for long-running conversations:

spec:
  sessionAffinity: ClientIP
  sessionAffinityConfig:
    clientIP:
      timeoutSeconds: 3600

Caution: Always validate generated service manifests against your cluster’s network policies before applying. AI-generated configurations may not account for your specific security requirements.

External Access Patterns

For external access, use an Ingress controller with TLS termination rather than exposing LoadBalancer services directly. This centralizes certificate management and allows rate limiting at the ingress layer. Set the OLLAMA_ORIGINS environment variable to restrict cross-origin requests:

env:
- name: OLLAMA_ORIGINS
  value: "https://ai-dashboard.example.com,https://internal-tools.example.com"

Service mesh implementations like Istio or Linkerd provide additional traffic management capabilities, including automatic retries and circuit breaking for failed model inference requests.

Installation and Configuration Steps

Start by building a custom Ollama container image with your required models pre-loaded. This approach reduces startup time and ensures consistent model availability across pod restarts.

FROM ollama/ollama:latest

ENV OLLAMA_HOST=0.0.0.0:11434
ENV OLLAMA_MODELS=/models

RUN mkdir -p /models

COPY preload-models.sh /usr/local/bin/
RUN chmod +x /usr/local/bin/preload-models.sh

CMD ["/usr/local/bin/preload-models.sh"]

Create the preload script to pull models during image build:

#!/bin/bash
ollama serve &
sleep 5
ollama pull llama3.2:3b
ollama pull mistral:7b
wait

Kubernetes Deployment Configuration

Deploy Ollama using a StatefulSet rather than a Deployment to maintain stable network identities and persistent storage for model caching:

apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: ollama
spec:
  serviceName: ollama
  replicas: 3
  selector:
    matchLabels:
      app: ollama
  template:
    metadata:
      labels:
        app: ollama
    spec:
      containers:
      - name: ollama
        image: your-registry/ollama-preloaded:latest
        ports:
        - containerPort: 11434
          name: http
        env:
        - name: OLLAMA_HOST
          value: "0.0.0.0:11434"
        - name: OLLAMA_NUM_GPU
          value: "1"
        - name: OLLAMA_ORIGINS
          value: "*"
        volumeMounts:
        - name: models
          mountPath: /models
        resources:
          limits:
            nvidia.com/gpu: 1
            memory: 16Gi
          requests:
            memory: 8Gi
  volumeClaimTemplates:
  - metadata:
      name: models
    spec:
      accessModes: ["ReadWriteOnce"]
      resources:
        requests:
          storage: 100Gi

Caution: Always validate resource limits against your actual GPU memory capacity before deploying. Test model loading times with your specific hardware configuration in a development namespace first.