Setting OLLAMA_NUM_GPU for Multi-GPU Local AI in 2026

TL;DR

The OLLAMA_NUM_GPU environment variable controls how many GPUs Ollama uses for inference, but setting it correctly requires understanding your hardware topology and workload patterns. Unlike single-GPU setups where Ollama auto-detects your card, multi-GPU configurations demand explicit tuning to avoid memory fragmentation and PCIe bottlenecks.

Set OLLAMA_NUM_GPU=2 to split model layers across two GPUs, or OLLAMA_NUM_GPU=4 for quad-GPU systems. Ollama distributes transformer layers sequentially – GPU 0 handles the first N layers, GPU 1 takes the next batch, and so on. This differs from data parallelism where each GPU processes different prompts simultaneously.

export OLLAMA_NUM_GPU=2
ollama serve

Most multi-GPU gains appear when running models that exceed single-GPU VRAM. A 70B parameter model needs roughly 40GB in FP16, so two 24GB cards working together enable inference that would otherwise require offloading to system RAM. Smaller models like Llama 3.1 8B see minimal benefit from multiple GPUs since they fit comfortably on one card.

Performance scaling is sublinear – two GPUs rarely double throughput due to inter-GPU communication overhead. Expect 1.6x to 1.8x speedup with two cards, and 2.5x to 3.2x with four cards, depending on PCIe generation and NVLink availability. Systems without NVLink rely on PCIe bandwidth, which becomes the bottleneck when layers exchange activations.

Common pitfall: setting OLLAMA_NUM_GPU higher than your physical GPU count causes Ollama to fail silently or fall back to CPU inference. Verify GPU detection with nvidia-smi or rocm-smi before configuring the variable. Another issue: mismatched GPU memory capacities lead to uneven layer distribution, where the smaller GPU fills up first and forces the rest onto system RAM.

This guide covers optimal OLLAMA_NUM_GPU values for different hardware configurations, memory allocation strategies across GPUs, and troubleshooting detection failures in mixed-GPU environments.

Understanding OLLAMA_NUM_GPU and Multi-GPU Architecture

The OLLAMA_NUM_GPU environment variable controls how many GPUs Ollama uses for model inference. Unlike layer-based distribution systems, this variable tells Ollama to spread computation across the specified number of devices. When you set OLLAMA_NUM_GPU=2, Ollama attempts to utilize two GPUs for parallel processing rather than confining the model to a single card.

Ollama uses tensor parallelism to split model layers across available GPUs. When running a 70B parameter model with OLLAMA_NUM_GPU=2, the framework divides attention heads and feed-forward layers between both cards. This differs from pipeline parallelism where sequential layers live on different devices – tensor parallelism splits individual operations horizontally.

The practical impact: memory requirements per GPU decrease while inter-GPU communication overhead increases. A model requiring 48GB VRAM on one GPU might need only 26GB per card across two GPUs, accounting for activation memory and communication buffers.

Setting the Variable

Configure OLLAMA_NUM_GPU before starting the Ollama service:

export OLLAMA_NUM_GPU=2
ollama serve

For systemd-managed installations, edit the service file:

sudo systemctl edit ollama

Add the override:

[Service]
Environment="OLLAMA_NUM_GPU=2"

Then restart:

sudo systemctl restart ollama

Verification and Detection

Check GPU detection after starting Ollama:

ollama run llama3.1:70b

Watch nvidia-smi or rocm-smi output in another terminal. Both GPUs should show memory allocation and utilization. If only one GPU activates, check CUDA_VISIBLE_DEVICES or ROCR_VISIBLE_DEVICES settings – these can override OLLAMA_NUM_GPU by hiding devices from the runtime.

Caution: Always verify GPU assignments match your hardware topology. Mismatched PCIe lanes or NVLink configurations can create bottlenecks that negate multi-GPU benefits.

Multi-GPU Load Balancing Strategies

Ollama’s multi-GPU support relies on the OLLAMA_NUM_GPU environment variable to distribute model layers across available GPUs. Unlike traditional load balancing that routes requests between workers, Ollama splits a single model’s computation across multiple devices simultaneously.

When you set OLLAMA_NUM_GPU to match your GPU count, Ollama automatically partitions the model’s transformer layers. For a 70B parameter model with 80 layers running on four GPUs, each device handles approximately 20 layers. This approach maximizes throughput for individual requests rather than parallelizing multiple concurrent requests.

export OLLAMA_NUM_GPU=4
ollama run llama3.1:70b

The distribution happens at model load time. Ollama examines available VRAM on each GPU and assigns layers accordingly. If GPU 0 has 24GB and GPU 1 has 16GB, the allocation favors the larger device until memory constraints force spillover.

Handling Asymmetric GPU Configurations

Mixed GPU setups require careful consideration. Running two RTX 4090s alongside two RTX 3090s creates uneven memory distribution. Ollama will use all four devices, but the 3090s with less VRAM may bottleneck layer processing.

# Force Ollama to use only the faster GPUs
export CUDA_VISIBLE_DEVICES=0,1
export OLLAMA_NUM_GPU=2
ollama serve

For heterogeneous clusters, limiting OLLAMA_NUM_GPU to your slowest GPU count often yields better results than including all devices. A four-GPU setup with one underperforming card may run faster as a three-GPU configuration.

Memory Pressure Management

Monitor VRAM usage during inference to identify imbalances. Use nvidia-smi in a watch loop to track per-GPU memory consumption:

watch -n 1 nvidia-smi --query-gpu=index,memory.used,memory.total --format=csv

If one GPU consistently maxes out while others remain underutilized, the model exceeds optimal size for your configuration. Drop to a smaller quantization or reduce OLLAMA_NUM_GPU to prevent thrashing.

Performance Benchmarks: 1 vs 2 vs 4 GPU Configurations

Testing multi-GPU configurations reveals significant differences in throughput and response latency. Real-world benchmarks using llama3.1:70b on identical AMD Radeon RX 7900 XTX cards show how OLLAMA_NUM_GPU affects performance.

With OLLAMA_NUM_GPU=1, a 70B parameter model generates approximately 12 tokens per second with 48GB VRAM fully utilized. Model loading takes 45-60 seconds as weights transfer from system RAM. This configuration works well for single-user scenarios but struggles under concurrent requests.

OLLAMA_NUM_GPU=1 ollama serve
# Single user: smooth experience
# 3+ concurrent users: noticeable queuing delays

Dual GPU Configuration

Setting OLLAMA_NUM_GPU=2 distributes model layers across both cards. Token generation speed improves noticeably, reaching 20-22 tokens per second. More importantly, concurrent request handling improves dramatically – two users can query simultaneously without the severe slowdowns seen in single-GPU setups.

OLLAMA_NUM_GPU=2 OLLAMA_HOST=0.0.0.0:11434 ollama serve

Memory distribution becomes visible through rocm-smi monitoring. Each GPU typically holds 35-40GB of model weights, leaving headroom for context windows.

Quad GPU Scaling

Four GPUs show diminishing returns for single requests but excel at parallel workloads. Token generation reaches 28-32 tokens per second – not a linear 4x improvement due to inter-GPU communication overhead. The real advantage emerges when running multiple models simultaneously or handling 6-8 concurrent users without degradation.

OLLAMA_NUM_GPU=4 ollama serve
# Run multiple models: llama3.1:70b + codellama:34b

Memory per GPU drops to 18-22GB for 70B models, enabling smaller cards to participate. Watch for uneven distribution if cards have different VRAM capacities – Ollama attempts balanced allocation but may favor larger cards for initial layers.

Troubleshooting GPU Detection and Memory Issues

When Ollama fails to detect all GPUs or reports memory allocation errors, start by verifying your system sees the hardware correctly. Run nvidia-smi or rocm-smi depending on your GPU vendor to confirm all devices appear with their full VRAM capacity. If a GPU is missing, check PCIe slot connections and power cables before troubleshooting software.

Check which GPUs Ollama recognizes by examining its startup logs. Run Ollama in the foreground to see initialization messages:

OLLAMA_NUM_GPU=4 ollama serve

Look for lines indicating GPU discovery. If Ollama only detects fewer GPUs than expected, verify your driver installation supports all cards. Mixed GPU models sometimes cause detection failures – ensure all GPUs use compatible driver versions.

Memory Allocation Conflicts

Out-of-memory errors during model loading typically indicate VRAM fragmentation across GPUs. Ollama distributes model layers based on available memory, but existing processes can block allocation. Kill competing GPU workloads:

sudo fuser -k /dev/nvidia*

For persistent memory issues, reduce concurrent model instances. Running multiple models simultaneously exhausts VRAM faster than sequential loading. Set OLLAMA_NUM_GPU to match your actual free VRAM capacity rather than physical GPU count.

Driver and Runtime Mismatches

CUDA toolkit version mismatches cause silent GPU detection failures. Verify your installed CUDA version matches Ollama’s requirements:

nvcc --version
nvidia-smi | grep "CUDA Version"

For ROCm systems, check that rocminfo reports all GPUs with compute capability enabled. Outdated ROCm versions frequently fail to initialize secondary GPUs in multi-card setups.

Caution: AI-generated diagnostic scripts may suggest commands that reset GPU state or modify driver configurations. Always verify such commands against official documentation before running them on production systems. Test configuration changes with small models like llama3.2:1b before loading larger models that consume substantial VRAM.

Optimal Memory Allocation Patterns

When distributing model layers across multiple GPUs, memory allocation becomes the primary bottleneck for inference performance. Ollama automatically splits model layers based on available VRAM, but understanding the allocation pattern helps you choose appropriate models for your hardware configuration.

Ollama divides transformer layers sequentially across GPUs specified by OLLAMA_NUM_GPU. For a 70B parameter model with 80 layers running on two GPUs, Ollama typically allocates 40 layers to each device. The allocation happens at model load time, not dynamically during inference.

Check actual memory usage after loading a model:

nvidia-smi --query-gpu=index,memory.used,memory.total --format=csv

For AMD GPUs using ROCm:

rocm-smi --showmeminfo vram

Avoiding Memory Fragmentation

The most common allocation mistake is running models that exceed total VRAM when accounting for context windows. A 34B model might fit in 24GB during initial load but fail when processing long contexts. Reserve at least 20% of total VRAM for KV cache and temporary tensors.

Test memory headroom before production deployment:

export OLLAMA_NUM_GPU=2
ollama run llama3.1:70b "Generate a 4000 word technical document about distributed systems"

Monitor both GPUs during generation. If one GPU maxes out while others remain underutilized, the model size exceeds optimal distribution for your configuration.

Balancing Unequal VRAM

Systems with mixed GPU memory capacities require manual testing. Set OLLAMA_NUM_GPU to match your smallest GPU count, then verify layer distribution doesn’t overflow lower-capacity cards. Ollama does not currently support asymmetric layer allocation – it assumes equal VRAM across all GPUs.

For a system with one 24GB and one 16GB GPU, limit models to sizes that fit the 16GB constraint when split evenly, typically 30B parameter models or smaller with 4-bit quantization.

Installation and Configuration Steps

Start with a clean Ollama installation on your multi-GPU system. The official installer handles driver detection automatically:

curl -fsSL https://ollama.com/install.sh | sh

Verify GPU detection immediately after installation:

ollama serve &
sleep 5
nvidia-smi

Check that all GPUs appear in nvidia-smi output. If any GPU shows zero processes, driver issues may prevent Ollama from accessing that device.

Configuring OLLAMA_NUM_GPU

The OLLAMA_NUM_GPU environment variable controls how many GPUs Ollama distributes model layers across. Set this before starting the Ollama service:

export OLLAMA_NUM_GPU=2
ollama serve

For persistent configuration, create a systemd override:

sudo mkdir -p /etc/systemd/system/ollama.service.d
sudo tee /etc/systemd/system/ollama.service.d/override.conf << EOF
[Service]
Environment="OLLAMA_NUM_GPU=4"
Environment="OLLAMA_HOST=0.0.0.0:11434"
EOF
sudo systemctl daemon-reload
sudo systemctl restart ollama

Validation Testing

Pull a large model to test multi-GPU distribution:

ollama pull llama3.1:70b
ollama run llama3.1:70b "Explain quantum computing"

Monitor GPU memory allocation during inference:

watch -n 1 nvidia-smi

You should see memory usage distributed across the number of GPUs specified in OLLAMA_NUM_GPU. Uneven distribution often indicates PCIe bandwidth bottlenecks or NVLink topology issues.

Caution: Always validate GPU memory capacity before setting OLLAMA_NUM_GPU. A 70B parameter model requires approximately 40GB VRAM at 4-bit quantization. Setting OLLAMA_NUM_GPU=4 with four 12GB GPUs will cause out-of-memory errors during model loading.