TL;DR

llama.cpp supports heterogeneous multi-GPU configurations, letting you mix NVIDIA, AMD, and even Intel Arc cards in the same system for local LLM inference. Unlike Ollama’s automatic GPU detection, llama.cpp requires explicit layer distribution using the -ngl flag combined with --split-mode and --tensor-split parameters. This gives you fine-grained control over which layers run on which card, essential when mixing a high-VRAM card with lower-capacity GPUs.

The key advantage: you can combine whatever hardware you already own. A typical homelab setup might pair an RTX 3090 (24GB) with an older GTX 1080 Ti (11GB), distributing model layers proportionally to available VRAM. llama.cpp handles the memory allocation and inter-GPU communication, though performance depends heavily on PCIe bandwidth and whether cards share the same vendor’s driver stack.

Configuration syntax differs significantly from Ollama. Instead of setting OLLAMA_NUM_GPU, you pass arguments directly to llama-server:

./llama-server -m models/llama-2-13b.Q5_K_M.gguf \
  -ngl 43 \
  --split-mode row \
  --tensor-split 0.6,0.4

This offloads 43 layers across two GPUs, with the first GPU handling 60% of tensor operations and the second handling 40%. The split ratio should roughly match VRAM capacity ratios to avoid bottlenecks.

Mixed-vendor setups introduce complexity. NVIDIA and AMD cards require separate driver installations and may not achieve optimal memory transfer speeds. Practical testing shows matched GPUs from the same generation typically deliver better throughput than mismatched cards, even when total VRAM is identical. Memory allocation failures are common when tensor splits don’t align with actual layer memory requirements – start conservative and monitor GPU utilization with nvidia-smi or radeontop during inference.

Always validate configuration changes with test prompts before production deployment. AI-generated llama.cpp commands may suggest invalid split ratios or incompatible quantization levels for your specific hardware combination.

Understanding llama.cpp’s Heterogeneous GPU Architecture

llama.cpp handles mixed GPU configurations through its layer distribution system, which splits transformer layers across available compute devices regardless of vendor or model. Unlike frameworks that require matched hardware, llama.cpp’s architecture allows you to combine an NVIDIA RTX 3060 with an AMD RX 6700 XT, or pair a high-end card with older hardware in the same inference pipeline.

The engine detects available GPUs at runtime and assigns layers based on VRAM capacity rather than compute capability. When you specify -ngl 40 (number of GPU layers), llama.cpp distributes those 40 layers across detected devices proportionally to their memory. A system with 12GB and 8GB cards might place 24 layers on the first GPU and 16 on the second, though exact distribution depends on model architecture and available VRAM after overhead.

Each GPU maintains its own context buffer and KV cache allocation. For a 7B parameter model at Q4_K_M quantization, expect roughly 4GB base model size plus 1-2GB per GPU for runtime overhead. Mixed configurations require careful calculation – if your combined VRAM totals 20GB but one card only has 6GB, you cannot load a model requiring 8GB minimum per device.

The critical difference from Ollama’s OLLAMA_NUM_GPU approach: llama.cpp does not abstract GPU selection. You control layer distribution through -ngl and optionally -mg (main GPU selection) flags, but the engine handles cross-device tensor transfers automatically. This means you can run inference across an NVIDIA card for primary compute and an AMD card for overflow layers without manual tensor routing.

Caution: Verify GPU detection with llama-server --verbose before production deployment. AI-generated configuration commands may suggest invalid layer counts that exceed your actual VRAM capacity, causing out-of-memory crashes during inference.

Configuration Syntax for Mixed GPU Setups

llama.cpp uses the --split-mode and --tensor-split flags to control how model layers distribute across heterogeneous GPUs. Unlike Ollama’s single environment variable approach, llama.cpp requires explicit per-device memory allocation ratios at runtime.

The --split-mode flag determines distribution strategy. Use layer for most mixed GPU scenarios:

./llama-server -m mistral-7b-instruct-v0.2.Q5_K_M.gguf \
  --split-mode layer \
  --tensor-split 3,1 \
  -ngl 35

The --tensor-split ratio 3,1 allocates 75% of layers to GPU 0 and 25% to GPU 1. This syntax differs fundamentally from Ollama’s device count specification – you’re defining proportional memory allocation, not device selection.

Handling VRAM Disparities

When mixing a 12GB card with a 6GB card, match the ratio to available VRAM:

./llama-server -m llama-3.1-8b-instruct.Q4_K_M.gguf \
  --split-mode layer \
  --tensor-split 2,1 \
  -ngl 40 \
  --host 0.0.0.0 \
  --port 8080

The -ngl value (number of GPU layers) must not exceed what your combined VRAM supports. For Q4_K_M quantization, estimate roughly 0.5GB per billion parameters plus 2GB overhead. Start conservative and increase -ngl until you see CUDA out-of-memory errors.

Vendor Mixing Limitations

Mixing NVIDIA and AMD GPUs requires separate llama.cpp builds. The CUDA and ROCm backends cannot coexist in a single binary. For NVIDIA+AMD setups, run two llama-server instances on different ports, each compiled for its respective backend.

Caution: AI-generated configuration commands may suggest invalid --tensor-split syntax or incompatible flag combinations. Always verify flags against the official llama.cpp documentation before deploying to production inference servers. Test with small models first to validate your hardware configuration responds correctly.

Memory Allocation Across Dissimilar Cards

When running llama.cpp with mixed GPU hardware, memory allocation requires explicit configuration to prevent the inference engine from attempting to load more layers than your smallest card can handle. Unlike matched GPU setups where automatic distribution works reliably, heterogeneous configurations need manual layer splits.

Start by identifying VRAM capacity for each card. A system with an NVIDIA RTX 3060 (12GB) and an AMD RX 6600 (8GB) cannot split layers evenly. Use the --split-mode parameter with layer to control distribution:

./llama-server -m mistral-7b-instruct-v0.2.Q5_K_M.gguf \
  --split-mode layer \
  --tensor-split 60,40 \
  -ngl 35

The --tensor-split ratio (60,40) allocates proportionally more layers to the larger card. The -ngl 35 value offloads 35 layers total – llama.cpp distributes these according to your split ratio. For a 32-layer model, this puts approximately 21 layers on the 3060 and 14 on the 6600.

Handling VRAM Exhaustion

If llama-server crashes with CUDA out-of-memory errors, reduce -ngl incrementally by 5 layers until stable. Monitor actual VRAM usage with nvidia-smi or radeontop during inference. The context size (-c parameter) also impacts memory – a 4096 token context uses substantially more VRAM than 2048.

For mixed vendor setups (NVIDIA + AMD), compile llama.cpp with both CUDA and ROCm support enabled. Set CUDA_VISIBLE_DEVICES and HIP_VISIBLE_DEVICES to control which cards participate:

export CUDA_VISIBLE_DEVICES=0
export HIP_VISIBLE_DEVICES=1
./llama-server -m model.gguf --split-mode layer --tensor-split 55,45 -ngl 30

Caution: Always validate memory allocation commands in a test environment before production deployment. AI-generated configuration suggestions may not account for your specific driver versions or model architecture requirements.

Performance Benchmarks: Mixed vs Matched GPU Configurations

Testing llama.cpp with heterogeneous GPU setups reveals consistent patterns across different hardware combinations. A system pairing an NVIDIA RTX 3090 (24GB) with an RTX 3060 (12GB) demonstrates how llama.cpp distributes layers based on available VRAM rather than compute capability.

For a Llama-2-13B model at Q4_K_M quantization, the matched dual-3090 configuration processes prompts noticeably faster than the mixed 3090/3060 setup. The performance gap widens with larger context windows – 8K token contexts show more pronounced differences than 2K contexts. The bottleneck emerges during layer synchronization when the slower card must catch up before the next inference step begins.

Memory Allocation Efficiency

Mixed AMD and NVIDIA configurations introduce additional overhead. A system combining an AMD RX 7900 XTX with an NVIDIA RTX 4070 requires explicit layer splitting since llama.cpp cannot automatically balance across different driver stacks. You must manually specify layer counts with the -ngl parameter and monitor actual VRAM usage with nvidia-smi and radeontop.

# Force specific layer distribution
./llama-server -m llama-2-13b.Q4_K_M.gguf -ngl 28 -c 4096

The mixed AMD/NVIDIA setup typically shows reduced throughput compared to dual-NVIDIA configurations due to PCIe bandwidth competition and driver-level coordination overhead.

Practical Recommendations

Matched GPU pairs deliver more predictable performance. If building a new system, prioritize identical cards over mixing generations or vendors. For existing mixed setups, assign the faster GPU to handle more layers and reserve the slower card for overflow when context length exceeds the primary card’s capacity.

Caution: Always validate layer distribution with actual inference tests. AI-generated layer split recommendations may not account for your specific model size and quantization level. Monitor VRAM usage during initial runs to prevent out-of-memory errors.

Installation and Configuration Steps

Start by cloning the llama.cpp repository and building with GPU acceleration enabled. For mixed GPU setups, compile with both CUDA and ROCm support if you’re running NVIDIA and AMD cards together:

git clone https://github.com/ggerganov/llama.cpp
cd llama.cpp
mkdir build && cd build
cmake .. -DLLAMA_CUDA=ON -DLLAMA_CUBLAS=ON
cmake --build . --config Release

For AMD cards, replace the CUDA flags with -DLLAMA_HIPBLAS=ON. Systems with both vendors require separate builds or runtime library switching.

Configuring GPU Layer Distribution

The -ngl parameter controls how many model layers load onto GPU memory. With heterogeneous cards, llama.cpp distributes layers sequentially across available devices. Launch llama-server with explicit GPU selection:

./llama-server -m models/mistral-7b-instruct-v0.2.Q5_K_M.gguf \
  -ngl 35 \
  --split-mode layer \
  -t 8 \
  --port 8080

The --split-mode layer option ensures layer-wise distribution rather than tensor splitting. For a system with an RTX 3060 (12GB) and GTX 1660 Ti (6GB), allocate layers proportionally. A 7B model at Q5_K_M quantization uses roughly 5GB, so the 3060 can handle most layers while the 1660 Ti takes overflow.

Verifying GPU Utilization

Monitor actual GPU usage during inference with nvidia-smi or radeontop:

watch -n 1 nvidia-smi

Check llama-server startup logs for device detection. You should see both cards listed with their VRAM capacity. If one GPU shows zero utilization, reduce -ngl until both devices activate, or manually specify device IDs through CUDA_VISIBLE_DEVICES environment variable.

Caution: AI-generated configuration commands may suggest invalid parameter combinations. Always verify flags against the official llama.cpp documentation before production deployment, especially when mixing GPU architectures with different compute capabilities.

Verification and Testing

After configuring llama-server for heterogeneous GPU setups, verify that both cards are recognized and actively processing layers. Run llama-server with verbose logging to see device enumeration:

./llama-server --model models/mistral-7b-instruct-v0.2.Q5_K_M.gguf \
  --n-gpu-layers 35 \
  --split-mode row \
  --verbose

Check the startup output for lines showing VRAM allocation per device. You should see entries like “CUDA0: 8192 MB” and “CUDA1: 6144 MB” reflecting your actual hardware. If only one GPU appears, verify your CUDA toolkit installation supports multi-device contexts.

Load Testing Mixed Configurations

Send concurrent requests to measure actual throughput improvements. Use a simple Python script to generate parallel inference calls:

import requests
import concurrent.futures

def send_prompt(text):
    response = requests.post('http://localhost:8080/v1/completions',
        json={'prompt': text, 'max_tokens': 100})
    return response.json()

prompts = ["Explain quantum computing"] * 10

with concurrent.futures.ThreadPoolExecutor(max_workers=5) as executor:
    results = list(executor.map(send_prompt, prompts))

Monitor GPU utilization with nvidia-smi during the test. Both cards should show active compute processes. Uneven utilization often indicates layer distribution issues – try adjusting –n-gpu-layers or switching to layer split mode.

Troubleshooting Memory Allocation Failures

When mixing cards with different VRAM capacities, llama-server may fail to allocate layers optimally. If you encounter “CUDA out of memory” errors despite having sufficient total VRAM, reduce –n-gpu-layers incrementally until stable. The smaller card becomes the bottleneck in row split mode.

For AMD and NVIDIA combinations, ensure ROCm and CUDA libraries coexist without conflicts. Build llama.cpp with both backends enabled, then specify devices explicitly using CUDA_VISIBLE_DEVICES and HIP_VISIBLE_DEVICES environment variables before launching llama-server.