Multi-Gpu

TL;DR llama.cpp supports heterogeneous multi-GPU configurations, letting you mix NVIDIA, AMD, and even Intel Arc cards in the same system for local LLM inference. Unlike Ollama’s automatic GPU detection, llama.cpp requires explicit layer distribution using the -ngl flag combined with --split-mode and --tensor-split parameters. This gives you fine-grained control over which layers run on which card, essential when mixing a high-VRAM card with lower-capacity GPUs. ...

TL;DR The OLLAMA_NUM_GPU environment variable controls how many GPUs Ollama uses for inference, but setting it correctly requires understanding your hardware topology and workload patterns. Unlike single-GPU setups where Ollama auto-detects your card, multi-GPU configurations demand explicit tuning to avoid memory fragmentation and PCIe bottlenecks. Set OLLAMA_NUM_GPU=2 to split model layers across two GPUs, or OLLAMA_NUM_GPU=4 for quad-GPU systems. Ollama distributes transformer layers sequentially – GPU 0 handles the first N layers, GPU 1 takes the next batch, and so on. This differs from data parallelism where each GPU processes different prompts simultaneously. ...

TL;DR This guide shows you how to build a peer-to-peer GPU sharing network using Go-based tools that let idle machines serve AI inference requests across your local network or homelab. Instead of leaving GPUs idle on workstations overnight, you can pool them into a distributed inference cluster that routes requests to available hardware. ...

TL;DR MegaTrain represents a breakthrough in democratizing large language model training by enabling full-precision training of models exceeding 100 billion parameters on consumer-grade hardware without cloud dependencies. Traditional training approaches require expensive GPU clusters with hundreds of gigabytes of VRAM, but MegaTrain employs aggressive memory optimization techniques including gradient checkpointing, CPU offloading, and dynamic tensor swapping to fit massive models into systems with as little as 24GB of VRAM. The framework integrates seamlessly with local AI stacks like Ollama and LM Studio, allowing you to train custom models on your own hardware while maintaining complete data privacy. Unlike cloud-based training services that charge recurring fees and expose your training data to third parties, MegaTrain runs entirely on your infrastructure using standard PyTorch backends. The system achieves this through a combination of mixed-precision computation scheduling, intelligent layer freezing, and memory-mapped parameter storage that keeps most weights on NVMe drives while actively training only small subsets in GPU memory. For homelab operators and privacy-focused teams, this means you can fine-tune models like Llama 3 70B or Mixtral 8x22B using your existing hardware setup without compromising on training quality or sending proprietary data off-premises. The framework supports distributed training across multiple consumer GPUs using standard networking, so you can scale from a single RTX 4090 to a cluster of gaming cards as your needs grow. MegaTrain outputs standard safetensors and GGUF formats compatible with llama.cpp and Open WebUI, ensuring your trained models integrate directly into your existing local AI deployment pipeline without conversion headaches. ...

Multi-GPU Ollama Setup: Running 70B Models on Dual GPUs TL;DR A single 24GB GPU cannot run a 70B parameter LLM. The model requires approximately 40GB of VRAM at Q4 quantization. Two GPUs solve this by splitting the model across both cards. This guide covers the hardware, configuration, and performance expectations for running 70B models on dual RTX 3090s with Ollama. ...