Gpu

TL;DR llama.cpp remains the fastest way to run quantized LLMs locally in 2026, but choosing the right command-line flags makes the difference between a sluggish 2 tokens/second and a responsive 30+ tokens/second experience. This guide covers the essential flags you need for optimal performance on consumer hardware. The most impactful flags control resource allocation: --n-gpu-layers offloads model layers to your GPU (start with -ngl 35 for 8GB VRAM), --threads sets CPU cores for processing (use physical cores minus 2), and --ctx-size defines context window length (2048 for chat, 8192 for document analysis). Getting these three right solves most performance issues. ...

TL;DR llama.cpp supports heterogeneous multi-GPU configurations, letting you mix NVIDIA, AMD, and even Intel Arc cards in the same system for local LLM inference. Unlike Ollama’s automatic GPU detection, llama.cpp requires explicit layer distribution using the -ngl flag combined with --split-mode and --tensor-split parameters. This gives you fine-grained control over which layers run on which card, essential when mixing a high-VRAM card with lower-capacity GPUs. ...

TL;DR DeepSeek v4 runs locally through Ollama with Open WebUI providing a chat interface. This guide covers installation, model-specific configuration for DeepSeek’s extended context window, and performance tuning for the model’s unique reasoning architecture. Install Ollama first, then pull the DeepSeek v4 model: curl -fsSL https://ollama.com/install.sh | sh ollama pull deepseek-v4 DeepSeek v4 requires specific memory allocation due to its 128K token context window. Set OLLAMA_NUM_GPU to control GPU layer offloading – most systems benefit from full GPU utilization with this model’s architecture: ...

TL;DR The OLLAMA_NUM_GPU environment variable controls how many GPUs Ollama uses for inference, but setting it correctly requires understanding your hardware topology and workload patterns. Unlike single-GPU setups where Ollama auto-detects your card, multi-GPU configurations demand explicit tuning to avoid memory fragmentation and PCIe bottlenecks. Set OLLAMA_NUM_GPU=2 to split model layers across two GPUs, or OLLAMA_NUM_GPU=4 for quad-GPU systems. Ollama distributes transformer layers sequentially – GPU 0 handles the first N layers, GPU 1 takes the next batch, and so on. This differs from data parallelism where each GPU processes different prompts simultaneously. ...

TL;DR Deploying Ollama on Kubernetes transforms local AI inference into a production-grade service with horizontal scaling, persistent model storage, and service mesh integration. This guide covers container orchestration patterns specifically for LLM workloads running on self-hosted infrastructure. The core deployment uses StatefulSets rather than Deployments to maintain stable network identities and persistent volume claims for model storage. Each Ollama pod serves the REST API on port 11434 and requires GPU node affinity when using NVIDIA runtime. Configure OLLAMA_HOST=0.0.0.0:11434 to bind the service to all interfaces within the pod network, and set OLLAMA_MODELS=/models pointing to your PersistentVolume mount path. ...

TL;DR This guide walks you through deploying Stable Diffusion on your own Linux machine using ComfyUI and Automatic1111 (A1111), giving you complete control over your image generation pipeline without sending prompts or outputs to third-party services. You need an NVIDIA GPU with at least 6GB VRAM for basic operation. Cards like the RTX 3060 work well for standard 512x512 images, while RTX 4090 or A6000 cards handle larger resolutions and batch processing. AMD GPUs work through ROCm but require additional configuration. CPU generation is possible but extremely slow. ...

TL;DR Ollama now supports image generation models through its standard API on port 11434, letting you run Stable Diffusion and similar models entirely offline. Install Ollama with curl -fsSL https://ollama.com/install.sh | sh, then pull an image model like ollama pull stable-diffusion. Generate images by sending prompts to the same REST endpoint you use for text models – no separate services required. ...

TL;DR LM Studio is a desktop GUI application for running large language models locally on Ubuntu 2026. Unlike command-line tools, it provides a graphical interface for downloading models from Hugging Face and running them without sending data to external servers. The application includes a local OpenAI-compatible API server, making it useful for developers who want to test AI integrations privately. ...

TL;DR This guide shows you how to build a peer-to-peer GPU sharing network using Go-based tools that let idle machines serve AI inference requests across your local network or homelab. Instead of leaving GPUs idle on workstations overnight, you can pool them into a distributed inference cluster that routes requests to available hardware. ...

TL;DR GAIA (Generative AI Integration Architecture) is an open-source framework that lets you build autonomous AI agents running entirely on your local hardware using Ollama, LM Studio, or llama.cpp as the inference backend. Unlike cloud-based agent frameworks, GAIA keeps your data on-premises and gives you full control over model selection, resource allocation, and execution policies. ...