TL;DR MTP (Model Transfer Protocol) is an experimental feature in Ollama that enables direct model transfers between Ollama instances without re-downloading from the registry. Instead of each server pulling a 7GB model from ollama.com, one instance can push it directly to another over your local network. This matters for air-gapped environments, bandwidth-constrained deployments, and multi-node setups where you want consistent model versions across machines. ...
TL;DR llama.cpp remains the fastest way to run quantized LLMs locally in 2026, but choosing the right command-line flags makes the difference between a sluggish 2 tokens/second and a responsive 30+ tokens/second experience. This guide covers the essential flags you need for optimal performance on consumer hardware. The most impactful flags control resource allocation: --n-gpu-layers offloads model layers to your GPU (start with -ngl 35 for 8GB VRAM), --threads sets CPU cores for processing (use physical cores minus 2), and --ctx-size defines context window length (2048 for chat, 8192 for document analysis). Getting these three right solves most performance issues. ...
TL;DR Running Llama models on AMD GPUs requires ROCm-specific optimizations that differ significantly from NVIDIA CUDA workflows. This guide covers the complete setup for self-hosting Llama 2, Llama 3, and Code Llama variants on AMD hardware using ROCm 6.0+, with focus on memory management, compilation flags, and performance tuning that existing NVIDIA guides do not address. ...
TL;DR llama.cpp supports heterogeneous multi-GPU configurations, letting you mix NVIDIA, AMD, and even Intel Arc cards in the same system for local LLM inference. Unlike Ollama’s automatic GPU detection, llama.cpp requires explicit layer distribution using the -ngl flag combined with --split-mode and --tensor-split parameters. This gives you fine-grained control over which layers run on which card, essential when mixing a high-VRAM card with lower-capacity GPUs. ...
TL;DR When you switch between models in Ollama, the service unloads the current GGUF file from memory and loads the new one from disk. Large models like llama3.1:70b or mixtral:8x7b can exceed 40GB, causing sustained disk reads that pin your SSD at maximum utilization. This becomes especially problematic when multiple users or applications trigger rapid model switches, creating a cascade of disk I/O that degrades system responsiveness. ...
TL;DR DeepSeek v4 runs locally through Ollama with Open WebUI providing a chat interface. This guide covers installation, model-specific configuration for DeepSeek’s extended context window, and performance tuning for the model’s unique reasoning architecture. Install Ollama first, then pull the DeepSeek v4 model: curl -fsSL https://ollama.com/install.sh | sh ollama pull deepseek-v4 DeepSeek v4 requires specific memory allocation due to its 128K token context window. Set OLLAMA_NUM_GPU to control GPU layer offloading – most systems benefit from full GPU utilization with this model’s architecture: ...
TL;DR The OLLAMA_NUM_GPU environment variable controls how many GPUs Ollama uses for inference, but setting it correctly requires understanding your hardware topology and workload patterns. Unlike single-GPU setups where Ollama auto-detects your card, multi-GPU configurations demand explicit tuning to avoid memory fragmentation and PCIe bottlenecks. Set OLLAMA_NUM_GPU=2 to split model layers across two GPUs, or OLLAMA_NUM_GPU=4 for quad-GPU systems. Ollama distributes transformer layers sequentially – GPU 0 handles the first N layers, GPU 1 takes the next batch, and so on. This differs from data parallelism where each GPU processes different prompts simultaneously. ...
TL;DR llama.cpp now handles inverse kinematics calculations through specialized GGUF models that generate joint angles and motion paths for robotic systems. You run llama-server with an IK-trained model, send it target positions as JSON prompts, and receive executable motion commands. This works entirely offline without cloud dependencies. The typical workflow involves loading a quantized IK model (Q4_K_M or Q5_K_M recommended for speed), sending coordinate targets through the OpenAI-compatible HTTP API, and parsing the structured output into robot control commands. Models like CodeLlama-IK and specialized Llama variants trained on robotics datasets handle 6-DOF arm calculations, path planning with obstacle avoidance, and real-time trajectory adjustments. ...
TL;DR Deploying Ollama on Kubernetes transforms local AI inference into a production-grade service with horizontal scaling, persistent model storage, and service mesh integration. This guide covers container orchestration patterns specifically for LLM workloads running on self-hosted infrastructure. The core deployment uses StatefulSets rather than Deployments to maintain stable network identities and persistent volume claims for model storage. Each Ollama pod serves the REST API on port 11434 and requires GPU node affinity when using NVIDIA runtime. Configure OLLAMA_HOST=0.0.0.0:11434 to bind the service to all interfaces within the pod network, and set OLLAMA_MODELS=/models pointing to your PersistentVolume mount path. ...
TL;DR Ollama Cloud offers managed hosting with zero infrastructure overhead, while local self-hosting gives you complete control and predictable costs after initial hardware investment. The decision hinges on your request volume, data sensitivity requirements, and whether you already own suitable hardware. For teams processing fewer than several thousand requests daily, Ollama Cloud eliminates the need to manage GPU servers, handle model updates, or troubleshoot CUDA driver conflicts. You pay per API call without worrying about idle capacity. Local hosting becomes cost-effective when you have consistent high-volume workloads that would generate substantial API bills – think continuous document processing pipelines or customer service chatbots handling hundreds of concurrent sessions. ...
