Local AI on Apple Silicon: Optimizing Ollama for M-Series Macs

TL;DR

# Install Ollama on macOS
brew install ollama
# Or download from https://ollama.com

# Start the server
ollama serve &

# Pull and run a model
ollama pull llama3.1:8b
ollama run llama3.1:8b

# Check Metal GPU utilization
sudo powermetrics --samplers gpu_power -i 1000 -n 1

Apple Silicon’s unified memory means your entire RAM pool is available as VRAM. An M1 with 16 GB can comfortably run 7B-13B models. An M3 Max with 96 GB can run 70B models at interactive speeds. Ollama uses Metal acceleration automatically – no configuration required.

Why Apple Silicon Is Good for Local AI

Apple Silicon has an architectural advantage for LLM inference that is not immediately obvious: unified memory. On traditional systems, a CPU has system RAM and a GPU has separate VRAM. Moving data between them is slow and creates bottlenecks. On Apple Silicon, the CPU, GPU, and Neural Engine all share the same memory pool.

For LLM inference, this means:

• No VRAM limit separate from RAM. A Mac with 64 GB of unified memory can load a model that would require a 64 GB GPU on a traditional system. An NVIDIA RTX 4090 tops out at 24 GB VRAM.
• No CPU-to-GPU memory transfer overhead. The model weights sit in memory once and both CPU and GPU access them directly.
• Metal GPU acceleration. Ollama uses Apple’s Metal framework for GPU-accelerated inference. The GPU cores handle matrix operations while the CPU handles other parts of the inference pipeline.

The tradeoff: Apple Silicon GPU cores are slower per-core than NVIDIA CUDA cores for this workload. A Mac compensates with memory capacity, not raw throughput. You can run bigger models, but at lower tokens/sec compared to an equivalent-cost NVIDIA setup.

Installation

Homebrew

brew install ollama

Direct Download

Download the macOS app from ollama.com. The app installs the ollama CLI and runs a menu bar service that starts the server automatically.

Verify Metal Support

After installation, confirm Ollama detects your GPU:

ollama run llama3.1:8b "hello" --verbose 2>&1 | grep -i "metal\|gpu"

Ollama uses Metal by default on Apple Silicon. There is no manual GPU configuration needed.

Chip Comparison and Model Capacity

Apple Silicon comes in a wide range of configurations. The key variable for LLM inference is unified memory – it determines the largest model you can load.

Apple Silicon Chip Reference

Key insight: Memory bandwidth correlates more strongly with tokens/sec than GPU core count. The jump from base M-series to Pro/Max doubles or quadruples bandwidth, which directly translates to faster token generation. This is because LLM inference is memory-bandwidth-bound, not compute-bound.

Model Sizing Rule of Thumb

For Q4_K_M quantized models (Ollama’s default):

• A model with N billion parameters requires approximately N x 0.6 GB of memory
• Add 1-2 GB overhead for context and KV cache
• Leave at least 4 GB free for macOS and applications

Examples:

• 7B model: ~4.2 GB + 2 GB overhead = ~6 GB needed (works on 8 GB machines, tight)
• 13B model: ~7.8 GB + 2 GB overhead = ~10 GB needed (needs 16 GB)
• 34B model: ~20 GB + 3 GB overhead = ~23 GB needed (needs 32 GB)
• 70B model: ~42 GB + 4 GB overhead = ~46 GB needed (needs 64 GB)

Performance Optimization

1. Close Memory-Hungry Applications

Unified memory is shared between macOS, applications, and the model. Every GB used by Chrome or Docker is a GB unavailable for inference. Before running large models:

# Check current memory pressure
memory_pressure
# or
vm_stat | head -10

If you are running a model that fills most of available memory, close browsers, Docker, and other heavy applications.

2. Set Context Size Appropriately

Larger context windows use more memory. If you do not need a 128K context window, reduce it:

# Default context is model-dependent. Override for lower memory use:
ollama run llama3.1:8b --ctx-size 4096

Or set it in a Modelfile:

FROM llama3.1:8b
PARAMETER num_ctx 4096

3. Keep Models Loaded

Model loading takes 5-15 seconds depending on size. Keep frequently-used models in memory:

# Keep loaded indefinitely (until manually unloaded)
export OLLAMA_KEEP_ALIVE=-1

# Or set via launchd for the service
# ~/Library/LaunchAgents/com.ollama.server.plist

Check what is loaded:

ollama ps

4. Use the Right Quantization

For Apple Silicon, Q4_K_M (the default) offers the best balance. Higher quantizations like Q8_0 produce marginally better output but use significantly more memory:

On an M1/M2 with 16 GB, the memory savings of Q4_K_M allow you to run a 13B model instead of an 8B Q8_0 – the 13B Q4 will produce better results than the 8B Q8 in most benchmarks.

5. Monitor GPU Utilization

Verify that Metal acceleration is active:

# Check GPU activity
sudo powermetrics --samplers gpu_power -i 2000 -n 3

# Watch in real time with Activity Monitor:
# Open Activity Monitor > Window > GPU History

During inference, GPU utilization should be high (60-90%). If it stays near 0%, Metal acceleration is not working – reinstall Ollama.

Comparison with NVIDIA GPU Setups

The NVIDIA setup wins on raw speed for models that fit in VRAM. Apple Silicon wins on:

• Running models too large for any single consumer GPU
• Power efficiency (10x less power per token for large models)
• Silence (no fans at full speed)
• Integrated system (no separate GPU, PSU, case)

For a developer who wants to run 70B models locally, a 96 GB Mac is currently the most practical consumer option. The only NVIDIA alternative would be dual RTX 3090s with NVLink or an A100, both requiring a dedicated workstation.

Running Open WebUI on Mac

Open WebUI provides a browser-based chat interface for Ollama. On macOS:

# Using Docker
docker run -d --name open-webui \
  -p 3000:8080 \
  -v open-webui:/app/backend/data \
  -e OLLAMA_BASE_URL=http://host.docker.internal:11434 \
  ghcr.io/open-webui/open-webui:main

# Or using pip (no Docker needed)
pip install open-webui
open-webui serve --port 3000

Access at http://localhost:3000. Open WebUI auto-discovers models from Ollama and provides conversation history, model switching, and document upload for RAG.

Note on Docker memory: Docker Desktop on macOS runs in a Linux VM with a separate memory allocation. By default, it gets 8 GB. If you run Open WebUI in Docker, that memory is taken from the pool available to Ollama. Running Open WebUI via pip avoids this overhead.

macOS-Specific Considerations

Memory Pressure and Swap

macOS aggressively uses swap (written to the SSD). When a model exceeds available memory, macOS does not crash – it swaps. This makes the model “work” but at dramatically reduced speed because SSD bandwidth (~5 GB/s) is far lower than memory bandwidth (100-800 GB/s).

Watch for memory pressure:

# Should show "System-wide memory free percentage: X%"
memory_pressure

# Check swap usage
sysctl vm.swapusage

If swap is being used during inference, the model is too large. Use a smaller model or a lower quantization.

Energy Efficiency

Apple Silicon is remarkably power-efficient for inference. Running a 13B model on an M2 Pro laptop draws roughly 25-30W total system power. Compare that to 300-450W for an NVIDIA GPU setup. For always-on local AI services, the Mac pays for itself in electricity savings over time.

Thermal Management

MacBooks thermal-throttle under sustained load. Inference on large models can sustain high temperatures for minutes or hours. For long batch processing jobs:

• Use a laptop stand that allows airflow underneath
• Keep ambient temperature reasonable
• Consider a Mac Mini or Mac Studio for always-on inference workloads – they have better thermal headroom than laptops

# Monitor CPU temperature
sudo powermetrics --samplers smc -i 5000 -n 1 | grep -i "die temp"

Troubleshooting

Model loads but inference is very slow: Check sysctl vm.swapusage. If swap is active, the model is too large. Switch to a smaller model or lower quantization.

“Metal not available” or GPU not used: Ensure you are running a native ARM64 build of Ollama, not an x86 build under Rosetta. Check: file $(which ollama) should show arm64.

Ollama server not starting: Check if another instance is running: pkill ollama && ollama serve. Also verify no port conflict on 11434: lsof -i :11434.

Slow first response after idle: Ollama unloads models after 5 minutes by default. Set OLLAMA_KEEP_ALIVE=-1 to keep models resident, or accept the 5-15 second reload time.

macOS firewall blocking access from other machines: If you want to access Ollama from other devices on your network, set OLLAMA_HOST=0.0.0.0 and allow incoming connections in System Settings > Network > Firewall.

Security note: If you set OLLAMA_HOST=0.0.0.0, Ollama accepts connections from any device on your network. Ollama has no authentication. Only do this on a trusted private network.