LocalAI Setup: OpenAI API-Compatible Local Inference

TL;DR

# Docker (quickest start)
docker run -d --name localai -p 8080:8080 \
  -v localai-models:/build/models \
  localai/localai:latest-gpu-nvidia-cuda-12

# Install a model from the gallery
curl http://localhost:8080/models/apply -d '{"id": "llama-3.1-8b-instruct"}'

# Test chat completions (same as OpenAI API)
curl http://localhost:8080/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{"model": "llama-3.1-8b-instruct", "messages": [{"role": "user", "content": "Hello"}]}'

# Generate embeddings
curl http://localhost:8080/v1/embeddings \
  -H "Content-Type: application/json" \
  -d '{"model": "text-embedding-ada-002", "input": "The quick brown fox"}'

Caution: LocalAI has no built-in authentication. Any process that can reach port 8080 can use the API. Use firewall rules, bind to localhost only, or put a reverse proxy with auth in front before exposing to a network.

What Is LocalAI

LocalAI is a self-hosted API server that replicates the OpenAI API surface. It handles text generation, embeddings, image generation, audio transcription, and text-to-speech – all running locally without sending data to external services.

The core value proposition: if you have application code that calls the OpenAI API, you can point it at LocalAI by changing only the base URL. The request and response formats are identical. This makes LocalAI useful as a development proxy (test against local models before paying for API calls), a privacy layer (keep data on-premise), or a cost elimination tool (no per-token charges).

Under the hood, LocalAI wraps multiple inference backends. Text generation uses llama.cpp (for GGUF models) or Hugging Face Transformers. Embeddings use sentence-transformers or llama.cpp. Image generation uses Stable Diffusion. Audio uses Whisper. Each capability maps to the corresponding OpenAI API endpoint.

Installation

Docker (Recommended)

Docker is the primary installation method. LocalAI provides images for different hardware configurations.

NVIDIA GPU:

docker run -d --name localai \
  --gpus all \
  -p 8080:8080 \
  -v localai-models:/build/models \
  localai/localai:latest-gpu-nvidia-cuda-12

AMD GPU (ROCm):

docker run -d --name localai \
  --device /dev/kfd --device /dev/dri \
  -p 8080:8080 \
  -v localai-models:/build/models \
  localai/localai:latest-gpu-hipblas

CPU only:

docker run -d --name localai \
  -p 8080:8080 \
  -v localai-models:/build/models \
  localai/localai:latest-cpu

Docker Compose

# docker-compose.yml
services:
  localai:
    image: localai/localai:latest-gpu-nvidia-cuda-12
    deploy:
      resources:
        reservations:
          devices:
            - driver: nvidia
              count: all
              capabilities: [gpu]
    ports:
      - "8080:8080"
    volumes:
      - localai-models:/build/models
      - ./config:/build/config  # Custom model configs
    environment:
      - THREADS=8
      - CONTEXT_SIZE=4096
      - DEBUG=false
    restart: unless-stopped

volumes:
  localai-models:

docker compose up -d
docker compose logs -f localai

Binary Install

For bare-metal deployments without Docker:

# Download the latest release
curl -Lo localai https://github.com/mudler/LocalAI/releases/latest/download/local-ai-linux-amd64-cuda12

chmod +x localai

# Run
./localai --models-path ./models --address :8080 --threads 8

The binary is a statically compiled Go executable. It bundles the llama.cpp inference engine and other backends. GPU support depends on which binary variant you download (CUDA, ROCm, or CPU).

Loading Models

Model Gallery

LocalAI includes a model gallery – a curated index of pre-configured models that can be installed with a single API call.

# List available models
curl http://localhost:8080/models/available

# Install from gallery
curl http://localhost:8080/models/apply \
  -d '{"id": "llama-3.1-8b-instruct"}'

# Check installation progress
curl http://localhost:8080/models/jobs

Gallery models come with pre-configured YAML definitions that set context size, prompt templates, and backend options. This is the simplest way to get started.

Manual GGUF Models

Download any GGUF model and create a configuration file.

# Download model
wget -P models/ https://huggingface.co/bartowski/Meta-Llama-3.1-8B-Instruct-GGUF/resolve/main/Meta-Llama-3.1-8B-Instruct-Q4_K_M.gguf

Create a YAML config file in the models directory:

# models/llama-3.1-8b.yaml
name: llama-3.1-8b
backend: llama-cpp
parameters:
  model: Meta-Llama-3.1-8B-Instruct-Q4_K_M.gguf
  temperature: 0.7
  top_p: 0.9
context_size: 4096
gpu_layers: 99
threads: 8
template:
  chat_message: |
    <|start_header_id|>{{.RoleName}}<|end_header_id|>

    {{.Content}}<|eot_id|>
  chat: |
    <|begin_of_text|>{{.Input}}
    <|start_header_id|>assistant<|end_header_id|>

stopwords:
  - <|eot_id|>
  - <|end_of_text|>

Restart LocalAI or wait for it to detect the new config:

docker restart localai

# Verify the model is loaded
curl http://localhost:8080/v1/models

Hugging Face Models (Transformers Backend)

For models not available in GGUF format, LocalAI can load Hugging Face Transformers models directly. This requires more VRAM (FP16) but supports the full Hugging Face ecosystem.

# models/phi-3-mini.yaml
name: phi-3-mini
backend: transformers
parameters:
  model: microsoft/Phi-3-mini-4k-instruct
context_size: 4096
cuda: true

API Compatibility

LocalAI implements the following OpenAI API endpoints:

Chat Completions

curl http://localhost:8080/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "llama-3.1-8b",
    "messages": [
      {"role": "system", "content": "You are a Linux systems administrator."},
      {"role": "user", "content": "How do I set up fail2ban?"}
    ],
    "temperature": 0.7,
    "max_tokens": 512,
    "stream": true
  }'

Text Completions

curl http://localhost:8080/v1/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "llama-3.1-8b",
    "prompt": "The main advantage of container orchestration is",
    "max_tokens": 200
  }'

Embeddings

curl http://localhost:8080/v1/embeddings \
  -H "Content-Type: application/json" \
  -d '{
    "model": "text-embedding-ada-002",
    "input": "Kubernetes pod scheduling"
  }'

LocalAI maps the text-embedding-ada-002 model name to a local embedding model (e.g., all-MiniLM-L6-v2). You can configure this mapping in the model YAML.

Image Generation

curl http://localhost:8080/v1/images/generations \
  -H "Content-Type: application/json" \
  -d '{
    "model": "stablediffusion",
    "prompt": "A server room with blinking lights",
    "size": "512x512"
  }'

Requires a Stable Diffusion model installed. LocalAI uses the diffusers backend for image generation.

Audio Transcription

curl http://localhost:8080/v1/audio/transcriptions \
  -F file=@recording.wav \
  -F model=whisper-1

Uses OpenAI Whisper for speech-to-text. The whisper-1 model name maps to a local Whisper model.

Text-to-Speech

curl http://localhost:8080/v1/audio/speech \
  -H "Content-Type: application/json" \
  -d '{"model": "tts-1", "input": "Hello from LocalAI", "voice": "alloy"}' \
  --output speech.mp3

Full Endpoint Reference

Endpoint	OpenAI Equivalent	Backend
`/v1/chat/completions`	Chat	llama-cpp, transformers
`/v1/completions`	Completions	llama-cpp, transformers
`/v1/embeddings`	Embeddings	llama-cpp, sentence-transformers
`/v1/images/generations`	DALL-E	diffusers (Stable Diffusion)
`/v1/audio/transcriptions`	Whisper	whisper
`/v1/audio/speech`	TTS	piper, bark
`/v1/models`	Models list	–

GPU Acceleration

NVIDIA (CUDA)

GPU offloading is controlled per-model in the YAML config:

# models/my-model.yaml
gpu_layers: 99  # Offload all layers to GPU
f16: true       # Use FP16 on GPU
main_gpu: 0     # GPU index (for multi-GPU systems)

Monitor GPU usage:

watch -n 1 nvidia-smi

Multi-GPU

For models that span multiple GPUs:

gpu_layers: 99
tensor_split: "0.5,0.5"  # Split evenly across 2 GPUs

Memory Planning

Model Size	Quantization	VRAM Required	Recommended GPU
7-8B	Q4_K_M	5-6 GB	RTX 3060 12GB
7-8B	FP16	14-16 GB	RTX 4090 24GB
13B	Q4_K_M	8-10 GB	RTX 3090 24GB
70B	Q4_K_M	36-40 GB	2x RTX 3090 or A100

Add 1-3 GB for context (KV cache) depending on context_size setting.

Comparison with Ollama and vLLM

Feature	LocalAI	Ollama	vLLM
Primary goal	OpenAI API drop-in	Simple model runner	Max throughput
API coverage	Chat, embeddings, images, audio, TTS	Chat, embeddings	Chat, completions, embeddings
Model formats	GGUF, HF Transformers, diffusers	GGUF	HF Transformers, AWQ, GPTQ
Model management	Gallery + manual config	Built-in pull/list	Manual (HF Hub)
Image generation	Yes (Stable Diffusion)	No	No
Audio transcription	Yes (Whisper)	No	No
Text-to-speech	Yes (Piper, Bark)	No	No
Concurrent throughput	Moderate	Basic	High (PagedAttention)
GPU requirement	Optional	Optional	Required
Config approach	YAML per model	Modelfile	CLI flags

When to Use Each Tool

LocalAI is the right choice when you need to replace multiple OpenAI API endpoints (not just chat) with local inference. If your application uses embeddings, image generation, or audio transcription alongside text generation, LocalAI handles all of them behind a single API server. It is also useful as a development proxy – point your OpenAI client at LocalAI during development to avoid API costs, then switch to OpenAI for production.

Ollama is better for simple model management and single-purpose text generation. It handles model downloads, version management, and quantization with less configuration overhead than LocalAI. If you only need chat completions and embeddings, Ollama is simpler to operate.

vLLM wins on throughput for multi-user text generation. If your workload is exclusively text inference at high concurrency, vLLM’s PagedAttention and continuous batching outperform both LocalAI and Ollama. It does not handle images, audio, or other modalities.

Practical Configuration

Environment Variables

# Core settings
THREADS=8                # CPU threads for inference
CONTEXT_SIZE=4096        # Default context size
DEBUG=false              # Enable debug logging
CORS=true                # Enable CORS headers
GALLERIES='[{"name": "model-gallery", "url": "github:mudler/LocalAI/gallery/index.yaml"}]'

Running Multiple Models

LocalAI serves all configured models from a single instance. Each model gets its own YAML config file in the models directory. The model parameter in API requests routes to the correct backend.

models/
  llama-3.1-8b.yaml      # Text generation
  embedding-model.yaml    # Embeddings
  whisper.yaml            # Audio transcription
  stablediffusion.yaml    # Image generation

All models share the same port and base URL. The client specifies which model to use in the request body.

Python Client Example

from openai import OpenAI

client = OpenAI(
    base_url="http://localhost:8080/v1",
    api_key="not-needed"
)

# Chat completion
chat = client.chat.completions.create(
    model="llama-3.1-8b",
    messages=[{"role": "user", "content": "Explain RAID levels."}],
    temperature=0.5
)
print(chat.choices[0].message.content)

# Embedding
embedding = client.embeddings.create(
    model="text-embedding-ada-002",
    input="Linux kernel modules"
)
print(f"Embedding dimensions: {len(embedding.data[0].embedding)}")

# Image generation
image = client.images.generate(
    model="stablediffusion",
    prompt="A diagram of a microservices architecture",
    size="512x512"
)
print(image.data[0].url)

Health Monitoring

# Basic health check
curl http://localhost:8080/readyz

# List loaded models
curl http://localhost:8080/v1/models | python3 -m json.tool

# Prometheus metrics (if enabled)
curl http://localhost:8080/metrics

Systemd Service (Binary Install)

# /etc/systemd/system/localai.service
[Unit]
Description=LocalAI Inference Server
After=network.target

[Service]
Type=simple
User=localai
Environment=THREADS=8
Environment=CONTEXT_SIZE=4096
ExecStart=/usr/local/bin/localai --models-path /var/lib/localai/models --address 127.0.0.1:8080
Restart=on-failure
RestartSec=10

[Install]
WantedBy=multi-user.target

sudo systemctl daemon-reload
sudo systemctl enable --now localai.service

Limitations

Throughput: LocalAI is not optimized for high-concurrency serving. It uses llama.cpp under the hood for text generation, which processes requests sequentially. For multi-user production workloads, vLLM is a better choice.

Model configuration: Each model requires a YAML config file with correct prompt templates and settings. Getting these wrong produces garbled output. The model gallery avoids this problem for popular models, but custom configurations require trial and error.

Resource usage: Running multiple backends (text + images + audio) simultaneously consumes significant RAM and VRAM. Plan your resource allocation carefully if you enable all capabilities.

Update cadence: LocalAI tracks upstream projects (llama.cpp, whisper.cpp, stable-diffusion.cpp) but may lag behind their latest releases. If you need cutting-edge inference features, running the backends directly gives you faster access to updates.

LocalAI is the most comprehensive OpenAI API replacement available for local deployment. It trades some throughput and simplicity for breadth of functionality. If your application touches multiple OpenAI endpoints, LocalAI lets you consolidate them behind a single self-hosted server.

LocalAI Setup: OpenAI API-Compatible Local Inference#

TL;DR#

What Is LocalAI#

Installation#

Docker (Recommended)#

Docker Compose#

Binary Install#

Loading Models#

Model Gallery#

Manual GGUF Models#

Hugging Face Models (Transformers Backend)#

API Compatibility#

Chat Completions#

Text Completions#

Embeddings#

Image Generation#

Audio Transcription#

Text-to-Speech#

Full Endpoint Reference#

GPU Acceleration#

NVIDIA (CUDA)#

Multi-GPU#

Memory Planning#

Comparison with Ollama and vLLM#

When to Use Each Tool#

Practical Configuration#

Environment Variables#

Running Multiple Models#

Python Client Example#

Health Monitoring#

Systemd Service (Binary Install)#

Limitations#

Related Local AI Guides

can ollama models access the internet

TL;DR

How to Move Ollama Models to Another Drive in 2026

TL;DR

Odysseus: Complete Self-Hosted AI Workspace with Ollama

TL;DR

Running Llama.cpp with Inverse Kinematics AI Models in 2026

TL;DR

Mac Mini Local LLM Setup Guide: Ollama & Open WebUI 2026

TL;DR

LM Studio API Key Setup Guide for Local AI Models 2026

TL;DR

LocalAI Setup: OpenAI API-Compatible Local Inference

TL;DR

What Is LocalAI

Installation

Docker (Recommended)

Docker Compose

Binary Install

Loading Models

Model Gallery

Manual GGUF Models

Hugging Face Models (Transformers Backend)

API Compatibility

Chat Completions

Text Completions

Embeddings

Image Generation

Audio Transcription

Text-to-Speech

Full Endpoint Reference

GPU Acceleration

NVIDIA (CUDA)

Multi-GPU

Memory Planning

Comparison with Ollama and vLLM

When to Use Each Tool

Practical Configuration

Environment Variables

Running Multiple Models

Python Client Example

Health Monitoring

Systemd Service (Binary Install)

Limitations