KoboldCpp Quick Start: Run GGUF Models with One Binary

TL;DR

# Download the latest release (Linux, CUDA)
wget https://github.com/LostRuins/koboldcpp/releases/latest/download/koboldcpp-linux-x64-cuda1150
chmod +x koboldcpp-linux-x64-cuda1150

# Run a GGUF model
./koboldcpp-linux-x64-cuda1150 --model llama-3.1-8b-instruct.Q4_K_M.gguf \
  --gpulayers 99 --contextsize 4096 --port 5001

# Web UI opens at http://localhost:5001
# KoboldAI API at http://localhost:5001/api/
# OpenAI-compatible API at http://localhost:5001/v1/chat/completions

Caution: KoboldCpp binds to localhost by default. If you use --host 0.0.0.0 to allow network access, there is no built-in authentication. Restrict access with firewall rules or a reverse proxy.

What Is KoboldCpp

KoboldCpp is a self-contained inference engine for GGUF models. It is a fork of llama.cpp packaged as a single executable with a built-in web interface. No Python environment, no Docker, no package manager – download one binary, point it at a GGUF file, and start generating text.

The project originated from the KoboldAI community (focused on creative writing and roleplay with LLMs) but has evolved into a general-purpose local inference tool. It competes directly with Ollama and llama.cpp server for the “simplest way to run a GGUF model” niche.

Key characteristics:

• Single binary: one file, no dependencies beyond GPU drivers
• Multi-backend GPU support: CUDA, ROCm (AMD), Vulkan, CLBlast
• Built-in web UI: chat interface included, no separate frontend needed
• Dual API: both KoboldAI-native and OpenAI-compatible endpoints
• GUI launcher: optional graphical configuration window on supported systems

Download and Run

Getting the Binary

Go to the KoboldCpp releases page and download the build matching your hardware.

# Download (example: CUDA 11.5)
wget https://github.com/LostRuins/koboldcpp/releases/latest/download/koboldcpp-linux-x64-cuda1150
chmod +x koboldcpp-linux-x64-cuda1150

Getting a Model

Download any GGUF model from Hugging Face. Q4_K_M quantization is the standard starting point for quality/size balance.

# Example: download Llama 3.1 8B Instruct Q4_K_M
wget https://huggingface.co/bartowski/Meta-Llama-3.1-8B-Instruct-GGUF/resolve/main/Meta-Llama-3.1-8B-Instruct-Q4_K_M.gguf

Launching

./koboldcpp-linux-x64-cuda1150 \
  --model Meta-Llama-3.1-8B-Instruct-Q4_K_M.gguf \
  --gpulayers 99 \
  --contextsize 4096 \
  --port 5001

The server starts, loads the model, and opens the web UI. You can also launch with --config to use a saved configuration file, or run with no arguments to get the GUI launcher (on systems with a display).

GUI Launcher

If you run the binary with no arguments on a system with a graphical environment, KoboldCpp opens a configuration window where you can:

• Browse for the GGUF file
• Set GPU layers, context size, and threads
• Select the GPU backend
• Configure network settings
• Save/load launch configurations

This is useful for desktop users who prefer not to work with CLI flags.

GPU Acceleration

KoboldCpp supports multiple GPU acceleration backends. The right choice depends on your hardware.

CUDA (NVIDIA)

The fastest option for NVIDIA GPUs. Use the CUDA build matching your driver version.

# Check your CUDA version
nvidia-smi | head -3

# Full GPU offload
./koboldcpp-linux-x64-cuda1150 \
  --model model.gguf \
  --gpulayers 99 \
  --usecublas normal

The --usecublas flag enables cuBLAS matrix multiplication. Options are normal (balanced) and mmq (uses custom quantized matrix multiplication kernels, sometimes faster for quantized models).

Vulkan (Cross-Platform)

Vulkan works on NVIDIA, AMD, and Intel GPUs. It is slower than CUDA on NVIDIA hardware but provides broad compatibility.

./koboldcpp-linux-x64-vulkan \
  --model model.gguf \
  --gpulayers 99 \
  --usevulkan

Vulkan is the recommended choice for AMD GPUs on systems where ROCm is not available or difficult to install.

ROCm (AMD)

Native AMD GPU acceleration. Requires ROCm drivers installed on the host system.

./koboldcpp-linux-x64-rocm \
  --model model.gguf \
  --gpulayers 99

CLBlast (Legacy)

OpenCL-based acceleration. Slower than CUDA and Vulkan but works on older hardware. Use only if other backends are not available.

./koboldcpp-linux-x64 \
  --model model.gguf \
  --gpulayers 99 \
  --useclblast 0 0  # platform_id device_id

Partial GPU Offloading

If your model does not fit entirely in VRAM, reduce --gpulayers to offload only some layers to the GPU. The remaining layers run on CPU.

# Check model layer count (printed during load)
# Then set gpulayers to what fits in your VRAM

# Example: 32-layer model, only 20 fit in 8GB VRAM
./koboldcpp-linux-x64-cuda1150 \
  --model model.gguf \
  --gpulayers 20 \
  --contextsize 2048 \
  --threads 8

Rule of thumb for GGUF Q4_K_M models: each layer of a 7B model uses roughly 200-250MB of VRAM. A 32-layer 7B model at Q4_K_M needs about 5-6GB for full GPU offload, plus 0.5-2GB for context (depending on --contextsize).

The Built-In Web UI

KoboldCpp includes a web interface at the server’s root URL (e.g., http://localhost:5001). It provides:

• Chat mode: multi-turn conversation with system prompt support
• Story mode: the original KoboldAI creative writing interface
• Instruct mode: send raw instruct-formatted prompts
• Settings panel: temperature, top-p, top-k, repetition penalty, min-p, typical-p
• Context viewer: inspect the full prompt being sent to the model
• Token counter: real-time token count for input and output

The UI is functional but minimal compared to Open WebUI or text-generation-webui. It works well for quick testing and single-user chat. For a more polished interface, connect Open WebUI to KoboldCpp’s API endpoint.

API Compatibility

KoboldCpp exposes two API styles simultaneously.

OpenAI-Compatible API

curl http://localhost:5001/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "koboldcpp",
    "messages": [
      {"role": "system", "content": "You are a helpful assistant."},
      {"role": "user", "content": "How do I list open ports on Linux?"}
    ],
    "max_tokens": 256,
    "temperature": 0.7
  }'

This endpoint works with any OpenAI client library:

from openai import OpenAI

client = OpenAI(base_url="http://localhost:5001/v1", api_key="not-needed")
response = client.chat.completions.create(
    model="koboldcpp",
    messages=[{"role": "user", "content": "Explain systemd timers."}]
)
print(response.choices[0].message.content)

KoboldAI Native API

The original KoboldAI API at /api/v1/generate:

curl http://localhost:5001/api/v1/generate \
  -H "Content-Type: application/json" \
  -d '{
    "prompt": "The three most important Linux commands are",
    "max_length": 200,
    "temperature": 0.7,
    "top_p": 0.9
  }'

The native API is used by KoboldAI-compatible frontends like SillyTavern. The OpenAI-compatible API is preferred for general use.

API Endpoints Summary

When to Use KoboldCpp vs Ollama vs llama.cpp

All three run GGUF models locally. The differences are in packaging and focus.

Choose KoboldCpp when you want zero-dependency setup, need Vulkan GPU support, want a built-in web UI without running a separate frontend, or use SillyTavern/KoboldAI-compatible tools.

Choose Ollama when you want seamless model management (ollama pull, ollama list), plan to run multiple models that swap in and out, or want the simplest possible CLI experience.

Choose llama.cpp server when you want maximum control over inference parameters, need the absolute latest llama.cpp features, or are building custom tooling against the API.

Performance Characteristics

KoboldCpp’s inference speed is essentially identical to llama.cpp (it tracks the upstream project closely). Differences come from backend selection and configuration.

Typical generation speeds (Llama-3.1-8B Q4_K_M, single user):

Context length impact: Generation speed drops as context fills up. A 4096-token context is roughly 10-15% slower at full capacity compared to the start of a conversation. At 8192+ tokens, the slowdown becomes noticeable.

Recommended Launch Configuration

For a general-purpose setup on a system with 8-12GB VRAM:

./koboldcpp-linux-x64-cuda1150 \
  --model Meta-Llama-3.1-8B-Instruct-Q4_K_M.gguf \
  --gpulayers 99 \
  --contextsize 4096 \
  --threads $(nproc) \
  --usecublas normal \
  --flashattention \
  --port 5001

The --flashattention flag enables Flash Attention, which reduces memory usage for the KV cache and speeds up long-context generation. It is supported on CUDA and Vulkan backends.

Running as a Background Service

# Simple background launch with nohup
nohup ./koboldcpp-linux-x64-cuda1150 \
  --model model.gguf --gpulayers 99 --contextsize 4096 --port 5001 \
  > /var/log/koboldcpp.log 2>&1 &

# Or create a systemd service
cat <<'EOF' | sudo tee /etc/systemd/system/koboldcpp.service
[Unit]
Description=KoboldCpp Inference Server
After=network.target

[Service]
Type=simple
User=llm
WorkingDirectory=/home/llm
ExecStart=/home/llm/koboldcpp-linux-x64-cuda1150 \
  --model /home/llm/models/Meta-Llama-3.1-8B-Instruct-Q4_K_M.gguf \
  --gpulayers 99 --contextsize 4096 --port 5001
Restart=on-failure
RestartSec=10

[Install]
WantedBy=multi-user.target
EOF

sudo systemctl daemon-reload
sudo systemctl enable --now koboldcpp.service

KoboldCpp is the easiest path from “I have a GGUF file” to “I am chatting with a model.” No runtime installation, no configuration files, no container orchestration. Download, run, done. The tradeoff is that you manage model files yourself and get a simpler (not worse, just simpler) interface compared to text-generation-webui or Open WebUI. For many users, that simplicity is exactly the point.