Text Generation WebUI Setup Guide for Local LLM Inference

TL;DR

# Clone and run the one-click installer
git clone https://github.com/oobabooga/text-generation-webui
cd text-generation-webui
bash start_linux.sh  # Installs conda env, dependencies, launches UI

# Or use Docker
docker compose up -d

# Access the web interface
# http://localhost:7860

# Enable API mode
python server.py --api --listen
# API available at http://localhost:5000/v1/chat/completions

Caution: The web interface has no authentication by default. Do not use --listen (which binds to 0.0.0.0) on networks you do not control. Use --listen --api-key YOUR_SECRET if exposing the API, and put a reverse proxy with auth in front for production use.

What Is Text Generation WebUI

Text-generation-webui (commonly called “oobabooga” after its creator’s GitHub handle) is a Gradio-based web interface for running large language models locally. It supports nearly every model format in the local LLM ecosystem: GGUF (llama.cpp), GPTQ, ExLlamaV2, AWQ, HQQ, Transformers, and more.

The project fills a specific niche. Where Ollama optimizes for CLI simplicity and vLLM targets production throughput, text-generation-webui provides a feature-rich GUI for interactive model experimentation. You get a ChatGPT-like chat interface, a notebook mode for long-form generation, granular parameter control (temperature, top-p, top-k, repetition penalty, mirostat), and an extensions system for adding functionality like RAG, multimodal input, and character personas.

It is the Swiss Army knife of local LLM interfaces – not the fastest or simplest, but the most configurable.

Installation

One-Click Installer (Recommended)

The one-click installer handles conda environment creation, dependency installation, and GPU detection automatically.

git clone https://github.com/oobabooga/text-generation-webui
cd text-generation-webui

# Linux
bash start_linux.sh

# The installer will ask:
# 1. GPU type (NVIDIA, AMD, Intel, Apple M, CPU only)
# 2. Whether to use CUDA 12.1 (recommended for modern GPUs)

On first run, it creates a conda environment in installer_files/, downloads PyTorch with the appropriate CUDA version, and installs all Python dependencies. This takes 5-15 minutes depending on your connection. Subsequent launches reuse the existing environment.

The script launches the web UI at http://localhost:7860 when ready.

Manual Installation

For more control over the environment:

# Create conda environment
conda create -n textgen python=3.11
conda activate textgen

# Install PyTorch with CUDA 12.1
pip install torch torchvision torchaudio --index-url https://download.pytorch.org/whl/cu121

# Clone and install
git clone https://github.com/oobabooga/text-generation-webui
cd text-generation-webui
pip install -r requirements.txt

# Launch
python server.py

Docker Deployment

The project includes Docker Compose files for different GPU configurations.

cd text-generation-webui

# NVIDIA GPU
docker compose --profile default up -d

# AMD GPU (ROCm)
docker compose --profile amd up -d

# CPU only
docker compose --profile cpu up -d

The Docker setup mounts volumes for models, characters, presets, and extensions so your data persists between container recreations.

Custom Docker Compose for NVIDIA:

# docker-compose.yml
services:
  textgen:
    image: atinoda/text-generation-webui:default-nvidia
    deploy:
      resources:
        reservations:
          devices:
            - driver: nvidia
              count: all
              capabilities: [gpu]
    ports:
      - "7860:7860"  # Web UI
      - "5000:5000"  # API
    volumes:
      - ./models:/app/models
      - ./characters:/app/characters
      - ./loras:/app/loras
      - ./extensions:/app/extensions
    environment:
      - EXTRA_LAUNCH_ARGS=--api --verbose
    restart: unless-stopped

Loading Models

Downloading Models

The web UI has a built-in model downloader on the Model tab. Enter a Hugging Face model ID and click Download.

From the command line:

# Download a GGUF model
cd text-generation-webui/models
wget https://huggingface.co/TheBloke/Llama-2-7B-Chat-GGUF/resolve/main/llama-2-7b-chat.Q4_K_M.gguf

# Or use huggingface-cli for full repos
huggingface-cli download TheBloke/Mistral-7B-Instruct-v0.2-GPTQ \
  --local-dir models/Mistral-7B-Instruct-v0.2-GPTQ

Place model files in the models/ directory. The web UI auto-detects them on the Model tab.

Model Format Selection

Text-generation-webui auto-selects the appropriate loader based on model format, but you can override it.

For most users: GGUF Q4_K_M files with the llama.cpp loader offer the best balance of quality, speed, and memory efficiency. They also allow CPU offloading if your GPU VRAM is limited.

Loading a GGUF Model

1. Go to the Model tab
2. Select the model from the dropdown
3. Set loader to llama.cpp
4. Configure GPU layers (n-gpu-layers): set to 99 to offload everything to GPU, or a lower number for partial offload
5. Set context length (n_ctx): 4096 is a safe default
6. Click Load

Equivalent CLI launch:

python server.py --model llama-2-7b-chat.Q4_K_M.gguf \
  --loader llama.cpp \
  --n-gpu-layers 99 \
  --n_ctx 4096

Loading a GPTQ/AWQ Model

python server.py --model Mistral-7B-Instruct-v0.2-GPTQ \
  --loader exllamav2 \
  --max_seq_len 4096 \
  --gpu-memory 20  # Max VRAM in GB

ExLlamaV2 is significantly faster than the Transformers loader for quantized models. Always prefer it for GPTQ and AWQ formats when using NVIDIA GPUs.

Web Interface Features

Chat Mode

The default tab. Supports system prompts, multi-turn conversation, and streaming output. You can create and switch between character cards (JSON persona definitions) for different use cases.

Key parameters accessible from the sidebar:

• temperature: 0.1-2.0 (lower = more deterministic)
• top_p: nucleus sampling threshold
• top_k: limits token pool per step
• repetition_penalty: 1.0 = off, 1.1-1.3 typical range
• max_new_tokens: output length cap

Notebook Mode

A plain text editor for non-conversational generation. Useful for text completion, creative writing, and testing raw model behavior without chat formatting. You type a prompt and the model continues from where you left off.

Parameters Presets

The UI ships with presets for common use cases:

Preset              Temperature   top_p   top_k   rep_penalty
-----------------------------------------------------------------
Default             0.7           0.9     40      1.15
Deterministic       0.1           0.1     1       1.0
Creative            1.2           0.95    100     1.05
Contrastive Search  0.0           ---     4       1.0 (uses penalty_alpha)

You can save custom presets in presets/ as YAML files.

Session Management

Conversations are saved automatically and can be exported. The UI supports multiple concurrent chat sessions in separate browser tabs (each maintains its own conversation history, but they share the same model instance).

API Mode

Enable the API to use text-generation-webui as a backend for other applications.

python server.py --api --api-key mysecretkey123

The API is OpenAI-compatible at http://localhost:5000/v1/:

curl http://localhost:5000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer mysecretkey123" \
  -d '{
    "model": "llama-2-7b-chat.Q4_K_M",
    "messages": [
      {"role": "user", "content": "What is the capital of France?"}
    ],
    "temperature": 0.7
  }'

from openai import OpenAI

client = OpenAI(
    base_url="http://localhost:5000/v1",
    api_key="mysecretkey123"
)

response = client.chat.completions.create(
    model="llama-2-7b-chat.Q4_K_M",
    messages=[{"role": "user", "content": "List 3 Linux distros for servers."}]
)
print(response.choices[0].message.content)

The API also supports /v1/completions for text completion and /v1/models for listing available models.

Extensions System

Extensions add functionality through a plugin architecture. Enable them from the Session tab or via command-line flags.

Notable extensions:

• multimodal – enables image input for vision models (LLaVA, etc.)
• openai – the OpenAI-compatible API (enabled with --api)
• superbooga – basic RAG with document ingestion and vector search
• long_term_memory – stores conversation history in a vector database for recall across sessions
• whisper_stt – voice input using OpenAI Whisper
• elevenlabs_tts – text-to-speech output

# Enable specific extensions at launch
python server.py --extensions openai superbooga whisper_stt

Third-party extensions can be installed by cloning them into the extensions/ directory.

Comparison with Open WebUI

Both provide browser-based interfaces for local LLMs, but they serve different roles.

Use text-generation-webui when you want direct control over model loading, need to test multiple model formats, or want to fine-tune sampling parameters. Use Open WebUI when you want a polished multi-user chat interface backed by Ollama or another inference server.

They are not mutually exclusive. A common setup is Open WebUI for daily chat use (backed by Ollama) and text-generation-webui for model evaluation and parameter experimentation.

Performance Tips

GPU layer offloading: For GGUF models, increase n-gpu-layers until you run out of VRAM. Each layer offloaded to GPU improves speed significantly. Check memory with nvidia-smi while loading.

Context length vs memory: Doubling context length roughly doubles KV cache memory usage. If you are running out of VRAM, reduce n_ctx before reducing GPU layers.

ExLlamaV2 for GPTQ/AWQ: Always use the ExLlamaV2 loader instead of Transformers for quantized models. It is 2-3x faster on NVIDIA GPUs.

Batch size: The --n_batch parameter (llama.cpp loader) controls how many tokens are processed per step during prompt evaluation. Higher values (512-1024) speed up long prompt processing but use more memory.

# Optimized launch for 24GB VRAM, GGUF model
python server.py --model llama-3.1-8b-instruct.Q4_K_M.gguf \
  --loader llama.cpp \
  --n-gpu-layers 99 \
  --n_ctx 8192 \
  --n_batch 512 \
  --threads 8

Disable Gradio analytics: Add --no-gradio-analytics to prevent telemetry requests that can cause occasional UI lag.

Text-generation-webui is not the fastest or simplest way to run local LLMs, but it is the most comprehensive. If you need to load a model in any format, tweak every parameter, and access results through both a web UI and an API, this is the tool.