GGUF Quantization Explained: Choosing the Right Format for Local AI

TL;DR

# Check quantization of an Ollama model
ollama show llama3.2:3b --modelfile | grep -i quant

# Inspect a GGUF file directly
python3 -c "from gguf import GGUFReader; r = GGUFReader('model.gguf'); print([kv for kv in r.fields])"

# Or use llama.cpp's built-in info
./llama-quantize --help

# Convert and quantize with llama.cpp
./llama-quantize input.gguf output-Q4_K_M.gguf Q4_K_M

GGUF is the standard file format for running quantized LLMs locally. Quantization reduces model size and VRAM usage by representing weights with fewer bits. The tradeoff is a small reduction in output quality. Choosing the right quantization level depends on your available VRAM, the model size, and your quality requirements.

What Is GGUF

GGUF (GPT-Generated Unified Format) is a binary file format for storing quantized large language models. It was introduced by the llama.cpp project in August 2023 as a successor to the GGML format. GGUF addresses several limitations of GGML:

• Self-contained metadata: Model architecture, tokenizer, and parameters are embedded in the file header. No separate config files needed.
• Extensibility: New metadata fields can be added without breaking backward compatibility.
• Alignment: Data is properly aligned in memory for efficient loading.
• Single-file distribution: One .gguf file contains everything needed to run inference.

Every model you pull through Ollama is stored internally as GGUF. When you download a model from Hugging Face for use with llama.cpp, you typically download a .gguf file.

GGUF vs GGML

If you encounter .ggml files, they are from older conversions and should be reconverted to GGUF. Most tools no longer support GGML directly.

How Quantization Works

A full-precision (FP32) language model stores each weight as a 32-bit floating-point number. A 7-billion-parameter model at FP32 requires:

7B x 4 bytes = 28 GB

That does not fit in most consumer GPUs. Quantization reduces the bit width of each weight:

• FP16 / BF16: 16 bits per weight. Half the size, negligible quality loss.
• Q8_0: 8 bits per weight. Minor quality loss, significant size reduction.
• Q6_K: 6 bits. Good quality, roughly 44% of FP16 size.
• Q5_K_M: 5 bits. Sweet spot for many use cases.
• Q4_K_M: 4 bits. Most popular for consumer hardware. Noticeable but acceptable quality loss.
• Q4_0: 4 bits, simpler algorithm. Slightly worse than Q4_K_M.
• Q3_K_M: 3 bits. Significant quality loss. Use only when VRAM is extremely limited.
• Q2_K: 2 bits. Severe quality degradation. Not recommended for production use.

The “K” variants (Q4_K_M, Q5_K_S, etc.) use k-quant methods that apply different quantization levels to different layers based on their sensitivity. Attention layers and output layers are quantized less aggressively than feed-forward layers, preserving quality where it matters most.

K-Quant Suffixes

• _S (small): More aggressive quantization. Smaller file, lower quality.
• _M (medium): Balanced. The default recommendation for most users.
• _L (large): Less aggressive. Larger file, better quality.

Quantization Comparison Table (7B Model)

VRAM figures include approximately 1-1.5 GB overhead for KV cache at 2048 context length. Larger context windows increase VRAM usage significantly.

Quality retention percentages are approximate, based on perplexity benchmarks (lower perplexity = better). Actual perceived quality depends on the task. Coding and math tasks degrade faster than conversational tasks at lower quantization levels.

Choosing the Right Quantization

Decision Framework

Start with Q4_K_M. It is the most widely tested quantization level and offers the best balance of size, speed, and quality for consumer hardware. Move up or down from there based on your situation.

Use Q5_K_M or Q6_K when:

• You have VRAM headroom (12+ GB GPU for a 7B model)
• The task requires precision (coding, math, structured output)
• Quality matters more than speed
• You are running the model as a primary assistant

Use Q4_K_M when:

• Running on 8 GB VRAM (7B models)
• Running on 24 GB VRAM (70B models with Q4_K_M)
• General-purpose chat, summarization, creative writing

Use Q8_0 when:

• You have substantial VRAM (24+ GB for 7B, 80+ GB for 70B)
• Maximum quality is required
• You are benchmarking or evaluating model capabilities

Use Q3_K_M or lower when:

• Fitting a larger model matters more than quality (e.g., 13B at Q3 vs 7B at Q5)
• VRAM is extremely constrained (6 GB GPU)
• The task is simple (classification, yes/no answers, keyword extraction)

Model Size vs Quantization

A common question: is it better to run a larger model at lower quantization or a smaller model at higher quantization?

General rule: a larger model at Q4_K_M usually outperforms a smaller model at Q8_0, assuming both fit in VRAM. A 13B model at Q4_K_M typically produces better results than a 7B at Q8_0. The additional parameters provide more capacity for knowledge and reasoning than the extra precision per weight.

The exception is highly precise tasks (math, code with exact syntax requirements) where quantization noise in larger models can cause more errors than a smaller but more precisely quantized model.

Checking Quantization in Ollama

When you pull a model from the Ollama library, the default tag usually provides a Q4_K_M quantization. To verify:

# Show model details
ollama show llama3.2:3b

Look for the quantization_version and architecture details. For more specifics:

# Show the full modelfile which includes the FROM layer digest
ollama show llama3.2:3b --modelfile

To check what quantization variants are available for a model, check the model page on ollama.com or use:

# List available tags (requires web access)
curl -s https://ollama.com/api/tags/llama3.2 | jq '.tags[].name'

Some models include the quantization in the tag name:

ollama pull llama3.1:8b-instruct-q5_K_M
ollama pull llama3.1:8b-instruct-q8_0

Inspecting GGUF Files Directly

If you have a standalone GGUF file (downloaded from Hugging Face or converted yourself), you can inspect it with the gguf Python package:

pip install gguf

python3 << 'EOF'
from gguf import GGUFReader

reader = GGUFReader("model.Q4_K_M.gguf")
for key in reader.fields:
    field = reader.fields[key]
    if key in ["general.name", "general.architecture", "general.quantization_version",
               "general.file_type", "llama.context_length", "llama.embedding_length"]:
        print(f"{key}: {field.parts[-1][0] if field.parts else 'N/A'}")
EOF

Or use llama.cpp’s built-in inspection:

./llama-cli --model model.Q4_K_M.gguf --verbose-prompt -n 1 2>&1 | head -30

This prints model metadata including architecture, context length, and quantization details.

Converting Models to GGUF

To convert a Hugging Face model (safetensors/PyTorch) to GGUF, use the conversion script from llama.cpp:

Prerequisites

git clone https://github.com/ggerganov/llama.cpp
cd llama.cpp
pip install -r requirements/requirements-convert_hf_to_gguf.txt

Step 1: Convert to FP16 GGUF

python3 convert_hf_to_gguf.py /path/to/huggingface-model/ \
  --outfile model-f16.gguf \
  --outtype f16

Step 2: Quantize

# Build llama.cpp first
cmake -B build && cmake --build build --config Release -j$(nproc)

# Quantize to Q4_K_M
./build/bin/llama-quantize model-f16.gguf model-Q4_K_M.gguf Q4_K_M

# Quantize to Q5_K_M
./build/bin/llama-quantize model-f16.gguf model-Q5_K_M.gguf Q5_K_M

Step 3: Test

./build/bin/llama-cli -m model-Q4_K_M.gguf -p "Hello, how are you?" -n 100

Step 4: Import into Ollama

Create a Modelfile:

FROM ./model-Q4_K_M.gguf

Then:

ollama create my-model -f Modelfile
ollama run my-model

Quantization and CPU Offloading

When a model does not fully fit in GPU VRAM, Ollama and llama.cpp automatically offload some layers to CPU RAM. Understanding quantization helps optimize this split:

# In llama.cpp, specify GPU layers
./llama-cli -m model.gguf -ngl 20  # 20 layers on GPU, rest on CPU

# In Ollama, GPU offloading is automatic
# Check allocation with:
ollama ps

The PROCESSOR column in ollama ps shows the GPU/CPU split (e.g., “80% GPU / 20% CPU”). Lower quantization means more layers fit on the GPU, reducing the CPU-offloaded portion and improving speed.

Caution: CPU-offloaded layers are significantly slower than GPU layers. A model that is 50% offloaded to CPU will run roughly 3-5x slower than a fully GPU-resident model. It is often better to use a smaller model that fits entirely in VRAM than a larger model that requires heavy CPU offloading.

Perplexity Testing

To measure the actual quality impact of quantization on your specific use case, run a perplexity test with llama.cpp:

./build/bin/llama-perplexity -m model-Q4_K_M.gguf \
  -f /path/to/test-text.txt \
  --chunks 100

Compare the perplexity score across quantization levels. A difference of less than 0.5 in perplexity is generally imperceptible in real-world use.

Summary

For most self-hosted AI setups on consumer hardware:

1. Default choice: Q4_K_M. Works well on 8 GB GPUs for 7B models, 24 GB GPUs for 70B models.
2. Quality upgrade: Q5_K_M if you have the VRAM. Measurably better on coding and reasoning tasks.
3. Maximum quality: Q8_0 or Q6_K when VRAM is not a constraint.
4. Tight VRAM: Q3_K_M as a last resort, but prefer a smaller model at Q4_K_M instead.
5. Always prefer _K_M variants over plain Q4_0/Q5_0. The k-quant methods are strictly better.