TL;DR

Fine-tuning local LLMs in 2026 means adapting pre-trained models to your specific use case without cloud dependencies. Both Ollama and llama.cpp support running fine-tuned models, but the actual training happens with separate tools like Unsloth, Axolotl, or llama.cpp’s built-in training capabilities.

The typical workflow: train or fine-tune using a framework that outputs GGUF format, then serve the resulting model through Ollama or llama-server. Ollama pulls base models from its library, but you can import custom GGUF files using ollama create with a Modelfile. For llama.cpp, point llama-server directly at your fine-tuned GGUF file.

After fine-tuning produces custom-model-q4.gguf, serve it with llama.cpp:

./llama-server -m custom-model-q4.gguf --port 8080 --host 0.0.0.0

Or import into Ollama:

cat > Modelfile <<EOF
FROM ./custom-model-q4.gguf
PARAMETER temperature 0.7
PARAMETER top_p 0.9
EOF

ollama create my-custom-model -f Modelfile
ollama run my-custom-model

The model now runs on port 11434 through Ollama’s API or port 8080 through llama-server’s OpenAI-compatible endpoint.

Key Considerations

Fine-tuning requires significant VRAM – most practitioners use cloud GPU instances for training, then deploy the resulting GGUF locally. Quantization happens post-training: Q4_K_M offers good quality-to-size ratio for most applications, while Q8_0 preserves more accuracy at double the file size.

Caution: Always validate fine-tuned model outputs in a test environment before production deployment. Models trained on narrow datasets can produce unexpected responses outside their training domain. Test thoroughly with diverse inputs that represent real-world usage patterns.

Understanding Fine-Tuning vs. Inference in Self-Hosted Environments

When running LLMs locally, understanding the distinction between inference and fine-tuning determines your hardware requirements and workflow design. Inference means running a pre-trained model to generate responses – what happens when you query Ollama on port 11434 or send requests to llama-server. Fine-tuning means continuing training on a base model with your own dataset to specialize its behavior.

Inference with Ollama requires enough VRAM or RAM to load the model weights. A 7B parameter model quantized to Q4_K_M typically needs 4-5GB of memory. You can run inference on modest hardware – a desktop with 16GB RAM handles most 7B models comfortably.

Fine-tuning demands significantly more resources. You need memory for the model, optimizer states, gradients, and training batches. Fine-tuning that same 7B model often requires 24GB+ VRAM even with parameter-efficient methods. Most self-hosted setups use consumer GPUs like RTX 4090 or rent cloud instances for training, then export the fine-tuned weights back to local inference.

Workflow Separation

A practical self-hosted workflow separates these concerns. Use llama.cpp or Ollama for inference – they excel at serving models efficiently with GGUF quantization. For fine-tuning, use dedicated frameworks like Axolotl or Unsloth that support LoRA adapters and gradient checkpointing. After training, convert your fine-tuned model to GGUF format and load it into Ollama:

ollama create my-custom-model -f Modelfile
ollama run my-custom-model

Your Modelfile references the converted GGUF weights. This separation lets you fine-tune on powerful hardware while serving the result on efficient inference infrastructure.

Caution: Fine-tuning requires validating training data quality and hyperparameters. AI-generated training examples should be manually reviewed before use – poor training data degrades model performance permanently.

Preparing Your Training Environment and Dataset

Before fine-tuning, you need a clean dataset and the right dependencies. Most fine-tuning workflows use Python libraries like Hugging Face Transformers or Unsloth, which handle the training loop while llama.cpp converts the result to GGUF format afterward.

Start with a Python virtual environment and install the training stack:

python3 -m venv finetune-env
source finetune-env/bin/activate
pip install torch transformers datasets accelerate peft bitsandbytes

For GPU training, verify CUDA is available:

python3 -c "import torch; print(torch.cuda.is_available())"

If this returns False, check your NVIDIA driver and CUDA toolkit installation.

Dataset Preparation

Fine-tuning requires structured data in JSONL format. Each line should contain a conversation or instruction-response pair:

{"instruction": "Explain Docker networking", "response": "Docker creates isolated networks for containers..."}
{"instruction": "Debug Python memory leak", "response": "Use tracemalloc to identify growing objects..."}

Clean your dataset by removing duplicates, fixing encoding issues, and validating JSON structure:

jq -c . training_data.jsonl > validated.jsonl

Most practitioners recommend at least several hundred examples for meaningful fine-tuning, though quality matters more than quantity. Domain-specific datasets work better than generic ones.

Converting Models to GGUF

After training completes, you’ll have PyTorch weights that need conversion for llama.cpp. Clone the llama.cpp repository and use the conversion script:

git clone https://github.com/ggerganov/llama.cpp
cd llama.cpp
python3 convert.py /path/to/trained-model --outtype f16 --outfile model-f16.gguf

Then quantize to reduce size:

./quantize model-f16.gguf model-q4_k_m.gguf Q4_K_M

Caution: Always validate AI-generated training scripts before running them. Incorrect hyperparameters can waste GPU hours or corrupt your base model. Test on a small subset first.

Fine-Tuning with llama.cpp: LoRA Training Workflow

llama.cpp includes native LoRA (Low-Rank Adaptation) training capabilities through the finetune binary. This approach lets you adapt base models to specific tasks without retraining the entire model, making it practical for consumer hardware.

Convert your training examples to JSONL format with instruction-response pairs:

{"prompt": "Translate to Spanish: Hello", "completion": "Hola"}
{"prompt": "Translate to Spanish: Goodbye", "completion": "Adios"}

llama.cpp expects plain text format for the finetune binary. Convert JSONL to the required format:

cat training_data.jsonl | jq -r '.prompt + " " + .completion' > training.txt

Running LoRA Training

Build llama.cpp from source to get the finetune binary:

git clone https://github.com/ggerganov/llama.cpp
cd llama.cpp
cmake -B build
cmake --build build --config Release

Start training with a base GGUF model:

./build/bin/finetune \
  --model-base models/llama-2-7b.Q8_0.gguf \
  --train-data training.txt \
  --lora-out lora-adapter.bin \
  --threads 8 \
  --ctx-size 2048

Training time varies based on dataset size and hardware. Most homelab setups complete small datasets (under 1000 examples) in hours rather than days.

Applying LoRA Adapters

Merge the LoRA adapter with your base model or load it dynamically:

./build/bin/llama-server \
  --model models/llama-2-7b.Q8_0.gguf \
  --lora lora-adapter.bin \
  --port 8080

The merged model retains the base model’s general knowledge while incorporating your fine-tuned behavior. Test thoroughly with validation examples before deploying to production workloads.

Caution: Validate training data quality before starting long training runs. Poor data quality produces models that hallucinate or repeat training artifacts. Always keep your base model file separate from LoRA adapters for easy rollback.

Converting and Merging LoRA Adapters to GGUF

LoRA adapters trained with frameworks like Axolotl or Unsloth produce PyTorch checkpoint files that need conversion before use with llama.cpp or Ollama. The conversion process merges the adapter weights with the base model and outputs a GGUF file ready for local inference.

Start by installing the llama.cpp Python conversion tools:

git clone https://github.com/ggerganov/llama.cpp
cd llama.cpp
pip install -r requirements.txt

Merge your LoRA adapter with the base model using the convert script:

python convert-lora-to-ggml.py \
  --base ./models/llama-2-7b \
  --lora ./fine-tuned/adapter_model \
  --outfile ./merged-model.gguf \
  --outtype q8_0

The --outtype parameter controls quantization. Use q8_0 for initial testing to preserve quality, then experiment with q4_k_m or q5_k_m for production deployments where memory constraints matter.

Loading Merged Models in Ollama

Create a Modelfile to import your converted GGUF:

FROM ./merged-model.gguf
PARAMETER temperature 0.7
PARAMETER top_p 0.9

Import and run:

ollama create custom-model -f Modelfile
ollama run custom-model

Validation and Testing

Always validate merged models before production use. Run test prompts comparing outputs against your training data expectations. Check for degraded performance or unexpected behavior – quantization can introduce subtle quality changes that only surface with domain-specific queries.

When using AI assistants to generate conversion commands, verify all file paths and parameter names match your actual directory structure. The conversion process is destructive if you overwrite files, so maintain backups of both base models and LoRA checkpoints.

For models larger than 13B parameters, consider splitting the conversion across multiple steps or using a machine with sufficient RAM to hold the full unquantized weights during merge operations.

Serving Your Fine-Tuned Model with Ollama

Once you have a fine-tuned GGUF model, Ollama provides the simplest path to serving it locally. Create a Modelfile that references your quantized model and defines the runtime parameters:

FROM ./mistral-7b-finance-q4_k_m.gguf

TEMPLATE """[INST] {{ .Prompt }} [/INST]"""

PARAMETER temperature 0.7
PARAMETER top_p 0.9
PARAMETER stop "[INST]"
PARAMETER stop "[/INST]"

Save this as Modelfile in the same directory as your GGUF file, then create the model:

ollama create finance-assistant -f Modelfile

Start serving immediately:

ollama run finance-assistant

For production deployments, run Ollama as a background service and configure GPU allocation:

export OLLAMA_NUM_GPU=1
export OLLAMA_HOST=0.0.0.0:11434
ollama serve

The REST API on port 11434 accepts OpenAI-compatible requests, making integration straightforward for existing applications. Test with curl:

curl http://localhost:11434/api/generate -d '{
  "model": "finance-assistant",
  "prompt": "Explain the difference between EBITDA and net income",
  "stream": false
}'

Your fine-tuned model’s quantization level directly impacts inference speed and quality. Q4_K_M offers the best balance for most use cases – reasonable quality with manageable memory requirements. Q5_K_M preserves more detail if you have extra RAM. Q8_0 approaches full precision but doubles memory usage compared to Q4_K_M.

Test multiple quantization levels with your specific prompts before committing to production. The quality difference varies significantly depending on your fine-tuning domain and task complexity.

Caution: Always validate model outputs in your specific domain before production deployment. Fine-tuned models can exhibit unexpected behavior on edge cases not represented in training data.

Installation and Configuration Steps

Start by installing Ollama to handle model downloads and serving. On Linux systems, run the official install script:

curl -fsSL https://ollama.com/install.sh | sh

Pull a base model suitable for fine-tuning. Smaller models like Mistral 7B work well for most tasks:

ollama pull mistral:7b-instruct

Verify the installation by checking the API endpoint:

curl http://localhost:11434/api/tags

Setting Up llama.cpp for Fine-Tuning

Clone and build llama.cpp from source for access to training utilities:

git clone https://github.com/ggerganov/llama.cpp
cd llama.cpp
mkdir build && cd build
cmake .. -DLLAMA_CUBLAS=ON
cmake --build . --config Release

The CUBLAS flag enables GPU acceleration on NVIDIA hardware. Omit it for CPU-only builds.

Preparing Your Training Environment

Create a dedicated directory structure for fine-tuning work:

mkdir -p ~/llm-finetune/{models,datasets,output}
export OLLAMA_MODELS=~/llm-finetune/models

Convert your base model from Ollama’s format to a working checkpoint. Export the model first:

ollama show mistral:7b-instruct --modelfile > ~/llm-finetune/Modelfile

Configuration Validation

Before proceeding with training, verify your GPU is accessible:

nvidia-smi

Set the OLLAMA_NUM_GPU environment variable to control GPU allocation:

export OLLAMA_NUM_GPU=1

Caution: Always validate AI-generated training commands against official documentation before executing them. Incorrect hyperparameters can corrupt models or waste significant compute time. Test configuration changes on small dataset samples first, then scale to full training runs once validated.