Advanced LLM Parameter Tuning for Production Workloads

TL;DR

This guide covers advanced parameter tuning techniques beyond basic temperature and top-p settings. For foundational concepts, installation, and basic parameter explanations, see our Complete Guide to Running Local LLMs.

Advanced topics covered: dynamic temperature scheduling based on task type, repeat penalty optimization for long-form content, mirostat sampling for consistent output quality, batch processing configuration, and A/B testing parameter combinations in production.

Dynamic Temperature Scheduling

Different tasks require different temperature settings. Instead of using fixed values, implement dynamic selection based on request context.

Task-Based Parameter Routing

Create a routing function that maps request types to optimal configurations:

def get_parameters_for_task(task_type):
    """Return optimal parameters for specific task types."""
    configs = {
        'code_generation': {
            'temperature': 0.1,
            'top_p': 0.9,
            'repeat_penalty': 1.15,
            'num_predict': 2048
        },
        'code_review': {
            'temperature': 0.2,
            'top_p': 0.95,
            'repeat_penalty': 1.1,
            'num_predict': 1024
        },
        'technical_documentation': {
            'temperature': 0.4,
            'top_p': 0.9,
            'repeat_penalty': 1.05,
            'num_predict': 2048
        },
        'brainstorming': {
            'temperature': 0.9,
            'top_p': 0.95,
            'repeat_penalty': 1.0,
            'num_predict': 1024
        },
        'factual_qa': {
            'temperature': 0.2,
            'top_p': 0.85,
            'repeat_penalty': 1.1,
            'num_predict': 512
        },
        'creative_writing': {
            'temperature': 0.8,
            'top_p': 0.95,
            'repeat_penalty': 1.02,
            'num_predict': 2048
        }
    }
    return configs.get(task_type, {'temperature': 0.7, 'top_p': 0.9})

# Integration with Ollama
import requests

def query_ollama(prompt, task_type='factual_qa'):
    params = get_parameters_for_task(task_type)

    response = requests.post('http://localhost:11434/api/generate', json={
        'model': 'llama3.2',
        'prompt': prompt,
        'options': params
    })

    return response.json()

Context-Aware Parameter Adjustment

Detect task type from prompt content:

import re

def detect_task_type(prompt):
    """Detect task type from prompt patterns."""
    prompt_lower = prompt.lower()

    if re.search(r'(write|implement|create).*code', prompt_lower):
        return 'code_generation'
    elif re.search(r'(review|analyze|check).*code', prompt_lower):
        return 'code_review'
    elif re.search(r'(explain|document|describe)', prompt_lower):
        return 'technical_documentation'
    elif re.search(r'(brainstorm|ideas|possibilities)', prompt_lower):
        return 'brainstorming'
    elif 'creative' in prompt_lower or 'story' in prompt_lower:
        return 'creative_writing'
    else:
        return 'factual_qa'

# Automatic routing
prompt = "Write a Python function to parse JSON"
task = detect_task_type(prompt)
response = query_ollama(prompt, task)

Repeat Penalty Optimization

Repeat penalty prevents models from reusing phrases excessively. Critical for long-form generation where loops become obvious.

Finding Optimal Values

Test different penalties with your actual workload:

#!/bin/bash
# Test repeat penalty values

PROMPT="Write a comprehensive guide to Docker networking covering bridges, host networking, and overlay networks."

for penalty in 1.0 1.05 1.1 1.15 1.2 1.25; do
  echo "Testing repeat_penalty: $penalty"

  curl -s http://localhost:11434/api/generate -d "{
    \"model\": \"llama3.2\",
    \"prompt\": \"$PROMPT\",
    \"options\": {
      \"temperature\": 0.7,
      \"repeat_penalty\": $penalty,
      \"num_predict\": 1000
    }
  }" | jq -r '.response' > "test_penalty_${penalty}.txt"

  # Count unique vs total words
  total=$(wc -w < "test_penalty_${penalty}.txt")
  unique=$(tr ' ' '\n' < "test_penalty_${penalty}.txt" | sort -u | wc -l)
  ratio=$(echo "scale=2; $unique / $total" | bc)

  echo "Penalty: $penalty | Total words: $total | Unique: $unique | Ratio: $ratio"
done

Analyze output files for repetitive patterns. Calculate unique word ratio to quantify repetition.

Workload-Specific Settings

Based on production testing:

Use CaseRepeat PenaltyRationale
Technical docs1.05-1.1Slight penalty prevents loops without hurting technical terms
Code generation1.15-1.2Higher penalty prevents copying patterns verbatim
Creative writing1.0-1.05Low penalty allows natural repetition (rhythm, style)
Chat/Q&A1.1Balanced - prevents loops in multi-turn conversations
API documentation1.08Allows repeated parameter names while avoiding redundancy

Combining with Frequency and Presence Penalties

llama.cpp supports additional controls:

curl http://localhost:8080/v1/chat/completions -d '{
  "messages": [{"role": "user", "content": "Explain microservices patterns"}],
  "temperature": 0.7,
  "repeat_penalty": 1.1,
  "frequency_penalty": 0.5,
  "presence_penalty": 0.2,
  "max_tokens": 1000
}'
  • repeat_penalty: Per-token penalty multiplier
  • frequency_penalty: Penalizes proportional to token frequency
  • presence_penalty: Fixed penalty for any token that appeared before

Combine all three for maximum repetition control in long documents.

Mirostat Sampling

Mirostat dynamically adjusts sampling to maintain target perplexity, producing consistent output length and quality. Unlike fixed temperature, it adapts based on model confidence.

Configuration

Ollama:

curl http://localhost:11434/api/generate -d '{
  "model": "llama3.2",
  "prompt": "Write a detailed explanation of Kubernetes networking",
  "options": {
    "mirostat": 2,
    "mirostat_tau": 5.0,
    "mirostat_eta": 0.1
  }
}'

llama.cpp:

./llama-server -m model.gguf \
  --mirostat 2 \
  --mirostat-tau 5.0 \
  --mirostat-eta 0.1

Parameter Meanings

  • mirostat: Mode (0=off, 1=v1, 2=v2 recommended)
  • mirostat_tau: Target entropy/perplexity (3.0-8.0, default 5.0)
    • Lower values (3-4): More focused, deterministic output
    • Higher values (6-8): More creative, varied output
  • mirostat_eta: Learning rate (0.05-0.2, default 0.1)
    • Controls how quickly sampling adapts
    • Higher = more aggressive adjustments

When to Use Mirostat

Good for:

  • Long-form article generation (1000+ tokens)
  • Documentation where consistent quality matters
  • Preventing quality degradation in extended outputs
  • Maintaining coherent narrative across paragraphs

Avoid for:

  • Short responses (<200 tokens) - adds overhead
  • Code generation - need determinism
  • Tasks where you need precise temperature control
  • Real-time chat - may add latency

Tuning Tau for Your Workload

# Test different tau values
for tau in 3.0 4.0 5.0 6.0 7.0; do
  curl http://localhost:11434/api/generate -d "{
    \"model\": \"llama3.2\",
    \"prompt\": \"Explain container orchestration\",
    \"options\": {
      \"mirostat\": 2,
      \"mirostat_tau\": $tau,
      \"num_predict\": 500
    }
  }" | jq -r '.response' > "mirostat_tau_${tau}.txt"
done

Lower tau produces more focused technical writing. Higher tau adds creativity but may reduce coherence.

Batch Processing Optimization

When processing multiple requests, batch configuration significantly impacts throughput.

llama.cpp Batch Settings

./llama-server -m model.gguf \
  --batch-size 512 \
  --ubatch-size 256 \
  --ctx-size 4096 \
  -ngl 35
  • batch-size: Prompt processing batch (larger = faster prompt ingestion)
  • ubatch-size: Generation batch (affects token generation speed)

For high-throughput scenarios with many concurrent requests, increase batch sizes. For low-latency single requests, decrease them.

Ollama Concurrency

Control concurrent requests via environment:

# In /etc/systemd/system/ollama.service
Environment="OLLAMA_NUM_PARALLEL=4"
Environment="OLLAMA_MAX_LOADED_MODELS=2"
  • OLLAMA_NUM_PARALLEL: Max concurrent inference requests
  • OLLAMA_MAX_LOADED_MODELS: Models kept in memory

Higher parallelism increases throughput but consumes more VRAM. Monitor with nvidia-smi.

A/B Testing Parameters in Production

Systematically test parameter combinations with real traffic.

Simple A/B Test Framework

import random
import json
from datetime import datetime

class ParameterTester:
    def __init__(self):
        self.results = []

    def run_ab_test(self, prompt, variant_a, variant_b, iterations=10):
        """Run A/B test comparing two parameter sets."""
        for i in range(iterations):
            variant = 'A' if random.random() < 0.5 else 'B'
            params = variant_a if variant == 'A' else variant_b

            start = datetime.now()
            response = requests.post('http://localhost:11434/api/generate', json={
                'model': 'llama3.2',
                'prompt': prompt,
                'options': params
            })
            duration = (datetime.now() - start).total_seconds()

            result = {
                'variant': variant,
                'duration': duration,
                'response_length': len(response.json().get('response', '')),
                'params': params
            }
            self.results.append(result)

        return self.analyze_results()

    def analyze_results(self):
        """Compare performance of variants."""
        a_results = [r for r in self.results if r['variant'] == 'A']
        b_results = [r for r in self.results if r['variant'] == 'B']

        return {
            'variant_a': {
                'avg_duration': sum(r['duration'] for r in a_results) / len(a_results),
                'avg_length': sum(r['response_length'] for r in a_results) / len(a_results)
            },
            'variant_b': {
                'avg_duration': sum(r['duration'] for r in b_results) / len(b_results),
                'avg_length': sum(r['response_length'] for r in b_results) / len(b_results)
            }
        }

# Usage
tester = ParameterTester()
results = tester.run_ab_test(
    prompt="Explain Docker networking",
    variant_a={'temperature': 0.5, 'repeat_penalty': 1.1},
    variant_b={'temperature': 0.7, 'repeat_penalty': 1.05},
    iterations=20
)
print(json.dumps(results, indent=2))

Track metrics: response time, output length, user satisfaction ratings.

Performance Monitoring

Monitor parameter impact on system resources and response quality.

Resource Tracking

import psutil
import requests
from datetime import datetime

def monitor_inference(prompt, params):
    """Track CPU, RAM, VRAM during inference."""
    # Get initial state
    initial_ram = psutil.virtual_memory().percent

    start = datetime.now()
    response = requests.post('http://localhost:11434/api/generate', json={
        'model': 'llama3.2',
        'prompt': prompt,
        'options': params
    })
    duration = (datetime.now() - start).total_seconds()

    final_ram = psutil.virtual_memory().percent

    return {
        'duration': duration,
        'ram_delta': final_ram - initial_ram,
        'response_length': len(response.json().get('response', '')),
        'tokens_per_second': response.json().get('eval_count', 0) / response.json().get('eval_duration', 1e9) * 1e9
    }

Log these metrics to identify parameter combinations that cause performance degradation.

Production Best Practices

Based on running local LLMs at scale:

  1. Start conservative, tune incrementally

    • Begin with temperature 0.5, standard repeat penalty 1.1
    • Adjust in 0.1 increments based on output analysis

  2. Test with real prompts

    • Synthetic benchmarks don’t reflect actual workload
    • Use production prompt samples for parameter tuning

  3. Monitor quality and performance together

    • Faster inference is worthless if output quality degrades
    • Track both metrics, optimize for balance

  4. Version parameter sets

    • Store configs in git with descriptive commit messages
    • Tag known-good configurations

  5. Implement gradual rollout

    • Test new parameters on 5% of traffic first
    • Increase gradually if metrics improve

Caution: When using AI assistants to generate parameter recommendations, always validate against your specific hardware and workload. Parameters that work for one model or use case may perform poorly for another. Test thoroughly before production deployment.

Further Reading

For foundational concepts not covered here: