TL;DR # Install Ollama and pull models curl -fsSL https://ollama.com/install.sh | sh ollama pull qwen2.5-coder:7b ollama pull codellama:7b # Verify Ollama is running curl http://localhost:11434/api/tags Install the Continue extension from the VS Code marketplace, open ~/.continue/config.json, point it at your local Ollama instance, and start coding with zero cloud dependencies. ...
LocalAI Setup: OpenAI API-Compatible Local Inference TL;DR # Docker (quickest start) docker run -d --name localai -p 8080:8080 \ -v localai-models:/build/models \ localai/localai:latest-gpu-nvidia-cuda-12 # Install a model from the gallery curl http://localhost:8080/models/apply -d '{"id": "llama-3.1-8b-instruct"}' # Test chat completions (same as OpenAI API) curl http://localhost:8080/v1/chat/completions \ -H "Content-Type: application/json" \ -d '{"model": "llama-3.1-8b-instruct", "messages": [{"role": "user", "content": "Hello"}]}' # Generate embeddings curl http://localhost:8080/v1/embeddings \ -H "Content-Type: application/json" \ -d '{"model": "text-embedding-ada-002", "input": "The quick brown fox"}' Caution: LocalAI has no built-in authentication. Any process that can reach port 8080 can use the API. Use firewall rules, bind to localhost only, or put a reverse proxy with auth in front before exposing to a network. ...
KoboldCpp Quick Start: Run GGUF Models with One Binary TL;DR # Download the latest release (Linux, CUDA) wget https://github.com/LostRuins/koboldcpp/releases/latest/download/koboldcpp-linux-x64-cuda1150 chmod +x koboldcpp-linux-x64-cuda1150 # Run a GGUF model ./koboldcpp-linux-x64-cuda1150 --model llama-3.1-8b-instruct.Q4_K_M.gguf \ --gpulayers 99 --contextsize 4096 --port 5001 # Web UI opens at http://localhost:5001 # KoboldAI API at http://localhost:5001/api/ # OpenAI-compatible API at http://localhost:5001/v1/chat/completions Caution: KoboldCpp binds to localhost by default. If you use --host 0.0.0.0 to allow network access, there is no built-in authentication. Restrict access with firewall rules or a reverse proxy. ...
Text Generation WebUI Setup Guide for Local LLM Inference TL;DR # Clone and run the one-click installer git clone https://github.com/oobabooga/text-generation-webui cd text-generation-webui bash start_linux.sh # Installs conda env, dependencies, launches UI # Or use Docker docker compose up -d # Access the web interface # http://localhost:7860 # Enable API mode python server.py --api --listen # API available at http://localhost:5000/v1/chat/completions Caution: The web interface has no authentication by default. Do not use --listen (which binds to 0.0.0.0) on networks you do not control. Use --listen --api-key YOUR_SECRET if exposing the API, and put a reverse proxy with auth in front for production use. ...
vLLM Local Setup: High-Throughput LLM Serving Guide TL;DR # Install vLLM (requires CUDA 12.1+ and Python 3.9+) pip install vllm # Serve a model with OpenAI-compatible API vllm serve meta-llama/Llama-3.1-8B-Instruct --port 8000 # Test the endpoint curl http://localhost:8000/v1/chat/completions \ -H "Content-Type: application/json" \ -d '{"model": "meta-llama/Llama-3.1-8B-Instruct", "messages": [{"role": "user", "content": "Hello"}]}' # Docker deployment docker run --gpus all -p 8000:8000 vllm/vllm-openai:latest \ --model meta-llama/Llama-3.1-8B-Instruct Caution: vLLM requires a Hugging Face account with accepted model licenses for gated models like Llama. Set HF_TOKEN in your environment before serving. Never expose the API port to untrusted networks without authentication – vLLM has no built-in auth layer. ...
TL;DR LM Studio’s plugin architecture transforms the desktop application from a simple model runner into an extensible AI platform. While the base application handles model loading and inference, plugins add custom workflows, integrate external tools, and automate complex tasks without writing server code from scratch. The plugin system uses a JavaScript-based API that hooks into LM Studio’s model lifecycle, request pipeline, and UI components. Developers can create plugins that preprocess prompts, post-process responses, connect to external databases, or trigger actions based on model outputs. Unlike building a separate application that calls LM Studio’s OpenAI-compatible API, plugins run inside the application context with direct access to model state and configuration. ...
Building a Local RAG Pipeline with Ollama and Open WebUI TL;DR Retrieval-augmented generation (RAG) lets your local LLM answer questions using your own documents instead of relying on its training data. This guide walks through building a fully local RAG pipeline: document ingestion, embedding, vector storage, and retrieval through Open WebUI. ...
TL;DR Hamilton-Jacobi-Bellman equations provide the mathematical foundation for optimal control in reinforcement learning, but implementing them locally requires combining numerical solvers with LLM-assisted code generation. This guide shows you how to use Ollama running locally to generate HJB solver implementations, validate discretization schemes, and debug boundary conditions without sending your research code to cloud APIs. ...
TL;DR Qwen3 Coder runs locally via Ollama with a single command after installing Ollama using curl -fsSL https://ollama.com/install.sh | sh. The model excels at code completion, refactoring, and multi-language support with context windows up to 32K tokens in the larger variants. Unlike general-purpose models, Qwen3 Coder is specifically trained on code repositories and technical documentation, making it competitive with DeepSeek Coder and CodeLlama for local development workflows. ...
TL;DR Running llama.cpp in Docker containers solves the deployment complexity of local LLM inference while maintaining reproducibility across different host systems. This guide covers production-ready containerization patterns specifically for llama.cpp, focusing on aspects not typically addressed in basic setup tutorials. You’ll learn to build multi-stage Docker images that compile llama.cpp with optimal flags for your target hardware, then copy only the runtime binaries to a minimal production image. The approach reduces final image size while preserving GPU acceleration support through CUDA or ROCm layers. ...
