Vllm

TL;DR Major enterprises are moving production AI workloads from GPT-4 to self-hosted Llama and Mistral models, achieving substantial cost reductions while maintaining acceptable quality for most use cases. This migration requires careful planning around API compatibility, prompt engineering adjustments, and performance validation. The typical migration path involves running both systems in parallel during a transition period, using an API compatibility layer that translates OpenAI-formatted requests to local model endpoints. Tools like LiteLLM and OpenAI-compatible servers in Ollama handle this translation, letting teams test self-hosted models without rewriting application code. ...

vLLM Local Setup: High-Throughput LLM Serving Guide TL;DR # Install vLLM (requires CUDA 12.1+ and Python 3.9+) pip install vllm # Serve a model with OpenAI-compatible API vllm serve meta-llama/Llama-3.1-8B-Instruct --port 8000 # Test the endpoint curl http://localhost:8000/v1/chat/completions \ -H "Content-Type: application/json" \ -d '{"model": "meta-llama/Llama-3.1-8B-Instruct", "messages": [{"role": "user", "content": "Hello"}]}' # Docker deployment docker run --gpus all -p 8000:8000 vllm/vllm-openai:latest \ --model meta-llama/Llama-3.1-8B-Instruct Caution: vLLM requires a Hugging Face account with accepted model licenses for gated models like Llama. Set HF_TOKEN in your environment before serving. Never expose the API port to untrusted networks without authentication – vLLM has no built-in auth layer. ...