TL;DR

Serving a 70B parameter model across multiple GPUs requires careful orchestration of tensor parallelism, memory management, and framework selection. The key challenge is that a 70B model in FP16 precision needs approximately 140GB of memory just for weights, plus additional overhead for KV cache and activations – far exceeding single GPU capacity.

Your best options are vLLM for production throughput, TensorRT-LLM for lowest latency, or Text Generation Inference (TGI) for ease of deployment. Each framework handles tensor parallelism differently: vLLM uses Megatron-style tensor parallel with automatic pipeline scheduling, TensorRT-LLM offers fused kernels with custom CUDA optimizations, and TGI provides a simpler API with good defaults.

For a typical 4x A100 40GB setup, you can expect to serve Llama 2 70B at roughly 20-30 tokens per second with batch size 1, scaling to higher throughput with continuous batching. Memory allocation breaks down to approximately 35GB per GPU for model weights in FP16, leaving 5GB per device for KV cache – enough for context lengths around 2048 tokens per request when serving multiple concurrent users.

Critical optimization techniques include KV cache sharding across GPUs to distribute memory pressure, pipeline parallelism to overlap computation with communication, and PagedAttention to reduce memory fragmentation. You must also tune the number of pipeline stages, tensor parallel degree, and batch scheduling parameters based on your latency versus throughput requirements.

The practical workflow involves profiling your specific model with each framework, measuring actual GPU memory usage with nvidia-smi, and load testing with realistic request patterns. Avoid common pitfalls like underestimating KV cache growth with long contexts or ignoring inter-GPU communication bottlenecks on PCIe-connected systems versus NVLink.

Understanding 70B Model Memory Requirements

A 70B parameter model in FP16 precision requires approximately 140GB of memory just for the weights – each parameter consumes 2 bytes. Add another 20-30GB for the KV cache during inference, activation memory, and framework overhead, pushing total requirements to 160-170GB under typical serving conditions.

When distributing a 70B model across multiple GPUs, tensor parallelism splits individual weight matrices across devices. For a 4xA100 setup (40GB each), you have 160GB total VRAM – barely sufficient for FP16 serving with minimal KV cache headroom. The model weights consume roughly 35GB per GPU, leaving only 5GB per device for dynamic allocations.

Quantization dramatically changes this equation. INT8 quantization cuts weight memory in half to 70GB total, while 4-bit formats like GPTQ or AWQ reduce it to approximately 35GB. A 4-bit quantized 70B model fits comfortably on 2x48GB GPUs with substantial room for KV cache and concurrent requests.

Calculating KV Cache Requirements

The KV cache grows linearly with context length and batch size. For a 70B model with 8192 token context:

# Approximate KV cache size calculation
num_layers = 80
hidden_size = 8192
num_heads = 64
head_dim = hidden_size / num_heads
context_length = 8192
batch_size = 4

kv_cache_per_token = 2 * num_layers * hidden_size * 2  # 2 for K+V, 2 bytes FP16
total_kv_cache = kv_cache_per_token * context_length * batch_size / (1024**3)
print(f"KV cache: {total_kv_cache:.2f} GB")

This yields roughly 26GB for a batch of 4 requests at full context. With tensor parallelism across 4 GPUs, each device stores approximately 6.5GB of KV cache data. Plan your GPU memory budget accordingly – insufficient KV cache space forces smaller batch sizes and reduces throughput substantially.

Tensor Parallelism vs Pipeline Parallelism Strategies

When serving 70B models across multiple GPUs, you face a fundamental choice between two parallelism strategies that handle model distribution differently.

Tensor parallelism splits individual weight matrices across GPUs. Each transformer layer’s attention and feed-forward operations execute simultaneously on all devices, requiring constant inter-GPU communication. For a 70B model on four A100 GPUs, each device holds roughly 17.5B parameters worth of weights.

vLLM implements tensor parallelism through Ray, automatically sharding attention heads and MLP layers. Launch with:

python -m vllm.entrypoints.openai.api_server \
  --model meta-llama/Llama-2-70b-hf \
  --tensor-parallel-size 4 \
  --gpu-memory-utilization 0.90

This approach minimizes memory per GPU but introduces latency from NVLink bandwidth constraints. Expect roughly 15-20ms additional overhead per forward pass on PCIe-connected GPUs versus NVLink systems.

Pipeline Parallelism Trade-offs

Pipeline parallelism assigns consecutive layers to different GPUs, reducing communication to layer boundaries. TensorRT-LLM supports this through:

from tensorrt_llm import Pipeline
pipeline = Pipeline(
    model_path="llama-70b",
    pipeline_parallel_size=4,
    micro_batch_size=1
)

Each GPU processes its assigned layers sequentially, creating pipeline bubbles where devices wait idle. For interactive workloads with batch size one, pipeline parallelism wastes GPU cycles. However, batch sizes above eight can achieve better throughput than tensor parallelism by overlapping computation across pipeline stages.

Hybrid Approaches

Text Generation Inference combines both strategies. On an eight-GPU server, configure tensor parallelism across pairs and pipeline parallelism between pairs:

text-generation-launcher \
  --model-id meta-llama/Llama-2-70b-hf \
  --num-shard 8 \
  --tensor-parallel-size 2 \
  --pipeline-parallel-size 4

This balances communication overhead with hardware utilization. Test both pure and hybrid configurations with your actual query patterns before committing to production infrastructure.

Serving Framework Comparison: vLLM, TGI, and TensorRT-LLM

When serving 70B models across multiple GPUs, your framework choice directly impacts throughput and memory efficiency. Each option handles tensor parallelism and memory management differently.

vLLM excels at batch processing with PagedAttention, which treats KV cache like virtual memory pages. For a 70B model on four A100s, vLLM typically achieves better throughput than alternatives when handling concurrent requests. Install with:

pip install vllm
vllm serve meta-llama/Llama-2-70b-hf --tensor-parallel-size 4 --gpu-memory-utilization 0.9

The --gpu-memory-utilization flag controls how much VRAM vLLM reserves for KV cache versus model weights. Setting this to 0.9 leaves headroom for CUDA operations while maximizing cache capacity.

Text Generation Inference for Hugging Face Models

TGI integrates tightly with the Hugging Face ecosystem and supports flash attention out of the box. Launch a 70B model with:

docker run --gpus all --shm-size 1g -p 8080:80 \
  ghcr.io/huggingface/text-generation-inference:latest \
  --model-id meta-llama/Llama-2-70b-hf --num-shard 4

TGI automatically handles tensor parallelism across shards. The shared memory size matters for inter-GPU communication – increase --shm-size if you see NCCL errors.

TensorRT-LLM for Lowest Latency

TensorRT-LLM requires model conversion but delivers the fastest single-request latency. Build and serve:

python build.py --model_dir ./llama-70b --dtype float16 \
  --tp_size 4 --output_dir ./trt_engines
mpirun -n 4 python run.py --engine_dir ./trt_engines --tokenizer_dir ./llama-70b

The compilation step takes significant time but produces optimized engines. TensorRT-LLM works best for real-time applications where minimizing first-token latency matters more than maximizing batch throughput.

Caution: Always validate framework-specific configuration flags in a test environment before production deployment. Memory settings that work for one model size may cause OOM errors with different architectures.

KV Cache Sharding and Memory Optimization

When serving 70B models across multiple GPUs, the key-value cache becomes your primary memory bottleneck. Each token in the context window stores key and value tensors for every attention layer, and with 70B models typically having 80 layers, this accumulates rapidly.

For a 70B model with 8192 context length, each token requires approximately 1.5 MB of KV cache storage in FP16 precision. A full context window consumes roughly 12 GB per request. With four concurrent requests, you need 48 GB dedicated solely to KV cache – before accounting for model weights.

Sharding Strategies with vLLM

vLLM implements automatic KV cache sharding through tensor parallelism. When you launch with --tensor-parallel-size 4, the framework distributes both model weights and KV cache across GPUs proportionally. Each GPU stores one-quarter of the attention heads, reducing per-device memory pressure.

from vllm import LLM, SamplingParams

llm = LLM(
    model="meta-llama/Llama-2-70b-hf",
    tensor_parallel_size=4,
    gpu_memory_utilization=0.90,
    max_num_seqs=8,
    max_model_len=4096
)

The gpu_memory_utilization parameter reserves memory for KV cache growth. Setting this to 0.90 leaves approximately 10 percent headroom for dynamic allocation during inference spikes.

PagedAttention for Memory Efficiency

vLLM’s PagedAttention mechanism breaks KV cache into fixed-size blocks, similar to virtual memory paging. This eliminates fragmentation and enables dynamic sharing of cached prefixes across requests. When multiple users query with similar system prompts, the shared prefix blocks are stored once and referenced by multiple sequences.

TensorRT-LLM offers comparable functionality through its --use_paged_kv_cache flag, though configuration requires more manual tuning of block sizes and memory pools.

Caution: Always validate memory calculations against your specific hardware before production deployment. Monitor GPU memory usage with nvidia-smi dmon during load testing to identify actual peak consumption patterns.

Multi-GPU Communication Patterns and NVLink Topology

When serving 70B models across multiple GPUs, the physical interconnect between cards determines your maximum throughput. NVLink provides substantially higher bandwidth than PCIe, making it the preferred topology for tensor parallelism workloads.

Before deploying a 70B model, verify your GPU topology:

nvidia-smi topo -m

Look for NVLink connections marked as NV12 or NV18 between GPUs. PCIe-only connections appear as PIX or PHB. A server with four A100 GPUs connected via NVLink will show direct NV connections between all pairs, while consumer setups often rely on PCIe switching.

Bandwidth Impact on Serving Performance

Tensor parallelism splits model layers across GPUs, requiring constant synchronization during inference. Each token generation involves multiple all-reduce operations to aggregate activations. NVLink’s 600 GB/s bidirectional bandwidth per link enables this communication with minimal latency overhead.

For PCIe Gen4 x16 connections, the 64 GB/s theoretical maximum becomes a bottleneck. You will observe higher time-to-first-token and reduced throughput as the framework waits for cross-GPU transfers to complete.

Framework-Specific Topology Awareness

vLLM automatically detects NVLink topology and adjusts its pipeline parallelism strategy:

from vllm import LLM

llm = LLM(
    model="meta-llama/Llama-2-70b-hf",
    tensor_parallel_size=4,
    gpu_memory_utilization=0.95
)

The framework will prefer placing sequential pipeline stages on NVLink-connected GPUs to minimize communication overhead. TensorRT-LLM requires explicit topology specification in its build configuration, while TGI handles detection automatically but allows manual override via environment variables.

Caution: Always validate GPU assignments match your physical topology. Misconfigurations can route traffic through PCIe when NVLink paths exist, degrading performance without obvious errors. Use nvidia-smi nvlink -s during inference to monitor actual link utilization and confirm your serving framework uses the optimal paths.

Installation and Configuration Steps

For 70B models, vLLM offers the best balance of throughput and ease of use. Install it with tensor parallelism support:

pip install vllm==0.3.2
pip install ray==2.9.0

TensorRT-LLM provides lower latency but requires more configuration. Text Generation Inference works well for Hugging Face models but has higher memory overhead for large context windows.

Configuring Tensor Parallelism

Launch a 70B model across four GPUs using vLLM’s tensor parallel mode:

python -m vllm.entrypoints.openai.api_server \
  --model meta-llama/Llama-2-70b-chat-hf \
  --tensor-parallel-size 4 \
  --gpu-memory-utilization 0.90 \
  --max-model-len 4096 \
  --dtype float16

The tensor-parallel-size must match your GPU count. Each GPU holds roughly 18GB of model weights for a 70B FP16 model, plus KV cache overhead.

Memory Optimization Settings

Enable KV cache sharding to reduce per-GPU memory pressure:

python -m vllm.entrypoints.openai.api_server \
  --model meta-llama/Llama-2-70b-chat-hf \
  --tensor-parallel-size 4 \
  --enable-prefix-caching \
  --max-num-seqs 128 \
  --max-num-batched-tokens 8192

The max-num-seqs controls concurrent requests. Start conservative and increase based on your available VRAM after model loading.

Validation and Testing

Test your deployment with a simple request:

import openai
openai.api_base = "http://localhost:8000/v1"
openai.api_key = "dummy"

response = openai.ChatCompletion.create(
    model="meta-llama/Llama-2-70b-chat-hf",
    messages=[{"role": "user", "content": "Explain tensor parallelism"}],
    max_tokens=100
)
print(response.choices[0].message.content)

Caution: Always verify AI-generated serving configurations match your hardware specifications before deploying to production. Monitor GPU memory usage with nvidia-smi dmon during initial load testing to prevent OOM crashes.