vLLM Serving SOP

1. 何时激活 (When to activate)

Activate this skill when any of the following hold:

• The user wants to serve an LLM in production (multi-user, concurrent requests, throughput-oriented) and has GPU infrastructure.
• The user is comparing inference engines (vLLM vs TGI vs SGLang vs TensorRT-LLM vs llama.cpp/Ollama).
• The user reports a vLLM operational issue: CUDA OOM, low throughput, high TTFT, request preemption, multi-GPU setup, quantization choice.
• The user is sizing hardware for an open-weights model (Llama / Qwen / Mixtral / DeepSeek-V3) and asking about tensor/pipeline parallelism.
• The user mentions PagedAttention, prefix caching, continuous batching, chunked prefill, or speculative decoding.

Do NOT activate for: training/fine-tuning (use accelerate/deepspeed/trl), CPU-only edge inference (use llama.cpp/Ollama), Apple Silicon production (vLLM Metal/MPS is experimental, not production-ready as of 2026) [aimadetools.com 2026], API-only consumption of hosted models (just call the OpenAI/Anthropic SDK).

2. 核心心智模型 (Core Mental Model)

2.1 The OS abstraction: KV cache as virtual memory