serving-llms-vllm

Malicious tool definition detected

Tool: SKILL.md [1/2] Description: --- name: serving-llms-vllm description: Serves LLMs with high throughput using vLLM's PagedAttention and continuous batching.

Tool: SKILL.md [2/2] Description: faster generation: ```bash vllm serve MODEL --speculative-model DRAFT_MODEL ``` ## Advanced topics **Server deployment patterns**: See [references/server-deployment.md](references/server-deployment.md) for Docker, Kubernetes, and load balancing configurations.

Malicious tool definition detected

Tool: references/optimization.md Description: # Performance Optimization ## Contents - PagedAttention explained - Continuous batching mechanics - Prefix caching strategies - Speculative decoding setup - Benchmark results and comparisons - Performance tuning guide ## PagedAttention explained **Traditional attention problem**: - KV cache stored in contiguous memory - Wastes ~50% GPU memory due to fragmentation - Cannot dynamically reallocate for varying sequence lengths **PagedAttention solution**

Malicious tool definition detected

Tool: references/quantization.md Description: # Quantization Guide ## Contents - Quantization methods comparison - AWQ setup and usage - GPTQ setup and usage - FP8 quantization (H100) - Model preparation - Accuracy vs compression trade-offs ## Quantization methods comparison | Method | Compression | Accuracy Loss | Speed | Best For | |--------|-------------|---------------|-------|----------| | **AWQ** | 4-bit (75%) | <1% | Fast | 70B models, production | | **GPTQ** | 4-bit (75%) | 1-2% | Fast |

Malicious tool definition detected

Tool: references/server-deployment.md Description: # Server Deployment Patterns ## Contents - Docker deployment - Kubernetes deployment - Load balancing with Nginx - Multi-node distributed serving - Production configuration examples - Health checks and monitoring ## Docker deployment **Basic Dockerfile**: ```dockerfile FROM nvidia/cuda:12.1.0-devel-ubuntu22.04 RUN apt-get update && apt-get install -y python3-pip RUN pip install vllm EXPOSE 8000 CMD ["vllm", "serve", "meta-llama/Llama-3-8B-Instru

Malicious tool definition detected

Tool: references/troubleshooting.md [1/2] Description: # Troubleshooting Guide ## Contents - Out of memory (OOM) errors - Performance issues - Model loading errors - Network and connection issues - Quantization problems - Distributed serving issues - Debugging tools and commands ## Out of memory (OOM) errors ### Symptom: `torch.cuda.OutOfMemoryError` during model loading **Cause**: Model + KV cache exceeds available VRAM **Solutions (try in order)**: 1.

Tool: references/troubleshooting.md [2/2]