GPU Kubernetes Operations

Run resilient and cost-efficient GPU clusters for production AI workloads.

Key Capabilities

• NVIDIA device plugin and GPU operator lifecycle
• MIG partitioning for multi-workload efficiency
• GPU-aware autoscaling (KEDA/cluster autoscaler)
• Node health checks and proactive remediation

Cluster Baseline

• Dedicated GPU node pools with taints and tolerations
• Runtime class and driver/toolkit compatibility checks
• Local SSD or high-throughput network storage for model weights
• DCGM metrics exported to Prometheus

Scheduling Patterns

• Use node affinity by GPU type (A10/L4/A100/H100).
• Separate latency-critical inference from batch training.
• Pin model replicas with anti-affinity for availability.
• Reserve headroom for failover and rolling updates.

Autoscaling Strategy

• Scale on queue depth + GPU utilization, not CPU alone.
• Warm spare replicas for large model cold-start mitigation.
• Cap burst scaling to avoid quota exhaustion.

Reliability Checks

• ECC error and Xid monitoring
• GPU memory pressure alerts
• Driver mismatch detection during upgrades
• Pod preemption impact analysis

Cost Optimization

• Prefer MIG slices for smaller inference services.
• Schedule batch jobs in off-peak windows.
• Route low-priority traffic to cheaper model tiers.

Related Skills

• llm-inference-scaling - Autoscale inference workloads
• model-serving-kubernetes - Production model serving patterns
• gpu-server-management - Host-level GPU management fundamentals