EKS Networking

Comprehensive guide for configuring Amazon EKS networking including VPC CNI plugin, load balancers, network policies, and security.

Overview

EKS networking involves several key components working together:

VPC CNI Plugin - Assigns real VPC IP addresses to pods
Load Balancers - ALB for Layer 7, NLB for Layer 4 traffic
Network Policies - Control pod-to-pod and pod-to-external traffic
Security Groups for Pods - AWS-level network security
DNS - CoreDNS for in-cluster, ExternalDNS for external records
Service Discovery - AWS Cloud Map for multi-cluster

Quick Start

1. Enable VPC CNI with Prefix Mode

# Update VPC CNI addon with prefix delegation
aws eks update-addon \
  --cluster-name my-cluster \
  --addon-name vpc-cni \
  --addon-version v1.19.2-eksbuild.1 \
  --configuration-values '{
    "env": {
      "ENABLE_PREFIX_DELEGATION": "true",
      "WARM_PREFIX_TARGET": "1"
    }
  }'

# Verify configuration
kubectl get daemonset -n kube-system aws-node -o yaml | grep ENABLE_PREFIX_DELEGATION

2. Install AWS Load Balancer Controller

# Create IAM policy
curl -O https://raw.githubusercontent.com/kubernetes-sigs/aws-load-balancer-controller/v2.11.0/docs/install/iam_policy.json

aws iam create-policy \
  --policy-name AWSLoadBalancerControllerIAMPolicy \
  --policy-document file://iam_policy.json

# Create IRSA
eksctl create iamserviceaccount \
  --cluster=my-cluster \
  --namespace=kube-system \
  --name=aws-load-balancer-controller \
  --attach-policy-arn=arn:aws:iam::ACCOUNT_ID:policy/AWSLoadBalancerControllerIAMPolicy \
  --approve

# Install via Helm
helm repo add eks https://aws.github.io/eks-charts
helm repo update

helm install aws-load-balancer-controller eks/aws-load-balancer-controller \
  -n kube-system \
  --set clusterName=my-cluster \
  --set serviceAccount.create=false \
  --set serviceAccount.name=aws-load-balancer-controller

3. Create ALB Ingress

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: app-ingress
  annotations:
    alb.ingress.kubernetes.io/scheme: internet-facing
    alb.ingress.kubernetes.io/target-type: ip
    alb.ingress.kubernetes.io/group.name: shared-alb
    alb.ingress.kubernetes.io/listen-ports: '[{"HTTP": 80}, {"HTTPS": 443}]'
    alb.ingress.kubernetes.io/ssl-redirect: '443'
    alb.ingress.kubernetes.io/certificate-arn: arn:aws:acm:region:account:certificate/xxx
spec:
  ingressClassName: alb
  rules:
  - host: app.example.com
    http:
      paths:
      - path: /
        pathType: Prefix
        backend:
          service:
            name: app-service
            port:
              number: 80

4. Enable Network Policies

# VPC CNI v1.14+ supports network policies natively
kubectl set env daemonset -n kube-system aws-node ENABLE_NETWORK_POLICY=true

# Apply a network policy
kubectl apply -f - <<EOF
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: api-network-policy
  namespace: default
spec:
  podSelector:
    matchLabels:
      app: api
  policyTypes:
  - Ingress
  - Egress
  ingress:
  - from:
    - podSelector:
        matchLabels:
          app: frontend
    ports:
    - protocol: TCP
      port: 8080
  egress:
  - to:
    - podSelector:
        matchLabels:
          app: database
    ports:
    - protocol: TCP
      port: 5432
  - to:
    - namespaceSelector: {}
      podSelector:
        matchLabels:
          k8s-app: kube-dns
    ports:
    - protocol: UDP
      port: 53
EOF

Workflow: Complete Networking Setup

Phase 1: VPC and Subnet Planning

1.1 Calculate IP Requirements

# Formula: IPs per node = (max-pods × 2) + ENIs + buffer
# Example for m5.large:
# - Max pods: 29
# - ENIs: 3
# - IPs needed: (29 × 2) + 3 + 5 = 66 IPs per node

# For a 10-node cluster: 660 IPs minimum
# Recommended subnet: /24 (254 IPs) or /23 (510 IPs)

1.2 Tag Subnets for EKS

# Public subnets (for internet-facing ALB/NLB)
aws ec2 create-tags \
  --resources subnet-xxx \
  --tags Key=kubernetes.io/role/elb,Value=1

# Private subnets (for internal ALB/NLB and worker nodes)
aws ec2 create-tags \
  --resources subnet-yyy \
  --tags Key=kubernetes.io/role/internal-elb,Value=1

# Cluster-specific tags (required)
aws ec2 create-tags \
  --resources subnet-xxx subnet-yyy \
  --tags Key=kubernetes.io/cluster/my-cluster,Value=shared

Phase 2: Configure VPC CNI

2.1 Choose Configuration Mode

# Option 1: Standard Mode (default)
# - One IP per pod
# - Limited by ENI capacity

# Option 2: Prefix Delegation Mode (recommended for high pod density)
kubectl set env daemonset -n kube-system aws-node ENABLE_PREFIX_DELEGATION=true
kubectl set env daemonset -n kube-system aws-node WARM_PREFIX_TARGET=1

# Option 3: IPv6 (recommended for IP exhaustion issues)
# - Virtually unlimited IPs
# - Must create IPv6-enabled cluster

# Option 4: Custom Networking (for secondary CIDR)
kubectl set env daemonset -n kube-system aws-node AWS_VPC_K8S_CNI_CUSTOM_NETWORK_CFG=true
kubectl set env daemonset -n kube-system aws-node ENI_CONFIG_LABEL_DEF=topology.kubernetes.io/zone

2.2 Configure IP Management

# Warm pool settings (reduce pod startup time)
kubectl set env daemonset -n kube-system aws-node WARM_IP_TARGET=5
kubectl set env daemonset -n kube-system aws-node MINIMUM_IP_TARGET=10

# Maximum IPs per node
kubectl set env daemonset -n kube-system aws-node MAX_ENI=3

# Monitor IP usage
kubectl apply -f https://raw.githubusercontent.com/aws/amazon-vpc-cni-k8s/master/config/master/cni-metrics-helper.yaml

Phase 3: Load Balancer Setup

3.1 Configure IngressClass

apiVersion: networking.k8s.io/v1
kind: IngressClass
metadata:
  name: alb
spec:
  controller: ingress.k8s.aws/alb
---
apiVersion: networking.k8s.io/v1
kind: IngressClass
metadata:
  name: nginx
spec:
  controller: k8s.io/ingress-nginx

3.2 Create Shared ALB with IngressGroups

# App 1
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: app1-ingress
  annotations:
    alb.ingress.kubernetes.io/group.name: shared-alb
    alb.ingress.kubernetes.io/scheme: internet-facing
    alb.ingress.kubernetes.io/target-type: ip
    alb.ingress.kubernetes.io/healthcheck-path: /health
spec:
  ingressClassName: alb
  rules:
  - host: app1.example.com
    http:
      paths:
      - path: /
        pathType: Prefix
        backend:
          service:
            name: app1-service
            port:
              number: 80
---
# App 2 (shares same ALB)
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: app2-ingress
  annotations:
    alb.ingress.kubernetes.io/group.name: shared-alb
    alb.ingress.kubernetes.io/scheme: internet-facing
    alb.ingress.kubernetes.io/target-type: ip
spec:
  ingressClassName: alb
  rules:
  - host: app2.example.com
    http:
      paths:
      - path: /
        pathType: Prefix
        backend:
          service:
            name: app2-service
            port:
              number: 80

3.3 Create NLB for TCP Services

apiVersion: v1
kind: Service
metadata:
  name: tcp-service
  annotations:
    service.beta.kubernetes.io/aws-load-balancer-type: external
    service.beta.kubernetes.io/aws-load-balancer-nlb-target-type: ip
    service.beta.kubernetes.io/aws-load-balancer-scheme: internet-facing
    service.beta.kubernetes.io/aws-load-balancer-healthcheck-protocol: TCP
    service.beta.kubernetes.io/aws-load-balancer-cross-zone-load-balancing-enabled: "true"
spec:
  type: LoadBalancer
  selector:
    app: tcp-app
  ports:
  - port: 3306
    targetPort: 3306
    protocol: TCP

Phase 4: Network Security

4.1 Enable Network Policies

# For VPC CNI v1.14+
kubectl set env daemonset -n kube-system aws-node ENABLE_NETWORK_POLICY=true

# OR install Calico for enhanced policies
kubectl apply -f https://raw.githubusercontent.com/projectcalico/calico/v3.28.0/manifests/calico-policy-only.yaml

4.2 Implement Security Groups for Pods

# Enable SGP in VPC CNI
kubectl set env daemonset -n kube-system aws-node ENABLE_POD_ENI=true

# Create security group
aws ec2 create-security-group \
  --group-name pod-security-group \
  --description "Security group for pods" \
  --vpc-id vpc-xxx

# Create SecurityGroupPolicy
kubectl apply -f - <<EOF
apiVersion: vpcresources.k8s.aws/v1beta1
kind: SecurityGroupPolicy
metadata:
  name: database-pods-sg
  namespace: default
spec:
  podSelector:
    matchLabels:
      app: database
  securityGroups:
    groupIds:
    - sg-xxx
EOF

4.3 Default Deny Network Policy

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: default-deny-all
  namespace: production
spec:
  podSelector: {}
  policyTypes:
  - Ingress
  - Egress

Phase 5: DNS and Service Discovery

5.1 Configure CoreDNS

# Scale CoreDNS for large clusters
kubectl scale deployment coredns -n kube-system --replicas=3

# Custom DNS forwarding
kubectl edit configmap coredns -n kube-system
# Add custom forwarding rules in Corefile

5.2 Install ExternalDNS

# Create IAM policy for Route 53
cat > external-dns-policy.json <<EOF
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": [
        "route53:ChangeResourceRecordSets"
      ],
      "Resource": "arn:aws:route53:::hostedzone/*"
    },
    {
      "Effect": "Allow",
      "Action": [
        "route53:ListHostedZones",
        "route53:ListResourceRecordSets"
      ],
      "Resource": "*"
    }
  ]
}
EOF

# Create IRSA
eksctl create iamserviceaccount \
  --cluster=my-cluster \
  --namespace=kube-system \
  --name=external-dns \
  --attach-policy-arn=arn:aws:iam::ACCOUNT_ID:policy/ExternalDNSPolicy \
  --approve

# Deploy ExternalDNS
helm repo add external-dns https://kubernetes-sigs.github.io/external-dns/
helm install external-dns external-dns/external-dns \
  -n kube-system \
  --set serviceAccount.create=false \
  --set serviceAccount.name=external-dns \
  --set provider=aws \
  --set policy=sync \
  --set txtOwnerId=my-cluster

Phase 6: Optimization and Monitoring

6.1 Reduce Cross-AZ Traffic

# Use topology-aware routing
apiVersion: v1
kind: Service
metadata:
  name: app-service
  annotations:
    service.kubernetes.io/topology-mode: Auto
spec:
  internalTrafficPolicy: Local
  selector:
    app: myapp
  ports:
  - port: 80
    targetPort: 8080

6.2 Deploy VPC Endpoints

# Create VPC endpoints for AWS services
aws ec2 create-vpc-endpoint \
  --vpc-id vpc-xxx \
  --service-name com.amazonaws.us-west-2.ecr.api \
  --vpc-endpoint-type Interface \
  --subnet-ids subnet-xxx subnet-yyy \
  --security-group-ids sg-zzz

# Common endpoints to create:
# - com.amazonaws.region.ecr.api
# - com.amazonaws.region.ecr.dkr
# - com.amazonaws.region.s3
# - com.amazonaws.region.logs
# - com.amazonaws.region.sts

6.3 Monitor Network Metrics

# Deploy CNI metrics helper
kubectl apply -f https://raw.githubusercontent.com/aws/amazon-vpc-cni-k8s/master/config/master/cni-metrics-helper.yaml

# View metrics
kubectl top nodes
kubectl get --raw /apis/metrics.k8s.io/v1beta1/nodes

Common Tasks

Check Available IPs

# View CNI metrics
kubectl get daemonset aws-node -n kube-system -o yaml | grep -A 10 WARM

# Check CloudWatch metrics
aws cloudwatch get-metric-statistics \
  --namespace AWS/EKS \
  --metric-name cluster_ip_allocation \
  --dimensions Name=ClusterName,Value=my-cluster \
  --statistics Average \
  --start-time 2025-01-01T00:00:00Z \
  --end-time 2025-01-27T00:00:00Z \
  --period 3600

Troubleshoot Pod Networking

# Check pod IP assignment
kubectl get pods -o wide

# Describe pod networking
kubectl describe pod <pod-name> | grep IP

# Check CNI logs
kubectl logs -n kube-system -l k8s-app=aws-node --tail=50

# Verify ENI attachments
aws ec2 describe-network-interfaces \
  --filters "Name=attachment.instance-id,Values=i-xxx"

Verify Load Balancer Configuration

# Check ingress status
kubectl get ingress -A
kubectl describe ingress <ingress-name>

# Check ALB controller logs
kubectl logs -n kube-system deployment/aws-load-balancer-controller

# Verify target groups
aws elbv2 describe-target-groups
aws elbv2 describe-target-health --target-group-arn <arn>

Test Network Policies

# Create test pod
kubectl run test-pod --image=nicolaka/netshoot -it --rm -- /bin/bash

# Test connectivity
curl http://service-name:port
nc -zv service-name port

# Verify policy applied
kubectl get networkpolicy
kubectl describe networkpolicy <policy-name>

Reference Documentation

For detailed information, see:

VPC CNI: references/vpc-cni.md - CNI plugin configuration, modes, and optimization
Load Balancers: references/load-balancers.md - ALB, NLB, and AWS Load Balancer Controller
Network Policies: references/network-policies.md - Network policies, security groups, and segmentation

Best Practices

IP Address Management

Use prefix delegation mode for high pod density clusters
Consider IPv6 for new clusters to avoid IP exhaustion
Monitor IP usage with CNI metrics helper
Plan subnets with 2x expected IP requirements

Load Balancing

Use IngressGroups to share ALBs and reduce costs
Set target-type to ip for best performance
Enable cross-zone load balancing for high availability
Use NLB for static IP requirements

Network Security

Enable network policies (VPC CNI v1.14+ or Calico)
Combine NetworkPolicies with Security Groups for Pods (defense-in-depth)
Default deny, explicit allow policy approach
Use private subnets for worker nodes

Performance and Cost

Deploy VPC endpoints for AWS services (reduces NAT costs)
Use topology-aware routing to minimize cross-AZ traffic
Enable prefix delegation to reduce ENI pressure
Monitor cross-AZ traffic with Container Network Observability

High Availability

Spread subnets across at least 3 availability zones
Use multiple replicas for critical services
Configure pod topology spread constraints
Enable cross-zone load balancing

Troubleshooting

Pod Can't Get IP Address

Symptoms: Pod stuck in ContainerCreating

Check:

# View CNI logs
kubectl logs -n kube-system -l k8s-app=aws-node

# Check available IPs
kubectl get nodes -o jsonpath='{.items[*].status.allocatable.pods}'

# Verify ENI limits not reached
aws ec2 describe-instances --instance-ids i-xxx

Solutions:

Enable prefix delegation mode
Use larger instance types (more ENIs)
Add more nodes to cluster

Ingress Not Creating ALB

Symptoms: No load balancer provisioned for Ingress

Check:

# Verify controller running
kubectl get pods -n kube-system | grep aws-load-balancer-controller

# Check controller logs
kubectl logs -n kube-system deployment/aws-load-balancer-controller

# Verify subnet tags
aws ec2 describe-subnets --subnet-ids subnet-xxx

Solutions:

Ensure subnets properly tagged
Verify IAM permissions for controller
Check IngressClass specified correctly

Network Policy Not Working

Symptoms: Traffic not blocked as expected

Check:

# Verify network policy enabled
kubectl get daemonset -n kube-system aws-node -o yaml | grep ENABLE_NETWORK_POLICY

# Check policy applied
kubectl get networkpolicy -A
kubectl describe networkpolicy <name>

Solutions:

Enable network policy support in CNI
Verify label selectors match pods
Check policy has both ingress and egress rules

DNS Resolution Failures

Symptoms: Pods can't resolve service names

Check:

# Test DNS from pod
kubectl run -it --rm debug --image=busybox --restart=Never -- nslookup kubernetes.default

# Check CoreDNS status
kubectl get pods -n kube-system -l k8s-app=kube-dns

# View CoreDNS logs
kubectl logs -n kube-system -l k8s-app=kube-dns

Solutions:

Scale CoreDNS replicas
Check CoreDNS ConfigMap
Verify network policy allows DNS traffic

2025 Recommendations

CNI Selection

Default: VPC CNI with prefix delegation mode
IPv4 exhaustion: IPv6 clusters (AWS recommended)
Advanced policies: VPC CNI + Calico policy-only agent
Maximum features: Cilium (for EKS Hybrid Nodes or advanced use cases)

Load Balancing

AWS Native: AWS Load Balancer Controller (recommended)
Multi-cloud portability: NGINX Ingress with NLB
Service mesh: Istio Gateway (App Mesh deprecated Sept 2026)

Network Security

VPC CNI native network policies (v1.14+) for most use cases
Calico for enhanced policy features and observability
Cilium for eBPF-powered security and deep insights
Always combine with Security Groups for Pods (defense-in-depth)

Cost Optimization

VPC endpoints for all AWS services (ECR, S3, CloudWatch, etc.)
Topology-aware routing to minimize cross-AZ traffic
Single NAT Gateway per AZ (not per subnet)
Monitor network costs with Container Network Observability

eks-networking

EKS Networking

Overview

Quick Start

1. Enable VPC CNI with Prefix Mode

2. Install AWS Load Balancer Controller

3. Create ALB Ingress

4. Enable Network Policies

Workflow: Complete Networking Setup

Phase 1: VPC and Subnet Planning

Phase 2: Configure VPC CNI

Phase 3: Load Balancer Setup

Phase 4: Network Security

Phase 5: DNS and Service Discovery

Phase 6: Optimization and Monitoring

Common Tasks

Check Available IPs

Troubleshoot Pod Networking

Verify Load Balancer Configuration

Test Network Policies

Reference Documentation

Best Practices

IP Address Management

Load Balancing

Network Security

Performance and Cost

High Availability

Troubleshooting

Pod Can't Get IP Address

Ingress Not Creating ALB

Network Policy Not Working

DNS Resolution Failures

2025 Recommendations

CNI Selection

Load Balancing

Network Security

Cost Optimization