Change Management

Implement structured change management processes covering change classification, CAB workflows, emergency change procedures, and automation for compliance with SOC 2, ITIL, and regulatory frameworks.

When to Use

• Establishing change management processes for production environments
• Implementing change advisory board (CAB) workflows
• Defining change classification and approval requirements
• Configuring automated change tracking in CI/CD pipelines
• Handling emergency changes with proper controls and documentation

Change Classification

change_types:
  standard:
    risk: Low
    approval: Pre-approved (no per-change approval needed)
    lead_time: None (within maintenance window)
    examples:
      - Routine patching within tested patch sets
      - Certificate rotation with established procedure
      - Scaling operations (adding/removing instances within limits)
      - Pre-approved configuration changes
      - Log rotation and archival
    requirements:
      - Change must match an approved Standard Change template
      - Automated testing must pass
      - Documented rollback procedure exists
      - Within defined maintenance window

  normal_low:
    risk: Low
    approval: Peer review (1 approver)
    lead_time: 2 business days
    examples:
      - Non-critical configuration changes
      - Feature flag toggles
      - Documentation updates to production systems
      - Adding monitoring dashboards or alerts

  normal_medium:
    risk: Medium
    approval: Team lead + peer review (2 approvers)
    lead_time: 5 business days
    examples:
      - Application deployments with new features
      - Database schema changes (non-breaking)
      - Network rule modifications
      - Integration endpoint changes
      - Dependency version upgrades

  normal_high:
    risk: High
    approval: CAB review required
    lead_time: 10 business days
    examples:
      - Infrastructure migrations
      - Breaking database schema changes
      - Major version upgrades (OS, runtime, database engine)
      - Changes to authentication or authorization systems
      - Multi-service coordinated deployments
      - Changes affecting data processing or compliance controls

  emergency:
    risk: Variable
    approval: Emergency CAB (minimum 2 approvers from on-call)
    lead_time: None (immediate implementation)
    examples:
      - Security vulnerability remediation (active exploitation)
      - Production outage resolution
      - Data integrity emergency fixes
      - Regulatory compliance deadline fixes
    requirements:
      - Retroactive full documentation within 48 hours
      - Post-implementation review required
      - CAB retroactive review at next meeting

Change Request Template

change_request:
  metadata:
    id: "CR-YYYY-NNNN"
    title: ""
    requestor: ""
    date_submitted: ""
    target_date: ""
    change_type: ""  # standard | normal_low | normal_medium | normal_high | emergency

  description:
    summary: "Brief description of the change"
    detailed_description: "Full technical details of what will change"
    business_justification: "Why this change is needed"
    affected_systems: []
    affected_services: []
    affected_users: "Description of user impact"

  risk_assessment:
    risk_level: ""  # low | medium | high
    impact_if_failed: "What happens if the change fails"
    likelihood_of_failure: ""  # low | medium | high
    risk_mitigation: "Steps to reduce risk"
    dependencies: "Other systems or changes this depends on"

  implementation:
    change_window:
      start: ""
      end: ""
      maintenance_window: true
    implementation_steps:
      - step: "Step 1 description"
        responsible: "Person/team"
        estimated_duration: "X minutes"
      - step: "Step 2 description"
        responsible: "Person/team"
        estimated_duration: "X minutes"

  testing:
    pre_change_testing:
      - "Unit tests pass"
      - "Integration tests pass"
      - "Staging deployment verified"
    post_change_verification:
      - "Health check endpoints responding"
      - "Key transactions processing successfully"
      - "No error rate increase in monitoring"
      - "Performance metrics within baseline"

  rollback:
    rollback_plan: "Detailed steps to revert the change"
    rollback_trigger: "Conditions that trigger rollback"
    rollback_estimated_time: "X minutes"
    rollback_steps:
      - "Step 1: Revert deployment to previous version"
      - "Step 2: Verify rollback successful"
      - "Step 3: Notify stakeholders"
    data_rollback: "Describe any data migration rollback needed"

  communication:
    stakeholders_notified: []
    notification_sent_date: ""
    status_page_update: true
    customer_notification_required: false

  approvals:
    technical_reviewer: ""
    technical_approval_date: ""
    security_reviewer: ""
    security_approval_date: ""
    cab_approval_date: ""
    cab_notes: ""

  closure:
    implementation_date: ""
    implementation_result: ""  # success | partial | failed | rolled_back
    post_implementation_review: ""
    lessons_learned: ""
    follow_up_actions: []

CAB Workflow

cab_workflow:
  meeting_schedule:
    regular_cab: "Weekly, Thursday 2:00 PM"
    emergency_cab: "On-demand, minimum 2 members required"

  cab_members:
    permanent:
      - Engineering Manager (Chair)
      - Security Team Representative
      - Infrastructure/SRE Lead
      - Release Manager
    advisory:
      - Business stakeholder (invited per change)
      - Database administrator (for DB changes)
      - Network engineer (for network changes)

  agenda:
    1: "Review emergency changes from prior week"
    2: "Review high-risk change requests for upcoming window"
    3: "Review failed changes and lessons learned"
    4: "Discuss upcoming change freeze periods"
    5: "Review change metrics and trends"

  decision_criteria:
    approve_when:
      - Risk assessment is complete and accurate
      - Testing evidence is provided
      - Rollback plan is documented and feasible
      - Change window is appropriate
      - Required approvals obtained
      - No conflicts with other scheduled changes
    request_changes_when:
      - Rollback plan is missing or incomplete
      - Testing is insufficient for the risk level
      - Impact assessment needs clarification
      - Change conflicts with another scheduled change
    deny_when:
      - Risk is unacceptable without mitigation
      - Change window conflicts with freeze period
      - Dependencies are not resolved
      - Compliance concerns are unaddressed

Emergency Change Procedure

emergency_change_process:
  definition: "A change required to restore service or prevent imminent security compromise"

  step_1_declare:
    actions:
      - On-call engineer identifies need for emergency change
      - Incident commander approves emergency classification
      - Minimum 2 approvers from emergency CAB roster contacted
      - Document initial justification in incident channel

  step_2_approve:
    approval_method:
      - Slack/Teams approval with screenshots preserved
      - Verbal approval over bridge call (documented in notes)
      - Emergency approvers can be any 2 of the following roles:
        - Engineering Manager
        - SRE/Infrastructure Lead
        - Security Team Lead
        - VP of Engineering
    timeout: "If no response in 15 minutes, escalate to next tier"

  step_3_implement:
    actions:
      - Implement the minimum change needed to resolve the issue
      - Record all actions taken with timestamps
      - Monitor for successful resolution
      - Document any deviations from planned change

  step_4_verify:
    actions:
      - Confirm service restoration
      - Verify no unintended side effects
      - Run post-change verification checks
      - Update status page and stakeholders

  step_5_document:
    deadline: "Within 48 hours of implementation"
    required_documentation:
      - Complete change request form (retroactive)
      - Timeline of events and actions
      - Justification for emergency classification
      - Approval records (messages, emails)
      - Post-implementation verification results
      - Root cause analysis (what made it an emergency)
      - Preventive actions to avoid future emergency

  step_6_review:
    actions:
      - CAB review at next regular meeting
      - Assess if emergency classification was appropriate
      - Identify process improvements
      - Track emergency change trends

Pull Request Template for Changes

## Change Request

### Type
- [ ] Standard (pre-approved, low risk)
- [ ] Normal - Low Risk
- [ ] Normal - Medium Risk
- [ ] Normal - High Risk (CAB required)
- [ ] Emergency (retroactive documentation required)

### Description
<!-- What is being changed and why? -->

### Risk Assessment
**Impact if failed:** <!-- What breaks? -->
**Likelihood of failure:** Low / Medium / High
**Affected services:** <!-- List services -->
**User impact:** <!-- Will users notice? -->

### Testing Evidence
- [ ] Unit tests pass
- [ ] Integration tests pass
- [ ] Staging deployment verified
- [ ] Performance test completed (if applicable)
- [ ] Security scan clean (if applicable)

### Rollback Plan
<!-- How to revert if something goes wrong -->
**Estimated rollback time:** <!-- X minutes -->
**Data rollback needed:** Yes / No

### Deployment Plan
**Target window:** <!-- Date and time -->
**Estimated duration:** <!-- X minutes -->

### Post-Deployment Verification
- [ ] Health checks passing
- [ ] Error rates within baseline
- [ ] Key transactions working
- [ ] Monitoring dashboards reviewed

### Communication
- [ ] Team notified
- [ ] Stakeholders notified (if user-facing)
- [ ] Status page updated (if applicable)

### Approvals Required
- [ ] Peer review
- [ ] Team lead (medium+ risk)
- [ ] Security review (security-impacting changes)
- [ ] CAB approval (high risk)

CI/CD Change Tracking Automation

# GitHub Actions - Automated change tracking
name: Change Management
on:
  pull_request:
    types: [opened, synchronize, labeled]
  push:
    branches: [main]

jobs:
  classify-change:
    if: github.event_name == 'pull_request'
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4

      - name: Classify change risk
        id: classify
        run: |
          FILES_CHANGED=$(gh pr diff ${{ github.event.pull_request.number }} --name-only)

          # High risk indicators
          if echo "$FILES_CHANGED" | grep -qE 'terraform/|infrastructure/|migrations/|auth/|security/'; then
            echo "risk=high" >> $GITHUB_OUTPUT
            echo "::warning::High-risk change detected - CAB review may be required"
          elif echo "$FILES_CHANGED" | grep -qE 'config/|database/|api/'; then
            echo "risk=medium" >> $GITHUB_OUTPUT
          else
            echo "risk=low" >> $GITHUB_OUTPUT
          fi
        env:
          GH_TOKEN: ${{ secrets.GITHUB_TOKEN }}

      - name: Add risk label
        run: |
          gh pr edit ${{ github.event.pull_request.number }} \
            --add-label "risk:${{ steps.classify.outputs.risk }}"
        env:
          GH_TOKEN: ${{ secrets.GITHUB_TOKEN }}

      - name: Enforce approvals by risk
        if: steps.classify.outputs.risk == 'high'
        run: |
          APPROVALS=$(gh pr view ${{ github.event.pull_request.number }} \
            --json reviews --jq '[.reviews[] | select(.state=="APPROVED")] | length')
          if [ "$APPROVALS" -lt 2 ]; then
            echo "::error::High-risk changes require at least 2 approvals"
            exit 1
          fi
        env:
          GH_TOKEN: ${{ secrets.GITHUB_TOKEN }}

  record-deployment:
    if: github.event_name == 'push' && github.ref == 'refs/heads/main'
    runs-on: ubuntu-latest
    steps:
      - name: Record deployment
        run: |
          CHANGE_ID="CR-$(date +%Y)-$(printf '%04d' ${{ github.run_number }})"
          echo "Change ID: $CHANGE_ID"
          echo "Deployed at: $(date -u +%Y-%m-%dT%H:%M:%SZ)"
          echo "Commit: ${{ github.sha }}"
          echo "Author: ${{ github.actor }}"

          cat > /tmp/deployment-record.json <<EOF
          {
            "change_id": "$CHANGE_ID",
            "timestamp": "$(date -u +%Y-%m-%dT%H:%M:%SZ)",
            "commit": "${{ github.sha }}",
            "author": "${{ github.actor }}",
            "environment": "production",
            "status": "deployed"
          }
          EOF

Change Freeze Policy

change_freeze:
  definition: "Period during which non-emergency changes are prohibited"

  scheduled_freezes:
    year_end:
      start: "December 15"
      end: "January 3"
      scope: "All production changes"
    major_events:
      - "Black Friday through Cyber Monday (e-commerce)"
      - "Tax filing deadline periods (financial services)"
      - "Open enrollment periods (healthcare)"

  exceptions_during_freeze:
    allowed:
      - Security patches for actively exploited vulnerabilities
      - Changes required by regulatory deadline
      - Fixes for P1/SEV1 production incidents
    approval: "VP of Engineering + Security Lead"

  communication:
    announcement: "2 weeks before freeze"
    reminder: "1 week and 1 day before freeze"
    daily_status: "During freeze period"
    lift_notification: "When freeze ends"

Change Management Metrics

metrics:
  change_success_rate:
    description: "Percentage of changes implemented without rollback or incident"
    target: ">95%"
    formula: "(successful changes / total changes) * 100"

  emergency_change_rate:
    description: "Percentage of changes classified as emergency"
    target: "<5%"
    formula: "(emergency changes / total changes) * 100"

  rollback_rate:
    description: "Percentage of changes that required rollback"
    target: "<3%"

  mean_time_to_implement:
    description: "Average time from approval to implementation"
    target: "Varies by type"

  cab_approval_time:
    description: "Average time from submission to CAB decision"
    target: "<5 business days for normal changes"

Change Management Checklist

change_management_checklist:
  process_setup:
    - [ ] Change types defined with classification criteria
    - [ ] Approval matrix documented (who approves what)
    - [ ] CAB established with regular meeting schedule
    - [ ] Emergency change procedure documented
    - [ ] Change request template created
    - [ ] Change freeze policy defined

  tooling:
    - [ ] PR template includes change management fields
    - [ ] Automated risk classification in CI/CD
    - [ ] Branch protection enforces required approvals
    - [ ] Deployment records captured automatically
    - [ ] Change audit trail preserved (PR history, approvals)

  compliance:
    - [ ] All production changes have documented approval
    - [ ] Rollback plans exist for every change
    - [ ] Post-implementation reviews conducted for failures
    - [ ] Emergency changes documented retroactively within 48 hours
    - [ ] Change metrics reported monthly
    - [ ] Audit trail retained for compliance period (1-3 years)

Best Practices

• Classify changes by risk level to apply proportionate controls without slowing low-risk work
• Automate risk classification based on files changed, services affected, and deployment scope
• Use PR approvals as the native change approval mechanism for code-driven changes
• Require rollback plans for every change and test rollback procedures periodically
• Track emergency changes as a key metric: a high rate indicates systemic process issues
• Implement change freezes during critical business periods to protect stability
• Conduct post-implementation reviews for all failed changes to drive improvement
• Separate duty of implementation from duty of approval (no self-approving changes)
• Capture deployment records automatically in CI/CD rather than relying on manual entry
• Keep the CAB focused on high-risk decisions; do not bottleneck low-risk changes through CAB