Remediation Verification Skill

This skill verifies CVE remediation success by validating that vulnerabilities have been properly fixed on target systems.

Integration with Remediation Skill: The /remediation skill orchestrates this skill as part of its Step 6 (Verify Deployment) workflow. For standalone verification after manual remediation, you can invoke this skill directly.

When to Use This Skill

Use this skill directly when you need:

• Verify CVE remediation after playbook execution
• Confirm package updates were applied successfully
• Check service health after remediation
• Validate Kubernetes pod recovery after node updates
• Generate verification reports for compliance

Use the /remediation skill when you need:

• Full remediation workflow including verification
• Integrated remediation → execution → verification

How they work together: The /remediation skill invokes this skill after the user executes the remediation playbook, providing final confirmation that the CVE is resolved.

Workflow

1. CVE Status Verification

MCP Tool: get_cve or vulnerability__get_cve (from lightspeed-mcp)

Parameters for get_cve:

• cve_id: Exact CVE identifier (format: "CVE-YYYY-NNNNN")
  • Example: "CVE-2024-1234"
• include_details: true (retrieve complete metadata including remediation status)

Expected After Remediation:

• CVE metadata still exists (CVE doesn't disappear from database)
• Remediation marked as applied
• Fixed version recorded

MCP Tool: get_cve_systems or vulnerability__get_cve_systems (from lightspeed-mcp)

Parameters for get_cve_systems:

• cve_id: Exact CVE identifier (format: "CVE-YYYY-NNNNN")
  • Example: "CVE-2024-1234"
• limit: Optional number of systems to return (default: all)
  • Example: 100
• offset: Optional pagination offset (default: 0)
  • Example: 0

Expected After Remediation:

• Target systems removed from affected list OR
• Systems marked as "patched" status OR
• Systems show fixed package version

Verification Logic:

✓ System UUID not in affected systems list → PASS
✓ System status = "patched" → PASS
✗ System still in affected list with "vulnerable" status → FAIL

Important: Red Hat Lightspeed inventory updates may take time (up to 24 hours after remediation). Consider this timing when interpreting results.

2. Package Version Verification

MCP Tool: get_host_details or inventory__get_host_details (from lightspeed-mcp)

Parameters:

• system_id: UUID of the system to verify (from Red Hat Lightspeed inventory)
  • Example: "uuid-1"
  • Format: UUID string (get from system-context skill or get_cve_systems result)
• include_system_profile: true (retrieve installed packages and service status)
  • Example: true

Expected Output: System details including:

• system_profile.installed_packages - List of installed RPM packages with versions
• system_profile.enabled_services - Services enabled at boot
• system_profile.running_processes - Currently running processes

Verification Workflow:

For each target system:

1. Get current installed packages
   Tool: get_host_details(system_id="uuid-1", include_system_profile=true)
   Extract: system_profile.installed_packages

2. Compare against expected fixed versions
   CVE Fix Example: httpd-2.4.37-1.el8 → httpd-2.4.37-2.el8

   Installed Packages Check:
   ✓ httpd-2.4.37-2.el8 (or newer) installed → PASS
   ✗ httpd-2.4.37-1.el8 (old version) still present → FAIL
   ✗ httpd not found → FAIL (package removed unexpectedly)

3. Handle version comparison edge cases
   - Epoch numbers (e.g., 1:httpd-2.4.37)
   - Release suffixes (e.g., 2.4.37-2.el8_9.1)
   - Architecture (x86_64, aarch64)

Package Version Comparison Logic:

def verify_package_version(installed, expected_fixed):
    """
    installed: "httpd-2.4.37-2.el8.x86_64"
    expected_fixed: "httpd-2.4.37-2.el8"

    Returns: True if installed >= expected_fixed
    """
    # Parse version components using RPM version comparison
    # Account for epoch, version, release
    # Use >= comparison (newer versions are acceptable)

3. Service Health Verification

MCP Tool: get_host_details or inventory__get_host_details (from lightspeed-mcp)

Parameters: Same as Step 2 - system_id with include_system_profile=true

Verify affected services are running properly:

For each affected service (e.g., httpd):

1. Check service status
   Extract from: system_profile.enabled_services
   Extract from: system_profile.running_processes

   Service Health Checks:
   ✓ Service in enabled_services list → Service will start on boot
   ✓ Service process in running_processes → Service currently running
   ✗ Service not enabled → WARN (service won't start on reboot)
   ✗ Service process not running → FAIL (service down)

2. Check for service errors
   Look for: system_profile.systemd_failed_units
   ✓ Service not in failed units → PASS
   ✗ Service in failed units → FAIL (service failed to start)

3. Verify uptime (if applicable)
   Check: Service started recently (after remediation)
   ✓ Service uptime < remediation time + 10 minutes → Service restarted
   ⚠ Service uptime > remediation time → Service may not have restarted

4. Remediation Summary Generation

Generate comprehensive verification report:

{
  "verification_status": "success",  # or "partial_success", "failed"

  "cve_id": "CVE-2024-1234",
  "verification_date": "2024-01-20T15:30:00Z",

  "systems_remediated": {
    "total": 10,
    "successful": 10,
    "failed": 0,
    "partial": 0
  },

  "verification_details": [
    {
      "system_id": "uuid-1",
      "hostname": "web-server-01",
      "status": "verified",

      "checks": {
        "cve_status": {
          "result": "pass",
          "details": "System removed from affected systems list"
        },
        "package_version": {
          "result": "pass",
          "expected": "httpd-2.4.37-2.el8",
          "installed": "httpd-2.4.37-2.el8",
          "details": "Package updated successfully"
        },
        "service_health": {
          "result": "pass",
          "service": "httpd",
          "status": "running",
          "details": "Service restarted and healthy"
        }
      }
    }
  ],

  "compliance": {
    "all_systems_patched": true,
    "services_healthy": true
  },

  "recommendations": [
    "Remediation verified successfully on all systems",
    "Consider re-scanning with insights-client for updated inventory",
    "Document remediation in change management system"
  ]
}

6. Handle Verification Failures

If verification fails, provide troubleshooting guidance:

Package Version Mismatch:

Verification Failed: Package Not Updated

System: web-server-01 (uuid-1)
Expected: httpd-2.4.37-2.el8
Found: httpd-2.4.37-1.el8 (OLD VERSION)

Possible causes:
1. Playbook execution failed (check Ansible output)
2. Package repository doesn't have fixed version
3. Package update was skipped due to dependency conflict

Troubleshooting:
1. Check Ansible playbook output for errors
2. Verify package availability:
   sudo dnf info httpd-2.4.37-2.el8
3. Manually update package:
   sudo dnf update httpd
4. Check for package holds:
   sudo dnf versionlock list

Service Not Running:

Verification Failed: Service Not Running

System: web-server-01 (uuid-1)
Service: httpd
Status: Failed

Troubleshooting:
1. Check service status:
   sudo systemctl status httpd
2. View service logs:
   sudo journalctl -u httpd --since "10 minutes ago"
3. Check for configuration errors:
   sudo httpd -t
4. Restart service manually:
   sudo systemctl restart httpd

Output Template

When completing verification, provide output in this format:

# Remediation Verification Report

## CVE: CVE-YYYY-NNNNN
**Verification Date**: 2024-01-20 15:30 UTC
**Overall Status**: ✓ SUCCESS

## Summary
**Total Systems**: 10
**Successfully Remediated**: 10
**Failed**: 0
**Partial Success**: 0

## Verification Results

### System: web-server-01 (uuid-1)
**Status**: ✓ VERIFIED

**Checks Performed**:
✓ CVE Status: System removed from affected list
✓ Package Version: httpd-2.4.37-2.el8 (updated from 2.4.37-1.el8)
✓ Service Health: httpd running and healthy

---

### System: web-server-02 (uuid-2)
**Status**: ✓ VERIFIED

**Checks Performed**:
✓ CVE Status: System marked as patched
✓ Package Version: httpd-2.4.37-2.el8 installed
✓ Service Health: httpd running

---

[Additional systems...]

## Compliance Status
✓ All systems successfully patched
✓ All services running and healthy

## Recommendations
1. Remediation verified successfully on all 10 systems
2. Re-scan systems with Red Hat Lightspeed for updated inventory:
   ```bash
   sudo insights-client --check-results

3. Document remediation in change management system
4. Consider scheduling next vulnerability scan in 7 days

Next Steps

• Remediation complete, no further action required
• Monitor systems for 24 hours to ensure stability
• Update vulnerability tracking system

## Examples

### Example 1: Successful Verification

**User Request**: "Verify remediation for CVE-2024-1234 on 5 systems"

**Skill Response**:
1. Call `get_cve_systems` → 0 systems affected (down from 5)
2. Call `get_host_details` for each → All have httpd-2.4.37-2.el8
3. Check service status → All httpd services running
4. Return: "✓ All 5 systems verified, CVE remediated successfully"

### Example 2: Partial Success

**User Request**: "Verify batch remediation on 20 systems"

**Skill Response**:
1. Call `get_cve_systems` → 2 systems still affected (18 fixed)
2. Call `get_host_details` → 2 systems have old package version
3. Identify failed systems: web-server-18, web-server-19
4. Return: "⚠ 18/20 systems verified. 2 systems failed package update. Troubleshooting guidance provided."

## Dependencies

### Required MCP Servers
- `lightspeed-mcp` - Red Hat Lightspeed platform access

### Required MCP Tools
- `get_cve` or `vulnerability__get_cve` (from lightspeed-mcp) - Get CVE metadata and remediation status
  - Parameters: cve_id (string, format CVE-YYYY-NNNNN), include_details (boolean)
  - Returns: CVE metadata including remediation status
- `get_cve_systems` or `vulnerability__get_cve_systems` (from lightspeed-mcp) - List systems affected by CVE
  - Parameters: cve_id (string), limit (number), offset (number)
  - Returns: List of systems with vulnerability status
- `get_host_details` or `inventory__get_host_details` (from lightspeed-mcp) - Get system details including packages and services
  - Parameters: system_id (UUID string), include_system_profile (boolean)
  - Returns: System profile with installed_packages, enabled_services, running_processes

### Related Skills
- `playbook-generator` - Generates playbooks that this skill verifies
- `system-context` - Provides system context for verification scope
- `cve-impact` - Initial impact assessment to compare against verification results
- `playbook-executor` - Executes playbooks that this skill verifies

### Reference Documentation
- None required (verification skill uses MCP tool data)

## Best Practices

1. **Wait before verification** - Allow 5-10 minutes after playbook execution for system updates to register
2. **Check multiple indicators** - CVE status + package version + service health (defense in depth)
3. **Re-scan with Lightspeed** - Recommend `insights-client --check-results` to update inventory
4. **Document failures** - Provide detailed troubleshooting for any verification failures
5. **Consider timing** - Lightspeed inventory updates may take up to 24 hours to propagate
6. **Verify at scale** - Use batch verification for large deployments (call get_host_details in parallel)

## Integration with Other Skills

- **playbook-generator**: Generates playbooks that this skill verifies
- **system-context**: Provides system context for verification scope
- **cve-impact**: Initial impact assessment to compare against verification results

**Orchestration Example** (from `/remediation` skill):
1. User requests CVE remediation
2. Agent invokes playbook-generator → Creates playbook
3. User executes playbook manually
4. Agent invokes remediation-verifier skill → Confirms success
5. Agent reports: "✓ CVE remediated and verified on all systems"

**Verification-First Principle**:

Never assume remediation worked. Always verify:

1. CVE status in Lightspeed
2. Package versions updated
3. Services running

Trust, but verify.