Fix Review Skill

Overview

This skill implements Trail of Bits' fix review methodology for the agent-studio framework. When a security audit produces findings and developers commit fixes, this skill systematically verifies that each finding is properly addressed without introducing new vulnerabilities or regressions.

Source repository: https://github.com/trailofbits/skills License: CC-BY-SA-4.0 Output: Structured FIX_REVIEW_REPORT.md

When to Use

• After developers commit fixes for security audit findings
• During PR review of security-related changes
• When verifying remediation of vulnerability reports
• Before closing security findings in tracking systems
• When reviewing patches for CVE remediations
• After automated security tool findings are addressed

Iron Law

NO FINDING CLOSED WITHOUT FIX VERIFICATION

A finding is not fixed until the fix has been reviewed against the original finding description, verified to address the root cause, and confirmed not to introduce new issues.

Fix Status Categories

Status	Meaning
FIXED	Finding is fully addressed. Root cause eliminated. No regressions introduced.
PARTIALLY_FIXED	Some aspects addressed but gaps remain. More dangerous than unfixed.
NOT_ADDRESSED	Fix does not relate to the finding, or finding location unchanged.
CANNOT_DETERMINE	Insufficient context or code complexity prevents definitive assessment.
NEW_ISSUE	Fix introduces a new vulnerability or regression.

Anti-Pattern Detection

The following anti-patterns indicate potentially incorrect or incomplete fixes:

Anti-Pattern 1: Validation Removed

ALERT: Input validation was removed or weakened in the fix.

Indicators:

• Deleted input sanitization calls
• Relaxed regex patterns
• Removed length/type checks
• Weakened allowlist to blocklist

Anti-Pattern 2: Access Control Weakened

ALERT: Authorization checks were reduced or bypassed in the fix.

Indicators:

• Removed authentication middleware
• Changed role checks from strict to permissive
• Added bypass conditions to access control
• Removed rate limiting

Anti-Pattern 3: Error Handling Reduced

ALERT: Error handling was simplified or removed, potentially hiding failures.

Indicators:

• Replaced specific catch blocks with generic catch-all
• Added empty catch blocks
• Removed error logging
• Changed error responses from specific to generic (hiding useful diagnostics)

Anti-Pattern 4: External Call Reordering

ALERT: Order of external calls changed, potentially creating race conditions or TOCTOU.

Indicators:

• Authorization check moved after data access
• Validation moved after processing
• Lock acquisition moved or removed
• Transaction boundaries changed

Anti-Pattern 5: Integer Operation Changes

ALERT: Integer arithmetic modified, check for overflow/underflow/truncation.

Indicators:

• Changed integer types (int64 to int32, uint to int)
• Added/removed overflow checks
• Changed comparison operators (> to >=, == to !=)
• Modified loop bounds

Anti-Pattern 6: Crypto Downgrade

ALERT: Cryptographic operations were changed in a potentially weakening way.

Indicators:

• Algorithm changed (AES-256 to AES-128, SHA-256 to SHA-1)
• Key length reduced
• Removed salt/nonce/IV
• Changed from authenticated to unauthenticated encryption

Workflow

Step 1: Load Findings

Read the original findings report to understand what was found:

# Read findings file
cat findings-report.md

# Or extract from issue tracker
gh issue view <issue-number>

Step 2: Identify Fix Commits

Map each finding to its fix commit(s):

# View commits since the findings report
git log --oneline --since="<audit-date>"

# View a specific fix commit
git show <commit-hash>

# View PR diff
gh pr diff <pr-number>

Step 3: Analyze Each Fix

For each finding-fix pair:

1. Re-read the finding description and affected code location
2. Read the fix diff line by line
3. Check root cause: Does the fix address the root cause or just the symptom?
4. Check completeness: Are all instances of the pattern fixed?
5. Check anti-patterns: Does the fix match any anti-pattern?
6. Check regressions: Does the fix break adjacent functionality?
7. Assign status: FIXED / PARTIALLY_FIXED / NOT_ADDRESSED / CANNOT_DETERMINE / NEW_ISSUE

Step 4: Generate Report

FIX_REVIEW_REPORT Format

# Fix Review Report

**Date**: YYYY-MM-DD
**Reviewer**: [agent-type]
**Audit Reference**: [original audit report path]
**Commits Reviewed**: [list of commit hashes]

## Summary

| Status           | Count |
| ---------------- | ----- |
| FIXED            | N     |
| PARTIALLY_FIXED  | N     |
| NOT_ADDRESSED    | N     |
| CANNOT_DETERMINE | N     |
| NEW_ISSUE        | N     |

## Finding Reviews

### Finding F-001: [Title]

- **Original Severity**: Critical/High/Medium/Low/Informational
- **Original Location**: `file:line`
- **Fix Commit**: `<hash>`
- **Fix Status**: FIXED | PARTIALLY_FIXED | NOT_ADDRESSED | CANNOT_DETERMINE
- **Anti-Patterns Detected**: None | [list]

**Original Finding Summary**:
[Brief description of the finding]

**Fix Analysis**:
[Line-by-line analysis of what the fix does]

**Root Cause Addressed**: Yes/No/Partial
[Explanation]

**Regression Risk**: None/Low/Medium/High
[Explanation of potential regressions]

**Remaining Gaps** (if PARTIALLY_FIXED):

- [ ] Gap 1: [description]
- [ ] Gap 2: [description]

---

### Finding F-002: [Title]

[Same structure as above]

---

## New Issues Introduced

### NI-001: [Title]

- **Introduced By**: `<commit-hash>`
- **Location**: `file:line`
- **Severity**: Critical/High/Medium/Low
- **Description**: [What the new issue is]
- **Recommendation**: [How to fix it]

## Recommendations

1. [Prioritized list of remaining actions]
2. [Re-audit recommendations if needed]

Integration with Agent-Studio

Recommended Workflow

1. Run audit-context-building to deeply analyze the code
2. After fixes are committed, invoke fix-review to verify
3. Feed report to code-reviewer for additional review perspectives
4. Use variant-analysis to check if similar patterns exist elsewhere

Complementary Skills

Skill	Relationship
audit-context-building	Provides deep context for understanding fixes
differential-review	Security-focused diff review (lower granularity)
variant-analysis	Finds unpatched variants of fixed vulnerabilities
static-analysis	Automated confirmation of fix effectiveness
code-reviewer	General code quality review of fix commits

Iron Laws

1. NO FINDING CLOSED WITHOUT FIX VERIFICATION — A finding is not fixed until the fix has been reviewed against the original finding, verified to address root cause, and confirmed not to introduce new issues.
2. ALWAYS compare fix against original finding — reviewing the diff in isolation misses context; the fix must address the specific vulnerability described.
3. ALWAYS check for all instances — if a pattern is fixed in one location, verify that all other instances of the same pattern are also fixed.
4. NEVER close PARTIALLY_FIXED findings — partial fixes are more dangerous than unfixed findings because they create false confidence.
5. ALWAYS check adjacent code paths — fixes that pass all anti-pattern checks can still introduce regressions in code that depends on the changed behavior.

Anti-Patterns

Anti-Pattern	Why It Fails	Correct Approach
Reviewing diff without reading finding	Fix may address wrong issue or only surface symptom	Always re-read original finding before reviewing fix diff
Closing finding after any code change	Change may be unrelated or insufficient	Verify fix status is FIXED with root cause elimination confirmed
Ignoring partial fixes	PARTIALLY_FIXED is more dangerous than NOT_ADDRESSED (false security)	Track remaining gaps explicitly; keep finding open
Skipping anti-pattern checklist	Subtle regressions (weakened validation, reordered checks) go unseen	Run all 6 anti-pattern checks on every fix diff
Not checking for variant vulnerabilities	Same bug pattern likely exists elsewhere in codebase	Invoke variant-analysis after confirming fix

Memory Protocol (MANDATORY)

Before starting:

Read .claude/context/memory/learnings.md

Check for:

• Original findings report and any prior fix review reports
• Known anti-pattern frequencies from previous fix reviews
• Previous fix quality patterns for this codebase

After completing:

• Anti-pattern frequency data -> .claude/context/memory/learnings.md
• Fix quality concern -> .claude/context/memory/issues.md
• Decision about finding status -> .claude/context/memory/decisions.md

ASSUME INTERRUPTION: If it's not in memory, it didn't happen.