Regex Log Parsing Skill

This skill provides a systematic approach for constructing complex regular expressions that extract and validate structured data from log files.

When to Use This Skill

This skill applies when:

Building regex patterns to extract data from log entries
Validating specific formats (IPv4 addresses, dates, timestamps) within logs
Handling requirements for first/last occurrence selection
Enforcing word boundary conditions
Combining multiple validation constraints in a single pattern

Approach: Decomposition Strategy

Complex log parsing regex should be built by decomposing the problem into sub-patterns:

Step 1: Identify All Requirements

Before writing any regex, create a complete list of requirements:

What data needs to be validated (present but not captured)?
What data needs to be captured?
What boundary conditions apply (word boundaries, line anchors)?
Are there positional requirements (first, last, nth occurrence)?
What constitutes an invalid match?

Step 2: Build Sub-Patterns Independently

Construct each validation pattern separately before combining:

IPv4 Address Pattern

For valid IPv4 addresses (0-255 per octet, no leading zeros except for 0 itself):

Octet pattern: (?:25[0-5]|2[0-4][0-9]|1[0-9]{2}|[1-9][0-9]|[0-9])
Order alternatives from most specific to least specific
Full IPv4: (?:(?:25[0-5]|2[0-4][0-9]|1[0-9]{2}|[1-9][0-9]|[0-9])\.){3}(?:25[0-5]|2[0-4][0-9]|1[0-9]{2}|[1-9][0-9]|[0-9])

Date Pattern (YYYY-MM-DD)

For valid dates with proper month-day validation:

31-day months: (?:0[13578]|1[02])-(?:0[1-9]|[12][0-9]|3[01])
30-day months: (?:0[469]|11)-(?:0[1-9]|[12][0-9]|30)
February (up to 29): 02-(?:0[1-9]|1[0-9]|2[0-9])
Combine with year: [0-9]{4}-(?:...combined month-day patterns...)

Step 3: Apply Positional Requirements

Selecting Last Occurrence

To capture the last valid pattern in a line:

^.*<pattern>(?!.*<pattern>)

Use ^.* to greedily consume characters
Use negative lookahead (?!.*<pattern>) to ensure no pattern follows

Selecting First Occurrence

To capture the first valid pattern:

^(?:(?!<pattern>).)*<pattern>

Or simply rely on regex engines returning the first match by default.

Step 4: Apply Validation Without Capture

To require presence of a pattern without capturing it:

Use lookahead: (?=.*<pattern>) at the start of the regex
This validates the line contains the pattern without affecting the capture

Step 5: Apply Word Boundaries

For patterns that must not be adjacent to alphanumeric characters:

Use \b word boundaries: \b<pattern>\b
Be aware that \b matches between word and non-word characters

Verification Strategy

Create Comprehensive Test Cases

Organize tests by category:

Valid cases: Confirm expected matches
- Minimum/maximum valid values (e.g., 0.0.0.0, 255.255.255.255)
- Edge values for each component
Invalid format cases: Confirm rejection
- Out-of-range values (e.g., 256.0.0.0)
- Invalid formatting (leading zeros where prohibited)
- Invalid months (00, 13) or days (32)
Boundary condition cases:
- Pattern at start/end of line
- Pattern adjacent to alphanumeric characters (should fail with word boundaries)
- Pattern adjacent to punctuation (should pass with word boundaries)
Positional cases:
- Multiple valid patterns in one line (verify correct one is captured)
- Single pattern in line
- No valid pattern in line

Test File Structure

Create a structured test file that:

Groups tests by category
Uses clear naming for each test case
Reports pass/fail status for each test
Summarizes overall results

Example structure:

test_cases = {
    "valid_ipv4": [...],
    "invalid_ipv4": [...],
    "valid_dates": [...],
    "invalid_dates": [...],
    "last_occurrence": [...],
    "boundary_conditions": [...]
}

Common Pitfalls

1. Incomplete First Attempt

Problem: Creating incomplete or truncated test files
Solution: Plan the full test structure before writing; validate file completeness before execution

2. Environment Assumptions

Problem: Assuming python command exists when only python3 is available
Solution: Check the Python environment first or use python3 explicitly

3. Scattered Reasoning

Problem: Disorganized thought process leading to repeated work
Solution: Follow the decomposition strategy linearly; complete each sub-pattern before moving to the next

4. Duplicate Patterns Without Abstraction

Problem: Same regex pattern repeated multiple times, increasing error risk
Solution: Define complex sub-patterns once in reasoning, then reference them; in code, use variables

5. Missing Edge Cases

Problem: Focusing only on happy path validation
Solution: Explicitly test:
- Boundary values (min/max for each component)
- Invalid values just outside valid range
- Empty and null cases
- Patterns at different positions in the line

6. Order of Alternatives

Problem: Less specific alternatives matching before more specific ones
Solution: Order regex alternatives from most specific to least specific (e.g., 25[0-5] before 2[0-4][0-9] before [0-9])

7. Greedy vs Non-Greedy Matching

Problem: Unexpected capture due to greedy quantifiers
Solution: Understand when to use .* vs .*?; for "last occurrence" patterns, greedy .* is typically correct

Workflow Summary

List all requirements explicitly
Build and test sub-patterns independently
Combine sub-patterns with appropriate anchors and lookaheads
Create comprehensive test cases covering all categories
Run tests and verify all pass
Clean up test files after validation

regex-log