Python Regression Test Generator

Overview

This skill automatically generates regression tests for Python code by analyzing differences between old and new versions, migrating existing tests, and creating new tests for modified or added functionality. It ensures previously tested behavior still works while covering new code paths.

Test Generation Workflow

Follow these steps to generate regression tests:

1. Analyze Code Changes

Compare old and new versions to identify:

Function Signature Changes:

Added parameters (with or without defaults)
Removed parameters
Changed parameter types or names
Changed return types

Logic Changes:

Modified implementation
New validation rules
Changed error handling
Algorithm improvements

Structural Changes:

Added functions/classes/methods
Removed functions/classes/methods
Renamed functions/classes/methods
Moved code between modules

Example Analysis:

# Old version
def calculate_total(items):
    return sum(item.price for item in items)

# New version
def calculate_total(items, tax_rate=0.0):
    subtotal = sum(item.price for item in items)
    return subtotal * (1 + tax_rate)

Changes identified:

Added parameter: tax_rate with default value 0.0
Modified logic: now applies tax calculation
Backward compatible: old calls still work

2. Analyze Existing Tests

Review old tests to understand:

Test structure (unittest vs pytest)
Test coverage (which functions/paths are tested)
Test patterns (fixtures, mocks, parametrization)
Dependencies and setup requirements

Example Existing Test:

def test_calculate_total():
    items = [Item(price=10), Item(price=20)]
    assert calculate_total(items) == 30

Analysis:

Uses pytest style
Tests basic functionality
No mocking needed
Simple assertion

3. Migrate Existing Tests

Update tests to work with new code:

For backward-compatible changes:

Keep original test (ensures backward compatibility)
Add new tests for new functionality

For breaking changes:

Update test to match new signature/behavior
Document what changed in test docstring

Migrated Tests:

def test_calculate_total_no_tax():
    """Regression: ensure backward compatibility with no tax"""
    items = [Item(price=10), Item(price=20)]
    assert calculate_total(items) == 30  # Uses default tax_rate=0.0

def test_calculate_total_with_tax():
    """New: test tax calculation functionality"""
    items = [Item(price=10), Item(price=20)]
    assert calculate_total(items, tax_rate=0.1) == 33.0

4. Generate Tests for New Functionality

Create tests for:

New parameters and their effects
New functions/methods
New code paths
Edge cases and boundaries
Error conditions

Generated Tests:

def test_calculate_total_zero_tax():
    """Test explicit zero tax rate"""
    items = [Item(price=100)]
    assert calculate_total(items, tax_rate=0.0) == 100

def test_calculate_total_high_tax():
    """Test high tax rate"""
    items = [Item(price=100)]
    assert calculate_total(items, tax_rate=0.5) == 150

def test_calculate_total_empty_items():
    """Edge case: empty items list"""
    assert calculate_total([], tax_rate=0.1) == 0

def test_calculate_total_negative_tax():
    """Edge case: negative tax (discount)"""
    items = [Item(price=100)]
    assert calculate_total(items, tax_rate=-0.1) == 90

5. Create Mocks for Dependencies

Identify dependencies that need mocking:

External APIs and HTTP requests
Database operations
File I/O operations
Email/notification services
Time-dependent operations
Random number generation

Example with External Dependency:

New Code:

def send_notification(user_id, message):
    user = database.get_user(user_id)
    email_service.send_email(
        to=user.email,
        subject='Notification',
        body=message
    )
    return True

Generated Test with Mocks:

from unittest.mock import Mock, patch

def test_send_notification():
    """Test notification sending with mocked dependencies"""
    # Mock database
    mock_db = Mock()
    mock_user = Mock()
    mock_user.email = 'user@example.com'
    mock_db.get_user.return_value = mock_user

    # Mock email service
    with patch('myapp.email_service.send_email') as mock_send:
        mock_send.return_value = True

        with patch('myapp.database', mock_db):
            result = send_notification(user_id=123, message="Hello")

            assert result is True
            mock_db.get_user.assert_called_once_with(123)
            mock_send.assert_called_once_with(
                to='user@example.com',
                subject='Notification',
                body='Hello'
            )

6. Add Setup and Teardown

Determine if tests need:

Shared fixtures
Database setup/cleanup
File creation/deletion
State initialization

unittest Setup:

class TestCalculateTotal(unittest.TestCase):
    def setUp(self):
        """Run before each test"""
        self.items = [
            Item(price=10),
            Item(price=20),
            Item(price=30)
        ]

    def test_no_tax(self):
        result = calculate_total(self.items)
        self.assertEqual(result, 60)

    def test_with_tax(self):
        result = calculate_total(self.items, tax_rate=0.1)
        self.assertAlmostEqual(result, 66.0)

pytest Fixtures:

@pytest.fixture
def items():
    """Fixture providing test items"""
    return [
        Item(price=10),
        Item(price=20),
        Item(price=30)
    ]

def test_no_tax(items):
    result = calculate_total(items)
    assert result == 60

def test_with_tax(items):
    result = calculate_total(items, tax_rate=0.1)
    assert result == pytest.approx(66.0)

7. Ensure Test Quality

Verify generated tests are:

Deterministic: Same input always produces same output
Isolated: Tests don't depend on each other
Readable: Clear test names and docstrings
Maintainable: Follow project conventions
Executable: Can run without errors

Complete Example

Scenario: Adding Tax Calculation

Old Code (calculator.py):

class Calculator:
    def calculate_total(self, items):
        """Calculate total price of items"""
        return sum(item.price for item in items)

Old Test (test_calculator.py):

import unittest
from calculator import Calculator
from models import Item

class TestCalculator(unittest.TestCase):
    def test_calculate_total(self):
        calc = Calculator()
        items = [Item(price=10), Item(price=20)]
        result = calc.calculate_total(items)
        self.assertEqual(result, 30)

New Code (calculator.py):

class Calculator:
    def calculate_total(self, items, tax_rate=0.0, discount=0.0):
        """Calculate total price with tax and discount"""
        if tax_rate < 0 or tax_rate > 1:
            raise ValueError("Tax rate must be between 0 and 1")
        if discount < 0 or discount > 1:
            raise ValueError("Discount must be between 0 and 1")

        subtotal = sum(item.price for item in items)
        after_discount = subtotal * (1 - discount)
        total = after_discount * (1 + tax_rate)
        return round(total, 2)

Generated Regression Tests (test_calculator.py):

import unittest
from unittest.mock import Mock
from calculator import Calculator
from models import Item

class TestCalculator(unittest.TestCase):
    def setUp(self):
        """Set up test fixtures"""
        self.calc = Calculator()
        self.items = [Item(price=10), Item(price=20)]

    def test_calculate_total_backward_compatibility(self):
        """Regression: ensure old behavior still works"""
        result = self.calc.calculate_total(self.items)
        self.assertEqual(result, 30)

    def test_calculate_total_with_tax(self):
        """New: test tax calculation"""
        result = self.calc.calculate_total(self.items, tax_rate=0.1)
        self.assertAlmostEqual(result, 33.0)

    def test_calculate_total_with_discount(self):
        """New: test discount calculation"""
        result = self.calc.calculate_total(self.items, discount=0.2)
        self.assertAlmostEqual(result, 24.0)

    def test_calculate_total_with_tax_and_discount(self):
        """New: test combined tax and discount"""
        result = self.calc.calculate_total(
            self.items,
            tax_rate=0.1,
            discount=0.2
        )
        self.assertAlmostEqual(result, 26.4)

    def test_calculate_total_empty_items(self):
        """Edge case: empty items list"""
        result = self.calc.calculate_total([])
        self.assertEqual(result, 0)

    def test_calculate_total_invalid_tax_rate_negative(self):
        """Error case: negative tax rate"""
        with self.assertRaises(ValueError) as context:
            self.calc.calculate_total(self.items, tax_rate=-0.1)
        self.assertIn("Tax rate must be between 0 and 1", str(context.exception))

    def test_calculate_total_invalid_tax_rate_too_high(self):
        """Error case: tax rate > 1"""
        with self.assertRaises(ValueError):
            self.calc.calculate_total(self.items, tax_rate=1.5)

    def test_calculate_total_invalid_discount_negative(self):
        """Error case: negative discount"""
        with self.assertRaises(ValueError):
            self.calc.calculate_total(self.items, discount=-0.1)

    def test_calculate_total_invalid_discount_too_high(self):
        """Error case: discount > 1"""
        with self.assertRaises(ValueError):
            self.calc.calculate_total(self.items, discount=1.5)

if __name__ == '__main__':
    unittest.main()

Framework-Specific Examples

pytest Example

Old Code:

def fetch_user_data(user_id):
    response = requests.get(f'https://api.example.com/users/{user_id}')
    return response.json()

Old Test:

def test_fetch_user_data():
    with patch('requests.get') as mock_get:
        mock_get.return_value.json.return_value = {'id': 1, 'name': 'Alice'}
        result = fetch_user_data(1)
        assert result['name'] == 'Alice'

New Code:

async def fetch_user_data(user_id, include_posts=False):
    async with aiohttp.ClientSession() as session:
        url = f'https://api.example.com/users/{user_id}'
        if include_posts:
            url += '?include=posts'
        async with session.get(url) as response:
            return await response.json()

Generated Regression Tests:

import pytest
from unittest.mock import AsyncMock, patch

@pytest.mark.asyncio
async def test_fetch_user_data_backward_compatibility():
    """Regression: basic user fetch still works"""
    with patch('aiohttp.ClientSession') as mock_session:
        mock_response = AsyncMock()
        mock_response.json.return_value = {'id': 1, 'name': 'Alice'}
        mock_session.return_value.__aenter__.return_value.get.return_value.__aenter__.return_value = mock_response

        result = await fetch_user_data(1)
        assert result['name'] == 'Alice'

@pytest.mark.asyncio
async def test_fetch_user_data_with_posts():
    """New: test include_posts parameter"""
    with patch('aiohttp.ClientSession') as mock_session:
        mock_response = AsyncMock()
        mock_response.json.return_value = {
            'id': 1,
            'name': 'Alice',
            'posts': [{'id': 1, 'title': 'Post 1'}]
        }
        mock_session.return_value.__aenter__.return_value.get.return_value.__aenter__.return_value = mock_response

        result = await fetch_user_data(1, include_posts=True)
        assert 'posts' in result
        assert len(result['posts']) == 1

@pytest.mark.asyncio
@pytest.mark.parametrize("user_id,include_posts", [
    (1, False),
    (1, True),
    (999, False),
])
async def test_fetch_user_data_parametrized(user_id, include_posts):
    """Parametrized test for various inputs"""
    with patch('aiohttp.ClientSession') as mock_session:
        mock_response = AsyncMock()
        mock_response.json.return_value = {'id': user_id, 'name': 'User'}
        mock_session.return_value.__aenter__.return_value.get.return_value.__aenter__.return_value = mock_response

        result = await fetch_user_data(user_id, include_posts=include_posts)
        assert result['id'] == user_id

Test Generation Guidelines

When to Migrate vs Create New

Migrate existing test when:

Function signature changed but is backward compatible
Return type changed (update assertions)
Exception type changed
Implementation changed but interface is same

Create new test when:

New parameters added
New functionality added
New code paths introduced
New edge cases discovered

Mark as obsolete when:

Function removed from codebase
Functionality completely replaced
Test no longer relevant

Naming Conventions

Regression tests:

test_<function>_backward_compatibility
test_<function>_<old_behavior>
Include "Regression:" in docstring

New functionality tests:

test_<function>_<new_feature>
test_<function>_with_<parameter>
Include "New:" in docstring

Edge case tests:

test_<function>_empty_input
test_<function>_boundary_<condition>
Include "Edge case:" in docstring

Error tests:

test_<function>_invalid_<parameter>
test_<function>_raises_<exception>
Include "Error case:" in docstring

Mock Generation Rules

Always mock:

External API calls (requests, aiohttp)
Database operations
File system operations
Email/SMS services
Time-dependent functions (datetime.now, time.sleep)

Consider mocking:

Complex object creation
Expensive computations
Non-deterministic operations

Don't mock:

Simple data structures (lists, dicts)
Pure functions without side effects
The function under test
Standard library functions (unless they have side effects)

Constraints

MUST:

Analyze both old and new code versions
Preserve existing valid tests
Generate tests for new functionality
Create appropriate mocks for dependencies
Follow existing test framework (unittest or pytest)
Ensure tests are deterministic and isolated
Include docstrings explaining test purpose
Use proper setup/teardown or fixtures

MUST NOT:

Remove valid tests without justification
Generate redundant tests
Create tests that depend on external state
Mock the function being tested
Generate non-executable test code
Ignore breaking changes
Create flaky or non-deterministic tests

Output Format

Complete test module structure:

"""
Regression tests for <module_name>

Generated tests ensure backward compatibility and cover new functionality
added in the latest version.
"""

import unittest  # or pytest
from unittest.mock import Mock, patch, MagicMock
# Other imports

class Test<ClassName>(unittest.TestCase):  # or pytest functions
    def setUp(self):
        """Set up test fixtures"""
        # Setup code

    def test_<name>_backward_compatibility(self):
        """Regression: <description of old behavior>"""
        # Test code

    def test_<name>_<new_feature>(self):
        """New: <description of new functionality>"""
        # Test code

    # More tests...

if __name__ == '__main__':
    unittest.main()

Resources

references/test_generation_patterns.md

Comprehensive patterns and techniques including:

Change analysis patterns (signature changes, logic changes, structural changes)
Test generation strategies (migration, behavior preservation, edge cases)
Testing framework support (unittest and pytest)
Mocking and stubbing patterns
Test structure patterns (AAA, Given-When-Then, setup/teardown)
Coverage strategies (path coverage, boundary testing, state-based testing)
Test migration patterns with complete examples

python-regression-test-generator