Python Regression Test Generator

Same idea as → java-regression-test-generator — capture current behavior, lock it in — but Python's tooling makes capture easier: pickle snapshots, syrupy/pytest-snapshot, and duck typing means fewer fixture gymnastics.

Capture approaches by effort

Approach	Effort	When
Snapshot library (syrupy)	Lowest	Outputs are dicts/lists/strings — most cases
Pickle round-trip	Low	Complex custom objects with sane __eq__
Explicit field assertions	Medium	You want readable diffs when things break
Golden files	Medium	Large outputs (rendered HTML, CSVs)

Default to snapshot libraries for the bulk, explicit assertions for the handful of truly critical paths.

Snapshot-based capture

# First run with --snapshot-update: records current output
# Subsequent runs: compares against recorded
def test_price_two_items_us(snapshot):
    order = Order(
        items=[LineItem("sku-1", 2, Decimal("10.00")),
               LineItem("sku-2", 1, Decimal("5.50"))],
        region="US",
        discount=None,
    )
    assert engine.price(order) == snapshot

Snapshot lives in __snapshots__/test_pricing.ambr. When behavior changes, diff is shown. Accept with --snapshot-update or fix the regression.

Explicit assertions — when readability matters

For the few paths you really care about, write it out:

def test_price_enterprise_eu_discount():
    """
    Captured 2024-01-15. The EU enterprise discount is 15% — as observed,
    no spec found. If this breaks, verify with product before updating.
    """
    order = Order(
        items=[LineItem("sku", 1, Decimal("100.00"))],
        region="EU",
        tier="enterprise",
    )
    invoice = engine.price(order)

    assert invoice.subtotal == Decimal("100.00")
    assert invoice.discount == Decimal("15.00")   # 15% — observed, not spec'd
    assert invoice.tax == Decimal("17.00")        # 20% VAT on post-discount 85
    assert invoice.total == Decimal("102.00")

Explicit fields → readable failure: assert Decimal('17.00') == Decimal('20.00') tells you exactly what changed.

Input generation via tracing

Python lets you record real inputs cheaply:

# Wrap the real function — deploy briefly to staging, collect inputs
_captured = []

def capturing_price(self, order):
    result = _original_price(self, order)
    _captured.append((copy.deepcopy(order), copy.deepcopy(result)))
    return result

PricingEngine.price = capturing_price

# ... run realistic traffic ...

# Dump
with open("regression_cases.pkl", "wb") as f:
    pickle.dump(_captured, f)

Then generate:

with open("regression_cases.pkl", "rb") as f:
    CASES = pickle.load(f)

@pytest.mark.parametrize("order,expected", CASES, ids=lambda c: repr(c)[:60])
def test_price_regression(order, expected):
    assert engine.price(order) == expected

Real traffic → realistic edge cases you wouldn't think of.

Non-determinism — Python specifics

Issue	Fix
datetime.now()	freezegun.freeze_time("2024-01-15") decorator
time.time()	Same — freezegun patches both
uuid.uuid4()	mocker.patch("uuid.uuid4", side_effect=[UUID("..."), ...]) or assert format
random	random.seed(0) at test start, or inject Random(0)
dict order	Guaranteed insertion-ordered since 3.7 — only a problem if code uses set or pre-3.7
set iteration	assert sorted(result) == sorted(expected)
Float precision	pytest.approx(expected, rel=1e-9)
PYTHONHASHSEED	Set to fixed value in conftest if hashing user strings

Known-wrong capture

@pytest.mark.xfail(
    reason="Bug #1234: tax computed on post-discount subtotal. "
           "Spec says pre-discount. Expected 9.00, currently 8.10. "
           "When fixed, flip strict=True → test must pass.",
    strict=False,
)
def test_discount_tax_base_KNOWN_BUG():
    ...
    assert invoice.tax == Decimal("9.00")   # the CORRECT value

xfail(strict=False) — currently fails, that's expected. When the bug is fixed, test passes → xpass is reported. Flip to strict=True and it becomes a normal passing test.

Alternative: assert the wrong value with a # TRACKING BUG comment. Less elegant (xfail is nicer) but works without xfail semantics.

Do not

• Do not pickle objects with open file handles, sockets, or lambdas in them — unpickleable. Use explicit field extraction for those.
• Do not snapshot-test functions with hundreds of call sites and one snapshot each. Review burden when things change is enormous. Parametrize and group.
• Do not forget to sanitize captured production inputs. Real Order objects might have real customer emails in them.
• Do not capture outputs that include absolute paths, hostnames, or PIDs. Normalize (path.relative_to(root), socket.gethostname() → "<host>") before snapshotting.
• Do not leave capture wrappers in production. Monkey-patching price for an hour in staging: fine. Forgetting to remove it: not fine.

Output format

## Capture target
<module.function(s)>

## Input set
| # | Input summary | Source | Branches hit |
| - | ------------- | ------ | ------------ |

## Capture approach
<snapshot | explicit | pickle-parametrize | golden>

## Non-determinism
| Source | Fix |
| ------ | --- |

## Generated tests
<pytest code>

## Known-wrong captures
| Test | Bug ref | xfail or tracking-comment |
| ---- | ------- | ------------------------- |

## Coverage delta
Before: <%>  After: <%>  (branch, via `pytest --cov --cov-branch`)