ABOUTME: Test quality assessment using Farley's 8 Properties of Good Tests

ABOUTME: Detects tautological tests, mock theatre, and structural test weaknesses

Test Design Reviewer

Quality Notes

• Read every test file thoroughly before scoring
• Quality over speed: analyze what each test actually verifies
• Do not skip the Tautology Theatre check

Process

Step 1: Collect test files

Identify all test files in scope. Use language-appropriate patterns:

• Go: *_test.go
• Python: test_*.py, *_test.py
• Ruby: *_spec.rb
• JS/TS: *.test.ts, *.spec.ts

Step 2: Score against Farley's 8 Properties

Rate each property 0-10 across the test suite. Provide evidence.

#	Property	Question to ask	Red flags
1	Understandable	Can you tell what's being tested in 5 seconds?	Cryptic names, no arrange/act/assert structure, shared state
2	Maintainable	Will this break when implementation changes?	Testing private methods, brittle selectors, hardcoded values
3	Repeatable	Same result every run, any order, any machine?	Time-dependent, filesystem-dependent, test ordering, shared DB state
4	Atomic	One reason to fail?	Multiple assertions testing different behaviors, setup-heavy
5	Necessary	Does this test earn its keep?	Duplicate coverage, testing framework/language behavior
6	Granular	Pinpoints the failure location?	Coarse assertions (assert result), catch-all tests
7	Fast	Runs in milliseconds?	Real HTTP calls, sleep/wait, full DB setup per test
8	First	Written before production code?	Tests that mirror implementation structure, not behavior

Scoring methodology:

• Static scoring: compute 0-10 per property using sigmoid normalization on signal densities (negative signals / test methods, positive signals / test methods). Use lib/cli_calculator.py for deterministic math (JSON in, JSON out).
• LLM scoring: assess holistically per property, focusing on semantic aspects static analysis misses (naming quality, assertion appropriateness, tautology theatre)
• Blend: final_property_score = 0.60 * static_score + 0.40 * llm_score per property
• Conservative default: when no signals detected for a property, default to 5.0 (unknown quality, not good quality)

Per-property scoring rubrics (anchor scores to these bands):

Property	9-10	7-8	5-6	3-4	1-2
U	Reads like specs; behavior clear without reading impl	Clear with minor ambiguities	Requires code inspection to understand	Cryptic; relies on impl details	test1/test2; magic numbers throughout
M	Proper abstractions; verifies behavior not impl	Good separation; occasional brittleness	Some impl coupling; some over-specified mocks	Tightly coupled; verify with exact counts	Reflection for private fields; mirrors impl exactly
R	Fully deterministic; no external deps	Rarely flaky; minimal env deps	Occasional flakiness; timing deps	Filesystem, timing, env deps present	sleep, file I/O, network, system time, unseeded random
A	Fully isolated; no shared state; parallelizable	Mostly isolated; minor shared setup	Some shared state; order sometimes matters	Heavy interdeps; must run in order	Shared mutable statics; ordering annotations
N	Every test adds unique value; parameterized for variations	Most tests valuable; minor redundancy	Checkbox exercises; moderate redundancy	Redundant tests; framework testing; mock tautologies	assertTrue(true); disabled tests; tests verify only mocks
G	Each test verifies single outcome; pinpoints issues	Focused; occasional logical assertion groups	Multiple behaviors; failure diagnosis takes effort	Sprawling; multiple unrelated assertions	20+ assertions; testEverything() methods
F	Pure computation; no I/O; milliseconds	Quick; minor optimization opportunities	Some slow tests; noticeable suite time	File I/O or database calls	sleep, network calls, heavy setup/teardown
T	Clear test-first evidence; tests drive design	Likely test-first; good design influence	Unclear; tests may be afterthoughts	Mirrors impl; likely test-after; mock-heavy	Clearly written after code; coverage patches

Aggregation methodology:

• Per-test-method: collect signals at individual method level
• Per-test-file: mean for positive signals, P90 for negative signals (worst offenders must surface)
• Per-test-suite: LOC-weighted mean across files

Sampling for large suites:

• Under 50 test files: analyze all
• Over 50: SHA-256 deterministic selection (30%) plus all files exceeding 100 test methods

Weighted Farley Index = (U*1.5 + M*1.5 + R*1.25 + A*1.0 + N*1.0 + G*1.0 + F*0.75 + T*1.0) / 9.0

Divisor is 9.0 (sum of weights), not 8 (number of properties). U/M weighted highest (readability, coupling); F weighted lowest (speed is contextual).

Range	Rating	Interpretation
9.0-10.0	Exemplary	Model suite; tests serve as living documentation
7.5-8.9	Excellent	High quality with minor improvement opportunities
6.0-7.4	Good	Solid foundation with clear areas for improvement
4.5-5.9	Fair	Functional but needs significant attention to test design
3.0-4.4	Poor	Tests provide limited value; major refactoring needed
0.0-2.9	Critical	Tests may be harmful; consider rewriting from scratch

Step 3: Tautology Theatre Detection

The critical question: "Would this test still pass if all production code were deleted?"

Scan for these 4 patterns:

Pattern	What it looks like	Example
Mock tautology	Test verifies that a mock returns what it was told to return	mock.return_value = 42; assert service.get() == 42 (only tests the mock)
Mock-only test	Every dependency is mocked, nothing real executes	Test with 5 mocks and zero real objects
Trivial tautology	Assertion is always true regardless of code	assert isinstance(result, dict) when function signature guarantees dict
Framework test	Tests framework behavior, not application logic	Testing that Rails validations work, that pytest fixtures inject

Also scan for mock interaction anti-patterns (affect Maintainable score):

Pattern	What it looks like
Over-specified interactions	verify with exact call counts, call ordering, verifyNoMoreInteractions
Testing internal details	ArgumentCaptor deep inspection, verify(never()) mirroring branches, high verify-to-assert ratio

For each tautology or anti-pattern found: report the file, line, pattern type, and why it's problematic.

Step 4: Report

## Test Design Review

### Farley Index: X.X / 10.0 (Rating)

| Property | Static | LLM | Blended | Weight | Weighted | Key Evidence |
|----------|--------|-----|---------|--------|----------|--------------|
| Understandable | X.X | X.X | X.X | 1.50x | X.XX | ... |
| Maintainable | X.X | X.X | X.X | 1.50x | X.XX | ... |
| Repeatable | X.X | X.X | X.X | 1.25x | X.XX | ... |
| Atomic | X.X | X.X | X.X | 1.00x | X.XX | ... |
| Necessary | X.X | X.X | X.X | 1.00x | X.XX | ... |
| Granular | X.X | X.X | X.X | 1.00x | X.XX | ... |
| Fast | X.X | X.X | X.X | 0.75x | X.XX | ... |
| First (TDD) | X.X | X.X | X.X | 1.00x | X.XX | ... |

### Tautology Theatre Analysis

Each subsection always present; use "None detected." when empty.

#### Mock Tautologies
| Test Method | Line | Mock Setup | Assertion |
#### Mock-Only Tests
| Test Method | Line | Evidence |
#### Trivial Tautologies
| Test Method | Line | Assertion |
#### Framework Tests
| Test Method | Line | Assertion | What It Actually Tests |

**Summary**: {total} instances across {affected}/{total_methods} test methods.

### Top 3 Improvements
1. [Highest-impact fix targeting weakest high-weight property]
2. [Second priority]
3. [Third priority]

### Methodology Notes
- Static/LLM blend: 60/40
- Files analyzed: {count} ({sampling note})
- Language: {lang}, Framework: {framework}

Integration with Review Pipeline

This skill is invoked by the orchestrator when test files are in scope (see orchestrator-protocol.md, review routing step). Can also be invoked directly via /test-design-reviewer.

Deterministic Scoring Calculator

lib/cli_calculator.py provides JSON-in, JSON-out math for reproducible scores. Delegate all Farley Index arithmetic to this CLI to avoid LLM rounding drift.

Commands: normalize-property, blend-scores, compute-farley, get-rating, aggregate-file, aggregate-suite, full-pipeline.

# Normalize a single property from signal counts
python lib/cli_calculator.py normalize-property '{"prop":"U","neg_count":2,"pos_count":8,"total_methods":20}'

# Compute Farley Index from 8 blended scores
python lib/cli_calculator.py compute-farley '{"U":8.5,"M":7.0,"R":9.0,"A":8.0,"N":7.5,"G":8.0,"F":6.0,"T":7.0}'

# End-to-end: raw signals + optional LLM scores -> index + rating
python lib/cli_calculator.py full-pipeline '{"properties":{"U":{"neg_count":2,"pos_count":8,"total_methods":20},...},"llm_scores":{"U":8.0,...}}'

Common Issues

Issue	Solution
No test files found	Check file patterns; some projects use non-standard locations
High mock count ≠ bad	Mocks are fine when testing boundaries; flag only when they replace all real logic
Property scores vary by test type	Score unit tests and integration tests separately if the suite is mixed
Legacy test suite scores low	Focus improvements on the top 3, not a full rewrite