coverage-analysis

---

Discovery Questions

1. What coverage tooling is configured? Check for jest.config.* (coverageProvider), vitest.config.* (coverage), .nycrc, c8 in scripts, or [tool.coverage] in pyproject.toml.
2. What is the current coverage level? Run the existing coverage command and note line, branch, and function percentages. This is the baseline for the ratchet.
3. Is coverage gated in CI? Check GitHub Actions or GitLab CI for --coverage, coverageThreshold, fail_under, or --cov-fail-under flags.
4. What is the target, and who set it? A target without rationale ("the VP said 80%") leads to gaming. Targets should reflect risk tolerance and codebase maturity.
5. Check .agents/qa-project-context.md first. Respect existing coverage conventions and thresholds.

---

Core Principles

1. Coverage measures breadth, not depth. A line of code being executed does not mean it is tested correctly. expect(true).toBe(true) executes the function but asserts nothing meaningful. Coverage tells you what code ran, not whether the tests would catch a bug.

2. Branch coverage matters more than line coverage. A ternary condition ? a : b on one line counts as "covered" in line coverage even if only one branch executes. Branch coverage exposes untested paths.

// Line coverage: 100% (the line executed)
// Branch coverage: 50% (only the 'true' branch ran)
function discount(price: number, isPremium: boolean): number {
  return isPremium ? price * 0.8 : price;
}

// Test only covers isPremium = true
expect(discount(100, true)).toBe(80);
// Missing: expect(discount(100, false)).toBe(100);

3. Ratchet pattern: never decrease, only increase. Record the current coverage as the minimum threshold. Every PR must meet or exceed it. Coverage goes up over time without forcing artificial targets.

4. Focus on gaps, not numbers. A project at 85% coverage is not "better" than one at 75%. What matters is whether the untested 15% or 25% contains critical business logic. Analyze gaps by risk, not by percentage.

5. New code has a higher bar than legacy code. Require 90%+ coverage on new code in PRs even if the overall project is at 65%. This prevents coverage decay without demanding a rewrite of legacy code.

---

Coverage Tools

V8 / c8 (Node.js Built-in)

V8's built-in code coverage is faster than Istanbul because it does not instrument source code. Use c8 as the CLI wrapper.

npm i -D c8

// package.json
{
  "scripts": {
    "test": "vitest run",
    "test:coverage": "vitest run --coverage"
  }
}

// vitest.config.ts
import { defineConfig } from "vitest/config";

export default defineConfig({
  test: {
    coverage: {
      provider: "v8",
      reporter: ["text", "html", "lcov", "json-summary"],
      reportsDirectory: "./coverage",
      include: ["src/**/*.ts"],
      exclude: [
        "src/**/*.test.ts",
        "src/**/*.spec.ts",
        "src/**/*.d.ts",
        "src/**/index.ts",       // Barrel exports
        "src/**/types.ts",       // Type-only files
        "src/**/*.stories.ts",   // Storybook
        "src/generated/**",      // Generated code
      ],
      thresholds: {
        lines: 80,
        branches: 80,
        functions: 80,
        statements: 80,
      },
    },
  },
});

Istanbul / nyc (JavaScript/TypeScript)

Istanbul instruments source code for coverage tracking. Slower than V8 but more widely compatible.

npm i -D nyc

// .nycrc.json
{
  "all": true,
  "include": ["src/**/*.ts"],
  "exclude": [
    "src/**/*.test.ts",
    "src/**/*.spec.ts",
    "src/**/*.d.ts",
    "src/**/index.ts",
    "src/generated/**"
  ],
  "reporter": ["text", "html", "lcov", "json-summary"],
  "report-dir": "./coverage",
  "check-coverage": true,
  "branches": 80,
  "lines": 80,
  "functions": 80,
  "statements": 80,
  "watermarks": {
    "lines": [70, 90],
    "functions": [70, 90],
    "branches": [70, 90],
    "statements": [70, 90]
  }
}

// package.json (with Jest)
{
  "scripts": {
    "test:coverage": "jest --coverage"
  },
  "jest": {
    "coverageProvider": "v8",
    "collectCoverageFrom": [
      "src/**/*.ts",
      "!src/**/*.{test,spec,d}.ts",
      "!src/**/index.ts",
      "!src/generated/**"
    ],
    "coverageThreshold": {
      "global": {
        "branches": 80,
        "functions": 80,
        "lines": 80,
        "statements": 80
      }
    }
  }
}

coverage.py (Python)

pip install pytest-cov

# pyproject.toml
[tool.coverage.run]
source = ["src"]
branch = true
omit = [
    "src/**/test_*.py",
    "src/**/conftest.py",
    "src/**/__init__.py",
    "src/generated/*",
]

[tool.coverage.report]
fail_under = 80
show_missing = true
skip_covered = true
precision = 1
exclude_lines = [
    "pragma: no cover",
    "if TYPE_CHECKING:",
    "if __name__ == .__main__.",
    "raise NotImplementedError",
    "@overload",
    "\\.\\.\\.",     # Ellipsis in abstract methods
]

[tool.coverage.html]
directory = "coverage/html"

[tool.coverage.xml]
output = "coverage/coverage.xml"

# Run tests with coverage
pytest --cov=src --cov-report=term-missing --cov-report=html --cov-report=xml

# Fail if coverage drops below threshold
pytest --cov=src --cov-fail-under=80

Coverage Report Types

Reporter	Output	Use Case
text	Terminal table	Quick local check
html	Interactive HTML	Detailed local analysis, clicking through files
lcov	lcov.info file	SonarQube, Codecov, Coveralls integration
json-summary	coverage-summary.json	CI scripts, PR comments, dashboard metrics
cobertura	cobertura-coverage.xml	GitLab CI coverage visualization

---

Gap Analysis

Identify Untested Code Paths

Coverage reports show which lines and branches are not executed. But not all gaps are equal. Prioritize by risk.

Step 1: Generate the coverage report.

npm run test:coverage
# Open coverage/index.html in a browser

Step 2: Sort files by uncovered lines. Parse coverage-summary.json, sort files by (total - covered) descending, and focus on the top 20. A script that reads the JSON summary and outputs a table of file, line%, branch%, and uncovered count makes this repeatable.

Step 3: Map gaps to risk areas.

Gap Location	Risk Level	Action
Payment processing	Critical	Write tests immediately
Auth/permissions	Critical	Write tests immediately
Data validation	High	Add to next sprint
Error handling paths	High	Add to next sprint
Utility functions	Medium	Cover when modifying
UI formatting	Low	Skip unless regression-prone
Generated code	None	Exclude from coverage

Coverage Diff on PRs

Show coverage change per PR so reviewers see the impact of each change.

# GitHub Actions: coverage diff comment
- name: Run tests with coverage
  run: npm run test:coverage

- name: Coverage Report
  uses: davelosert/vitest-coverage-report-action@v2
  if: github.event_name == 'pull_request'
  with:
    json-summary-path: coverage/coverage-summary.json
    json-final-path: coverage/coverage-final.json
    vite-config-path: vitest.config.ts

Alternative: use marocchino/sticky-pull-request-comment@v2 with a script that reads coverage-summary.json, filters to changed files (via git diff --name-only origin/main...HEAD), and renders a markdown table showing per-file line and branch coverage.

---

Coverage as CI Gate

Threshold Configuration

Global threshold (minimum for the entire project):

// vitest.config.ts
coverage: {
  thresholds: {
    lines: 80,
    branches: 80,
    functions: 80,
    statements: 80,
  },
}

Per-directory thresholds (stricter for critical code):

// vitest.config.ts
coverage: {
  thresholds: {
    // Global baseline
    lines: 75,
    branches: 75,

    // Stricter for critical paths
    "src/payments/**": { lines: 95, branches: 90 },
    "src/auth/**": { lines: 90, branches: 85 },
    "src/utils/**": { lines: 80, branches: 80 },
  },
}

Jest per-file thresholds:

// jest.config.js
module.exports = {
  coverageThreshold: {
    global: { branches: 75, functions: 75, lines: 75, statements: 75 },
    "./src/payments/": { branches: 90, functions: 90, lines: 95, statements: 95 },
    "./src/auth/": { branches: 85, functions: 85, lines: 90, statements: 90 },
  },
};

Ratchet Pattern

Never let coverage decrease. Record the current level as the minimum and update it upward when coverage improves.

// scripts/coverage-ratchet.ts
import fs from "fs";
import coverageSummary from "../coverage/coverage-summary.json";

const RATCHET_FILE = ".coverage-ratchet.json";

interface Ratchet {
  lines: number;
  branches: number;
  functions: number;
  statements: number;
  updatedAt: string;
}

// Read current ratchet or initialize
let ratchet: Ratchet;
try {
  ratchet = JSON.parse(fs.readFileSync(RATCHET_FILE, "utf-8"));
} catch {
  ratchet = { lines: 0, branches: 0, functions: 0, statements: 0, updatedAt: "" };
}

const current = (coverageSummary as any).total;
const metrics = ["lines", "branches", "functions", "statements"] as const;
let failed = false;

for (const metric of metrics) {
  const currentPct = current[metric].pct;
  const ratchetPct = ratchet[metric];

  if (currentPct < ratchetPct) {
    console.error(
      `FAIL: ${metric} coverage dropped from ${ratchetPct}% to ${currentPct}% (delta: ${(currentPct - ratchetPct).toFixed(1)}%)`
    );
    failed = true;
  } else if (currentPct > ratchetPct) {
    console.log(`IMPROVED: ${metric} coverage increased from ${ratchetPct}% to ${currentPct}%`);
    ratchet[metric] = Math.floor(currentPct); // Floor to avoid ratcheting on decimals
  } else {
    console.log(`STABLE: ${metric} coverage at ${currentPct}%`);
  }
}

if (failed) {
  console.error("\nCoverage ratchet failed. Coverage must not decrease.");
  console.error("If this is intentional (e.g., removing dead code), update .coverage-ratchet.json manually.");
  process.exit(1);
}

// Update ratchet file
ratchet.updatedAt = new Date().toISOString();
fs.writeFileSync(RATCHET_FILE, JSON.stringify(ratchet, null, 2) + "\n");
console.log(`\nRatchet updated: ${JSON.stringify(ratchet)}`);

Commit .coverage-ratchet.json to the repo (e.g., { "lines": 82, "branches": 78, ... }). In CI, run the ratchet script after tests. On main branch merges, auto-commit the updated ratchet file if coverage improved.

PR Diff Coverage Gate

Require that new code in a PR meets a higher threshold (e.g., 90%) than the project baseline. In CI, use git diff --numstat origin/main...HEAD to identify changed files, then check their coverage from coverage-summary.json. Fail the pipeline if the average coverage of changed files falls below the threshold. This prevents coverage decay without demanding a rewrite of legacy code.

---

Meaningful vs Vanity Coverage

Why 100% Coverage Is Usually Wrong

100% coverage requires testing every branch of every line, including:

• Error handling for impossible states
• Default cases in exhaustive switches
• Framework lifecycle methods that are never called directly
• Defensive checks against corrupted data

Tests written to hit 100% are often trivial, brittle, and do not catch real bugs.

Diminishing Returns

Coverage Range	Value	Effort
0% to 60%	High -- covers main paths, catches obvious regressions	Low
60% to 80%	Medium -- covers error paths, edge cases	Medium
80% to 90%	Lower -- covers unusual combinations, defensive code	High
90% to 100%	Minimal -- covers unreachable code, framework internals	Very high

The sweet spot is 75-85% for most projects. Critical paths (payments, auth) should aim higher (90%+).

What NOT to Cover

Exclude these from coverage calculations. They inflate the denominator without adding value.

// vitest.config.ts or jest.config.js -- exclude patterns
exclude: [
  "**/*.d.ts",              // Type definitions
  "**/index.ts",            // Barrel exports (re-exports only)
  "**/*.stories.{ts,tsx}",  // Storybook stories
  "**/generated/**",        // Auto-generated code (GraphQL, Prisma)
  "**/migrations/**",       // Database migrations
  "**/__mocks__/**",        // Test mocks
  "**/types/**",            // Type-only modules
]

Quality Indicators Beyond Percentage

Coverage percentage alone is insufficient. Combine it with:

Indicator	What It Measures	How to Get It
Mutation score	Would tests catch a real bug?	Stryker / mutmut
Branch coverage	Are all conditional paths tested?	V8/Istanbul with branch reporting
Critical path coverage	Are payment/auth/data flows fully covered?	Per-directory thresholds
Defect escape rate	Do production bugs occur in tested code?	Post-incident analysis
Coverage delta	Is coverage improving or declining?	Ratchet pattern tracking

---

Anti-Patterns

Treating coverage as proof of quality. "We have 90% coverage so we are well-tested" is a dangerous statement. Coverage says code executed, not that behavior was verified. A test with no assertions contributes to coverage but catches zero bugs.

Excluding files to inflate numbers. Adding files to the exclude list because they are hard to test (e.g., error handlers, integration modules) hides the most important gaps. Only exclude genuinely untestable code: generated files, type definitions, and barrel exports.

Writing trivial tests to hit targets. Tests like it("should exist", () => { expect(MyClass).toBeDefined(); }) add coverage without value. Every test should verify a behavior that, if broken, would affect users.

Global threshold without per-module analysis. An 80% global threshold can pass even if the payment module is at 30% coverage -- as long as utility functions inflate the average. Use per-directory thresholds for critical paths.

Coverage threshold set once, never adjusted. The ratchet should increase over time. A team that has been at 78% for 6 months is not improving. Review the ratchet quarterly and investigate why it stagnates.

Ignoring branch coverage. Line coverage reports 100% on const result = condition ? a : b even if only one branch runs. Always report and gate on branch coverage alongside line coverage. Branch coverage is the more honest metric.

Coverage in E2E tests only. E2E tests execute many lines but cover broad, shallow paths. A single E2E test might touch 60% of the codebase without testing any edge case. Unit tests provide targeted, deep coverage of logic branches. Measure coverage from unit and integration tests separately from E2E.

---

Done When

• Coverage is instrumented and running automatically in CI on every push (no manual steps to generate the report).
• Coverage threshold is enforced in CI configuration (build fails when line or branch coverage drops below the defined minimum).
• Coverage report is published as a CI artifact (HTML report and json-summary) and linked from PR comments showing per-PR coverage delta.
• Meaningful vs. vanity coverage distinction is documented for the codebase: excluded paths are listed and justified (generated code, barrel exports, type definitions), and per-directory thresholds are set higher for critical paths such as payments and auth.
• Coverage-as-ratchet is configured so the threshold only goes up: .coverage-ratchet.json is committed, the ratchet script runs in CI, and the build fails if coverage regresses from the recorded baseline.

Related Skills

• unit-testing -- Test writing patterns, mocking strategies, and framework-specific coverage configuration.
• ci-cd-integration -- Pipeline configuration for coverage gates, artifact storage, and PR comments.
• qa-metrics -- Quality KPIs including coverage trends, mutation scores, and defect escape rates.
• ai-qa-review -- AI-assisted identification of coverage gaps and undertested code paths.