convergence-monitoring
Installation
SKILL.md
Convergence Monitoring
Convergence monitoring answers the most important question in iterative AI development: when should you stop iterating? The answer is not a fixed number of iterations or a time limit -- it is convergence. Convergence means the agent's output is stabilizing; each iteration produces fewer and smaller changes than the last.
Core insight: You don't need a zero-diff -- you need the remaining modifications to be inconsequential.
1. What Is Convergence?
Convergence appears as a rapid, consistent decline in the volume of changes from one iteration to the next:
Iteration 1: ████████████████████████████████████████ 300 lines changed
Iteration 2: ████████████████ 120 lines changed
Iteration 3: ██████ 40 lines changed
Iteration 4: ██ 10 lines changed (cosmetic only)
^--- Convergence reached: the diff shrinks each pass until only cosmetic changes remain
Convergence indicators
| Signal | What It Means |
|---|---|
| Lines changed decreasing exponentially | Each iteration makes roughly half the changes of the previous one |
| Changes become trivial | Remaining changes are formatting, comments, imports -- not behavior |
| Tests stabilize | Test count stops increasing; pass rate approaches 100% |
| No new files created | The architecture has settled; only existing files are modified |
| Impl tracking updates shrink | Implementation tracking changes are status updates, not new findings |
| Completion signal emitted | Agent determines all exit criteria are met |
What convergence looks like in git
# Check lines changed per iteration
git log --oneline --stat
# Iteration 5: trivial changes
abc1234 Iteration 5: formatting and comment fixes
3 files changed, 8 insertions(+), 6 deletions(-)
# Iteration 4: minor adjustments
def5678 Iteration 4: edge case handling
5 files changed, 22 insertions(+), 8 deletions(-)
# Iteration 3: moderate changes
ghi9012 Iteration 3: complete API integration
12 files changed, 85 insertions(+), 31 deletions(-)
# Iteration 2: significant changes
jkl3456 Iteration 2: implement core features
18 files changed, 156 insertions(+), 42 deletions(-)
# Iteration 1: major initial work
mno7890 Iteration 1: initial implementation
25 files changed, 312 insertions(+), 15 deletions(-)
2. What Is a Ceiling?
A ceiling is when the agent cannot make further progress due to external constraints. Like convergence, it produces small diffs -- but for fundamentally different reasons.
Convergence: Agent is DONE -> small diffs because work is complete
Ceiling: Agent is STUCK -> small diffs because agent cannot proceed
Ceiling causes
| Cause | Example | How to Detect |
|---|---|---|
| Missing dependency | API not available, library not installed | Agent logs errors about unavailable resources |
| Ambiguous spec | Requirement can be interpreted multiple ways | Agent oscillates between implementations |
| Tooling limitation | Build tool does not support needed feature | Agent tries workarounds that do not converge |
| External service | Test requires network access, external API | Tests fail with connection/timeout errors |
| Context window exhaustion | Codebase too large for one session | Agent loses track of earlier work |
| Permission boundary | Agent cannot access needed files or systems | Repeated permission errors in logs |
How to tell them apart
| Dimension | Convergence (work is finishing) | Ceiling (work is stuck) |
|---|---|---|
| Size of diffs | Shrinking steadily toward zero | Staying small but not trending down |
| Nature of changes | Cosmetic -- whitespace, comments, naming | Functional but going in circles |
| Test results | Pass rate climbing toward full coverage | Pass rate plateaued below target |
| Agent stance | Wrapping up, marking exit criteria done | Retrying the same strategies repeatedly |
| Tracking status | Tasks moving to DONE | BLOCKED items piling up |
| Recommended action | Declare done, move to next phase | Diagnose the obstacle, resolve it, then continue |
How to distinguish them
Check 1: Are tests passing?
YES, and improving -> Convergence
NO, stuck at same failures -> Ceiling
Check 2: Is the agent trying new approaches?
NO, just polishing -> Convergence
YES, but they all fail similarly -> Ceiling
Check 3: Are there BLOCKED tasks in impl tracking?
NO -> Convergence
YES -> Ceiling (read the blockers)
Check 4: Is the agent producing meaningful error messages?
NO, just minor changes -> Convergence
YES, about dependencies/tools/access -> Ceiling
3. Non-Convergence Signals
Non-convergence means the agent is making changes, but they are NOT decreasing. The system is not stabilizing.
Non-convergence:
Iteration 1: ████████████████████████████████████████ 250 lines changed
Iteration 2: ██████████████████████████████████████ 230 lines changed
Iteration 3: ████████████████████████████████████████ 260 lines changed
Iteration 4: ██████████████████████████████████ 220 lines changed
^--- NOT converging: changes are flat/oscillating
Root causes of non-convergence
| Root Cause | Symptom | Fix |
|---|---|---|
| Fuzzy specs | Agent interprets requirements differently each iteration | Make specs more precise; add concrete acceptance criteria |
| Weak validation | Agent cannot verify correctness, so it keeps changing things | Add build/test/lint gates; strengthen acceptance criteria |
| Fighting sub-agents | Multiple agents change the same code in conflicting ways | Add file ownership tables; dispatch subagents via the Agent tool |
| Contradictory requirements | Spec A says X, spec B says not-X | Resolve contradictions in specs; add explicit priority/precedence |
| Missing exit criteria | Agent does not know when it is done | Add explicit exit criteria checklists and completion signals |
| Over-broad scope | Too much work for one prompt/iteration | Split into smaller, focused prompts with clear boundaries |
| Unstable dependencies | External library or API keeps changing | Pin dependencies; mock external services in tests |
The critical rule
When the loop isn't stabilizing, the problem is upstream -- fix the specifications, validation, or coordination rather than adding more passes.
Running more iterations when the system is not converging wastes time and compute. Instead:
- Stop the iteration loop
- Analyze the non-convergence pattern
- Fix the root cause (usually specs or validation)
- Resume the iteration loop
4. Test Pass Rate as Convergence Signal
Test pass rate is the most reliable quantitative convergence signal. Track these metrics:
Metrics to monitor
| Iteration | Tests | Pass | Fail | Skip | Pass Rate | Delta |
|-----------|-------|------|------|------|-----------|-------|
| 1 | 45 | 30 | 15 | 0 | 66.7% | -- |
| 2 | 62 | 50 | 12 | 0 | 80.6% | +13.9 |
| 3 | 78 | 70 | 8 | 0 | 89.7% | +9.1 |
| 4 | 85 | 82 | 3 | 0 | 96.5% | +6.8 |
| 5 | 88 | 87 | 1 | 0 | 98.9% | +2.4 |
What to look for
| Pattern | Meaning | Action |
|---|---|---|
| Test count increasing | Agent is adding coverage | Good -- system is maturing |
| Pass rate approaching 100% | Implementation matches specs | Good -- approaching convergence |
| Fewer failures per iteration | Each pass fixes more than it breaks | Good -- healthy convergence |
| Pass rate plateaus < 100% | Some tests consistently fail | Ceiling -- investigate failing tests |
| Test count decreasing | Agent is deleting tests | Bad -- investigate why; may be deleting inconvenient tests |
| Pass rate oscillating | Fixes in one area break another | Non-convergence -- check for conflicting specs |
Automated convergence check
# After each iteration, check convergence signals
echo "=== Convergence Check ==="
# 1. Lines changed (should be decreasing)
git diff --stat HEAD~1
# 2. Test results (should be improving)
{TEST_COMMAND} 2>&1 | tail -5
# 3. Build health (should always pass)
{BUILD_COMMAND} 2>&1 | tail -3
# 4. Files changed (should be decreasing)
git diff --name-only HEAD~1 | wc -l
5. Forward Progress Metrics
For large projects where full convergence takes many iterations, track forward progress toward eventual convergence.
Spec requirement coverage
The percentage of spec requirements with passing tests:
Spec Requirements Coverage:
spec-auth.md: ██████████████████████████████████████ 95% (19/20 requirements)
spec-data.md: ████████████████████████████████ 80% (16/20 requirements)
spec-ui.md: ██████████████████████ 55% (11/20 requirements)
spec-api.md: ████████████████████████████ 70% (14/20 requirements)
─────────────────────────────────────────────────────
Overall: ████████████████████████████ 75% (60/80 requirements)
Forward progress signals
| Metric | Healthy Trend | Unhealthy Trend |
|---|---|---|
| Requirements with passing tests | Increasing each iteration | Flat or decreasing |
| Total test count | Increasing | Flat or decreasing |
| DONE tasks in impl tracking | Increasing | Flat with BLOCKED tasks growing |
| Open issues | Decreasing | Increasing or flat |
| Dead ends documented | Increasing slightly (learning) | Exploding (thrashing) |
Iteration velocity
Track how much progress each iteration makes:
| Iteration | Requirements Met | New This Iteration | Velocity |
|-----------|-----------------|-------------------|----------|
| 1 | 15/80 | 15 | 15 |
| 2 | 30/80 | 15 | 15 |
| 3 | 48/80 | 18 | 18 |
| 4 | 60/80 | 12 | 12 |
| 5 | 68/80 | 8 | 8 |
| 6 | 73/80 | 5 | 5 |
| 7 | 76/80 | 3 | 3 |
Velocity should decrease over time (easy requirements first, hard ones last), but should never hit zero. Zero velocity = ceiling.
6. When to Stop Iterating
Stop conditions (convergence reached)
Stop the iteration loop when ANY of these are true:
- Completion signal emitted: Agent outputs
<all-tasks-complete> - Changes are trivial: Last iteration changed fewer than ~20 lines, all formatting/comments
- Test pass rate is stable: Pass rate has been 95%+ for 2+ consecutive iterations
- All exit criteria met: Every
[ ]in the exit criteria checklist is[x] - Forward progress stalled positively: All spec requirements have passing tests
Continue conditions (not yet converged)
Continue iterating when ALL of these are true:
- Changes are still substantial (behavior changes, not just formatting)
- Test pass rate is still improving
- There are still TODO or IN_PROGRESS tasks in impl tracking
- The iteration count is under the maximum
Investigate conditions (possible ceiling)
Pause and investigate when ANY of these are true:
- Changes are small but tests are NOT passing
- Agent is retrying the same approach repeatedly
- BLOCKED tasks are accumulating in impl tracking
- Test pass rate is oscillating (up-down-up-down)
- Agent is producing error messages about dependencies or tooling
7. Monitoring During Iteration Loops
What to monitor in real time
+------------------------------------------------------+
| Convergence Dashboard |
+------------------------------------------------------+
| Iteration: 4/10 |
| Lines changed: 45 (prev: 112, trend: decreasing) |
| Files changed: 3 (prev: 8, trend: decreasing) |
| Test pass rate: 94.2% (prev: 87.1%, trend: up) |
| Tests: 82 total (prev: 75, trend: up) |
| BLOCKED tasks: 0 (prev: 1, trend: down) |
| Status: CONVERGING |
+------------------------------------------------------+
Monitoring commands
# Quick convergence check after each iteration
echo "--- Lines changed ---"
git diff --stat HEAD~1 | tail -1
echo "--- Files changed ---"
git diff --name-only HEAD~1 | wc -l
echo "--- Test results ---"
{TEST_COMMAND} --summary 2>&1 | tail -3
echo "--- Impl tracking status ---"
grep -c "BLOCKED\|IN_PROGRESS\|TODO\|DONE" context/impl/impl-*.md
Automated alerts
Set up alerts for non-convergence signals:
| Alert | Trigger | Action |
|---|---|---|
| Oscillation | Lines changed increased vs previous iteration | Pause; check for conflicting changes |
| Stall | Lines changed < 5 but tests still failing | Pause; likely a ceiling |
| Regression | Test pass rate decreased | Pause; investigate what broke |
| Runaway | Lines changed > 500 for 3+ iterations | Pause; scope may be too broad |
8. Non-Convergence Recovery
When you detect non-convergence, follow this recovery process:
Step 1: Stop the iteration loop
Do not keep running. More iterations will not help.
Step 2: Diagnose the root cause
## Non-Convergence Diagnosis
### Symptoms
- [ ] Changes are flat (not decreasing)
- [ ] Changes are oscillating (up-down-up-down)
- [ ] Agent is retrying failed approaches
- [ ] Tests are oscillating (passing then failing)
- [ ] Multiple agents changing the same files
### Root Cause Analysis
1. Check specs: Are requirements clear and unambiguous?
2. Check validation: Can the agent verify correctness?
3. Check file ownership: Are agents conflicting?
4. Check scope: Is the prompt trying to do too much?
5. Check dependencies: Are external resources available?
Step 3: Fix the root cause
| Root Cause | Fix |
|---|---|
| Fuzzy specs | Rewrite ambiguous requirements with concrete acceptance criteria |
| Weak validation | Add build/test/lint gates to the prompt |
| File conflicts | Add file ownership tables; dispatch subagents via the Agent tool |
| Over-broad scope | Split into smaller prompts; reduce concurrent agents |
| External dependency | Mock the dependency; or resolve it before resuming |
Step 4: Resume the iteration loop
After fixing the root cause, resume from where you stopped. Do NOT restart from scratch -- git history preserves all progress.
# Resume with the same prompt, possibly fewer remaining iterations
iteration-loop context/prompts/003-generate-impl-from-plans.md -n 5 -t 1h
9. Convergence and Revision
Revision directly improves convergence by making specs more complete:
Without revision:
Iteration 1: 200 lines, 5 manual fixes -> specs unchanged
Iteration 2: 180 lines, 4 manual fixes -> specs unchanged
Iteration 3: 170 lines, 4 manual fixes -> NOT converging
With revision:
Iteration 1: 200 lines, 5 manual fixes -> specs updated with 5 new requirements
Iteration 2: 100 lines, 2 manual fixes -> specs updated with 2 new requirements
Iteration 3: 50 lines, 0 manual fixes -> CONVERGING
Frequent manual fixes without revision = non-convergence. The iteration loop keeps producing the same bugs because nothing in the specs prevents them.
Cross-References
- Convergence patterns reference: See
references/convergence-patterns.mdfor the complete convergence pattern catalog with examples. - Revision: See
ck:revisionskill for how tracing bugs to specs improves convergence. - Prompt pipeline: See
ck:prompt-pipelineskill for designing prompts with proper exit criteria and completion signals. - Validation-first design: See
ck:validation-firstskill for building validation gates that provide convergence signals. - Impl tracking: See
ck:impl-trackingskill for tracking progress and detecting ceiling conditions.