Skill Improver

Process user feedback from skill retrospectives and update skill files to improve them over time.

When to Use

User asks to "review skill feedback" or "improve skills based on usage"
You notice feedback files in .claude/feedback/
User mentions a skill didn't work well or missed something
Periodic review (monthly) to incorporate learnings

How It Works

Step 1: Gather Feedback

Read all feedback files in .claude/feedback/:

ls -la .claude/feedback/retro-*.md

Look for patterns:

Multiple users reporting same missing step → add to skill
Benchmarks don't match user's context → add context-specific ranges
Workflow confusing → restructure or add clarifications
Skill incomplete → add missing sections

Step 2: Identify High-Impact Changes

Prioritize updates based on:

High Priority (do first):

Missing critical steps that users had to figure out themselves
Incorrect benchmarks or numbers
Confusing workflow that requires clarification
Safety issues or errors

Medium Priority:

Additional examples or templates
Better explanations of existing steps
Alternative approaches for different contexts

Low Priority:

Nice-to-have additions
Stylistic improvements
Minor clarifications

Step 3: Update Skill Files

For each skill needing updates:

3a. Add a "Learnings" Section

If the skill doesn't have one, add at the end:

## Learnings from Use

**[Date]**: [Brief description of what was learned]
- **Feedback**: [What users reported]
- **Update**: [What we changed]
- **Result**: [Expected improvement]

3b. Update Main Content

If feedback suggests core changes:

Add missing steps to checklists
Update benchmarks with ranges (e.g., "20-30% for B2C, 50-70% for B2B")
Restructure workflow if confusing
Add "Common Pitfalls" section if users make same mistakes

3c. Version the Change

At the top of the skill, track versions:

---
name: skill-name
version: 1.2.0
last_updated: 2026-01-22
changelog:
  - v1.2.0 (2026-01-22): Added missing step for X based on user feedback
  - v1.1.0 (2026-01-15): Updated benchmarks for Y context
  - v1.0.0 (2026-01-01): Initial release
---

Step 4: Archive Processed Feedback

Move processed feedback to archive:

mkdir -p .claude/feedback/archive
mv .claude/feedback/retro-2026-01-22-*.md .claude/feedback/archive/

Keep a summary of learnings in .claude/feedback/SUMMARY.md:

# Feedback Summary

## [Skill Name]

**Total feedback sessions**: 12
**Last updated**: 2026-01-22

**Key learnings**:
- Added step for X (reported by 3 users)
- Updated benchmarks for B2B context (reported by 5 users)
- Clarified workflow around Y (reported by 2 users)

**Patterns**:
- Users in enterprise context need higher benchmarks
- Early-stage startups need more examples
- Non-technical users need clearer explanations of jargon

Example Workflow

Scenario: product-market-fit skill needs improvement

Step 1: Review Feedback

Read .claude/feedback/retro-2026-01-22-143022.md:

## Feedback

**Missed important steps?** yes

**Improvements needed:**
The Sean Ellis test threshold of 40% seems high for B2B enterprise products.
We're at 32% "very disappointed" but our retention is 85% D30 which is excellent.
Should the skill mention that thresholds vary by product type?

Step 2: Identify Pattern

Check other feedback files → 3 more users report B2B context needs different benchmarks.

Step 3: Update Skill

Edit .claude/skills/product-market-fit/SKILL.md:

Before:

## Sean Ellis Test (40% Rule)

"How would you feel if you could no longer use [product]?"
- ≥40% "Very disappointed" = Strong PMF

After:

## Sean Ellis Test (Context-Dependent Thresholds)

"How would you feel if you could no longer use [product]?"

**Thresholds by product type:**
- **Consumer B2C**: ≥40% "Very disappointed" = Strong PMF
- **SMB B2B**: ≥35% "Very disappointed" = Strong PMF
- **Enterprise B2B**: ≥30% "Very disappointed" = Strong PMF (longer sales cycles, different buying psychology)

**Why the difference?**
- Enterprise buyers are more rational than emotional
- Switching costs are higher (contracts, integrations)
- Retention is a better PMF signal for B2B (see Step 2)

Add to Learnings section:

## Learnings from Use

**2026-01-22**: Refined Sean Ellis thresholds by product type
- **Feedback**: 4 users reported 40% threshold too high for B2B enterprise
- **Update**: Added context-specific thresholds (B2C 40%, SMB 35%, Enterprise 30%)
- **Result**: More accurate PMF diagnosis for different product types

Step 4: Archive & Track

mv .claude/feedback/retro-2026-01-22-*.md .claude/feedback/archive/

Update .claude/feedback/SUMMARY.md:

## product-market-fit

**Total feedback sessions**: 4
**Last updated**: 2026-01-22

**Key learnings**:
- Added context-specific Sean Ellis thresholds (reported by 4 users)
- B2B needs different benchmarks than B2C

**Next improvements to consider**:
- Add industry-specific retention benchmarks
- Include examples from different verticals

Quality Checklist

Before updating any skill, ensure:

Feedback is from multiple users (pattern, not outlier)
Change makes skill more accurate, not just more complex
Benchmarks are sourced or validated (not anecdotal)
Update is backward compatible (doesn't break existing workflows)
Learnings section documents why we made the change
Version number incremented appropriately (semver)
Processed feedback archived, not deleted

Feedback Categories

Track feedback by type to identify systemic issues:

Category 1: Missing Steps

Example: "Skill forgot to mention we need to segment cohorts by acquisition channel" Action: Add step to checklist

Category 2: Incorrect Benchmarks

Example: "40% D30 retention is not 'strong' for our B2B SaaS, it's average" Action: Update benchmarks with context (B2C vs B2B vs Enterprise)

Category 3: Confusing Workflow

Example: "I didn't know whether to do cohort analysis before or after Sean Ellis test" Action: Number steps clearly, add workflow diagram

Category 4: Missing Context

Example: "Skill assumes I have 1000+ users, what if I only have 50?" Action: Add "Early Stage Adaptation" section

Category 5: Tool-Specific Issues

Example: "How do I calculate D30 retention in Google Analytics?" Action: Add "Implementation in Common Tools" section

Best Practices

Do:

✅ Look for patterns across multiple feedback sessions ✅ Update skills incrementally (small, tested changes) ✅ Document why changes were made (Learnings section) ✅ Preserve feedback history (archive, don't delete) ✅ Version skills so users know what changed

Don't:

❌ Update based on single piece of feedback (might be outlier) ❌ Make skills overly complex trying to cover every edge case ❌ Remove content without understanding why it was there ❌ Ignore feedback for more than 30 days (patterns emerge) ❌ Update without testing the new version

Automation Ideas

Weekly Digest (optional)

Create a script to summarize new feedback:

#!/bin/bash
# .claude/hooks/learning/weekly-feedback-digest.sh

echo "📊 Feedback Digest (Last 7 Days)"
echo "━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━"

find .claude/feedback -name "retro-*.md" -mtime -7 | while read -r file; do
    echo ""
    echo "File: $(basename $file)"
    grep "Skills Used" -A 5 "$file"
    grep "Improvements needed:" -A 3 "$file"
done

Auto-Tag for Review

When feedback mentions specific issues, auto-tag:

"missing step" → tag for immediate review
"wrong number" → tag for fact-check
"confusing" → tag for clarity rewrite

Success Metrics

Track improvement over time:

Feedback frequency: Decreasing = skills getting better
Repeated issues: Should approach zero over time
User satisfaction: Track "Did this skill help?" responses
Skill usage: Updated skills should see increased usage

Meta: This Skill Improves Itself

This skill should follow its own advice:

Learnings from Use:

[To be filled as this skill gets used and improved]

Version History:

v1.0.0 (2026-01-22): Initial release - framework for skill improvement

Next Steps:

Review feedback in .claude/feedback/
Identify patterns and prioritize updates
Update skill files with improvements
Document learnings
Archive processed feedback
Commit changes with clear message

The more you use this system, the better your skills become. It's a continuous improvement loop.

skill-improver