Usability Frameworks

Comprehensive frameworks and methodologies for planning, conducting, and analyzing usability tests to improve user experience.

When to Use This Skill

Auto-loaded by agents:

• research-ops - For usability testing and heuristic evaluation

Use when you need:

• Planning usability tests
• Conducting user testing sessions
• Evaluating interface designs
• Identifying usability problems
• Testing prototypes or live products
• Applying Nielsen's heuristics
• Measuring usability metrics

Core Concepts

What is Usability Testing?

Usability testing is a method for evaluating a product by testing it with representative users. Users attempt to complete typical tasks while observers watch, listen, and take notes.

Purpose: Identify usability problems, discover opportunities for improvement, and learn about user behavior and preferences.

When to use:

• Before development (testing prototypes)
• During development (iterative testing)
• After launch (validation and optimization)
• Before major redesigns

The Five Usability Quality Components (Jakob Nielsen)

1. Learnability: How easy is it for users to accomplish basic tasks the first time?
2. Efficiency: How quickly can users perform tasks once they've learned the design?
3. Memorability: Can users remember how to use it after time away?
4. Errors: How many errors do users make, how severe, and how easily can they recover?
5. Satisfaction: How pleasant is it to use the design?

Usability Testing Methodologies

1. Moderated Testing

Setup: Researcher guides participants through tasks in real-time Location: In-person or remote (video call)

Best for:

• Early-stage prototypes needing clarification
• Complex products requiring guidance
• Exploring "why" behind user behavior
• Uncovering emotional reactions

Process:

1. Welcome and set expectations
2. Pre-task questions (background, experience)
3. Task scenarios with think-aloud protocol
4. Post-task questions and discussion
5. Wrap-up and thank you

Advantages:

• Rich qualitative insights
• Can probe deeper into issues
• Observe non-verbal cues
• Clarify misunderstandings immediately

Limitations:

• More time-intensive (30-60 min per session)
• Researcher bias possible
• Smaller sample sizes
• Scheduling logistics

2. Unmoderated Testing

Setup: Participants complete tasks independently, recorded for later review Location: Remote, on participant's own schedule

Best for:

• Mature products with clear tasks
• Large sample sizes needed
• Quick turnaround required
• Benchmarking and metrics

Process:

1. Automated instructions and consent
2. Participants record screen/audio while completing tasks
3. Automated post-task surveys
4. Researcher reviews recordings later

Advantages:

• Faster data collection
• Larger sample sizes
• More natural environment
• Lower cost per participant

Limitations:

• Can't probe or clarify
• May miss nuanced insights
• Technical issues harder to resolve
• Participants may skip think-aloud

3. Hybrid Approaches

Combination methods:

• Moderated first impressions + unmoderated task completion
• Unmoderated testing + follow-up interviews with interesting cases
• Moderated pilot + unmoderated scale testing

Nielsen's 10 Usability Heuristics

Quick reference for evaluating interfaces. See references/nielsens-10-heuristics.md for detailed explanations and examples.

1. Visibility of system status - Keep users informed
2. Match between system and real world - Speak users' language
3. User control and freedom - Provide escape hatches
4. Consistency and standards - Follow platform conventions
5. Error prevention - Prevent problems before they occur
6. Recognition rather than recall - Minimize memory load
7. Flexibility and efficiency of use - Accelerators for experts
8. Aesthetic and minimalist design - Remove irrelevant information
9. Help users recognize, diagnose, and recover from errors - Plain language error messages
10. Help and documentation - Provide when needed

Think-Aloud Protocol

What It Is

Participants verbalize their thoughts while completing tasks, providing real-time insight into their mental model.

Types

Concurrent think-aloud: Speak while performing tasks

• More natural thought flow
• May affect task performance slightly

Retrospective think-aloud: Review recording and explain thinking after

• Doesn't disrupt natural behavior
• May forget or rationalize thoughts

Facilitating Think-Aloud

Prompts to use:

• "What are you thinking right now?"
• "What are you looking for?"
• "What would you expect to happen?"
• "Is this what you expected?"

Don't:

• Ask leading questions
• Provide hints or solutions
• Interrupt natural flow too often
• Make participants feel tested

See references/think-aloud-protocol-guide.md for detailed facilitation techniques.

Task Scenario Design

Good task scenarios are critical to meaningful usability test results.

Characteristics of Good Task Scenarios

Realistic: Based on actual user goals Specific: Clear endpoint/success criteria Self-contained: Provide all necessary context Actionable: Clear starting point Not prescriptive: Don't tell them how to do it

Example Transformation

Poor: "Click on the 'My Account' link and change your password"

• Too prescriptive, tells them exactly where to click

Good: "You've heard about recent security breaches and want to make your account more secure. Update your account to use a stronger password."

• Realistic motivation, clear goal, doesn't prescribe path

Task Complexity Levels

Simple tasks (1-2 steps): Establish baseline usability Medium tasks (3-5 steps): Test core workflows Complex tasks (6+ steps): Evaluate overall experience and error recovery

See assets/task-scenario-template.md for ready-to-use templates.

Severity Rating Framework

Not all usability issues are equal. Prioritize fixes based on severity.

Three-Factor Severity Rating

Frequency: How often does this issue occur?

• High: > 50% of users encounter
• Medium: 10-50% encounter
• Low: < 10% encounter

Impact: When it occurs, how badly does it affect users?

• High: Prevents task completion / causes data loss
• Medium: Causes frustration or delays
• Low: Minor annoyance

Persistence: Do users overcome it with experience?

• High: Problem doesn't go away
• Medium: Users learn to avoid/work around
• Low: One-time problem only

Combined Severity Ratings

Critical (P0): High frequency + High impact Serious (P1): High frequency + Medium impact, OR Medium frequency + High impact Moderate (P2): High frequency + Low impact, OR Medium frequency + Medium impact, OR Low frequency + High impact Minor (P3): Everything else

See assets/severity-rating-guide.md for detailed rating criteria and examples.

Usability Metrics

Quantitative Metrics

Task Success Rate: % of participants who complete task successfully

• Binary: Did they complete it? (yes/no)
• Partial credit: Did they complete most of it?

Time on Task: How long to complete (for successful completions)

• Compare to baseline or competitor benchmarks

Error Rate: Number of errors per task

• Define what counts as an error for each task

Clicks/Taps to Task Completion: Efficiency measure

• More relevant for well-defined tasks

Standardized Questionnaires

SUS (System Usability Scale):

• 10 questions, 5-point Likert scale
• Score 0-100 (industry avg ~68)
• Quick, reliable, easy to administer
• Good for comparing versions or benchmarking

UMUX (Usability Metric for User Experience):

• 4 questions, lighter than SUS
• Similar reliability
• Faster for participants

SEQ (Single Ease Question):

• "Overall, how difficult or easy was the task to complete?" (1-7)
• One question per task
• Immediate subjective difficulty rating

Other scales:

• SUPR-Q (for websites)
• PSSUQ (post-study)
• NASA-TLX (cognitive load)

Qualitative Insights

Observed behaviors:

• Hesitations and confusion
• Error patterns
• Unexpected paths
• Verbal frustrations

Verbalized thoughts (think-aloud):

• Mental model mismatches
• Expectation violations
• Pleasantly surprising discoveries

Sample Size Guidelines

For Qualitative Insights

Nielsen's recommendation: 5 users finds ~85% of usability problems

• Diminishing returns after 5
• Run 3+ small rounds instead of 1 large round
• Iterate between rounds

Reality check:

• 5 is a minimum, not ideal
• Complex products may need 8-10
• Multiple user types need 5 each

For Quantitative Metrics

Benchmarking: 20+ users per user group A/B testing: Depends on effect size and desired confidence Statistical significance: Use power analysis calculators

Planning Your Usability Test

1. Define Objectives

What decisions will this research inform?

• Redesign priorities?
• Feature cut decisions?
• Success of recent changes?

2. Identify User Segments

Who needs to be tested?

• New vs. experienced users?
• Different roles or use cases?
• Different devices or contexts?

3. Select Tasks

What tasks represent success?

• Most critical user goals
• Most frequent tasks
• Recently changed features
• Known problem areas

4. Choose Methodology

Moderated, unmoderated, or hybrid?

• Consider timeline, budget, research questions

5. Create Test Script

See assets/usability-test-script-template.md for a ready-to-use structure including:

• Welcome and consent
• Background questions
• Task instructions
• Probing questions
• Wrap-up

6. Recruit Participants

• Define screening criteria
• Aim for 5-10 per user segment
• Plan for no-shows (recruit 20% extra)
• Offer appropriate incentives

7. Conduct Pilot Test

• Test with colleague or friend
• Validate timing
• Check recording setup
• Refine unclear tasks

8. Run Sessions

• Stay neutral and encouraging
• Observe without interfering
• Take detailed notes
• Record if permitted

9. Analyze and Synthesize

• Code issues by severity
• Identify patterns across participants
• Link issues to heuristics violated
• Quantify task success and time

10. Report and Recommend

• Prioritized issue list
• Video clips of critical issues
• Recommendations with rationale
• Quick wins vs. strategic fixes

Integration with Product Development

When to Test

Discovery phase: Test competitors or analogous products Concept phase: Test paper prototypes or wireframes Design phase: Test high-fidelity mockups Development phase: Test working builds iteratively Pre-launch: Validate before release Post-launch: Identify optimization opportunities

Continuous Usability Testing

Build it into your process:

• Weekly or bi-weekly test sessions
• Rotating focus (new features, established flows, mobile vs. desktop)
• Standing recruiting panel
• Lightweight reporting to team

Ready-to-Use Templates

We provide templates to accelerate your usability testing:

In assets/:

• usability-test-script-template.md: Complete moderator script structure
• task-scenario-template.md: Framework for creating effective task scenarios
• severity-rating-guide.md: Detailed criteria for rating usability issues

In references/:

• nielsens-10-heuristics.md: Deep dive into each heuristic with examples
• think-aloud-protocol-guide.md: Advanced facilitation techniques and troubleshooting

Common Pitfalls to Avoid

1. Leading participants: "Was that easy?" → "How would you describe that experience?"
2. Testing the wrong tasks: Tasks that aren't real user goals
3. Over-explaining: Let users struggle and discover issues naturally
4. Ignoring severity: Fixing cosmetic issues while critical issues remain
5. Testing too late: After it's expensive to change
6. Not iterating: One-and-done testing instead of continuous improvement
7. Confusing usability with preference: "I like green" ≠ usability issue
8. Sample bias: Testing only power users or only complete novices

Further Learning

Books:

• "Rocket Surgery Made Easy" by Steve Krug
• "Handbook of Usability Testing" by Jeffrey Rubin
• "Moderating Usability Tests" by Joseph Dumas

Online resources:

• Nielsen Norman Group articles
• Usability.gov
• Baymard Institute research

---

Troubleshooting

"Participants aren't finding real issues": Your tasks are too easy or too guided. Write tasks as goals ("Find a flight under $300 to Chicago"), not instructions ("Click the search button, then enter Chicago"). Let participants struggle -- that's where the insights are.

"Stakeholders dismiss usability findings": Quantify severity. Use Nielsen's severity scale (0-4) and pair each finding with task success rate data. "3 of 5 users failed to complete checkout" is harder to dismiss than "checkout is confusing."

"We only have 3 users available for testing": That's enough for a guerrilla test. Nielsen's research shows 5 users find ~85% of issues, but 3 users still find the majority. Run the test, fix the top issues, test again with 3 more.

---

Related Skills

• interview-frameworks - User interview techniques for deeper qualitative research
• synthesis-frameworks - Synthesizing usability findings into actionable insights
• user-research-techniques - Broader research methods beyond usability testing
• validation-frameworks - Validating solutions after usability improvements