Usability Frameworks

Comprehensive frameworks and methodologies for planning, conducting, and analyzing usability tests to improve user experience.

When to Use This Skill

Auto-loaded by agents:

research-ops - For usability testing and heuristic evaluation

Use when you need:

Planning usability tests
Conducting user testing sessions
Evaluating interface designs
Identifying usability problems
Testing prototypes or live products
Applying Nielsen's heuristics
Measuring usability metrics

Core Concepts

What is Usability Testing?

Usability testing is a method for evaluating a product by testing it with representative users. Users attempt to complete typical tasks while observers watch, listen, and take notes.

Purpose: Identify usability problems, discover opportunities for improvement, and learn about user behavior and preferences.

When to use:

Before development (testing prototypes)
During development (iterative testing)
After launch (validation and optimization)
Before major redesigns

The Five Usability Quality Components (Jakob Nielsen)

Learnability: How easy is it for users to accomplish basic tasks the first time?
Efficiency: How quickly can users perform tasks once they've learned the design?
Memorability: Can users remember how to use it after time away?
Errors: How many errors do users make, how severe, and how easily can they recover?
Satisfaction: How pleasant is it to use the design?

Usability Testing Methodologies

1. Moderated Testing

Setup: Researcher guides participants through tasks in real-time Location: In-person or remote (video call)

Best for:

Early-stage prototypes needing clarification
Complex products requiring guidance
Exploring "why" behind user behavior
Uncovering emotional reactions

Process:

Welcome and set expectations
Pre-task questions (background, experience)
Task scenarios with think-aloud protocol
Post-task questions and discussion
Wrap-up and thank you

Advantages:

Rich qualitative insights
Can probe deeper into issues
Observe non-verbal cues
Clarify misunderstandings immediately

Limitations:

More time-intensive (30-60 min per session)
Researcher bias possible
Smaller sample sizes
Scheduling logistics

2. Unmoderated Testing

Setup: Participants complete tasks independently, recorded for later review Location: Remote, on participant's own schedule

Best for:

Mature products with clear tasks
Large sample sizes needed
Quick turnaround required
Benchmarking and metrics

Process:

Automated instructions and consent
Participants record screen/audio while completing tasks
Automated post-task surveys
Researcher reviews recordings later

Advantages:

Faster data collection
Larger sample sizes
More natural environment
Lower cost per participant

Limitations:

Can't probe or clarify
May miss nuanced insights
Technical issues harder to resolve
Participants may skip think-aloud

3. Hybrid Approaches

Combination methods:

Moderated first impressions + unmoderated task completion
Unmoderated testing + follow-up interviews with interesting cases
Moderated pilot + unmoderated scale testing

Nielsen's 10 Usability Heuristics

Quick reference for evaluating interfaces. See references/nielsens-10-heuristics.md for detailed explanations and examples.

Visibility of system status - Keep users informed
Match between system and real world - Speak users' language
User control and freedom - Provide escape hatches
Consistency and standards - Follow platform conventions
Error prevention - Prevent problems before they occur
Recognition rather than recall - Minimize memory load
Flexibility and efficiency of use - Accelerators for experts
Aesthetic and minimalist design - Remove irrelevant information
Help users recognize, diagnose, and recover from errors - Plain language error messages
Help and documentation - Provide when needed

Think-Aloud Protocol

What It Is

Participants verbalize their thoughts while completing tasks, providing real-time insight into their mental model.

Types

Concurrent think-aloud: Speak while performing tasks

More natural thought flow
May affect task performance slightly

Retrospective think-aloud: Review recording and explain thinking after

Doesn't disrupt natural behavior
May forget or rationalize thoughts

Facilitating Think-Aloud

Prompts to use:

"What are you thinking right now?"
"What are you looking for?"
"What would you expect to happen?"
"Is this what you expected?"

Don't:

Ask leading questions
Provide hints or solutions
Interrupt natural flow too often
Make participants feel tested

See references/think-aloud-protocol-guide.md for detailed facilitation techniques.

Task Scenario Design

Good task scenarios are critical to meaningful usability test results.

Characteristics of Good Task Scenarios

Realistic: Based on actual user goals Specific: Clear endpoint/success criteria Self-contained: Provide all necessary context Actionable: Clear starting point Not prescriptive: Don't tell them how to do it

Example Transformation

Poor: "Click on the 'My Account' link and change your password"

Too prescriptive, tells them exactly where to click

Good: "You've heard about recent security breaches and want to make your account more secure. Update your account to use a stronger password."

Realistic motivation, clear goal, doesn't prescribe path

Task Complexity Levels

Simple tasks (1-2 steps): Establish baseline usability Medium tasks (3-5 steps): Test core workflows Complex tasks (6+ steps): Evaluate overall experience and error recovery

See assets/task-scenario-template.md for ready-to-use templates.

Severity Rating Framework

Not all usability issues are equal. Prioritize fixes based on severity.

Three-Factor Severity Rating

Frequency: How often does this issue occur?

High: > 50% of users encounter
Medium: 10-50% encounter
Low: < 10% encounter

Impact: When it occurs, how badly does it affect users?

High: Prevents task completion / causes data loss
Medium: Causes frustration or delays
Low: Minor annoyance

Persistence: Do users overcome it with experience?

High: Problem doesn't go away
Medium: Users learn to avoid/work around
Low: One-time problem only

Combined Severity Ratings

Critical (P0): High frequency + High impact Serious (P1): High frequency + Medium impact, OR Medium frequency + High impact Moderate (P2): High frequency + Low impact, OR Medium frequency + Medium impact, OR Low frequency + High impact Minor (P3): Everything else

See assets/severity-rating-guide.md for detailed rating criteria and examples.

Usability Metrics

Quantitative Metrics

Task Success Rate: % of participants who complete task successfully

Binary: Did they complete it? (yes/no)
Partial credit: Did they complete most of it?

Time on Task: How long to complete (for successful completions)

Compare to baseline or competitor benchmarks

Error Rate: Number of errors per task

Define what counts as an error for each task

Clicks/Taps to Task Completion: Efficiency measure

More relevant for well-defined tasks

Standardized Questionnaires

SUS (System Usability Scale):

10 questions, 5-point Likert scale
Score 0-100 (industry avg ~68)
Quick, reliable, easy to administer
Good for comparing versions or benchmarking

UMUX (Usability Metric for User Experience):

4 questions, lighter than SUS
Similar reliability
Faster for participants

SEQ (Single Ease Question):

"Overall, how difficult or easy was the task to complete?" (1-7)
One question per task
Immediate subjective difficulty rating

Other scales:

SUPR-Q (for websites)
PSSUQ (post-study)
NASA-TLX (cognitive load)

Qualitative Insights

Observed behaviors:

Hesitations and confusion
Error patterns
Unexpected paths
Verbal frustrations

Verbalized thoughts (think-aloud):

Mental model mismatches
Expectation violations
Pleasantly surprising discoveries

Sample Size Guidelines

For Qualitative Insights

Nielsen's recommendation: 5 users finds ~85% of usability problems

Diminishing returns after 5
Run 3+ small rounds instead of 1 large round
Iterate between rounds

Reality check:

5 is a minimum, not ideal
Complex products may need 8-10
Multiple user types need 5 each

For Quantitative Metrics

Benchmarking: 20+ users per user group A/B testing: Depends on effect size and desired confidence Statistical significance: Use power analysis calculators

Planning Your Usability Test

1. Define Objectives

What decisions will this research inform?

Redesign priorities?
Feature cut decisions?
Success of recent changes?

2. Identify User Segments

Who needs to be tested?

New vs. experienced users?
Different roles or use cases?
Different devices or contexts?

3. Select Tasks

What tasks represent success?

Most critical user goals
Most frequent tasks
Recently changed features
Known problem areas

4. Choose Methodology

Moderated, unmoderated, or hybrid?

Consider timeline, budget, research questions

5. Create Test Script

See assets/usability-test-script-template.md for a ready-to-use structure including:

Welcome and consent
Background questions
Task instructions
Probing questions
Wrap-up

6. Recruit Participants

Define screening criteria
Aim for 5-10 per user segment
Plan for no-shows (recruit 20% extra)
Offer appropriate incentives

7. Conduct Pilot Test

Test with colleague or friend
Validate timing
Check recording setup
Refine unclear tasks

8. Run Sessions

Stay neutral and encouraging
Observe without interfering
Take detailed notes
Record if permitted

9. Analyze and Synthesize

Code issues by severity
Identify patterns across participants
Link issues to heuristics violated
Quantify task success and time

10. Report and Recommend

Prioritized issue list
Video clips of critical issues
Recommendations with rationale
Quick wins vs. strategic fixes

Integration with Product Development

When to Test

Discovery phase: Test competitors or analogous products Concept phase: Test paper prototypes or wireframes Design phase: Test high-fidelity mockups Development phase: Test working builds iteratively Pre-launch: Validate before release Post-launch: Identify optimization opportunities

Continuous Usability Testing

Build it into your process:

Weekly or bi-weekly test sessions
Rotating focus (new features, established flows, mobile vs. desktop)
Standing recruiting panel
Lightweight reporting to team

Ready-to-Use Templates

We provide templates to accelerate your usability testing:

In `assets/`:

usability-test-script-template.md: Complete moderator script structure
task-scenario-template.md: Framework for creating effective task scenarios
severity-rating-guide.md: Detailed criteria for rating usability issues

In `references/`:

nielsens-10-heuristics.md: Deep dive into each heuristic with examples
think-aloud-protocol-guide.md: Advanced facilitation techniques and troubleshooting

Common Pitfalls to Avoid

Leading participants: "Was that easy?" → "How would you describe that experience?"
Testing the wrong tasks: Tasks that aren't real user goals
Over-explaining: Let users struggle and discover issues naturally
Ignoring severity: Fixing cosmetic issues while critical issues remain
Testing too late: After it's expensive to change
Not iterating: One-and-done testing instead of continuous improvement
Confusing usability with preference: "I like green" ≠ usability issue
Sample bias: Testing only power users or only complete novices

Further Learning

Books:

"Rocket Surgery Made Easy" by Steve Krug
"Handbook of Usability Testing" by Jeffrey Rubin
"Moderating Usability Tests" by Joseph Dumas

Online resources:

Nielsen Norman Group articles
Usability.gov
Baymard Institute research

This skill provides the foundation for conducting effective usability testing. Use the templates in assets/ for quick starts and references/ for deeper dives into specific techniques.

usability-frameworks