Evaluation Rubrics

Workflow
Common Patterns
Guardrails
Quick Reference

Example

Scenario: Evaluating technical blog posts (1-5 scale)

Criterion	1 (Poor)	3 (Adequate)	5 (Excellent)
Technical Accuracy	Multiple factual errors, misleading	Mostly correct, minor inaccuracies	Fully accurate, technically rigorous
Clarity	Confusing, jargon-heavy, poor structure	Clear to experts, some structure	Accessible to target audience, well-organized
Practical Value	No actionable guidance, theoretical only	Some examples, limited applicability	Concrete examples, immediately applicable
Originality	Rehashes common knowledge, no new insight	Some fresh perspective, builds on existing	Novel approach, advances understanding

Scoring: Post A [4, 5, 3, 2] = 3.5 avg. Post B [5, 4, 5, 4] = 4.5 avg. Feedback for Post A: "Strong clarity (5) and good accuracy (4), but needs more practical examples (3) and offers less original insight (2)."

Workflow

Copy this checklist and track your progress:

Rubric Development Progress:
- [ ] Step 1: Define purpose and scope
- [ ] Step 2: Identify evaluation criteria
- [ ] Step 3: Design the scale
- [ ] Step 4: Write performance descriptors
- [ ] Step 5: Test and calibrate
- [ ] Step 6: Use and iterate

Step 1: Define purpose and scope

Clarify what you're evaluating, who evaluates, who uses results, what decisions depend on scores. See resources/template.md for scoping questions.

Step 2: Identify evaluation criteria

Brainstorm quality dimensions, prioritize most important/observable, balance coverage vs. simplicity (4-8 criteria typical). See resources/template.md for brainstorming framework.

Step 3: Design the scale

Choose number of levels (1-5, 1-4, 1-10), scale type (numeric, qualitative), anchors (what does each level mean?). See resources/methodology.md for scale selection guidance.

Step 4: Write performance descriptors

For each criterion × level, write observable description of what that performance looks like. See resources/template.md for writing guidelines.

Step 5: Test and calibrate

Have multiple reviewers score sample work, compare scores, discuss discrepancies, refine rubric. See resources/methodology.md for inter-rater reliability testing.

Step 6: Use and iterate

Apply rubric, collect feedback from evaluators and evaluatees, revise criteria/descriptors as needed. Validate using resources/evaluators/rubric_evaluation_rubrics.json. Minimum standard: Average score ≥ 3.5.

Common Patterns

Pattern 1: Analytic Rubric (Most Common)

Structure: Multiple criteria (rows), multiple levels (columns), descriptor for each cell
Use case: Detailed feedback needed, want to see performance across dimensions, diagnostic assessment
Pros: Specific feedback, identifies strengths/weaknesses by criterion, high reliability
Cons: Time-consuming to create and use, can feel reductive
Example: Code review rubric (Correctness, Efficiency, Readability, Maintainability × 1-5 scale)

Pattern 2: Holistic Rubric

Structure: Single overall score, descriptors integrate multiple criteria
Use case: Quick overall judgment, summative assessment, criteria hard to separate
Pros: Fast, intuitive, captures gestalt quality
Cons: Less actionable feedback, lower reliability, can't diagnose specific weaknesses
Example: Essay holistic scoring (1=poor essay, 3=adequate essay, 5=excellent essay with detailed descriptors)

Pattern 3: Single-Point Rubric

Structure: Criteria listed with only "meets standard" descriptor, space to note above/below
Use case: Growth mindset feedback, encourage self-assessment, less punitive feel
Pros: Emphasizes improvement not deficit, simpler to create, encourages dialogue
Cons: Less precision, requires written feedback to supplement
Example: Design critique (list criteria like "Visual hierarchy", "Accessibility", note "+Clear focal point, -Poor contrast")

Pattern 4: Checklist (Binary)

Structure: List of yes/no items, must-haves for acceptance
Use case: Compliance checks, minimum quality gates, pass/fail decisions
Pros: Very clear, objective, easy to use
Cons: No gradations, misses quality beyond basics, can feel rigid
Example: Pull request checklist (Tests pass? Code linted? Documentation updated? Security review?)

Pattern 5: Standards-Based Rubric

Structure: Criteria tied to learning objectives/competencies, levels = degree of mastery
Use case: Educational assessment, skill certification, training evaluation, criterion-referenced
Pros: Aligned to standards, shows progress toward mastery, diagnostic
Cons: Requires clear standards, can be complex to design
Example: Data science skills (Proficiency in: Data cleaning, Modeling, Visualization, Communication × Novice/Competent/Expert)

Guardrails

Criteria should be observable and measurable: Not "good attitude" (subjective), but "arrives on time, volunteers for tasks, helps teammates" (observable). Test: Can two independent reviewers score this criterion consistently?
Descriptors should distinguish levels clearly: Each level needs concrete differences from adjacent levels. Avoid "5=very good, 4=good, 3=okay". Better: "5=zero bugs, meets all requirements, 4=1-2 minor bugs, meets 90% requirements."
Use appropriate scale granularity: 1-3 is too coarse, 1-10 is too fine. Sweet spot: 1-4 (forced choice, no middle) or 1-5 (allows neutral middle). Match granularity to actual observable differences.
Balance comprehensiveness with simplicity: Aim for 4-8 criteria covering essential quality dimensions. If >10 criteria, consider grouping or prioritizing.
Calibrate for inter-rater reliability: Have multiple reviewers score same work, measure agreement (Kappa, ICC). If <70% agreement, refine descriptors.
Provide examples at each level: Include concrete examples of work at each level (anchor papers, reference designs, code samples) to calibrate reviewers.
Share rubric before evaluation: If evaluatees see the rubric only after being scored, it is grading not guidance. Share upfront so people know expectations and can self-assess.
Weight criteria appropriately: If "Security" matters more than "Code style", weight it (Security x3, Style x1). Or use thresholds (score >=4 on Security to pass, regardless of other scores).

Common pitfalls:

❌ Subjective language: "Shows effort", "creative", "professional" - not observable without concrete descriptors
❌ Overlapping criteria: "Clarity" and "Organization" often conflated - define boundaries clearly
❌ Hidden expectations: Rubric doesn't mention X, but evaluators penalize for missing X - document all criteria
❌ Central tendency bias: Reviewers avoid extremes (always score 3/5) - use even-number scales (1-4) to force choice
❌ Halo effect: High score on one criterion biases other scores up - score each criterion independently before looking at others
❌ Rubric drift: Descriptors erode over time, reviewers interpret differently - periodic re-calibration required

Quick Reference

Key resources:

resources/template.md: Purpose definition, criteria brainstorming, scale selection, descriptor templates, rubric formats
resources/methodology.md: Scale design principles, descriptor writing techniques, inter-rater reliability testing, bias mitigation
resources/evaluators/rubric_evaluation_rubrics.json: Quality criteria for rubric design (criteria clarity, scale appropriateness, descriptor specificity)

Scale Selection Guide:

Scale	Use When	Pros	Cons
1-3	Need quick categorization, clear tiers	Fast, forces clear decision	Too coarse, less feedback
1-4	Want forced choice (no middle)	Avoids central tendency, clear differentiation	No neutral option, feels binary
1-5	General purpose, most common	Allows neutral, familiar, good granularity	Central tendency bias (everyone gets 3)
1-10	Need fine gradations, large sample	Maximum differentiation, statistical analysis	False precision, hard to distinguish adjacent levels
Qualitative (Novice/Proficient/Expert)	Educational, skill development	Intuitive, growth-oriented	Less quantitative, harder to aggregate
Binary (Yes/No, Pass/Fail)	Compliance, gatekeeping	Objective, simple	No gradations, misses quality differences

Criteria Types:

Product criteria: Evaluate the artifact itself (correctness, clarity, completeness, aesthetics, performance)
Process criteria: How work was done (methodology followed, collaboration, iteration, time management)
Impact criteria: Outcomes/effects (user satisfaction, business value, learning achieved)
Meta criteria: Quality of quality (documentation, testability, maintainability, scalability)

Inter-Rater Reliability Benchmarks:

<50% agreement: Rubric unreliable, needs major revision
50-70% agreement: Marginal, refine descriptors and calibrate reviewers
70-85% agreement: Good, acceptable for most uses
>85% agreement: Excellent, highly reliable scoring

Typical Rubric Development Time:

Simple rubric (3-5 criteria, 1-4 scale, known domain): 2-4 hours
Standard rubric (5-7 criteria, 1-5 scale, some complexity): 6-10 hours + calibration session
Complex rubric (8+ criteria, multiple scales, novel domain): 15-25 hours + multiple calibration rounds

When to escalate beyond rubrics:

High-stakes decisions (hiring, admissions, awards) → Add structured interviews, portfolios, multi-method assessment
Subjective/creative work (art, poetry, design) → Supplement rubric with critique, discourse, expert judgment
Complex holistic judgment (leadership, cultural fit) → Rubrics help but don't capture everything, use thoughtfully → Rubrics are tools not replacements for human judgment. Use to structure thinking, not mechanize decisions.

Inputs required:

Artifact type (what are we evaluating? essays, code, designs, proposals?)
Criteria (quality dimensions to assess, 4-8 most common)
Scale (1-5 default, or specify 1-4, 1-10, qualitative labels)

Outputs produced:

evaluation-rubrics.md: Purpose, criteria definitions, scale with descriptors, usage instructions, weighting/thresholds, calibration notes