Research Claim Map

Workflow
Evidence Quality Framework
Source Credibility Assessment
Common Patterns
Guardrails
Quick Reference

Workflow

Copy this checklist and track your progress:

Research Claim Map Progress:
- [ ] Step 1: Define the claim precisely
- [ ] Step 2: Gather and categorize evidence
- [ ] Step 3: Rate evidence quality and source credibility
- [ ] Step 4: Identify limitations and gaps
- [ ] Step 5: Draw evidence-based conclusion

Step 1: Define the claim precisely

Restate the claim as a specific, testable assertion. Avoid vague language - use numbers, dates, and clear terms. See Common Patterns for claim reformulation examples.

Step 2: Gather and categorize evidence

Collect sources supporting and contradicting the claim. Organize into "Evidence For" and "Evidence Against". For straightforward verification → Use resources/template.md. For complex multi-source investigations → Study resources/methodology.md.

Step 3: Rate evidence quality and source credibility

Apply Evidence Quality Framework to rate each source (primary/secondary/tertiary). Apply Source Credibility Assessment to evaluate expertise, bias, and track record.

Step 4: Identify limitations and gaps

Document what's unknown, what assumptions were made, and where evidence is weak or missing. See resources/methodology.md for gap analysis techniques.

Step 5: Draw evidence-based conclusion

Synthesize findings into confidence level (0-100%) and actionable recommendation (believe/skeptical/reject claim). Self-check using resources/evaluators/rubric_research_claim_map.json before delivering. Minimum standard: Average score ≥ 3.5.

Evidence Quality Framework

Rating scale:

Primary Evidence (Strongest):

Direct observation or measurement
Original data or records
First-hand accounts from participants
Raw datasets, transaction logs
Example: Sales database showing 10,000 customer IDs

Secondary Evidence (Medium):

Analysis or interpretation of primary sources
Expert synthesis of multiple primary sources
Peer-reviewed research papers
Verified news reporting with primary source citations
Example: Industry analyst report analyzing public filings

Tertiary Evidence (Weakest):

Summaries of secondary sources
Textbooks, encyclopedias, Wikipedia
Press releases, marketing materials
Anecdotal reports without verification
Example: Company blog post claiming customer count

Non-Evidence (Unreliable):

Unverified social media posts
Anonymous claims
"Experts say" without attribution
Circular references (A cites B, B cites A)
Example: Viral tweet with no source

Source Credibility Assessment

Evaluate each source on:

Expertise (Does source have relevant knowledge?):

High: Domain expert with credentials, track record
Medium: Knowledgeable but not specialist
Low: No demonstrated expertise

Independence (Is source biased or conflicted?):

High: Independent, no financial/personal stake
Medium: Some potential bias, disclosed
Low: Direct financial interest, undisclosed conflicts

Track Record (Has source been accurate before?):

High: Consistent accuracy, corrections when wrong
Medium: Mixed record or unknown history
Low: History of errors, retractions, unreliability

Methodology (How did source obtain information?):

High: Transparent, replicable, rigorous
Medium: Some methodology disclosed
Low: Opaque, unverifiable, cherry-picked

Common Patterns

Pattern 1: Vendor Claim Verification

Claim type: Product performance, customer count, ROI
Approach: Seek independent verification (analysts, customers), test claims yourself
Red flags: Only vendor sources, vague metrics, "up to X%" ranges

Pattern 2: Academic Literature Review

Claim type: Research findings, causal claims
Approach: Check for replication studies, meta-analyses, competing explanations
Red flags: Single study, small sample, conflicts of interest, p-hacking

Pattern 3: News Fact-Checking

Claim type: Events, statistics, quotes
Approach: Trace to primary source, check multiple outlets, verify context
Red flags: Anonymous sources, circular reporting, sensational framing

Pattern 4: Statistical Claims

Claim type: Percentages, trends, correlations
Approach: Check methodology, sample size, base rates, confidence intervals
Red flags: Cherry-picked timeframes, denominator unclear, correlation ≠ causation

Guardrails

Avoid common biases:

Confirmation bias: Actively seek evidence against your hypothesis
Authority bias: Don't accept claims just because source is prestigious
Recency bias: Older evidence can be more reliable than latest claims
Availability bias: Vivid anecdotes ≠ representative data

Quality standards:

Rate confidence numerically (0-100%), not vague terms ("probably", "likely")
Document all assumptions explicitly
Distinguish "no evidence found" from "evidence of absence"
Update conclusions as new evidence emerges
Flag when evidence quality is insufficient for confident conclusion

Ethical considerations:

Respect source privacy and attribution
Avoid cherry-picking evidence to support desired conclusion
Acknowledge limitations and uncertainties
Correct errors promptly when found

Quick Reference

Resources:

Quick verification: resources/template.md
Complex investigations: resources/methodology.md
Quality rubric: resources/evaluators/rubric_research_claim_map.json

Evidence hierarchy: Primary > Secondary > Tertiary

Credibility factors: Expertise + Independence + Track Record + Methodology

Confidence calibration:

90-100%: Near certain, multiple primary sources, high credibility
70-89%: Confident, strong secondary sources, some limitations
50-69%: Uncertain, conflicting evidence or weak sources
30-49%: Skeptical, more evidence against than for
0-29%: Likely false, strong evidence against

research-claim-map