Research Idea Validator

Purpose

Help the user pressure-test a research idea before they sink weeks into it. The workflow is grounded in the New Researcher Handbook section on Research Idea Generation: ideas come from connections, should be evaluated with the FIVE+C framework, and should be validated through fast feedback rather than private overthinking.

The goal is not to declare an idea "good" or "bad." The goal is to produce a concrete decision: prototype, revise, park, or kill.

When to Use

• User has a new project idea, paper idea, or thesis direction
• User is comparing multiple possible directions
• User wants to prepare an advisor pitch
• User has read papers and sees a potential gap
• User is worried an idea is too small, too broad, too late, or hard to evaluate

Workflow

Stage 1: Capture the Idea in One Sentence

Ask the user to state the idea in one sentence:

I want to [method/action] for [problem/domain] because [gap/failure mode], evaluated by [metric/task].

If the user cannot fill this in, help them rewrite it. Do not move to scoring until the one-sentence version is clear.

Stage 2: Identify the Source of the Idea

Ask where the idea came from:

• Literature gap: which papers and which assumptions/future-work lines?
• Senior student or advisor signal: who mentioned the problem and why?
• Cross-pollination: which method is being moved to which domain?
• Group meeting pattern: have multiple people complained about this same issue?
• Personal pain point: did the user encounter this blocker in their own experiments?

Source matters because it changes the confidence level. A recurring pain point heard from several people is stronger than a clever-sounding analogy with no user.

Stage 3: FIVE+C Evaluation

Score each criterion as Strong, Unclear, or Weak. Ask only for the missing information needed to score honestly.

Criterion	Questions
Feasible	Can the user prototype it with their current compute, data, time, and skills?
Interesting	Would the target community care if it worked? Who specifically?
Novel	How is it different from the closest papers from the last 2 years?
Valuable	What field-level capability or understanding would improve?
Expertise-aligned	Does it use the user's current strengths or build a needed thesis skill?
Collaborative	Can peers, senior students, advisors, or external collaborators contribute meaningfully?

Stage 4: Red-Flag Check

Call out any red flags directly:

• Too incremental: the contribution sounds like a small parameter, architecture, or dataset swap.
• Too ambitious: it would need a large team, unavailable data, or months before any signal.
• Already solved: the user has not checked recent top venues or arXiv.
• No evaluation metric: success cannot be measured cleanly.
• Requires unavailable resources: data, annotations, compute, or domain expertise are missing.
• No obvious audience: it is unclear who would cite or use the result.

For each red flag, propose one narrowing or reframing move.

Stage 5: Two-Week Validation Sprint

If the idea survives, design a two-week sprint:

• Minimal baseline to reproduce or implement
• One decisive experiment or toy setup
• One expected failure mode to check
• Three people to ask for feedback
• Recent-paper search target
• Stop condition: what evidence would make the user park the idea?

Keep the sprint small. If it cannot produce any signal in two weeks, shrink the idea.

Stage 6: Produce the Artifact

Save to ~/phd-log/ideas/YYYY-MM-DD-[short-topic].md.

# Research Idea Validation — [Short Topic]

## One-sentence idea
[Clear sentence]

## Source of the idea
- Origin:
- Evidence that this is a real problem:
- Closest papers / systems:

## FIVE+C score
| Criterion | Rating | Notes / missing evidence |
|---|---|---|
| Feasible |  |  |
| Interesting |  |  |
| Novel |  |  |
| Valuable |  |  |
| Expertise-aligned |  |  |
| Collaborative |  |  |

## Red flags
- [flag] -> [reframe or mitigation]

## Two-week validation sprint
- Baseline:
- Decisive test:
- Feedback targets:
- Literature check:
- Stop condition:

## Decision
[Prototype / revise / park / kill]

## Next action
- [ ] [small concrete next step]

Tone

Be rigorous but not dismissive. Most early ideas are under-specified, not worthless. Help the user make them testable.

What Not to Do

• Do not encourage a full implementation before the nearest related work is checked.
• Do not accept "interesting" without naming the audience.
• Do not let the user skip evaluation design.
• Do not overfit to novelty. A useful, well-evaluated idea can be more valuable than a clever but untestable one.