Deep Research

Iterative multi-round research across Web, Reddit, X/Twitter, GitHub, Hacker News, Substack, Financial Media, LinkedIn, and more with structured output frameworks (comparison, landscape, deep-dive, decision).

Architecture

Source	Tool	Cost
Web	Claude Code `WebSearch` + xAI `web_search`	Free + $0.005/call
Reddit	Claude Code `WebSearch` (site:reddit.com) + xAI `web_search`	Free + $0.005/call
X/Twitter	xAI `x_search` only	$0.005/call
GitHub	Claude Code `WebSearch` (site:github.com) + xAI `web_search`	Free + $0.005/call
Hacker News	Claude Code `WebSearch` (site:news.ycombinator.com) + xAI `web_search`	Free + $0.005/call
Substack	Claude Code `WebSearch` (site:substack.com) + xAI `web_search`	Free + $0.005/call
Financial Media	Claude Code `WebSearch` (site-specific) + xAI `web_search`	Free + $0.005/call
Wall Street Oasis	Claude Code `WebSearch` (site:wallstreetoasis.com)	Free
LinkedIn	Claude Code `WebSearch` (site:linkedin.com) + xAI `web_search`	Free + $0.005/call
Crunchbase	Claude Code `WebSearch` (site:crunchbase.com)	Free
YouTube	Claude Code `WebSearch` (site:youtube.com)	Free
Tech Blogs	Claude Code `WebSearch` (site-specific)	Free

Results from multiple sources are merged and deduplicated for comprehensive coverage.

Prerequisites

Required Tools

Tool	Purpose	Install
uv	Python package manager (handles dependencies)	`curl -LsSf https://astral.sh/uv/install.sh \| sh`

Optional Tools

Tool	Purpose	Install
scrapling	Headless browser fallback for sites that block WebFetch (403, captcha, empty responses)	`uv tool install 'scrapling[all]'`

API Keys

Service	Purpose	Required	Get Key
xAI	X/Twitter search + supplemental web/GitHub/HN search	Recommended	https://console.x.ai

Note: The skill works without xAI key (web-only mode via Claude Code), but X/Twitter data and broader coverage require xAI.

Keychain Setup (One-Time, for xAI)

# 1. Create a dedicated keychain
security create-keychain -p 'YourPassword' ~/Library/Keychains/claude-keys.keychain-db

# 2. Add keychain to search list
security list-keychains -s ~/Library/Keychains/claude-keys.keychain-db ~/Library/Keychains/login.keychain-db /Library/Keychains/System.keychain

# 3. Store your xAI API key
echo -n "Enter xAI API key: " && read -s key && security add-generic-password -s "xai-api" -a "$USER" -w "$key" ~/Library/Keychains/claude-keys.keychain-db && unset key && echo

Before using: security unlock-keychain ~/Library/Keychains/claude-keys.keychain-db

Workflow Overview

Step	Action	Purpose
0	Detect xAI key	Determine Full vs Web-Only mode
1	Parse query	Extract TOPIC, FRAMEWORK, DEPTH
2	Round 1: Broad search	Discover entities, themes, initial findings
3	Gap analysis	Identify missing perspectives, unverified claims
4	Round 2: Targeted follow-up	Fill gaps, verify claims, deepen coverage
5	Round 3: Verification	(deep only) Primary source verification
6	Synthesis	Structure findings into framework template
7	Expert mode	Answer follow-ups from cached results

Step 0: Detect xAI Key

MANDATORY — run before every research session.

security find-generic-password -s "xai-api" -w ~/Library/Keychains/claude-keys.keychain-db 2>/dev/null && echo "XAI_AVAILABLE=true" || echo "XAI_AVAILABLE=false"

XAI_AVAILABLE=true: Use Full mode — Claude WebSearch AND xAI scripts in parallel.
XAI_AVAILABLE=false: Use Web-Only mode — Claude WebSearch only. Append note suggesting xAI setup.

This step is NOT optional. Always check before starting research.

Step 1: Parse Query

Extract from user input:

1a. TOPIC

The subject being researched. Strip framework indicators and depth modifiers.

1b. FRAMEWORK

Detect output framework from query patterns:

Framework	Detection Patterns	Example
COMPARISON	"X vs Y", "compare X and Y", "X or Y", "which is better"	"React vs Vue for enterprise apps"
LANDSCAPE	"landscape", "ecosystem", "market", "what's out there", "overview of"	"AI agent frameworks landscape"
DEEP_DIVE	"deep dive", "how does X work", "explain", "tell me about", "what is"	"Deep dive into WebAssembly"
DECISION	"should I/we", "evaluate", "which should we use", "recommend"	"Should we use Kafka or RabbitMQ?"

Explicit override: User can force a framework with [comparison], [landscape], [deep-dive], or [decision] anywhere in query.

Default: If no framework detected, use DEEP_DIVE.

1c. DEPTH

Depth	Trigger	Rounds	Target Sources
quick	"quick", "brief", "overview"	1	10-15
default	(none)	2	25-40
deep	"deep", "comprehensive", "thorough"	3	60-90

Step 2: Round 1 — Broad Search

Query Generation

Generate 6-9 queries covering different angles of the TOPIC:

Direct query: "{TOPIC}" — the topic as stated
Temporal query: "{TOPIC} 2026" or "{TOPIC} latest"
Reddit query: site:reddit.com "{TOPIC}"
GitHub query: site:github.com "{TOPIC}"
HN query: site:news.ycombinator.com "{TOPIC}"
Framework-specific query:
- COMPARISON: "{Alt A} vs {Alt B}"
- LANDSCAPE: "{TOPIC} ecosystem" OR "{TOPIC} landscape"
- DEEP_DIVE: "how {TOPIC} works" OR "{TOPIC} explained"
- DECISION: "{TOPIC}" experience OR recommendation
Substack query: site:substack.com "{TOPIC}"
Financial media query: site:tradingview.com OR site:benzinga.com OR site:seekingalpha.com "{TOPIC}" (for finance/economics topics)
LinkedIn query: site:linkedin.com "{TOPIC}" (when topic involves people or companies)

Parallel Execution

Run searches simultaneously:

Claude Code (free):

WebSearch: direct query
WebSearch: temporal query
WebSearch: Reddit-targeted query
WebSearch: GitHub-targeted query
WebSearch: HN-targeted query
WebSearch: Substack-targeted query
WebSearch: Financial media-targeted query (for finance/economics topics)
WebSearch: LinkedIn-targeted query (when topic involves people/companies)
WebSearch: YouTube-targeted query
WebSearch: WSO-targeted query (for finance topics)
WebSearch: Crunchbase-targeted query (for company/startup topics)

xAI scripts (if available, run as background Bash tasks):

uv run scripts/xai_search.py web "{TOPIC}" --json &
uv run scripts/xai_search.py reddit "{TOPIC}" --json &
uv run scripts/xai_search.py x "{TOPIC}" --json &
uv run scripts/xai_search.py github "{TOPIC}" --json &
uv run scripts/xai_search.py hn "{TOPIC}" --json &
uv run scripts/xai_search.py substack "{TOPIC}" --json &
uv run scripts/xai_search.py finance "{TOPIC}" --json &
uv run scripts/xai_search.py linkedin "{TOPIC}" --json &

Merge and Deduplicate

MERGED_WEB = dedupe(claude_web + xai_web)
MERGED_REDDIT = dedupe(claude_reddit + xai_reddit)
MERGED_GITHUB = dedupe(claude_github + xai_github)
MERGED_HN = dedupe(claude_hn + xai_hn)
MERGED_SUBSTACK = dedupe(claude_substack + xai_substack)
MERGED_FINANCE = dedupe(claude_finance + xai_finance)
MERGED_LINKEDIN = dedupe(claude_linkedin + xai_linkedin)
X_RESULTS = xai_x_results  (no Claude equivalent)
WSO_RESULTS = claude_wso  (WebSearch only)
CRUNCHBASE_RESULTS = claude_crunchbase  (WebSearch only)
YOUTUBE_RESULTS = claude_youtube  (WebSearch only)

Deduplication by URL, keeping the entry with more metadata.

Round 1 Internal Notes

Record (internally, NOT in output):

KEY_ENTITIES: [specific tools, companies, people discovered]
THEMES: [recurring themes across sources]
GAPS: [what's missing — feed into Step 3]
CONTRADICTIONS: [conflicting claims]
LEADS: [URLs worth deep-reading via WebFetch (or scrapling fallback) in Round 2]

Step 3: Gap Analysis

After Round 1, run gap analysis. See references/iterative-research.md for full checklist.

Gap Categories

Gap	Check	Action
Missing perspective	Have developer, operator, and business views?	Target missing perspective
Unverified claims	Any claims from only 1 source?	Seek corroboration
Shallow coverage	Any entity mentioned but unexplained?	Deep-search that entity
Stale data	Key facts > 12 months old?	Search for recent updates
Missing source type	Missing Reddit / GitHub / HN / X / blogs?	Target that platform
Missing financial/business context	Missing Substack / financial media / LinkedIn / Crunchbase?	Target that platform

Plan Round 2

Select top 4-6 gaps. Generate targeted queries for each. See references/search-patterns.md for multi-round refinement patterns.

Skip to Step 6 if depth is quick (single round only).

Step 4: Round 2 — Targeted Follow-Up

Query Rules

Never repeat Round 1 queries
Entity-specific queries — target names/tools discovered in Round 1
Source-type specific — target platforms underrepresented in Round 1
Framework-adapted — see targeting table in references/iterative-research.md

Execution

Same parallel pattern as Round 1, but with targeted queries.

Additionally, use WebFetch for high-value URLs discovered in Round 1:

Official documentation pages
Benchmark result pages
Engineering blog posts
Comparison articles with methodology

Maximum 4-6 WebFetch calls in Round 2.

Scrapling fallback: If WebFetch returns 403, empty content, a captcha page, or a blocked response, retry using the auto-escalation protocol from cli-web-scrape:

scrapling extract get "URL" /tmp/scrapling-fallback.md → Read → validate content
If content is thin (JS-only shell, no data, mostly nav links) → scrapling extract fetch "URL" /tmp/scrapling-fallback.md --network-idle --disable-resources → Read → validate
If still blocked → scrapling extract stealthy-fetch "URL" /tmp/scrapling-fallback.md --solve-cloudflare
All tiers fail → skip URL and note "scrapling blocked"

Confidence Update

After Round 2, re-assess all claims:

Before	New Evidence	After
[LOW]	Second source found	[MEDIUM]
[MEDIUM]	Third source found	[HIGH]
Any	Contradicted	Note conflict, present both sides

Skip to Step 6 if depth is default.

Step 5: Round 3 — Verification (Deep Only)

Round 3 is for verification only. No new discovery.

Budget

Maximum 6-10 WebFetch lookups targeting:

Target	Purpose	Max Calls
Primary sources for key claims	Verify accuracy	3-4
Independent benchmark sites	Validate performance claims	1-2
Both sides of contradictions	Resolve conflicts	1-2
Official sites for versions/dates	Confirm recency	1-2

Scrapling fallback: Same auto-escalation protocol as Round 2 — try HTTP tier, validate content, escalate to Dynamic/Stealthy if thin or blocked.

Rules

Verify, don't discover — no new topic exploration
Target highest-impact claims — those that would change the recommendation
Check primary sources — go to the original, not summaries
Update confidence — upgrade or downgrade based on findings
Trust primary over secondary — if primary contradicts secondary, note it

Step 6: Synthesis

Framework Selection

Load references/output-frameworks.md and select the template matching the detected FRAMEWORK.

Filling the Template

Header block — Framework type, topic, depth, source count, date
Core content — Fill framework sections with research findings
Confidence indicators — Mark each claim: [HIGH], [MEDIUM], or [LOW]
Community perspective — Synthesize Reddit/X/HN/GitHub sentiment
Statistics footer — Source counts and engagement metrics
Sources section — Organized by platform with URLs and metrics

Engagement-Weighted Synthesis

Weight sources by signal strength. See references/iterative-research.md for full weighting table.

Signal	Threshold	Weight
Reddit upvotes	100+	High
X engagement	50+ likes	High
GitHub stars	1000+	High
HN points	100+	High
Substack likes	50+	High
Multi-platform agreement	3+ sources	High
Dual-engine match	Claude + xAI	High
Seeking Alpha comments	20+	Medium
WSO upvotes	10+	Medium
YouTube views	10K+	Medium
Recent (< 7 days)	Any	Medium
Single source only	Any	Low

Statistics Footer Format

Research Statistics
├─ Reddit: {n} threads │ {upvotes} upvotes
├─ X: {n} posts │ {likes} likes │ {reposts} reposts
├─ GitHub: {n} repos │ {stars} total stars
├─ HN: {n} threads │ {points} total points
├─ Substack: {n} articles │ {likes} likes
├─ Financial: {n} articles │ {sources}
├─ LinkedIn: {n} profiles/articles
├─ YouTube: {n} videos
├─ WSO: {n} threads
├─ Crunchbase: {n} profiles
├─ Web: {n} pages │ {domains}
├─ Merged: {n} from Claude + {n} from xAI
└─ Top voices: r/{sub1} │ @{handle1} │ {blog1}

Web-Only Mode footer:

Research Statistics
├─ Reddit: {n} threads (via Claude WebSearch)
├─ GitHub: {n} repos (via Claude WebSearch)
├─ HN: {n} threads (via Claude WebSearch)
├─ Substack: {n} articles (via Claude WebSearch)
├─ Financial: {n} articles (via Claude WebSearch)
├─ LinkedIn: {n} profiles/articles (via Claude WebSearch)
├─ YouTube: {n} videos (via Claude WebSearch)
├─ Web: {n} pages
└─ Top sources: {site1}, {site2}

Add X/Twitter + broader coverage: Set up xAI API key (see Prerequisites)

Step 7: Expert Mode

After delivering research, enter Expert Mode:

Answer follow-ups from cached results
No new searches unless explicitly requested
Cross-reference between sources

New search triggers (exit Expert Mode):

"Search again for..."
"Find more about..."
"Update the research..."
"Look deeper into..."

Operational Modes

Mode	Sources	When
Full	Claude WebSearch + xAI (web + X + Reddit + GitHub + HN + Substack + Finance + LinkedIn)	Step 0 returns XAI_AVAILABLE=true
Web-Only	Claude WebSearch only	Step 0 returns XAI_AVAILABLE=false

Mode is determined by Step 0 — never skip it or assume Web-Only without checking.

Depth Control

Depth	Rounds	Sources	xAI Calls (Full)	Use Case
quick	1	10-15	8	Fast overview, time-sensitive
default	2	25-40	13	Balanced research
deep	3	60-90	18 + 6-10 WebFetch	Comprehensive analysis, important decisions

Cost Awareness

Action	Cost
Claude Code WebSearch	Free (subscription)
xAI search call (any type)	$0.005/call
WebFetch (built-in)	Free
scrapling fallback (optional)	Free

Estimated cost per research session:

Depth	Full Mode	Web-Only
quick	~$0.04 (8 xAI calls)	Free
default	~$0.065 (13 xAI calls)	Free
deep	~$0.09 (18 xAI calls)	Free

Cost-Saving Strategy:

Claude WebSearch handles most needs (free)
xAI adds X/Twitter (unique value) + broader coverage per platform
WebFetch for deep-reading specific URLs (free), with scrapling fallback for blocked sites

Critical Constraints

DO:

Run Step 0 (xAI key detection) before every research session
If xAI key exists: run Claude WebSearch AND xAI scripts in parallel (Full mode)
If xAI key missing: use Claude WebSearch only (Web-Only mode)
Run gap analysis between rounds — never skip it
Merge and deduplicate results by URL
Exclude archived GitHub repositories from results — they are unmaintained and read-only
Mark every claim with confidence: [HIGH], [MEDIUM], or [LOW]
Ground synthesis in actual research, not pre-existing knowledge
Cite specific sources with URLs
Use --json flag when calling xAI scripts for programmatic parsing
Load framework template from references/output-frameworks.md

DON'T:

Skip Claude Code WebSearch (it's free)
Run sequential searches when parallel is possible
Display duplicate results from different search engines
Quote more than 125 characters from any single source
Repeat queries across rounds — each round targets gaps from previous
Add Round 3 for quick or default depth — it's deep-only
Discover new topics in Round 3 — verification only

Troubleshooting

xAI key not found:

security find-generic-password -s "xai-api" ~/Library/Keychains/claude-keys.keychain-db

If not found, run keychain setup above.

Keychain locked:

security unlock-keychain ~/Library/Keychains/claude-keys.keychain-db

No X/Twitter results: Requires valid xAI API key. Check at https://console.x.ai

WebFetch blocked (403/captcha/empty): Install scrapling and follow the auto-escalation protocol from cli-web-scrape (HTTP → validate → Dynamic → Stealthy):

uv tool install 'scrapling[all]'
scrapling install  # one-time: install browser engines for Dynamic/Stealthy tiers

Script errors: Ensure uv is installed: which uv. If missing: curl -LsSf https://astral.sh/uv/install.sh | sh

References

references/output-frameworks.md — Framework templates (comparison, landscape, deep-dive, decision)
references/search-patterns.md — Search operators and multi-round query patterns
references/iterative-research.md — Gap analysis, round procedures, cross-referencing methodology

res-deep