Exploratory Data Analysis Skill

Run exploratory data analysis on any dataset. Generates distribution plots, time series, correlation matrices, summary statistics, and interactive HTML reports.

Quick Start

# Analyze a CSV file
/sc:eda data/sales.csv

# Analyze with specific output directory
/sc:eda data/users.json --output eda_results/users/

# Analyze with column focus
/sc:eda data/metrics.parquet --focus revenue,churn,signups

# Analyze a directory of JSON files
/sc:eda results/ --format json

# Quick summary only
/sc:eda data/logs.csv --depth quick

Behavioral Flow

1. Parse - Extract input path, output directory, focus columns, depth
2. Discover - Detect file format, schema, row count, column types
3. Profile - Generate summary statistics for all columns
4. Visualize - Create distribution plots, correlations, time series
5. Analyze - Identify patterns, outliers, missing data, correlations
6. Report - Generate interactive HTML report with findings
7. Summarize - Present key insights to user

Flags

Flag	Type	Default	Description
--output	string	eda_results/<name>_eda/	Output directory for plots and reports
--focus	string	-	Comma-separated columns to focus analysis on
--depth	string	standard	quick (stats only), standard (stats + plots), deep (full analysis)
--format	string	auto	Input format: csv, json, parquet, auto-detect
--time-col	string	auto	Column to use for time series (auto-detects datetime columns)
--group-by	string	-	Column to group analysis by (e.g., category, region)

Phase 1: Data Discovery

Detect and load the dataset:

Source	Detection	Loading
Single CSV	.csv extension	pandas.read_csv()
Single JSON	.json extension	pandas.read_json() or json_normalize()
Parquet	.parquet extension	pandas.read_parquet()
Directory	Multiple files	Glob + concatenate with source tracking
Nested JSON	Nested objects	json_normalize() with record paths

Report schema summary:

• Total rows and columns
• Column names, types, and non-null counts
• Memory usage
• Sample rows (first 5)

Phase 2: Statistical Profiling

Generate summary statistics:

Metric	Numeric Columns	Categorical Columns
Count, mean, std, min, max	Yes	-
Quartiles (25%, 50%, 75%)	Yes	-
Unique values, top values	-	Yes
Missing value counts	Yes	Yes
Skewness, kurtosis	Yes	-

Save to data/summary_stats.csv.

Phase 3: Visualizations

Generate plots based on depth level:

Quick Depth

• Missing value heatmap

Standard Depth (default)

• Distributions - Histograms/KDE for numeric columns, bar charts for categorical
• Correlation matrix - Heatmap of numeric column correlations
• Missing values - Heatmap showing missing data patterns

Deep Depth

All standard plots plus:

• Time series - Trends over time (if datetime column detected)
• Group analysis - Per-group comparisons (if --group-by specified)
• Outlier detection - Box plots and IQR-based outlier flagging
• Pairplot - Scatter matrix for top correlated columns (max 6)
• Category heatmap - Normalized metric comparison across categories

Output structure:

<output_dir>/
  plots/
    distributions.png
    correlation.png
    missing_values.png
    time_series.png      # deep only
    outliers.png         # deep only
    group_analysis.png   # if --group-by
    pairplot.png         # deep only
  data/
    summary_stats.csv
    correlation_matrix.csv
    outliers.csv         # deep only
  eda_report.html        # interactive report

Phase 4: Pattern Analysis

Identify and report:

Pattern	Method	Threshold
Strong correlations	Pearson r	|r| > 0.7
Missing data patterns	MCAR/MAR analysis	> 5% missing
Outliers	IQR method	> 1.5 * IQR
Skewed distributions	Skewness test	|skew| > 1.0
Categorical imbalance	Frequency ratios	Majority > 80%
Temporal trends	Rolling mean slope	Monotonic shift

Phase 5: Generate HTML Report

Create an interactive HTML report combining:

• Data overview and schema
• Summary statistics tables
• Embedded plot images (base64)
• Key findings and patterns
• Recommendations for further analysis

Phase 6: Present Summary

Report key findings:

• Total records and features analyzed
• Top correlations found
• Outlier counts and affected columns
• Missing data summary
• Notable patterns and anomalies
• Recommendations

Dependencies

The EDA script requires common data science packages:

• pandas - Data loading and manipulation
• matplotlib - Plot generation
• seaborn - Statistical visualizations
• numpy - Numerical computations

If not available, install via: pip install pandas matplotlib seaborn numpy

MCP Integration

PAL MCP (Optional)

Tool	When	Purpose
mcp__pal__thinkdeep	--depth deep	Hypothesis testing on patterns
mcp__pal__chat	Interpretation	Second opinion on findings

Rube MCP (Optional)

Tool	When	Purpose
mcp__rube__RUBE_SEARCH_TOOLS	Database source	Find DB query tools
mcp__rube__RUBE_REMOTE_WORKBENCH	Large datasets	Process in Python sandbox

Error Handling

Scenario	Action
File not found	Error with path suggestions
Unsupported format	Error listing supported formats
No numeric columns	Skip correlation/distribution, focus on categorical
Missing pandas/matplotlib	Prompt user to install
Dataset too large (>1M rows)	Sample for plots, full stats on complete data
All values missing in column	Skip column, note in report

Tool Coordination

• Read - Inspect data files and schemas
• Bash - Run Python scripts, install packages
• Write - Generate HTML reports and CSV outputs
• Glob - Find data files in directories
• PAL MCP - Pattern interpretation (deep mode)