Data Orchestrator - AI Trading Data Strategy Layer

Central nervous system for all data operations across the meme-times ecosystem. Implements a robust, AI-ready data strategy that precedes and enables all trading decisions.

Core Principle

Data strategy comes BEFORE AI. Clean governance, validated sources, and secure infrastructure enable useful trading insights.

Activation Triggers

Data Strategy Pillars

1. Diverse Data Sources

<data_sources> Price & Market Data:

Source	Data Type	Latency	Quality	Cost
Dexscreener API	DEX prices, pools	Real-time	High	Free tier
Birdeye API	Solana tokens, analytics	Real-time	High	Paid
Jupiter Price API	Solana swap prices	Real-time	High	Free
CoinGecko	Cross-chain prices	1-5 min	High	Free tier
CoinAPI	Historical + streaming	Configurable	Premium	Paid

On-Chain Data:

Source	Data Type	Latency	Quality	Cost
Helius RPC	Solana transactions	Real-time	High	Freemium
Solscan API	Token info, holders	Near real-time	High	Free tier
Dune Analytics	SQL queries	Minutes-hours	High	Freemium
Flipside Crypto	Pre-built datasets	Hours	High	Free

Sentiment & Social:

Source	Data Type	Latency	Quality	Cost
Twitter/X API	Social mentions	Real-time	Medium	Paid
LunarCrush	Social metrics	Near real-time	High	Paid
Telegram scraping	Community sentiment	Real-time	Low	DIY
Reddit API	Discussion sentiment	Minutes	Medium	Free

DeFi Protocol Data:

Source	Data Type	Latency	Quality	Cost
DefiLlama API	TVL, revenue, yields	15 min	High	Free
Token Terminal	Revenue, P/E ratios	Daily	High	Paid
DeFi Pulse	TVL rankings	Hourly	Medium	Free
</data_sources>

2. Data Quality & Governance

<data_governance> Quality Dimensions:

interface DataQualityMetrics {
  accuracy: number;      // 0-100: correctness vs ground truth
  completeness: number;  // 0-100: missing values ratio
  timeliness: number;    // 0-100: freshness score
  consistency: number;   // 0-100: cross-source agreement
  validity: number;      // 0-100: schema conformance
}

interface QualityThresholds {
  trading_signals: { min: 90, critical: 'timeliness' };
  historical_analysis: { min: 85, critical: 'completeness' };
  sentiment_analysis: { min: 70, critical: 'consistency' };
  backtesting: { min: 95, critical: 'accuracy' };
}

Validation Pipeline:

Raw Data → Schema Validation → Anomaly Detection → Cross-Source Check → Quality Score → Accept/Reject

Validation Rules:

Schema Validation: All data must match expected types/formats
Range Checks: Prices, volumes, percentages within valid bounds
Anomaly Detection: Flag outliers > 3 standard deviations
Cross-Source Verification: Compare with 2+ sources for critical data
Freshness Enforcement: Reject stale data beyond threshold

Automated Quality Monitoring:

# Run continuous quality checks
npx tsx .claude/skills/data-orchestrator/scripts/quality-monitor.ts \
  --sources "dexscreener,birdeye,jupiter" \
  --interval 60 \
  --alert-threshold 80

</data_governance>

3. Real-Time Data Pipeline Architecture

<pipeline_architecture>

┌─────────────────────────────────────────────────────────────────┐
│                     DATA INGESTION LAYER                        │
├─────────────────────────────────────────────────────────────────┤
│  WebSocket Feeds    │   REST Polling   │   RPC Subscriptions   │
│  (Dexscreener, WS) │   (Coingecko)    │   (Helius, Solana)    │
└──────────┬──────────┴────────┬─────────┴──────────┬────────────┘
           │                   │                    │
           ▼                   ▼                    ▼
┌─────────────────────────────────────────────────────────────────┐
│                    VALIDATION & ENRICHMENT                      │
├─────────────────────────────────────────────────────────────────┤
│  Schema Check  │  Anomaly Flag  │  Cross-Verify  │  Quality Score│
└──────────┬──────────────────────────────────────────────────────┘
           │
           ▼
┌─────────────────────────────────────────────────────────────────┐
│                      DATA STORAGE LAYER                         │
├─────────────────────────────────────────────────────────────────┤
│  Hot Store (Redis)   │  Warm Store (SQLite)  │  Cold (Parquet)  │
│  Real-time prices    │  Recent history (7d)  │  Historical data │
│  TTL: 5 minutes      │  Indexed, queryable   │  Compressed, ML  │
└──────────┬──────────────────────────────────────────────────────┘
           │
           ▼
┌─────────────────────────────────────────────────────────────────┐
│                    CONSUMPTION LAYER                            │
├─────────────────────────────────────────────────────────────────┤
│  Trading Signals  │  ML Training  │  Backtesting  │  Dashboards │
│  (meme-trader)    │  (llama-analyst) │ (meme-executor) │ (Reports)│
└─────────────────────────────────────────────────────────────────┘

Data Flow Configuration:

interface PipelineConfig {
  sources: DataSource[];
  validationRules: ValidationRule[];
  enrichmentSteps: EnrichmentFunction[];
  storageTargets: StorageTarget[];
  alertsEnabled: boolean;
  qualityThreshold: number;
}

const defaultPipeline: PipelineConfig = {
  sources: [
    { name: 'dexscreener', type: 'websocket', priority: 1 },
    { name: 'birdeye', type: 'rest', priority: 2 },
    { name: 'jupiter', type: 'rest', priority: 3 },
  ],
  validationRules: ['schema', 'range', 'anomaly', 'freshness'],
  enrichmentSteps: ['normalize', 'calculate_indicators', 'tag_quality'],
  storageTargets: ['redis:hot', 'sqlite:warm'],
  alertsEnabled: true,
  qualityThreshold: 85,
};

</pipeline_architecture>

4. ML/AI Integration Framework

<ml_integration> Data Preparation for ML:

interface MLReadyDataset {
  features: {
    price_data: TimeSeriesFeatures;
    volume_data: TimeSeriesFeatures;
    onchain_metrics: OnChainFeatures;
    sentiment_scores: SentimentFeatures;
    technical_indicators: TechnicalFeatures;
  };
  labels: {
    price_direction: 'up' | 'down' | 'sideways';
    price_magnitude: number;
    optimal_action: 'buy' | 'sell' | 'hold';
  };
  metadata: {
    timestamp: Date;
    token: string;
    quality_score: number;
    source_count: number;
  };
}

interface TimeSeriesFeatures {
  values: number[];
  timestamps: Date[];
  normalized: number[];  // Z-score normalized
  lagged: number[][];    // [lag_1, lag_5, lag_15, lag_60]
  rolling_stats: {
    mean_5: number[];
    std_5: number[];
    mean_15: number[];
    std_15: number[];
  };
}

Supported ML Techniques:

Anomaly Detection: Isolation forests for unusual price/volume patterns
NLP/Sentiment: BERT-based sentiment from social feeds
Time Series: LSTM/Transformer for price prediction
Classification: XGBoost for buy/sell signal classification
Reinforcement Learning: DQN for optimal trade execution

Continuous Learning Pipeline:

New Data → Feature Extraction → Model Inference → Signal Generation
                                      ↑
                                Performance Feedback
                                      ↓
                              Model Retraining (Weekly)

</ml_integration>

5. Backtesting Infrastructure

Backtest Execution:

# Run backtest with historical data
npx tsx .claude/skills/data-orchestrator/scripts/backtest-runner.ts \
  --strategy "momentum" \
  --token "BONK" \
  --start "2024-01-01" \
  --end "2024-12-01" \
  --initial-capital 1000 \
  --slippage 0.01

Backtest Report Output:

BACKTEST REPORT: Momentum Strategy on BONK
Period: 2024-01-01 to 2024-12-01

PERFORMANCE:
- Total Return: +234.5%
- Sharpe Ratio: 1.87
- Max Drawdown: -28.3%
- Win Rate: 62.4%
- Profit Factor: 2.15

TRADES:
- Total Trades: 156
- Avg Trade Duration: 4.2 hours
- Best Trade: +45.2%
- Worst Trade: -12.8%

DATA QUALITY:
- Coverage: 99.2%
- Missing Points: 847 / 105,120
- Quality Score: 94/100

CAVEATS:
- Historical results do not guarantee future performance
- Slippage model: 1% (actual may vary)
- Does not account for: MEV, extreme volatility periods

6. Risk & Portfolio Data Layer

<risk_data> Portfolio Metrics:

interface PortfolioData {
  positions: {
    token: string;
    entry_price: number;
    current_price: number;
    size: number;
    unrealized_pnl: number;
    allocation_pct: number;
    risk_score: number;
  }[];
  aggregate: {
    total_value: number;
    total_pnl: number;
    daily_var: number;  // Value at Risk (95%)
    beta_to_sol: number;
    concentration_score: number;  // Herfindahl index
  };
  limits: {
    max_position_size: number;
    max_daily_loss: number;
    max_correlation: number;
    stop_loss_pct: number;
  };
}

Risk Data Sources:

Position tracking from meme-executor
Price volatility from historical data
Correlation matrix from cross-asset analysis
Liquidity depth from DEX APIs </risk_data>

Implementation Scripts

Data Pipeline Manager

# Start the data pipeline
npx tsx .claude/skills/data-orchestrator/scripts/pipeline-manager.ts \
  --mode production \
  --sources all \
  --storage sqlite,redis

# Validate specific data source
npx tsx .claude/skills/data-orchestrator/scripts/validate-source.ts \
  --source dexscreener \
  --token "BONK" \
  --verbose

# Generate ML-ready dataset
npx tsx .claude/skills/data-orchestrator/scripts/ml-dataset-builder.ts \
  --token "BONK" \
  --features "price,volume,sentiment" \
  --lookback 30 \
  --output ./datasets/bonk_ml_ready.parquet

Integration with Other Skills

Data Orchestrator receives from:

All skills: Data quality feedback, missing data requests
meme-executor: Trade execution data for backtest validation

Quality Gates

<validation_rules>

All data must have quality score >= 80% for trading signals
Price data staleness: max 30 seconds for live trading
Historical data completeness: min 95% for backtesting
Cross-source agreement: min 2 sources for critical decisions
Schema validation: 100% compliance required
Anomaly flagging: auto-reject data points > 5 sigma </validation_rules>

Error Handling

<error_recovery>

Source unavailable: Failover to backup source within 5 seconds
Quality below threshold: Alert + fallback to last good data
Schema mismatch: Log error, use default values, alert
Rate limit hit: Exponential backoff, rotate API keys
Network timeout: Retry 3x with increasing delay </error_recovery>

Compliance & Security

Performance Targets

Metric	Target	Measurement
Data latency (real-time)	< 500ms	WebSocket to storage
Validation throughput	> 10K records/sec	Validation pipeline
Quality score accuracy	> 95%	vs manual audit
Backtest data coverage	> 99%	Historical completeness
ML feature freshness	< 5 min	Feature store update

<see_also>

references/data-sources.md - Complete API documentation
references/quality-standards.md - Validation rule definitions
references/ml-feature-catalog.md - Available ML features
scripts/pipeline-manager.ts - Main orchestration script
scripts/quality-monitor.ts - Continuous quality monitoring
scripts/backtest-runner.ts - Historical backtesting </see_also>