data-pipeline-engineering
Data Pipeline Engineering Skill
🔴 AI FIRST Quality Principle
Apply the AI FIRST principle: never accept first-pass quality. Minimum 2 iterations. Read all output, improve every section. No shortcuts.
Purpose
Expert knowledge in designing robust ETL (Extract, Transform, Load) pipelines for automated data processing, focusing on reliability, monitoring, and maintainability.
Core Principles
- Idempotency - Pipeline runs produce same results
- Observability - Full visibility into pipeline health
- Error Recovery - Graceful handling of failures
- Version Tracking - Track all data changes
- Monitoring - Real-time pipeline health checks
Enforces
- ETL workflow patterns (Extract → Transform → Load)
- Automated scheduling (cron, GitHub Actions)
- Data versioning and archival
- Pipeline health monitoring
- Error recovery strategies
- Audit logging
When to Use
- Building automated data pipelines
- Scheduling data fetching workflows
- Implementing data versioning
- Monitoring pipeline health
- Designing error recovery
References
Version: 1.0 | Last Updated: 2026-02-06 | Category: Development & Operations
More from hack23/riksdagsmonitor
osint-methodologies
OSINT collection, source evaluation, data integration, verification techniques for Swedish political intelligence
40economic-policy-analysis
Fiscal policy, budget analysis, economic forecasting, monetary policy, trade policy for political journalists
31electoral-analysis
Election forecasting models, campaign analysis, coalition prediction, voter behavior analysis for Swedish elections
25vulnerability-management
Vulnerability scanning, assessment, prioritization, and remediation processes following NIST and CIS Controls
25nist-csf-mapping
NIST Cybersecurity Framework 2.0 mapping for static HTML/CSS websites
24testing-strategy
Comprehensive testing strategy covering unit, integration, E2E, security, accessibility, and performance testing
23