Data Engineering Best Practices

Use this skill for production architecture and standards decisions: storage layout, lifecycle, incremental semantics, schema evolution, quality checks, and cost/performance tradeoffs.

When to use this skill

Use for:

Designing Bronze/Silver/Gold or equivalent data layers
Choosing append vs overwrite vs merge behavior
Partitioning and file-size strategy
Defining schema evolution policy
Setting testing/observability guardrails
Establishing retention + cost controls

Use domain skills for implementation details:

@data-engineering-core
@data-engineering-storage-lakehouse
@data-engineering-storage-formats
@data-engineering-storage-remote-access
@data-engineering-storage-authentication

Decision checklist (apply in order)

Data contract
- Required columns/types?
- Nullability and key uniqueness?
Layer semantics
- Bronze immutable?
- Silver deduplicated/validated?
- Gold business-ready aggregates?
Write mode
- Append, partition overwrite, or merge?
Layout
- Partition keys + target file size set?
Incremental logic
- Watermark/checkpoint strategy defined?
Evolution policy
- Additive-only by default?
Operational controls
- Tests + observability + retention + backfill process?

Core standards

1) Layering (Medallion)

Bronze: raw immutable ingestion; append-only
Silver: cleaned, validated, conformed schema
Gold: consumption-specific marts/features/aggregates

Do not skip Silver validation for convenience; silent quality drift is costly.

2) Write semantics

Operation	Use when	Notes
Append	strictly new immutable events	simplest, cheapest
Partition overwrite	deterministic reprocessing for date/key slice	safe for backfills
Merge/Upsert	corrections/late updates/deletes	needs key + conflict semantics

3) Partitioning

Good partition keys:

Frequently filtered dimensions (often date + low/moderate-cardinality dimension)

Avoid:

High-cardinality keys (e.g., user_id)
Over-partitioning creating tiny files

4) File sizing

Target file size: ~256MB–1GB (workload-dependent).

Too small → metadata/listing overhead + slow scans
Too large → poor parallelism and skewed processing

5) Schema evolution

Default policy:

✅ additive changes first (new nullable columns)
⚠️ type widening only when compatibility is clear
❌ destructive rename/drop in-place for shared production tables

6) Incremental processing

Persist watermark/checkpoint externally
Make re-runs idempotent
Include late-arriving data strategy (lag window/backfill)

7) Quality and reliability

Minimum controls:

Required columns + types
Primary key uniqueness (or dedupe policy)
Null thresholds on critical fields
Freshness/SLA checks
Run-level metrics (rows in/out, failures, latency)

Anti-patterns (reject in review)

Full table overwrite for small incremental changes
No checkpoint/watermark for recurring pipeline
Unbounded tiny-file generation
Dynamic SQL built from user values without parameter binding
Production credentials in code/config committed to repo
No backfill plan / no rollback strategy

Minimal production blueprint

Ingest raw to Bronze (append-only)
Validate + standardize to Silver
Build Gold outputs
Emit metrics + quality report
Persist checkpoint/watermark
Apply lifecycle rules + periodic compaction/maintenance

Progressive disclosure (read next as needed)

best-practices-detailed.md — comprehensive deep-dive examples
@data-engineering-core/patterns/incremental.md — incremental loading patterns
@data-engineering-storage-lakehouse — Delta/Iceberg/Hudi-specific behavior
@data-engineering-quality — validation framework implementation
@data-engineering-observability — metrics/tracing/alerting

data-engineering-best-practices