beam-dataflow-python
Apache Beam & Dataflow (Python) Best Practices
Implement Way's architectural patterns and modern (2025+) best practices when building Dataflow Python pipelines.
1. Unified Pipeline Architecture
- Mode-driven routing:
--mode streamingvs--mode batchflag conditionally injects I/O connectors (Pub/Sub vs. BigQuery/GCS) and windowing; transform logic is identical across modes - Layered files:
pipeline.py(PTransform wiring) →transforms.py(DoFn impls) →state_machine.py/ domain logic (pure Python, zero Beam imports) - Event-time first: always develop around event time so backfills produce consistent state
Reference: Way's Pipeline Patterns, Community Best Practices
2. Runner v2 + Streaming Engine
More from way-platform/skills
way-magefile
Build tool for Go projects. Use when the user wants to create, edit, or understand Way-specific Magefiles, build targets, or automate Go project tasks.
18way-go-style
Guide for writing idiomatic, effective, and standard Go code. Use this skill when writing, refactoring, or reviewing Go code to ensure adherence to established conventions and best practices.
18ileap
>-
17agents-md
This skill should be used when the user asks to "create AGENTS.md", "update AGENTS.md", "maintain agent docs", "set up CLAUDE.md", or needs to keep agent instructions concise. Guides discovery of local skills and enforces minimal documentation style.
11way-brand-identity
Write copy and use colors according to the Way brand.
11aep
AEP (API Enhancement Proposals) design standards. Use when designing, reviewing, or implementing APIs to ensure compliance with AEP conventions.
5