Gradle Build System in Apache Beam

Overview

Apache Beam is a mono-repo using Gradle as its build system. The entire project (Java, Python, Go, website) is managed as a single Gradle project.

Key Files

build.gradle.kts - Root build configuration
settings.gradle.kts - Project structure and module definitions
gradle.properties - Global properties and versions
buildSrc/ - Custom Gradle plugins including BeamModulePlugin

BeamModulePlugin

Located at buildSrc/src/main/groovy/org/apache/beam/gradle/BeamModulePlugin.groovy

Purpose

Manages Java dependencies centrally
Configures project types (Java, Python, Go, Proto, Docker, etc.)
Defines common custom tasks

Java Project Configuration

apply plugin: 'org.apache.beam.module'
applyJavaNature(
    automaticModuleName: 'org.apache.beam.sdk.io.kafka'
)

Common Commands

Build

# Build entire project
./gradlew build

# Build specific project
./gradlew :sdks:java:core:build
./gradlew -p sdks/java/core build

# Compile only (no tests)
./gradlew :sdks:java:core:compileJava

Test

# Run tests
./gradlew :sdks:java:core:test

# Run specific test
./gradlew :sdks:java:core:test --tests *MyTest

# Skip tests
./gradlew build -x test

Clean

# Clean specific project
./gradlew :sdks:java:core:clean

# Clean everything
./gradlew clean

Formatting

# Java formatting (Spotless)
./gradlew spotlessApply

# Check formatting
./gradlew spotlessCheck

# Format CHANGES.md
./gradlew formatChanges

Publishing

# Publish to Maven Local
./gradlew -Ppublishing :sdks:java:core:publishToMavenLocal

# Publish all Java artifacts
./gradlew -Ppublishing publishToMavenLocal

Pre-commit Tasks

Java

./gradlew javaPreCommit

Python

./gradlew pythonPreCommit

Combined

./gradlew :checkSetup  # Validates Go, Java, Python environments

Useful Flags

Flag	Description
`-p <path>`	Run task in specific project directory
`-x <task>`	Exclude task
`--tests <pattern>`	Filter tests
`-Ppublishing`	Enable publishing tasks
`-PdisableSpotlessCheck=true`	Disable formatting check
`-PdisableCheckStyle=true`	Disable checkstyle
`-PskipCheckerFramework`	Skip Checker Framework
`--continue`	Continue after failures
`--info`	Verbose output
`--debug`	Debug output
`--scan`	Generate build scan
`--parallel`	Parallel execution

GCP-related Properties

-PgcpProject=my-project
-PgcpRegion=us-central1
-PgcpTempRoot=gs://bucket/temp
-PgcsTempRoot=gs://bucket/temp

Docker Tasks

# Build Java SDK container
./gradlew :sdks:java:container:java11:docker

# Build Python SDK container
./gradlew :sdks:python:container:py312:docker

# Build and push a container into a custom repository
./gradlew :sdks:java:container:java11:docker \
  -Pdocker-repository-root=gcr.io/project \
  -Pdocker-tag=custom \
  -Ppush-containers

If a :docker task produces logs that contain the following:

WARNING: No output specified with docker-container driver.
Build result will only remain in the build cache.

then you must use -PuseDockerBuildx when running :docker tasks in this environment. For example:

# Build and push a go container into a custom repository
./gradlew :sdks:go:container:docker \
  -Pdocker-repository-root=gcr.io/project \
  -Pdocker-tag=custom \
  -Ppush-containers \
  -PuseDockerBuildx

Dependency Management

View Dependencies

./gradlew :sdks:java:core:dependencies
./gradlew :sdks:java:core:dependencies --configuration runtimeClasspath

Force Dependency Version

In build.gradle:

configurations.all {
    resolutionStrategy.force 'com.google.guava:guava:32.0.0-jre'
}

Troubleshooting

Clean Gradle Cache

rm -rf ~/.gradle/caches
rm -rf .gradle
rm -rf build

Common Errors

NoClassDefFoundError

Run ./gradlew clean
Delete gradle cache

Proto-related Errors

Regenerate protos: ./gradlew generateProtos

Dependency Conflicts

Check dependencies: ./gradlew dependencies
Use --scan for detailed analysis

Useful Tasks

# List all tasks
./gradlew tasks

# List tasks for a project
./gradlew :sdks:java:core:tasks

# Show project structure
./gradlew projects

gradle-build