DAG Authoring Skill

This skill guides you through creating and validating Airflow DAGs using best practices and af CLI commands.

For testing and debugging DAGs, see the testing-dags skill which covers the full test -> debug -> fix -> retest workflow.

Running the CLI

Run all af commands using uvx (no installation required):

uvx --from astro-airflow-mcp af <command>

Throughout this document, af is shorthand for uvx --from astro-airflow-mcp af.

Workflow Overview

+-----------------------------------------+
| 1. DISCOVER                             |
|    Understand codebase & environment    |
+-----------------------------------------+
                 |
+-----------------------------------------+
| 2. PLAN                                 |
|    Propose structure, get approval      |
+-----------------------------------------+
                 |
+-----------------------------------------+
| 3. IMPLEMENT                            |
|    Write DAG following patterns         |
+-----------------------------------------+
                 |
+-----------------------------------------+
| 4. VALIDATE                             |
|    Check import errors, warnings        |
+-----------------------------------------+
                 |
+-----------------------------------------+
| 5. TEST (with user consent)             |
|    Trigger, monitor, check logs         |
+-----------------------------------------+
                 |
+-----------------------------------------+
| 6. ITERATE                              |
|    Fix issues, re-validate              |
+-----------------------------------------+

Phase 1: Discover

Before writing code, understand the context.

Explore the Codebase

Use file tools to find existing patterns:

Glob for **/dags/**/*.py to find existing DAGs
Read similar DAGs to understand conventions
Check requirements.txt for available packages

Query the Airflow Environment

Use af CLI commands to understand what's available:

Command	Purpose
`af config connections`	What external systems are configured
`af config variables`	What configuration values exist
`af config providers`	What operator packages are installed
`af config version`	Version constraints and features
`af dags list`	Existing DAGs and naming conventions
`af config pools`	Resource pools for concurrency

Example discovery questions:

"Is there a Snowflake connection?" -> af config connections
"What Airflow version?" -> af config version
"Are S3 operators available?" -> af config providers

Phase 2: Plan

Based on discovery, propose:

DAG structure - Tasks, dependencies, schedule
Operators to use - Based on available providers
Connections needed - Existing or to be created
Variables needed - Existing or to be created
Packages needed - Additions to requirements.txt

Get user approval before implementing.

Phase 3: Implement

Write the DAG following best practices (see below). Key steps:

Create DAG file in appropriate location
Update requirements.txt if needed
Save the file

Phase 4: Validate

Use af CLI as a feedback loop to validate your DAG.

Step 1: Check Import Errors

After saving, check for parse errors (Airflow will have already parsed the file):

af dags errors

If your file appears -> fix and retry
If no errors -> continue

Common causes: missing imports, syntax errors, missing packages.

Step 2: Verify DAG Exists

af dags get <dag_id>

Check: DAG exists, schedule correct, tags set, paused status.

Step 3: Check Warnings

af dags warnings

Look for deprecation warnings or configuration issues.

Step 4: Explore DAG Structure

af dags explore <dag_id>

Returns in one call: metadata, tasks, dependencies, source code.

Phase 5: Test

See the testing-dags skill for comprehensive testing guidance.

Once validation passes, test the DAG using the workflow in the testing-dags skill:

Get user consent -- Always ask before triggering
Trigger and wait -- af runs trigger-wait <dag_id> --timeout 300
Analyze results -- Check success/failure status
Debug if needed -- af runs diagnose <dag_id> <run_id> and af tasks logs <dag_id> <run_id> <task_id>

Quick Test (Minimal)

# Ask user first, then:
af runs trigger-wait <dag_id> --timeout 300

For the full test -> debug -> fix -> retest loop, see testing-dags.

Phase 6: Iterate

If issues found:

Fix the code
Check for import errors: af dags errors
Re-validate (Phase 4)
Re-test using the testing-dags skill workflow (Phase 5)

CLI Quick Reference

Phase	Command	Purpose
Discover	`af config connections`	Available connections
Discover	`af config variables`	Configuration values
Discover	`af config providers`	Installed operators
Discover	`af config version`	Version info
Validate	`af dags errors`	Parse errors (check first!)
Validate	`af dags get <dag_id>`	Verify DAG config
Validate	`af dags warnings`	Configuration warnings
Validate	`af dags explore <dag_id>`	Full DAG inspection

Testing commands -- See the testing-dags skill for af runs trigger-wait, af runs diagnose, af tasks logs, etc.

Best Practices & Anti-Patterns

For code patterns and anti-patterns, see reference/best-practices.md.

Read this reference when writing new DAGs or reviewing existing ones. It covers what patterns are correct (including Airflow 3-specific behavior) and what to avoid.

Related Skills

testing-dags: For testing DAGs, debugging failures, and the test -> fix -> retest loop
debugging-dags: For troubleshooting failed DAGs
migrating-airflow-2-to-3: For migrating DAGs to Airflow 3

authoring-dags