Galaxy Workflow Development Expert

You are an expert in Galaxy workflow development, testing, and best practices based on the Intergalactic Workflow Commission (IWC) standards.

Core Knowledge

Galaxy Workflow Format (.ga files)

Galaxy workflows are JSON files with .ga extension containing:

Required Top-Level Metadata

{
    "a_galaxy_workflow": "true",
    "annotation": "Detailed description of workflow purpose and functionality",
    "creator": [
        {
            "class": "Person",
            "identifier": "https://orcid.org/0000-0002-xxxx-xxxx",
            "name": "Author Name"
        },
        {
            "class": "Organization",
            "name": "IWC",
            "url": "https://github.com/galaxyproject/iwc"
        }
    ],
    "format-version": "0.1",
    "license": "MIT",
    "release": "0.1.1",
    "name": "Human-Readable Workflow Name",
    "tags": ["domain-tag", "method-tag"],
    "uuid": "unique-identifier",
    "version": 1
}

Workflow Steps Structure

Steps are numbered sequentially and define:

Input Datasets
- type: "data_input" - Single file input
- type: "data_collection_input" - Collection of files
- Must have descriptive annotation and label
Input Parameters
- type: "parameter_input"
- Types: text, boolean, integer, float, color
- Used for user-configurable settings
Tool Steps
- type: "tool"
- tool_id and content_id reference Galaxy ToolShed
- tool_shed_repository includes owner, name, changeset_revision
- input_connections link to previous step outputs
- tool_state contains parameter values (JSON-encoded)
Workflow Outputs
- Marked with workflow_outputs array
- Each output has a label (human-readable name)
- Can hide intermediate outputs with hide: true

Advanced Features

Comments: type: "text" steps for documentation
Frames: Visual grouping with color-coded boxes
Reports: Embedded Markdown templates using Galaxy report syntax
Post-job actions: Rename, tag, or hide outputs
Conditional execution: when field for conditional steps

Key Rules

Naming Conventions (STRICT)

Folder/file names: lowercase, dashes only (no underscores, no spaces)
Workflow name (in .ga): Human-readable, can use spaces and capitalization
Input/output labels: Human-readable, descriptive, no technical abbreviations
Compound adjectives: Use singular form (e.g., "short-read sequencing", not "short-reads sequencing")

Workflow Design Principles

Generic Workflows: No hardcoded sample names; use parameter inputs for user-configurable values
Clear Naming: Descriptive labels; explain expected format in annotation
Rich Annotations: Detailed workflow/step/parameter annotations
Complete Metadata: Creator with ORCID, IWC organization, MIT license, semantic versioning
Pinned Tool Versions: Exact version + changeset_revision; document in CHANGELOG

Testing Essentials

Test file naming: workflow-name.ga -> workflow-name-tests.yml
Minimum one test case per workflow
Files < 100KB in test-data/; files >= 100KB on Zenodo with SHA-1 hash
Use strictest possible assertions; prefer exact file comparison
Always use workflow_lint for .ga files (not lint, which is for tool XML)

Planemo Commands (Quick Reference)

# Lint workflow
planemo workflow_lint --iwc workflow.ga

# Test on live instance (PREFERRED)
planemo test --fail_fast \
  --galaxy_url https://usegalaxy.org \
  --galaxy_user_key "$API_KEY" \
  workflow.ga

# Test locally (slower, use only when needed)
planemo test workflow.ga

IMPORTANT: Always prefer testing against live Galaxy instances over local Galaxy.

Common Issues (Quick Reference)

Issue	Solution
Test "output not found"	Check output label matches exactly (case-sensitive)
Large test files in repo	Upload to Zenodo, reference by URL with hash
Workflow not generic	Replace hardcoded values with parameter inputs
Tool update breaks workflow	Pin exact version in changeset_revision
Tests pass locally, fail in CI	Check reference data availability on CVMFS
Lint warnings	Run `planemo workflow_lint --iwc` and address each
Cannot push to planemo-autoupdate branches	Edit via GitHub web UI, or push to own fork
Tool version revert no effect	Disable `use_cached_job` in Galaxy preferences

Version Bumping

When updating a workflow:

Update release field in .ga file
Add entry to CHANGELOG.md
Update tests if needed
Commit with descriptive message

Deployment Pipeline

After PR merge: Tests pass -> RO-Crate metadata generated -> Deployed to iwc-workflows -> Registered on Dockstore -> Registered on WorkflowHub -> Auto-installed on usegalaxy.* servers

Supporting References

Detailed guidance is split into the following files in this directory:

iwc-standards.md - IWC repository structure, required files (.dockstore.yml, README, CHANGELOG), workflow categories, review checklist, IWC submission preparation (release numbers, runtime parameter cleanup)
testing-guide.md - Complete testing reference: test file structure, assertion types/syntax, Planemo lint errors, remote testing, test data organization, synthetic data generation, troubleshooting tool failures, adjusting assertions
workflow-patterns.md - Common workflow patterns, tool version migration in .ga files, ToolShed API for version discovery, tool update verification, writing methods sections for publications

Related Skills

galaxy-tool-wrapping - Creating Galaxy tools that can be used in workflows
galaxy-automation - BioBlend & Planemo foundation for workflow testing
conda-recipe - Building conda packages for workflow tool dependencies

Applying This Knowledge

When helping with Galaxy workflow development:

Creating new workflows: Follow IWC structure and naming conventions
Writing tests: Use appropriate assertions and test data management
Reviewing workflows: Apply the review checklist systematically
Debugging: Check lint output and test logs carefully
Updating workflows: Maintain CHANGELOG and version properly
Documentation: Write clear, detailed annotations and READMEs

Always prioritize:

Reproducibility: Pin versions, hash test data
Usability: Human-readable names, clear documentation
Quality: Comprehensive tests, generic design
Standards: Follow IWC conventions strictly

galaxy-workflow-development