Goldsky Mirror Pipelines

Mirror is Goldsky's original streaming pipeline product. It reads onchain data from a source (a subgraph entity or a direct-indexing dataset), optionally applies transforms, and writes the result to a sink (your database or message queue).

Mirror vs Turbo — which should you use?

	Mirror	Turbo
Subgraph sources	Yes	No
Speed & reliability	Good	Faster, more reliable
Sink variety	11 sink types	Growing — new sinks added regularly
Config complexity	Moderate	Simpler YAML
Dataset coverage	130+ chains	130+ chains, richer catalog

Use Turbo unless you need a subgraph source. Turbo is faster, more reliable, and actively gaining feature parity with Mirror — especially sink support. If you don't have a subgraph requirement, say "help me build a Turbo pipeline" and the /turbo-builder skill will guide you through a faster setup.

---

How Mirror Pipelines Work

Source (subgraph entity or direct-indexing dataset)
  ↓
Transforms (optional SQL or external handlers)
  ↓
Sink (PostgreSQL, ClickHouse, Kafka, S3, etc.)

A pipeline is defined in a YAML file (apiVersion: 3) and deployed with goldsky pipeline apply.

---

Pipeline YAML Structure

Top-level fields:

Field	Type	Required	Description
name	string	yes	Lowercase letters, numbers, hyphens only. Under 50 characters.
apiVersion	number	yes	Always 3
resource_size	string	no	s (default), m, l, xl, xxl
description	string	no	Pipeline description
sources	object	yes	At least one source
transforms	object	no	Use {} if none needed
sinks	object	yes	At least one sink

---

Sources

Source type	YAML type value	Description
Subgraph entity	subgraph_entity	Mirror data from Goldsky-hosted subgraphs
Dataset (direct indexing)	dataset	Raw onchain datasets (blocks, logs, transactions, traces, transfers)

Subgraph entity source

sources:
  subgraph_account:
    type: subgraph_entity
    name: account                # Entity name in your subgraph
    start_at: latest             # "earliest" or "latest" (default: latest)
    filter: ""                   # Optional SQL WHERE clause for fast scan
    subgraphs:
      - name: my-subgraph        # Deployed subgraph name
        version: 1.0.0
      - name: my-subgraph-arb    # Cross-chain: add more subgraphs
        version: 1.0.0

Fields: type (required: subgraph_entity), name (required: entity name), subgraphs (required: list of {name, version}), start_at (optional), filter (optional), description (optional).

Dataset source

sources:
  base_logs:
    type: dataset
    dataset_name: base.logs      # Use `goldsky dataset list` to discover names
    version: 1.0.0               # Use `goldsky dataset get <name>` for versions
    start_at: latest             # "earliest" or "latest" (default: latest)
    filter: "address = '0x...'"  # Optional — enables Fast Scan for backfills

Fields: type (required: dataset), dataset_name (required), version (required), start_at (optional), filter (optional), description (optional).

Fast Scan: When filter is defined on a dataset source with start_at: earliest, the filter is pre-applied at the source level, making historical backfill much faster. Use attributes that exist in the dataset schema (goldsky dataset get <dataset_name> to check).

See docs.goldsky.com/mirror/sources/supported-sources.

---

Sinks

Sink	YAML type value	Notes
PostgreSQL	postgres	Most common — OLTP, auto-creates tables, upsert via INSERT ON CONFLICT. Hosted option via NeonDB.
ClickHouse	clickhouse	OLAP — uses ReplacingMergeTree by default, append_only_mode for best performance
MySQL	mysql	OLTP workloads
Elasticsearch	elasticsearch	Real-time search and analytics
Kafka	kafka	High-throughput streaming to a topic, configurable topic_partitions
Object Storage	file	S3, GCS, or R2 — Parquet format, append-only, supports partition_columns
AWS SQS	sqs	Message queuing
Webhook	webhook	HTTP POST to an external endpoint

All sinks writing to user-managed destinations require a Goldsky Secret (secret_name). Create one with goldsky secret create.

Sinks support schema_override for casting column types at the sink level (e.g., string to jsonb).

Sink examples

# PostgreSQL
sinks:
  my_pg:
    type: postgres
    table: transfers
    schema: public
    secret_name: MY_PG_SECRET
    from: my_transform

# ClickHouse
sinks:
  my_ch:
    type: clickhouse
    table: transfers
    database: my_db
    secret_name: MY_CH_SECRET
    from: my_source

# Kafka
sinks:
  my_kafka:
    type: kafka
    topic: accounts
    topic_partitions: 2
    secret_name: MY_KAFKA_SECRET
    from: my_source

# Object Storage (S3/GCS/R2)
sinks:
  my_s3:
    type: file
    path: s3://bucket/path/
    format: parquet
    secret_name: MY_S3_SECRET
    from: my_source

# SQS
sinks:
  my_sqs:
    type: sqs
    url: https://sqs.us-east-1.amazonaws.com/123456/my-queue
    secret_name: MY_SQS_SECRET
    from: my_source

See docs.goldsky.com/mirror/sinks/supported-sinks.

---

Transforms

Type	YAML type value	Description
SQL	(none — default)	Filter, join, or reshape records with SQL
External handler	handler	POST records to an HTTP endpoint for custom logic

SQL transform

transforms:
  filtered_logs:
    sql: SELECT id, block_number, address FROM base_logs WHERE block_number > 1000
    primary_key: id

SQL transforms reference source or transform names as table names. Supports chaining (one transform reads from another).

Built-in decode functions:

• _gs_log_decode(abi, topics, data) — decode raw log events
• _gs_tx_decode(abi, input, output) — decode raw trace/transaction data
• _gs_fetch_abi(url, type) — fetch ABI from URL (etherscan-compatible or raw JSON); fetched once at pipeline start

External handler transform

transforms:
  my_handler:
    type: handler
    primary_key: id
    url: http://example.com/transform
    from: my_source
    batch_size: 100              # Records per batch (default: 100)
    batch_flush_interval: 1s     # Flush interval (default: 1s)
    payload_columns: [col1,col2] # Optional: send subset of columns
    headers:                     # Optional custom headers
      X-Api-Key: my-key

• At-least-once delivery with exponential backoff on failure
• Max response time: 5 minutes; max connection time: 1 minute
• Supports schema_override for return type casting

See docs.goldsky.com/mirror/transforms/sql-transforms.

---

Full YAML Examples

Subgraph entity to PostgreSQL

name: my-subgraph-sync
apiVersion: 3
resource_size: s
sources:
  subgraph_transfer:
    type: subgraph_entity
    name: Transfer
    subgraphs:
      - name: uniswap-v3
        version: 1.0.0
transforms: {}
sinks:
  my_postgres:
    type: postgres
    table: transfers
    schema: public
    secret_name: MY_PG_SECRET
    from: subgraph_transfer

Dataset (direct indexing) to PostgreSQL with SQL transform

name: base-logs-filtered
apiVersion: 3
resource_size: s
sources:
  base_logs:
    type: dataset
    dataset_name: base.logs
    version: 1.0.0
    start_at: earliest
    filter: "address = '0x833589fcd6edb6e08f4c7c32d4f71b54bda02913'"
transforms:
  select_fields:
    sql: SELECT id, block_number, transaction_hash, data FROM base_logs
    primary_key: id
sinks:
  pg_logs:
    type: postgres
    table: base_logs
    schema: public
    secret_name: MY_PG_SECRET
    from: select_fields

No subgraph source? You should almost certainly use Turbo instead — it's faster, more reliable, and has a richer dataset catalog with simpler syntax. Use /turbo-builder to get started.

---

CLI Reference — All Pipeline Commands

Global options available on every command: --token <string> (CLI auth token), --color (colorize output, default true), -h, --help.

goldsky pipeline apply <config-path>

Create or update a pipeline from a YAML config file. Idempotent.

Flag	Type	Description
--status	ACTIVE | INACTIVE | PAUSED	Desired pipeline status
--from-snapshot	string	Snapshot to start from: last, new, none, or a snapshot ID. last = latest available. new = create a fresh snapshot first. none = start from scratch. Default: new
--force	boolean	Skip confirmation prompts (useful for CI)
--skip-transform-validation	boolean	Skip transform validation on update
--save-progress	boolean	(deprecated, use --from-snapshot) Attempt snapshot before applying
--use-latest-snapshot	boolean	(deprecated, use --from-snapshot) Start from latest snapshot
--skip-validation	boolean	(deprecated) Same as --skip-transform-validation

goldsky pipeline apply my-pipeline.yaml --status ACTIVE
goldsky pipeline apply my-pipeline.yaml --status ACTIVE --from-snapshot last
goldsky pipeline apply my-pipeline.yaml --force   # CI/CD usage

goldsky pipeline start <nameOrConfigPath>

Start a pipeline (equivalent to apply with --status ACTIVE).

Flag	Type	Description
--from-snapshot	string	last, new, none, or snapshot ID
--use-latest-snapshot	boolean	(deprecated, use --from-snapshot)

goldsky pipeline stop <nameOrConfigPath>

Stop a pipeline without taking a snapshot. Sets status to INACTIVE, runtime to TERMINATED.

No additional flags beyond global options.

goldsky pipeline pause <nameOrConfigPath>

Pause a pipeline with a snapshot so it can resume from where it left off. Sets status to PAUSED, runtime to TERMINATED.

No additional flags beyond global options.

goldsky pipeline restart <nameOrConfigPath>

Restart a pipeline without configuration changes. Useful when the sink database was restarted, connection is stuck, etc.

Flag	Type	Description
--from-snapshot	string	Required. last, new, none, or snapshot ID
--disable-monitoring	boolean	Skip monitoring after restart (default: false)

goldsky pipeline restart my-pipeline --from-snapshot last
goldsky pipeline restart my-pipeline --from-snapshot none  # restart from scratch

goldsky pipeline get <nameOrConfigPath>

Get pipeline configuration and status.

Flag	Type	Description
--outputFormat, --output	json | table | yaml	Output format (default: yaml)
--definition	boolean	Print only the pipeline definition (sources, transforms, sinks)
-v, --version	string	Pipeline version (default: latest)

goldsky pipeline list

List all pipelines in the project.

Flag	Type	Description
--output, --outputFormat	json | table | yaml	Output format (default: table)
--outputVerbosity	summary | usablewithapplycmd | all	Detail level (default: summary)
--include-runtime-details	boolean	Include runtime status and errors (default: false)

goldsky pipeline list --output json
goldsky pipeline list --include-runtime-details

goldsky pipeline info <nameOrConfigPath>

Display pipeline information (status, config, runtime details).

Flag	Type	Description
-v, --version	string	Pipeline version (default: latest)

goldsky pipeline monitor <nameOrConfigPath>

Monitor pipeline runtime — status, metrics (records received/written), errors. Refreshes every 10 seconds.

Flag	Type	Description
--update-request	boolean	Monitor an in-flight update request
--max-refreshes, --maxRefreshes	number	Max number of data refreshes
-v, --version	string	Pipeline version (default: latest)

goldsky pipeline delete <nameOrConfigPath>

Delete a pipeline permanently.

Flag	Type	Description
-f, --force	boolean	Force deletion without confirmation prompt (default: false)

goldsky pipeline resize <nameOrConfigPath> <resourceSize>

Change the compute resources for a pipeline.

Positional	Description
resourceSize	One of: s, m, l, xl, xxl (default: s)

goldsky pipeline resize my-pipeline l

goldsky pipeline validate [config-path]

Validate a pipeline YAML config without deploying.

Flag	Type	Description
--definition	string	(deprecated) Inline JSON definition
--definition-path	string	(deprecated) Path to JSON/YAML definition

goldsky pipeline validate my-pipeline.yaml

goldsky pipeline export [name]

Export pipeline configuration.

Flag	Type	Description
--all	boolean	Export configs for all pipelines

goldsky pipeline cancel-update <nameOrConfigPath>

Cancel an in-flight update or snapshot request. Useful when a long-running snapshot blocks a needed update.

No additional flags beyond global options.

goldsky pipeline create <name> (interactive/guided)

Guided CLI experience for creating a pipeline interactively.

Flag	Type	Description
--resource-size, --resourceSize	s | m | l | xl | xxl	Resource size (default: s)
--use-dedicated-ip	boolean	Use dedicated egress IPs (default: false)
--skip-transform-validation	boolean	Skip transform validation
--status	ACTIVE | INACTIVE	(deprecated, use pipeline start/stop/pause)
--description	string	(deprecated, use pipeline apply)
--definition	string	(deprecated, use pipeline apply)
--definition-path	string	(deprecated, use pipeline apply)
--output, --outputFormat	json | table | yaml	Output format (default: yaml)

goldsky pipeline get-definition <name> (deprecated)

Get a shareable pipeline definition. Use goldsky pipeline get <name> --definition instead.

Snapshot Commands

# List snapshots for a pipeline
goldsky pipeline snapshots list <nameOrConfigPath> [-v <version>]

# Create a snapshot manually
goldsky pipeline snapshots create <nameOrConfigPath>

snapshots list supports -v, --version to filter by pipeline version (default: all versions).

---

Lifecycle Quick Reference

Action	Command
Deploy / start	goldsky pipeline apply <file.yaml> --status ACTIVE
Start (existing)	goldsky pipeline start <name>
Pause (with snapshot)	goldsky pipeline pause <name>
Stop (no snapshot)	goldsky pipeline stop <name>
Restart (no config change)	goldsky pipeline restart <name> --from-snapshot last
Update config	goldsky pipeline apply <file.yaml> (edit YAML first)
Resize	goldsky pipeline resize <name> <size>
Validate YAML	goldsky pipeline validate <file.yaml>
Monitor	goldsky pipeline monitor <name>
Get config	goldsky pipeline get <name> --definition
Export config	goldsky pipeline export <name>
Delete	goldsky pipeline delete <name> -f
Cancel in-flight op	goldsky pipeline cancel-update <name>
List snapshots	goldsky pipeline snapshots list <name>
Create snapshot	goldsky pipeline snapshots create <name>
List all pipelines	goldsky pipeline list

Pause vs. Stop:

• pause — takes a snapshot and suspends the pipeline (status: PAUSED + TERMINATED). Can resume from where it left off.
• stop — stops without taking a snapshot (status: INACTIVE + TERMINATED). Resuming may reprocess data.

Desired statuses: ACTIVE, INACTIVE, PAUSED Runtime statuses: STARTING, RUNNING, FAILING, TERMINATED

---

Snapshots

Snapshots capture a point-in-time state of a RUNNING pipeline for resumption. They contain progress on reading sources and SQL transform state — not sink state.

• Automatic snapshots are taken every 4 hours for healthy RUNNING pipelines.
• Before updates: a snapshot is created automatically before applying config changes to a RUNNING pipeline.
• On pause: a snapshot is created when pausing.
• Manual: goldsky pipeline snapshots create <name>.
• Resume: only the latest snapshot can be used. For older snapshots, contact support.

The --from-snapshot flag (on apply, start, restart) controls snapshot behavior:

• new — create a fresh snapshot, then start from it (default)
• last — use the latest existing snapshot (no new snapshot)
• none — start from scratch, no snapshot
• <snapshot-id> — use a specific snapshot

---

Resource Sizing

Set via resource_size in YAML or goldsky pipeline resize <name> <size>.

Size	Description
s	Default. Handles most use cases, backfill of small chains, up to 300K records/sec, up to ~8 subgraph sources
m, l, xl, xxl	Larger compute — for backfilling large chains or large JOINs

Start small and scale up if needed. Resource size affects pricing.

---

Networking

• Mirror pipelines write data from AWS us-west-2. Ensure your sink allows inbound connections from this region.
• IP addresses are dynamic by default.
• Dedicated egress IPs available on request — use --use-dedicated-ip on pipeline create, or contact support@goldsky.com.
• VPC peering available on request.
• For external handler transforms, deploy close to us-west-2 for best performance (aim for p95 < 100ms).

---

Dataset Discovery

# List available datasets
goldsky dataset list

# Get schema for a specific dataset
goldsky dataset get <dataset_name>

---

Common Questions

Can Mirror pipelines use subgraphs as a source? Yes — this is Mirror's primary advantage over Turbo. Set type: subgraph_entity in your source and reference your deployed subgraph.

Can Mirror handle multiple sources or cross-chain data? Yes — define multiple sources in the YAML and use SQL transforms to join or merge them. For subgraphs, you can list multiple subgraphs (different chains) in a single source's subgraphs array.

My pipeline needs more resources / is too slow? Run goldsky pipeline resize <name> l (or xl, xxl). Start small and scale up.

My pipeline is ACTIVE but TERMINATED — what happened? The desired status is ACTIVE but the runtime failed (e.g., bad secret, sink unavailable, resource issues). Check errors with goldsky pipeline monitor <name> --include-runtime-details or view the dashboard. Fix the issue and restart.

How do I update a pipeline without losing progress? Edit your YAML and run goldsky pipeline apply <file.yaml>. By default, a snapshot is taken before the update is applied. Use --from-snapshot last to skip creating a new snapshot and use the latest existing one.

A long snapshot is blocking my update — what do I do? Run goldsky pipeline cancel-update <name> to cancel the in-flight operation, then reapply with --from-snapshot last or --from-snapshot none.

---

Related

• /turbo-builder — Build a new Turbo pipeline (recommended for new projects not using subgraph sources)
• /subgraphs — Deploy and manage the subgraph you want to sync via Mirror
• /secrets — Create secrets for sink credentials
• /datasets — Browse available dataset names and chain prefixes
• Goldsky docs: docs.goldsky.com/mirror/introduction
• Pipeline config reference: docs.goldsky.com/mirror/reference/config-file/pipeline