Spice Data Accelerators

Data accelerators cache data locally for faster queries and reduced load on source systems.

Basic Configuration

datasets:
  - from: postgres:my_table
    name: my_table
    acceleration:
      enabled: true
      engine: duckdb           # arrow, duckdb, sqlite, postgres, cayenne
      mode: memory             # memory or file
      refresh_check_interval: 1h

Supported Engines

Engine	Mode	Best For
arrow	memory	Small datasets, fastest queries
duckdb	memory/file	Complex SQL, medium datasets
sqlite	memory/file	Simple queries, low overhead
cayenne	file	Large datasets (100GB+), analytics
postgres	N/A	External PostgreSQL integration

Refresh Modes

Mode	Description
full	Replace entire dataset on each refresh
append	Add new records based on time_column
changes	CDC-based incremental updates

Common Configurations

Memory Cache with Interval Refresh

acceleration:
  enabled: true
  engine: arrow
  refresh_check_interval: 5m

File-Based with Time Window

acceleration:
  enabled: true
  engine: duckdb
  mode: file
  refresh_mode: append
  refresh_check_interval: 1h
  refresh_data_window: 7d

With Retention Policy

datasets:
  - from: postgres:events
    name: events
    time_column: created_at
    acceleration:
      enabled: true
      engine: duckdb
      retention_check_enabled: true
      retention_period: 30d
      retention_check_interval: 1h

With Indexes

acceleration:
  enabled: true
  engine: sqlite
  indexes:
    user_id: enabled
    '(created_at, status)': unique
  primary_key: id

Engine-Specific Parameters

DuckDB

acceleration:
  engine: duckdb
  mode: file
  params:
    duckdb_file: ./data/cache.db

SQLite

acceleration:
  engine: sqlite
  mode: file
  params:
    sqlite_file: ./data/cache.sqlite

Documentation

• Data Accelerators Overview
• Datasets Reference
• Data Refresh
• Indexes