Data Storage Formats

Comprehensive guide to modern data serialization formats for analytics and machine learning: Parquet, Apache Arrow, Lance, Zarr, Avro, and ORC. Learn compression tradeoffs, partitioning strategies, and when to use each format.

Quick Comparison

Format	Type	Best For	Compression	Schema Evolution	Random Access
Parquet	Columnar	Analytics, data lakes	✅ (Snappy, Zstd, LZ4)	✅ (add/drop)	✅ (row groups)
Arrow/Feather	Columnar	In-memory, IPC, ML	✅ (LZ4, Zstd)	Limited	✅ (record batches)
Lance	Columnar	ML pipelines, vectors	✅ (Zstd, LZ4)	✅	✅ (multi-modal)
Zarr	Chunked arrays	ML, geospatial, N-dim	✅ (Blosc, gzip)	✅ (chunks)	✅ (chunk-level)
Avro	Row-based	Streaming, Kafka	✅ (deflate, snappy)	✅ (full schema)	❌ (sequential)
ORC	Columnar	Hive, Hadoop	✅ (ZLIB, Snappy)	Limited	✅ (stripe-level)

When to Use Which?

Choose Parquet when:

You need broad compatibility (Spark, DuckDB, Polars, pandas)
Analytics queries with filtering/aggregation
Data lake storage with partitioning
Mature ecosystem with best compression support

Choose Arrow/Feather when:

Zero-copy sharing between processes (IPC)
Fast serialization/deserialization for ML training
In-memory format persistence
Need Arrow ecosystem (Kernel, CUDA, etc.)

Choose Lance when:

Machine learning pipelines with embeddings/vectors
Need multi-modal data (text + images + audio + vectors)
Versioned datasets with Git-like branching
Cloud-native (S3/GCS) with no metadata catalog required

Choose Zarr when:

N-dimensional arrays (tensors, satellite imagery, medical scans)
Chunked, compressed storage for ML
Parallel reads/writes across chunks
Cloud-optimized (s3://, gs:// with fsspec)

Choose Avro when:

Streaming to Kafka/Kinesis
Schema evolution is critical (backward/forward)
Row-based access pattern
Need to serialize objects/records

Choose ORC when:

Working primarily with Hive/Hadoop
Hive ACID transactions
Legacy big data pipelines

Skill Dependencies

@data-engineering-core - Polars/DuckDB to read/write these formats
@data-engineering-storage-remote-access - Cloud storage backends
@data-engineering-storage-lakehouse - Table formats (Delta/Iceberg/Hudi) built on these

Detailed Guides

Parquet

See: parquet.md (detailed deep dive)

Parquet is the de facto standard for columnar analytics storage.

import polars as pl
import pyarrow.parquet as pq
import pyarrow as pa

# Write with Polars
df = pl.DataFrame({"id": [1, 2, 3], "value": [100.0, 200.0, 150.0]})
df.write_parquet("data.parquet", compression="zstd")

# Write with PyArrow (more control)
table = pa.Table.from_pandas(df)
pq.write_table(
    table,
    "data.parquet",
    compression="ZSTD",
    compression_level=3,
    row_group_size=100000,  # Target rows per row group
    use_dictionary=True     # Dictionary encoding for strings
)

# Read with column pruning
df = pl.read_parquet("data.parquet", columns=["id", "value"])

# Dataset scanning with predicate pushdown
lazy_df = pl.scan_parquet("s3://bucket/dataset/**/*.parquet")
result = lazy_df.filter(pl.col("value") > 100).collect()

Key concepts:

Row groups: Horizontal partitioning, enables skipping files
Column chunks: Within each row group, each column stored separately
Pages: Smallest unit (within column chunks), enables column pruning
Statistics: min/max/null count for predicate pushdown
Dictionary encoding: For low-cardinality strings

Apache Arrow (Feather/IPC)

Arrow is an in-memory columnar format. Feather (v1/v2) and IPC are on-disk/serialization formats.

import pyarrow as pa
import polars as pl

# Create Arrow table
table = pa.table({
    "id": [1, 2, 3],
    "value": [100.0, 200.0, 150.0],
    "category": ["A", "B", "A"]
})

# Write Feather file (Arrow IPC on disk)
pa.feather.write_feather(table, "data.feather")

# Read back
table2 = pa.feather.read_table("data.feather")

# Polars integration (zero-copy)
df = pl.from_arrow(table)  # No copy!
df.write_ipc("data.ipc")   # IPC format (stream or file)

# Arrow Flight RPC (network streaming)
from pyarrow.flight import FlightClient, FlightDescriptor

client = FlightClient("grpc+tcp://localhost:5005")
reader = client.do_get(descriptor)
table = reader.read_all()

When to use Arrow/Feather:

Fast serialization for ML (TensorFlow, PyTorch)
Inter-process communication (shared memory, files)
Zero-copy between Polars/Pandas/PyArrow
Not ideal for large-scale data lakes (no built-in partitioning)

Lance

Lance is an ML-native columnar format built on Arrow, with integrated vector search and versioning.

import lancedb
import polars as pl
from sentence_transformers import SentenceTransformer

# Create Lance dataset
db = lancedb.connect("./data.lance")
df = pl.DataFrame({
    "id": [1, 2, 3],
    "text": ["Hello world", "Goodbye world", "Machine learning"],
    "vector": [[0.1] * 128, [0.2] * 128, [0.3] * 128]  # Embeddings
})

# Write (creates .lance directory)
table = db.create_table("my_table", df)

# Append more data
table.add(df2)

# Vector search
results = table.search([0.1] * 128).limit(5).to_pandas()

# Versioned updates (like git)
table = db.create_table("versioned", df, mode="overwrite")
# Each overwrite creates a new version
table.checkout(version=1)  # Access previous version

# Cloud storage
db = lancedb.connect("s3://bucket/dataset/")

Lance advantages:

Built-in vector indexes (IVF_PQ, HNSW) - no separate DB needed
Multi-modal: store images, audio, text, vectors in same table
Version control for datasets
Zero-copy reads via memory mapping
No metadata catalog needed (self-contained)

Zarr

Zarr is a chunked, compressed N-dimensional array format, popular in ML, geospatial, and scientific computing.

import zarr
import numpy as np

# Create Zarr array (chunked, compressed)
z = zarr.open(
    'data.zarr',
    mode='w',
    shape=(1000000, 1000),  # Large 2D array
    chunks=(10000, 1000),    # Chunk size
    dtype='f4',
    compressor=zarr.Blosc(cname='zstd', clevel=3)
)

# Write chunks
z[:10000, :] = np.random.rand(10000, 1000).astype('f4')

# Read partial (only loads needed chunks)
slice = z[5000:6000, :]

# Group (like HDF5 groups)
g = zarr.open_group('experiment.zarr', mode='w')
g.create_dataset('images', data=image_array, chunks=True)
g.create_dataset('labels', data=label_array)

# Cloud storage (s3://)
import s3fs
fs = s3fs.S3FileSystem(anon=False)
store = s3fs.S3Map(root='mybucket/data.zarr', s3=fs)
zarr.open(store=store, mode='w', shape=(1000, 1000), chunks=(100, 100), dtype='f4')

Zarr advantages:

Parallel reads/writes across chunks
Cloud-optimized (each chunk is a separate object)
Schema flexibility (groups, hierarchies)
Good for terabyte-scale arrays
Growing ecosystem: xarray, dask, napari

Avro

Row-based format with rich schema evolution. Common in streaming (Kafka).

import fastavro
import json

# Define schema
schema = {
    "type": "record",
    "name": "Event",
    "fields": [
        {"name": "id", "type": "int"},
        {"name": "event_type", "type": "string"},
        {"name": "timestamp", "type": "long"}  # Unix epoch
    ]
}

# Write Avro file
with open("events.avro", "wb") as out:
    fastavro.writer(out, schema, [
        {"id": 1, "event_type": "click", "timestamp": 1700000000},
        {"id": 2, "event_type": "view", "timestamp": 1700000001}
    ])

# Read
with open("events.avro", "rb") as fo:
    records = list(fastavro.reader(fo))

# Kafka integration (confluent-kafka)
from confluent_kafka import SerializingProducer
from confluent_kafka.schema_registry.avro import AvroSerializer

# Schema Registry integration ensures compatibility

Avro vs Parquet:

Avro: row-based, append-only, streaming-friendly
Parquet: columnar, analytics-friendly, predicate pushdown
Convert: polars.read_avro() → write_parquet() for ETL

ORC

Optimized Row Columnar, primarily for Hive/Hadoop ecosystems.

import pyarrow.orc as orc

# Write
table = pa.table({
    "id": [1, 2, 3],
    "value": [100.0, 200.0, 150.0]
})
orc.write_table(table, "data.orc")

# Read
table = orc.read_table("data.orc")
df = table.to_pandas()

# Stripe-level statistics (similar to Parquet row groups)

ORC vs Parquet:

ORC: Hive-centric, ACID transactions, better compression for Hive queries
Parquet: More ecosystem support (Spark, DuckDB, Polars), better column pruning
Modern stacks typically prefer Parquet

Format Selection Guide

Use Case Matrix

Use Case	Recommended Format	Reason
Data lake analytics	Parquet	Mature, partitioning, ecosystem
ML training data	Arrow/Feather or Lance	Zero-copy, vector support
Geospatial arrays	Zarr	Chunked, N-dimensional, cloud-optimized
Streaming/Kafka	Avro	Schema evolution, row-based
Legacy Hive	ORC	Compatibility
Feature stores	Lance or Delta	Versioning, vectors
IPC between processes	Arrow IPC or Feather	Zero-copy, fast
Quick exports	Parquet (Zstd)	Good compression/decompression speed

Compression Codec Comparison

Codec	Compression Ratio	Speed (Compress/Decompress)	Best For
Snappy	Low (~2:1)	⚡⚡⚡ Fast	Fast analytics, default Parquet
Zstd	Medium-High (~4:1)	⚡⚡ Fast	General purpose, good balance
LZ4	Low-Medium (~2.5:1)	⚡⚡⚡ Very fast	Real-time streaming
Gzip	High (~5:1)	⚡ Slow	Archival, cold storage
Blosc (zstd)	Medium	⚡⚡	Zarr arrays

Advanced Patterns

Converting Formats

# Avro → Parquet (ETL)
import polars as pl
df = pl.read_avro("input.avro")
df.write_parquet("output.parquet")

# Arrow → Lance (ML pipeline)
import lancedb
table = pa.feather.read_table("data.feather")
db = lancedb.connect("./dataset.lance")
db.create_table("embeddings", table)

# Zarr → Parquet (geospatial to analytics)
import dask.array as da
z = da.from_zarr("sar.zarr")
df = z.mean(axis=0).to_dataframe()  # Aggregate and convert
df.to_parquet("summary.parquet")

Partitioning Strategies

Parquet partition discovery:

# Hive-style: year=2024/month=01/day=01/
dataset = ds.dataset(
    "s3://bucket/events/",
    filesystem=s3_fs,
    format="parquet",
    partitioning=ds.HivePartitioning.discover()
)

# Directory partitioning: year/2024/month/01/
dataset = ds.dataset(
    "s3://bucket/events/year=2024/month=01/",
    filesystem=s3_fs
)

Lance partitioning: Built-in via to_lance():

df.write_lance("data.lance", partition_by=["year", "month"])

Zarr chunking:

# Chunk by spatial region for geospatial
z = zarr.open(
    "satellite.zarr",
    mode='w',
    shape=(10000, 10000),  # 10000x10000 pixels
    chunks=(1000, 1000),   # 1000x1000 tiles
    dtype='float32'
)

Performance Tuning

Parquet

Row group size: 100K-1M rows for optimal skipping
Dictionary encoding: Enable for low-cardinality strings
Compression: Zstd level 3 for balance, Snappy for speed
Column order: Put high-selectivity columns first (min/max statistics better)

Lance

Vector index type: IVF_PQ for large datasets (>1M), HNSW for smaller/higher recall
Use create_index() after bulk load, not during writes
Batch writes for throughput

Zarr

Chunk size: Align with access pattern (e.g., time-series: chunk by time)
Compression: Blosc+zstd, tune clevel (3-5)
Consider Zipf+Shuffle filters for structured arrays

Emerging Formats (2024-2025)

Lance (2022): Gaining traction in ML community, integrated with RAPIDS, Polars, PyTorch
Soar (2023): Columnar format optimized for AI training (similar to Lance, different ecosystem)
Vortex: Not widely adopted yet - if you mean Arrow's compute kernels (not a format)
DuckDB's .duckdb format: Embedded SQLite-like for DuckDB persistence
Delta Lake / Iceberg (table formats): Already in lakehouse skill

Best Practices

✅ Default to Parquet - Broadest compatibility, good compression, ecosystem tooling
✅ Use Arrow/Feather for ML staging - Zero-copy between training frameworks
✅ Use Lance when vectors are first-class - No separate vector DB needed
✅ Use Zarr for N-dim arrays - Geospatial, video, 3D data
✅ Compress everything - Snappy (fast) or Zstd (balanced)
✅ Partition wisely - By date/region/tenant to enable pruning
❌ Don't use Avro for analytics (no column pruning, row-based)
❌ Don't use ORC unless in Hive Hadoop world
❌ Don't store wide tables in single Parquet files - Partition or use Delta/Iceberg

References

Apache Parquet Format
Apache Arrow Format
Lance Format
Zarr Format
Apache Avro Specification
@data-engineering-core - Reading/writing with Polars/DuckDB
@data-engineering-storage-lakehouse - Table formats built on these (Delta/Iceberg)
@data-engineering-storage-remote-access - Using these formats with cloud storage backends (S3, GCS, Azure)
@data-engineering-storage-authentication - Credential patterns for accessing cloud storage

data-engineering-storage-formats

Data Storage Formats

Quick Comparison

When to Use Which?

Choose Parquet when:

Choose Arrow/Feather when:

Choose Lance when:

Choose Zarr when:

Choose Avro when:

Choose ORC when:

Skill Dependencies

Detailed Guides

Parquet

Apache Arrow (Feather/IPC)

Lance

Zarr

Avro

ORC

Format Selection Guide

Use Case Matrix

Compression Codec Comparison

Advanced Patterns

Converting Formats

Partitioning Strategies

Performance Tuning

Parquet

Lance

Zarr

Emerging Formats (2024-2025)

Best Practices

References