Data Catalog Patterns

Reference patterns for managing the Dataiku data catalog via the Python API.

Key Concepts

Concept	What it is	Scope
Data Collection	A curated group of datasets, visible across projects	Instance-level (client)
Metadata	Label, description, tags, checklists, custom key-value pairs	Per dataset or project
Meaning	A semantic type for columns (e.g., "Email", "Country Code")	Instance-level (client)
Tags	Freeform labels on datasets or projects	Per dataset or project

Data Collections

List Collections

collections = client.list_data_collections(as_type="dict")
for c in collections:
    print(f"{c['displayName']} ({c['id']}) — {c['itemCount']} items")

Create a Collection

dc = client.create_data_collection(
    displayName="Customer Data",
    description="All customer-related datasets",
    tags=["customers", "production"]
)

Add Datasets to a Collection

dc = client.get_data_collection("collection_id")

# Add by dataset handle
ds = project.get_dataset("MY_DATASET")
dc.add_object(ds)

# Add by dict (for cross-project datasets)
dc.add_object({"type": "DATASET", "projectKey": "PROJECT_A", "id": "DATASET_NAME"})

List and Remove Objects

dc = client.get_data_collection("collection_id")
objects = dc.list_objects()
for obj in objects:
    raw = obj.get_raw()
    print(f"  {raw['projectKey']}.{raw['id']}")

    # Get as a dataset handle
    ds = obj.get_as_dataset()

    # Remove from collection
    obj.remove()

Update Collection Settings

dc = client.get_data_collection("collection_id")
settings = dc.get_settings()
settings.display_name = "Renamed Collection"
settings.description = "Updated description"
settings.tags = ["new-tag", "production"]
settings.save()

Dataset Metadata

Get and Set Metadata

ds = project.get_dataset("MY_DATASET")
metadata = ds.get_metadata()

# Metadata structure:
# {
#   "label": "...",
#   "description": "...",
#   "tags": ["tag1", "tag2"],
#   "checklists": {"checklists": [...]},
#   "custom": {"kv": {"key1": "value1"}}
# }

metadata["tags"] = ["cleaned", "production"]
metadata["custom"]["kv"]["owner"] = "data-team"
ds.set_metadata(metadata)

AI-Generated Descriptions

# Generate descriptions for dataset and columns (requires AI Services enabled)
result = ds.generate_ai_description(language="english", save_description=True)

Rate-limited: 1000 requests/day, then throttled to ~60s per call.

Meanings (Semantic Column Types)

List and Create Meanings

# List existing meanings
meanings = client.list_meanings()

# Create a values-list meaning
client.create_meaning(
    id="country_code",
    label="Country Code",
    type="VALUES_LIST",
    values=["US", "UK", "FR", "DE", "JP"],
    normalizationMode="EXACT",
    detectable=True
)

# Create a pattern-based meaning
client.create_meaning(
    id="email_address",
    label="Email Address",
    type="PATTERN",
    pattern=r"^[\w.-]+@[\w.-]+\.\w+$",
    detectable=True
)

Update a Meaning

meaning = client.get_meaning("country_code")
definition = meaning.get_definition()
definition["entries"].append({"value": "CA"})
meaning.set_definition(definition)

Catalog Indexing

Trigger re-indexing of connections so new tables appear in the catalog:

# Index specific connections
client.catalog_index_connections(connection_names=["my_snowflake", "my_postgres"])

# Index all connections
client.catalog_index_connections(all_connections=True)

Detailed References

• references/data-collections.md — Permissions, completeness checks, cross-project patterns
• references/metadata-and-tags.md — Full metadata structure, project tags, custom metadata
• references/meanings.md — All meaning types, normalization modes, detectable meanings