databricks-execution-compute
Installation
SKILL.md
Databricks Execution & Compute
Run code on Databricks. Four execution modes—choose based on workload. Most examples below use the Databricks CLI (Databricks Sandbox is SDK-driven); see the databricks-core skill for install and authentication.
Execution Mode Decision Matrix
| Aspect | Databricks Connect ⭐ | Serverless Job | Interactive Cluster | Databricks Sandbox |
|---|---|---|---|---|
| Use for | Spark code (ETL, data gen) | Heavy processing (ML) | State across tool calls, Scala/R | Untrusted / agent-generated code, isolated env |
| Startup | Instant | ~25-50s cold start | ~5min if stopped | ~1s (warm start) |
| State | Within Python process | None | Via context_id | Home dir persisted across stop/restart |
| Languages | Python (PySpark) | Python, SQL | Python, Scala, SQL, R | Any (general Linux shell) |
| Dependencies | withDependencies() |
CLI with environments spec | Install on cluster | Install in sandbox, or spec.environment.image_uri (UC container artifact) for a custom image |
Decision Flow
Main decision point: if you're using Declarative Automation Bundles (DABs) then follow the instructions of the databricks-dabs skill first. In short, you can use databricks bundle run to run code associated with jobs, pipelines, and other resources. This can be recognized by looking for a databricks.yml file in the project root. If these resources don't exist, or if you're not using DABs, then proceed with the below.