Event-Driven Declarative Orchestrator

Get started with Kestra in 4 minutes.

Kestra Databricks Plugin

Why

• What user problem does this solve? Teams need to databricks plugin for Kestra from orchestrated workflows instead of relying on manual console work, ad hoc scripts, or disconnected schedulers.
• Why would a team adopt this plugin in a workflow? It keeps Databricks steps in the same Kestra flow as upstream preparation, approvals, retries, notifications, and downstream systems.
• What operational/business outcome does it enable? It reduces manual handoffs and fragmented tooling while improving reliability, traceability, and delivery speed for processes that depend on Databricks.

What

• Provides plugin components under io.kestra.plugin.databricks.
• Includes classes such as DeleteCluster, CreateCluster, CreateJob, SubmitRun.

DatabricksCLI — Authentication

DatabricksCLI supports two mutually exclusive authentication modes. databricksToken takes precedence when both are provided.

PAT (personal access token)

tasks:
  - id: list_clusters
    type: io.kestra.plugin.databricks.cli.DatabricksCLI
    databricksHost: "{{ secret('DATABRICKS_HOST') }}"
    databricksToken: "{{ secret('DATABRICKS_TOKEN') }}"
    commands:
      - databricks clusters list

OAuth M2M (service principal)

tasks:
  - id: list_clusters
    type: io.kestra.plugin.databricks.cli.DatabricksCLI
    databricksHost: "{{ secret('DATABRICKS_HOST') }}"
    clientId: "{{ secret('DATABRICKS_CLIENT_ID') }}"
    clientSecret: "{{ secret('DATABRICKS_CLIENT_SECRET') }}"
    commands:
      - databricks clusters list

Documentation

• Full documentation can be found under kestra.io/docs
• Documentation for developing a plugin is included in the Plugin Developer Guide.

License

Apache 2.0 © Kestra Technologies

Stay up to date

We release new versions every month. Give the main repository a star to stay up to date with the latest releases and get notified about future updates.