Databricks SDK for JavaScript

[!WARNING]

⚠️ PREVIEW - NOT FOR PRODUCTION USE

This SDK is in preview and is subject to change without notice.

• ❌ Do NOT use in production environments
• ⚠️ Breaking changes may occur at any time
• 🔬 APIs are experimental and unstable
• 📝 Use for development and testing only

For production use cases, please wait for the stable 1.0.0 release.

The Databricks SDK for JavaScript provides a convenient way to interact with Databricks REST APIs in Node.js and TypeScript applications. It covers all public Databricks REST API operations and the SDK's internal HTTP client is robust and handles failures on different levels by performing intelligent retries.

Table of Contents

• Installation
• Getting Started
• Authentication
  • Default Authentication Flow
  • Databricks Native Authentication
  • Azure Native Authentication
  • Google Cloud Platform Native Authentication
  • Configuration File
  • Environment Variables
• Usage Examples
  • Listing Clusters
  • Creating and Waiting for Clusters
  • Submitting Jobs
  • Unity Catalog Operations
• Long-Running Operations
• Pagination
• Error Handling
• Configuration
  • Timeouts
  • HTTP Proxy
  • Rate Limiting
• Logging and Debugging
• Requirements
• Testing
• Contributing
• License

Installation

Install the SDK using your preferred package manager:

npm

npm install @databricks/sdk-experimental

pnpm

pnpm add @databricks/sdk-experimental

Getting Started

The simplest way to get started is to create a WorkspaceClient and let it automatically detect your credentials:

import {WorkspaceClient} from "@databricks/sdk-experimental";

// Create client (automatically detects credentials)
const client = new WorkspaceClient();

// List all clusters
for await (const cluster of client.clusters.list({})) {
    console.log(cluster.cluster_name);
}

// Get current user
const user = await client.currentUser.me();
console.log(`Logged in as: ${user.user_name}`);

Authentication

If you use Databricks configuration profiles or Databricks-specific environment variables for Databricks authentication, the only code required to start working with a Databricks workspace is:

import {WorkspaceClient} from "@databricks/sdk-experimental";
const client = new WorkspaceClient();

Default Authentication Flow

The SDK tries the following authentication methods in order until it succeeds:

1. Databricks Native Authentication (PAT tokens, OAuth M2M, Workload Identity Federation)
2. Azure Native Authentication (Azure CLI, Azure MSI, Azure Client Secret)
3. Google Cloud Platform Native Authentication (GCP credentials, default application credentials)
4. If unsuccessful, returns an authentication error

For each authentication method, the SDK searches for compatible credentials in the following locations:

1. Credentials explicitly passed to the Config constructor
2. Databricks-specific environment variables (e.g., DATABRICKS_HOST, DATABRICKS_TOKEN)
3. Configuration file (~/.databrickscfg)
4. Cloud provider authentication (Azure CLI, GCP default credentials, AWS instance profiles)

Databricks Native Authentication

Personal Access Token (PAT)

The most common authentication method for development:

import {Config, WorkspaceClient} from "@databricks/sdk-experimental";

const config = new Config({
    host: "https://your-workspace.cloud.databricks.com",
    token: "dapi...",
});
const client = new WorkspaceClient(config);

Or using environment variables:

export DATABRICKS_HOST="https://your-workspace.cloud.databricks.com"
export DATABRICKS_TOKEN="dapi..."

import {WorkspaceClient} from "@databricks/sdk-experimental";

// Automatically uses environment variables
const client = new WorkspaceClient();

OAuth Machine-to-Machine (M2M) - Service Principal

For production service-to-service authentication:

import {Config, WorkspaceClient} from "@databricks/sdk-experimental";

const config = new Config({
    host: "https://your-workspace.cloud.databricks.com",
    clientId: "your-client-id",
    clientSecret: "your-client-secret",
});
const client = new WorkspaceClient(config);

Environment variables:

export DATABRICKS_HOST="https://your-workspace.cloud.databricks.com"
export DATABRICKS_CLIENT_ID="your-client-id"
export DATABRICKS_CLIENT_SECRET="your-client-secret"

OAuth User-to-Machine (U2M) - External Browser

For interactive user authentication with SSO:

import {Config, WorkspaceClient} from "@databricks/sdk-experimental";

const config = new Config({
    host: "https://your-workspace.cloud.databricks.com",
    clientId: "your-client-id",
    authType: "oauth-u2m",
});
const client = new WorkspaceClient(config);

The SDK will automatically open a browser for user authentication.

Azure Native Authentication

Azure CLI

If you've authenticated with az login, the SDK will automatically use those credentials:

import {Config, WorkspaceClient} from "@databricks/sdk-experimental";

const config = new Config({
    host: "https://adb-<workspace-id>.<random>.azuredatabricks.net",
    authType: "azure-cli",
});
const client = new WorkspaceClient(config);

Azure Managed Service Identity (MSI)

For applications running in Azure:

import {Config, WorkspaceClient} from "@databricks/sdk-experimental";

const config = new Config({
    host: "https://adb-<workspace-id>.<random>.azuredatabricks.net",
    azureClientId: "your-managed-identity-client-id",
});
const client = new WorkspaceClient(config);

Azure Service Principal

import {Config, WorkspaceClient} from "@databricks/sdk-experimental";

const config = new Config({
    host: "https://adb-<workspace-id>.<random>.azuredatabricks.net",
    azureClientId: "your-client-id",
    azureClientSecret: "your-client-secret",
    azureTenantId: "your-tenant-id",
});
const client = new WorkspaceClient(config);

Google Cloud Platform Native Authentication

GCP Default Application Credentials

If you've authenticated with gcloud auth application-default login:

import {Config, WorkspaceClient} from "@databricks/sdk-experimental";

const config = new Config({
    host: "https://your-workspace.gcp.databricks.com",
    authType: "google-credentials",
});
const client = new WorkspaceClient(config);

GCP Service Account

import {Config, WorkspaceClient} from "@databricks/sdk-experimental";

const config = new Config({
    host: "https://your-workspace.gcp.databricks.com",
    googleServiceAccount: "[email protected]",
});
const client = new WorkspaceClient(config);

Configuration File

Create a ~/.databrickscfg file:

[DEFAULT]
host = https://your-workspace.cloud.databricks.com
token = dapi...

[PROD]
host = https://prod-workspace.cloud.databricks.com
token = dapi...

[STAGING]
host = https://staging-workspace.cloud.databricks.com
token = dapi...

Use profiles in your code:

import {Config, WorkspaceClient} from "@databricks/sdk-experimental";

// Use DEFAULT profile
const client = new WorkspaceClient();

// Use specific profile
const prodClient = new WorkspaceClient(new Config({profile: "PROD"}));
const stagingClient = new WorkspaceClient(new Config({profile: "STAGING"}));

Environment Variables

The SDK supports the following environment variables:

| Variable | Description | | ----------------------------------- | ---------------------------------------------------- | | DATABRICKS_HOST | Workspace or account URL | | DATABRICKS_TOKEN | Personal access token | | DATABRICKS_CLIENT_ID | OAuth client ID / Azure client ID | | DATABRICKS_CLIENT_SECRET | OAuth client secret / Azure client secret | | DATABRICKS_ACCOUNT_ID | Databricks account ID (for account-level operations) | | DATABRICKS_AZURE_TENANT_ID | Azure tenant ID | | DATABRICKS_GOOGLE_SERVICE_ACCOUNT | GCP service account email | | DATABRICKS_AUTH_TYPE | Force specific auth type |

Usage Examples

Listing Clusters

import {WorkspaceClient} from "@databricks/sdk-experimental";

const client = new WorkspaceClient();

// List all clusters
for await (const cluster of client.clusters.list({})) {
    console.log(`Cluster: ${cluster.cluster_name} (${cluster.cluster_id})`);
    console.log(`  State: ${cluster.state}`);
    console.log(`  Spark Version: ${cluster.spark_version}`);
}

Creating and Waiting for Clusters

import {WorkspaceClient} from "@databricks/sdk-experimental";

const client = new WorkspaceClient();

// Create cluster and wait for it to be running
const waiter = await client.clusters.create({
    cluster_name: "my-cluster",
    spark_version: "13.3.x-scala2.12",
    node_type_id: "i3.xlarge",
    num_workers: 2,
    autotermination_minutes: 60,
});

// Wait with progress callback
const cluster = await waiter.wait({
    timeout: 15 * 60 * 1000, // 15 minutes
    onProgress: (c) => {
        console.log(`Cluster state: ${c.state}`);
    },
});

console.log(`Cluster ${cluster.cluster_id} is running!`);

Submitting Jobs

import {WorkspaceClient} from "@databricks/sdk-experimental";

const client = new WorkspaceClient();

// Submit a one-time job run
const waiter = await client.jobs.submit({
    run_name: "my-job-run",
    tasks: [
        {
            task_key: "main",
            notebook_task: {
                notebook_path: "/Users/[email protected]/my-notebook",
                source: "WORKSPACE",
            },
            new_cluster: {
                spark_version: "13.3.x-scala2.12",
                node_type_id: "i3.xlarge",
                num_workers: 2,
            },
        },
    ],
});

// Wait for job to complete
const run = await waiter.wait({
    timeout: 30 * 60 * 1000, // 30 minutes
    onProgress: (r) => {
        console.log(`Job state: ${r.state?.life_cycle_state}`);
    },
});

console.log(`Job completed with result state: ${run.state?.result_state}`);

Unity Catalog Operations

import {WorkspaceClient} from "@databricks/sdk-experimental";

const client = new WorkspaceClient();

// List all catalogs
console.log("Catalogs:");
for await (const catalog of client.catalogs.list({})) {
    console.log(`  - ${catalog.name}`);
}

// Create a new schema
const schema = await client.schemas.create({
    name: "my_schema",
    catalog_name: "my_catalog",
    comment: "Schema for analytics",
});
console.log(`Created schema: ${schema.full_name}`);

// List tables in a schema
console.log("Tables in my_catalog.my_schema:");
for await (const table of client.tables.list({
    catalog_name: "my_catalog",
    schema_name: "my_schema",
})) {
    console.log(`  - ${table.full_name}`);
}

// Create a volume
const volume = await client.volumes.create({
    catalog_name: "my_catalog",
    schema_name: "my_schema",
    name: "my_volume",
    volume_type: "MANAGED",
});
console.log(`Created volume: ${volume.full_name}`);

Workspace Examples

The SDK includes 18+ complete, runnable examples covering common Databricks workspace operations. These examples demonstrate best practices and patterns for real-world use cases.

Available Examples

• Clusters (3 examples) - Create, list, start/stop clusters
• Jobs (4 examples) - Submit, create, trigger, and monitor jobs
• Unity Catalog (4 examples) - Manage catalogs, schemas, tables, volumes, and permissions
• SQL Warehouses (2 examples) - Create warehouses and execute queries
• ML/MLflow (3 examples) - Track experiments, log models, model registry
• Notebooks (2 examples) - Upload and export notebooks
• Files/DBFS (2 examples) - Upload and download files

Each example is self-contained, includes cleanup code, and demonstrates SDK best practices like the waiter pattern, pagination, and error handling.

See Workspace Examples for the complete catalog with usage instructions.

Quick Start

# Run any example with npx ts-node
npx ts-node examples/workspace/clusters/create-cluster.ts

# Or make executable and run directly
chmod +x examples/workspace/clusters/create-cluster.ts
./examples/workspace/clusters/create-cluster.ts

Example: Create and Monitor a Job

import {WorkspaceClient, Time, TimeUnits} from "@databricks/sdk-experimental";

const client = new WorkspaceClient({});

// Submit a one-time job run
const waiter = await client.jobs.submit({
    run_name: `my-job-${Date.now()}`,
    tasks: [
        {
            task_key: "main_task",
            notebook_task: {
                notebook_path: "/Shared/my-notebook",
                source: "WORKSPACE",
            },
            new_cluster: {
                spark_version: "13.3.x-scala2.12",
                node_type_id: "i3.xlarge",
                num_workers: 2,
            },
        },
    ],
});

// Wait for completion with progress updates
const result = await waiter.wait({
    timeout: new Time(20, TimeUnits.minutes),
    onProgress: async (run) => {
        console.log(`State: ${run.state?.life_cycle_state}`);
    },
});

console.log(
    `Job ${result.state?.result_state === "SUCCESS" ? "succeeded" : "failed"}`
);

For authentication examples and setup, see Authentication Examples.

Long-Running Operations

Many Databricks operations are asynchronous. The SDK provides a Waiter interface for these operations:

import {WorkspaceClient} from "@databricks/sdk-experimental";

const client = new WorkspaceClient();

// Start an async operation
const waiter = await client.clusters.create({
    cluster_name: "my-cluster",
    spark_version: "13.3.x-scala2.12",
    node_type_id: "i3.xlarge",
    num_workers: 2,
});

// Configure wait behavior
const cluster = await waiter.wait({
    timeout: 20 * 60 * 1000, // 20 minutes
    onProgress: (intermediate) => {
        console.log(`Current state: ${intermediate.state}`);
        console.log(`State message: ${intermediate.state_message}`);
    },
});

console.log(`Operation completed! Cluster ID: ${cluster.cluster_id}`);

Retrying Failed Operations

The SDK automatically retries transient errors (rate limits, temporary unavailability):

import {WorkspaceClient} from "@databricks/sdk-experimental";
import {Config} from "@databricks/sdk-experimental";

const config = new Config({
    host: "https://your-workspace.cloud.databricks.com",
    token: "dapi...",
    retryTimeoutSeconds: 600, // Retry for up to 10 minutes
});

const client = new WorkspaceClient(config);

Pagination

The SDK automatically handles pagination using async iterators:

import {WorkspaceClient} from "@databricks/sdk-experimental";

const client = new WorkspaceClient();

// Automatically handles pagination
for await (const job of client.jobs.list({limit: 25})) {
    console.log(`Job: ${job.settings?.name} (${job.job_id})`);
}

// Collect all results into an array
const allJobs = [];
for await (const job of client.jobs.list({})) {
    allJobs.push(job);
}
console.log(`Total jobs: ${allJobs.length}`);

Error Handling

The SDK provides structured error handling:

import {WorkspaceClient} from "@databricks/sdk-experimental";
import {ApiError} from "@databricks/sdk-experimental";

const client = new WorkspaceClient();

try {
    const cluster = await client.clusters.get({cluster_id: "invalid-id"});
} catch (error) {
    if (error instanceof ApiError) {
        console.error(`API Error: ${error.message}`);
        console.error(`Status Code: ${error.statusCode}`);
        console.error(`Error Code: ${error.errorCode}`);

        // Check for specific error types
        if (error.statusCode === 404) {
            console.error("Cluster not found");
        } else if (error.statusCode === 403) {
            console.error("Permission denied");
        } else if (error.statusCode === 429) {
            console.error("Rate limited - will automatically retry");
        }
    } else {
        console.error("Unexpected error:", error);
    }
}

The SDK automatically retries the following errors:

• 429: Too Many Requests (rate limiting)
• 503: Service Temporarily Unavailable
• Connection errors: Network issues, timeouts

Configuration

Timeouts

Configure HTTP and retry timeouts:

import {Config, WorkspaceClient} from "@databricks/sdk-experimental";

const config = new Config({
    host: "https://your-workspace.cloud.databricks.com",
    token: "dapi...",
    httpTimeoutSeconds: 60, // HTTP request timeout (default: 5s)
    retryTimeoutSeconds: 300, // Total retry timeout (default: 300s)
});

const client = new WorkspaceClient(config);

HTTP Proxy

Configure an HTTP proxy for all requests:

import {Config, WorkspaceClient} from "@databricks/sdk-experimental";

const config = new Config({
    host: "https://your-workspace.cloud.databricks.com",
    token: "dapi...",
    proxy: "http://proxy.example.com:8080",
});

const client = new WorkspaceClient(config);

Or using environment variables:

export HTTPS_PROXY="http://proxy.example.com:8080"
export HTTP_PROXY="http://proxy.example.com:8080"

Rate Limiting

Configure client-side rate limiting:

import {Config, WorkspaceClient} from "@databricks/sdk-experimental";

const config = new Config({
    host: "https://your-workspace.cloud.databricks.com",
    token: "dapi...",
    rateLimitPerSecond: 10, // Maximum 10 requests per second
});

const client = new WorkspaceClient(config);

Logging and Debugging

Enable debug logging to troubleshoot issues:

import {setLogger, WorkspaceClient} from "@databricks/sdk-experimental";

// Set up custom logger
setLogger({
    log: (level, message, context) => {
        console.log(`[${level}] ${message}`, context);
    },
});

const client = new WorkspaceClient();

Use environment variables for debugging:

# Enable header debugging (redacts sensitive data)
export DATABRICKS_DEBUG_HEADERS=true

# Limit debug output truncation
export DATABRICKS_DEBUG_TRUNCATE_BYTES=2000

Requirements

• Node.js: 18.0 or higher
• TypeScript: 4.5 or higher (if using TypeScript)

Authentication Examples

The SDK supports 10+ authentication methods. See Authentication Examples for complete, runnable examples covering:

• Personal Access Tokens (PAT)
• OAuth Machine-to-Machine (Service Principals)
• OAuth User-to-Machine (SSO)
• Configuration file profiles
• Azure authentication (CLI, MSI, Service Principal)
• Google Cloud Platform authentication
• AWS instance profiles
• Custom credentials providers

Each example includes setup instructions and can be run with npx ts-node.

Authentication Examples

The SDK supports 10+ authentication methods. See Authentication Examples for complete, runnable examples covering:

• Personal Access Tokens (PAT)
• OAuth Machine-to-Machine (Service Principals)
• OAuth User-to-Machine (SSO)
• Configuration file profiles
• Azure authentication (CLI, MSI, Service Principal)
• Google Cloud Platform authentication
• AWS instance profiles
• Custom credentials providers

Each example includes setup instructions and can be run with npx ts-node.

Testing

Unit Tests

Run unit tests:

npm run test

Integration Tests

Most tests are integration tests and must be run against a live cluster. Configure the following environment variables:

| Name | Value | | ------------------------- | --------------------------------------------------------------------------------------------------- | | DATABRICKS_HOST | Hostname of the Databricks workspace (starts with https://) | | DATABRICKS_TOKEN | Personal access token | | TEST_DEFAULT_CLUSTER_ID | (optional) ID of a cluster to run tests against. If missing, tests will create a cluster on demand. |

Run integration tests:

export DATABRICKS_HOST="https://your-workspace.cloud.databricks.com"
export DATABRICKS_TOKEN="dapi..."
npm run test

Contributing

See CONTRIBUTING.md for development setup and contribution guidelines.

The Databricks SDK for JavaScript is developed as part of the broader Databricks SDK ecosystem. We welcome contributions from the community!

License

Apache License 2.0. See LICENSE for details.