@fabric-harness/databricks

First-party Databricks integration helpers for Fabric Harness, with mock-first local development and a protected live certification workflow for target workspaces.

Install

npm install @fabric-harness/databricks @fabric-harness/sdk

Node 22 or newer is required. The package exact-pins every reviewed @databricks/sdk-* module to SDK release 0.21.0, so generated service clients own stable API serialization and authentication.

Add @fabric-harness/node and pg only when using the Node persistence helpers.

Fast path

import { init } from '@fabric-harness/sdk';
import { databricks } from '@fabric-harness/databricks';

const dbx = databricks({
  host: process.env.DATABRICKS_HOST!,
  principal: {
    kind: 'service-principal',
    host: process.env.DATABRICKS_HOST!,
    clientId: process.env.DATABRICKS_CLIENT_ID!,
    clientSecret: process.env.DATABRICKS_CLIENT_SECRET!,
  },
  model: process.env.DATABRICKS_MODEL ?? 'system.ai.gpt-oss-20b',
  warehouseId: process.env.DATABRICKS_WAREHOUSE_ID,
  sqlRead: true,
  aiSearch: {
    index: 'main.support.docs_index',
    textColumn: 'chunk',
    strategy: 'hybrid',
  },
  governance: { stewardAudience: 'data-steward' },
});

const fabric = await init({
  modelProvider: dbx.modelProvider,
  tools: dbx.tools,
  policy: dbx.policy,
});

The bundle threads a Databricks CLI OAuth profile, PAT, OAuth service principal, or on-behalf-of token provider through generated Databricks service clients, Unity AI Gateway, and custom Model Serving endpoints. Authentication/profile resolution comes from the official SDK credential chain. system.ai.* model services use the OpenAI-compatible /ai-gateway/mlflow/v1 route; other endpoint names retain /serving-endpoints. Unity Catalog remains the authorization source of truth.

import { databricksIdentity } from '@fabric-harness/databricks';

const token = databricksIdentity({
  kind: 'cli-profile',
  profile: process.env.DATABRICKS_CONFIG_PROFILE ?? 'DEFAULT',
});

host is optional here — when omitted, the workspace host is resolved from the named CLI profile in ~/.databrickscfg (or DATABRICKS_CONFIG_FILE). An explicit host (or DATABRICKS_HOST) takes precedence as the workspace origin used for API calls, but CLI token acquisition always follows the named profile — host and the profile must point at the same workspace.

For direct typed access, use the same generated clients the bundle uses:

import { databricksJobs, databricksSdk } from '@fabric-harness/databricks';

const sdk = databricksSdk({ host, principal });
const jobs = databricksJobs(sdk.jobs, { runPolicy: { allowedJobIds: [42] } });
await jobs.runJob({ jobId: 42, idempotencyToken: submission.id });

databricksRunJobTool(client, runPolicy) and databricksNotebookTool(client, notebookPolicy) require that bound as a positional argument, so a model-callable run tool cannot be constructed without one. An out-of-policy job id or notebook path throws DatabricksRunNotAllowedError before the Jobs API is called.

The same construction-time boundary applies to model-callable SQL, AI Functions, Lakeflow start/stop, and MLflow writes. warehouseId is connectivity only; it exposes no SQL tool by itself. Use sqlRead: true for the SELECT-only beginner path, or supply sqlExecute, aiFunctions, lakeflow.runPolicy, and factory-level MLflow run policies explicitly. Every out-of-policy call fails before the generated Databricks client is invoked.

databricks({ ... }).sdk exposes the principal-bound generated clients to custom agent runs. Fabric adds governance, approval, lifecycle safety, retries for explicitly idempotent operations, lineage, and durable state; it does not reimplement stable Databricks service APIs. A small private raw protocol transport remains only for APIs absent from the modular SDK: Agent Services, managed memory, Genie Agent Mode streaming, Workspace object import/export, MLflow 3 tracing, AI Gateway/model-service discovery, and the ResponsesAgent custom request schema. Its exhaustive method-and-path allowlist rejects every other route before credential resolution or network I/O. Callers use typed bundle capabilities rather than constructing that transport.

Fabric verticals must import this package instead of implementing local Databricks authentication, clients, credential exchange, polling, or retry logic. Applications retain only domain resource names, policies, and readiness composition.

Server application bundles should import control-plane, generated SDK, SQL, identity, and Lakebase helpers from @fabric-harness/databricks/runtime. That subpath intentionally excludes Harness build, deployment, and agent-authoring modules, so frameworks such as Next.js do not trace build-time Node tooling into request handlers. Agent definitions and release tooling continue to use the root export.

import {
  databricksControlPlane,
  databricksPrincipalFromEnv,
  databricksSdk,
} from '@fabric-harness/databricks/runtime';

const principal = databricksPrincipalFromEnv(process.env);
const sdk = databricksSdk({ host: process.env.DATABRICKS_HOST!, principal });
const controlPlane = databricksControlPlane({
  host: process.env.DATABRICKS_HOST!,
  principal,
});
const warehouses = await controlPlane.sqlWarehouses.list();

Fabric Platform integration

The optional @fabric-harness/databricks/platform subpath maps Databricks resource identity and execution results into the provider-neutral contracts shipped by @fabricorg/platform@^0.8.0. It records workspace resource references and effective principal delegation without tokens or secrets. Platform remains the canonical business-mutation and approval ledger; Harness returns technical execution attestations to that invocation.

Two compatibility exceptions are deliberately quarantined in SDK 0.21. The generated Genie query-result parser rejects Databricks' real row-array payload, and the shared JSON decoder attempts to convert a floating-point AI Search score to BigInt. Fabric retries only those read-only responses through the raw protocol transport when the exact decoder error is raised. Authentication, lifecycle, and unrelated failures continue through or fail from the official clients. Source-boundary and runtime-denial tests prevent these exceptions from spreading to other stable services.

Shipped surface

• AI Gateway and Model Serving: automatic system.ai.* routing, explicit mode/base URL selection, model-service discovery, gateway readiness, request tags, streaming, tools, and custom endpoints.
• Managed MCP: async databricksWithManagedMcp() discovery for Genie, AI Search, SQL, Unity Catalog functions, and registered Unity AI Gateway MCP Services; rotating OBO/M2M auth, same-origin token enforcement, explicit effect classification, token redaction, governed static resources, and request-scoped reconnection.
• Identity: Databricks CLI OAuth profiles, explicit PAT/pre-fetched bearer, cached OAuth M2M, Databricks Apps service principal, and forwarded user token helpers.
• Data and AI: SELECT-only databricksSqlReadTool(), policy-bound arbitrary SQL Warehouse execution, Unity Catalog discovery/admin, AI Search query/admin, bounded citation-validated RAG with token streaming, MLflow 3 evaluation records and a managed-judge release Job, embeddings, endpoint-bound ai_query(), typed Genie Agent conversations, beta normalized lifecycle/ACL authoring, explicit Beta Agent Mode SSE streaming, Feature Serving, typed Jobs 2.2 authoring/run APIs, and workspace read/write APIs.
• Agent governance: explicit-Beta Unity Catalog Agent Services registration, discovery, metadata updates, EXECUTE/READ_METADATA grants, and deletion for externally hosted Harness agents.
• Native managed agents: explicit-Beta Supervisor Agent and Knowledge Assistant discovery through the official generated SDK, guarded lifecycle access, Responses-compatible invocation, cancellation propagation, and fixed-resource model tool projection.
• Managed memory: explicit-Beta store/entry lifecycle and bounded search with mandatory trusted scope and /memories/ path isolation; it remains separate from Harness session persistence.
• Data engineering: read-only Lakeflow pipeline list/status by default, policy-bound start/stop, plus separately opt-in create/update/delete/event tools.
• Operations: System Tables consumption reporting, actual-cost budget source, MLflow tracing, serving usage capture, and Lakebase telemetry tables.
• Storage: Unity Catalog Volumes attachment store and Databricks Volume filesystem connector.
• Resource management: opt-in governed Jobs, Lakeflow, AI Search, serving, UC, workspace, secret-reference, and beta Genie Agent lifecycle tools; typed clients remain usable independently of model approval policy.
• Genie Agent Mode: explicit Beta SSE client with strict event ordering, idle/overall deadlines, cancellation, event callbacks, bounded model-tool projection, and conversation-item pagination. Workspace preview enrollment is required.
• Build/deploy: databricks-app and databricks-serving targets plus fh doctor --target databricks-*; Databricks App doctor output distinguishes contract support, exact live certification, preview reachability, and unestablished claims.
• Sandbox: databricksSqlSandbox() at @fabric-harness/databricks/sql-sandbox; file operations are in memory and exec() is SQL Statement Execution, not a general compute shell.

Deployment boundaries

• databricks-app runs the Node server inside Databricks Apps and emits app.yaml plus bundle assets. An optional databricks.app.genie binding emits a genie_space resource with CAN_RUN by default and injects DATABRICKS_GENIE_AGENT_ID through valueFrom; edit/manage requires authoring: true.
• databricks-app exposes durable persistent agents at the OpenAI-compatible /responses endpoint.
• databricks-serving emits an MLflow ResponsesAgent proxy that calls the App's /responses URL. It does not run the TypeScript agent inside Model Serving.
• --mock validates the local model loop only. It does not validate Databricks API contracts, OAuth scopes, Unity Catalog grants, SQL behavior, or a deployment.

Governed resource management

Model-exposed authoring flags fail closed unless governance.stewardAudience covers every enabled service. allowUnapprovedAuthoring: true is an explicit local-development escape hatch. Structured resource descriptors validate at bundle initialization and check every catalog-qualified object. Request-scoped bundle.forPrincipal({ tokenProvider, principal }) uses a verified principal and binds approval grants to the exact call, input digest, and executing identity. Secret write tools accept only SDK SecretRef values resolved by a server-side SecretProvider.

For an analytics copilot, analyticsCopilotGovernance() keeps ordinary Genie questions and SELECT-only sql_read calls approval-free while arbitrary SQL and Genie management stay routed to the steward audience. Its SQL/Genie service scope does not weaken authoring validation: enabling another authoring service fails initialization until approval routing explicitly includes it.

Existing clusters require an explicit allowedExistingClusterIds entry, and model-facing serving tools reject raw environmentVars. Genie management is enabled by genie.manage and requires a DatabricksManagedResourceStore; use the Lakebase implementation in production. Model tools accept normalized version-2 configuration and expose SQL examples, joins, snippets, and benchmarks only when a server-side genie.manage.sqlPolicy returns every referenced catalog object. Delete is limited to exact Harness-managed resources and treats an already-trashed retry as idempotent. Raw Genie exports remain typed-client-only.

Run pnpm test:coverage:authoring for the focused approval/governance/Jobs/certification gate. Protected workspace certification can create and remove reserved-prefix resources for all management surfaces; crash leftovers are handled by scripts/sweep-databricks-authoring-certification.mjs. Jobs, Lakeflow, AI Search administration, custom-model serving, managed-only UC administration, workspace writes, secret-reference writes, and Genie Agent management require retained protected workspace evidence for the exact release candidate before promotion. Provisioned throughput and AI Gateway administration remain separate partial capabilities. Agent Mode streaming is a separately configured Beta contract and requires Databricks preview enrollment; credential-free SSE regression tests do not imply protected-workspace support.

Capability metadata separates contractClouds (typed API targets) from clouds (clouds represented by linked retained live evidence). Do not use contract portability as proof that an unlisted cloud/region passed certification.

See Databricks resource management.

Unity Catalog Agent Services

import { databricks, databricksPrincipalFromEnv } from '@fabric-harness/databricks';

const workspace = databricks({
  host: process.env.DATABRICKS_HOST!,
  principal: databricksPrincipalFromEnv(process.env),
  agentServices: { acknowledgeBeta: true },
});
const services = workspace.agentServices!;

await services.create({
  catalog: 'main',
  schema: 'agents',
  id: 'support_agent',
  connection: 'fabric_support_connection',
  basePath: '/responses',
});
await services.grant('main.agents.support_agent', 'support-users', ['EXECUTE', 'READ_METADATA']);

The current Databricks Beta supports registration and permission management, not invocation through the Agent Service. Call the external Harness /responses endpoint for runtime requests. The package requires acknowledgeBeta: true and intentionally exposes no Agent Service invocation method.

Fabric Harness build manifests separate finite jobs from persistent agents. Node-derived targets, including databricks-app, run the same v2 server used during development. Cloudflare supports finite jobs and Durable Object-backed persistent agents.

Lakebase authentication

Lakebase Autoscaling uses two-step authentication: obtain a workspace OAuth token, then exchange it at POST /api/2.0/postgres/credentials for a database credential. lakebaseClient() performs this exchange, caches the credential, refreshes early with jitter, and single-flights concurrent refreshes. databricksApp().serverOptions() wires the Lakebase session, submission, and conversation-stream stores into startDevServer().

Set the full endpoint resource name (projects/.../branches/.../endpoints/...) with DATABRICKS_LAKEBASE_ENDPOINT or ENDPOINT_NAME. Local Postgres tests can use the explicit password option. The protected workspace workflow deploys the reference App and verifies Lakebase-backed restart recovery. See the Databricks Lakebase credential guide.

Run the opt-in credential/connection smoke with FABRIC_DATABRICKS_LAKEBASE_TEST=1 plus DATABRICKS_HOST, either DATABRICKS_TOKEN or M2M client credentials, ENDPOINT_NAME, PGHOST, PGDATABASE, and PGUSER.

Vertical application control-plane seams

Vertical applications import Databricks behavior from this package instead of maintaining their own token, SQL, or discovery clients. createDatabricksAuthenticatedFetch() adapts the canonical principal to third-party SDKs with a bounded authentication retry. runStatement() uses the generated Statement Execution client. databricks(...).aiGateway discovers both system model services and governed model-provider services, while databricks(...).sqlWarehouses.list() covers the reviewed SDK gap for warehouse discovery. Credentials remain outside returned evidence and application mutation parameters.

SQL Warehouse sandbox

import { init } from '@fabric-harness/sdk';
import { databricksSqlSandbox } from '@fabric-harness/databricks/sql-sandbox';

const fabric = await init({
  sandbox: databricksSqlSandbox({
    host: process.env.DATABRICKS_HOST!,
    principal: { kind: 'pat', token: process.env.DATABRICKS_TOKEN! },
    warehouseId: process.env.DATABRICKS_WAREHOUSE_ID!,
    catalog: 'main',
    schema: 'analytics',
    resultFormat: 'jsonl',
  }),
});

const result = await (await fabric.session()).shell('SELECT current_user()');
console.log(result.stdout);

Documentation

• Fabric Harness on Databricks
• Architecture and identity flow
• Integrations
• Resource management
• Unity Catalog Agent Services
• Connectors and sandboxes
• Brickbuilder submission readiness
• Databricks deployment guide
• Capability matrix

Keep Databricks credentials in environment variables or a secret store. Never place tokens in prompts, payloads, tool inputs, or lineage labels.

Host-based applications should compose databricksGovernanceRuntimeEvidence() from the platform entrypoint into createGovernedActionHost({ runtimeEvidence }). Each invocation then records the exact provider-bridge generation beside its Host and policy generations.