Connect SPSync to Databricks

This guide covers Databricks with Unity Catalog on Azure, AWS or Google Cloud. SPSync uses one destination kind, databricks, and writes through a SQL warehouse.

Before you start

A Databricks workspace with Unity Catalog and a running SQL warehouse. A stopped warehouse starts on the first statement, so the first test can be slow.
Nothing to install on the machine that runs SPSync.
Rights to create a schema and grant privileges in the catalog.
From the warehouse’s Connection details tab: the server host name and the HTTP path, which looks like /sql/1.0/warehouses/<id>.

Create the database and a least-privilege user

Every table is addressed as catalog.schema.table. databricks_catalog is the catalog and dataset_name is the schema inside it. Create the schema and give the principal SPSync signs in as the privileges below. The script is from the Databricks documentation and has not been run here against a workspace.

CREATE SCHEMA IF NOT EXISTS main.sharepoint_sync;

GRANT USE CATALOG ON CATALOG main TO `spsync-loader`;
GRANT USE SCHEMA, CREATE TABLE, SELECT, MODIFY,
      CREATE VOLUME, READ VOLUME, WRITE VOLUME
  ON SCHEMA main.sharepoint_sync TO `spsync-loader`;

Why each one:

CREATE TABLE, SELECT and MODIFY create the tables and run the merge.
The three volume privileges are needed because SPSync loads through a Unity Catalog volume it creates in the same schema, not through a storage bucket you configure.
CREATE SCHEMA on the catalog is needed only if you leave schema creation to SPSync.

The principal needs CAN USE on the SQL warehouse. For a service principal, add it to the workspace first. SPSync does not touch anything outside that one schema.

Allow SPSync to reach it

The workspace host name must be reachable over HTTPS (port 443) from the machine that runs SPSync. If the workspace restricts access with an IP access list, add the SPSync address to it. If it uses Private Link or has public access turned off, SPSync must run on a network that can reach the private endpoint. The IP access list and Private Link cases have not been tried here.

Encrypt the connection

The connection to the workspace is over HTTPS, with no setting to change.

Authentication SPSync supports today

Two forms, both in the connection string:

Personal access token: databricks://token:<PAT>@<workspace-host>/<http-path>. The user name is literally token.
OAuth machine-to-machine with a service principal: databricks://<client-id>:<client-secret>@<workspace-host>/<http-path>. Any user name other than token is read as the service principal’s application (client) id. SPSync gets and refreshes the access token itself.

A token or secret containing @, / or : must be percent-encoded. Prefer the service principal over a personal token: it is not tied to a person who may leave.

SPSync accepts nothing else for Databricks: there is no sign-in as a person and no Microsoft Entra ID token path.

Connect from SPSync

<workspace-host> is the host name only, such as adb-1234567890123456.7.azuredatabricks.net or dbc-xxxx.cloud.databricks.com.

databricks://token:<PAT>@<workspace-host>/sql/1.0/warehouses/<warehouse-id>

Desktop app: open the Destination screen, choose Databricks (Delta, Unity Catalog), paste the string into Connection string, enter the catalog in Unity Catalog, and choose Test connection.

CLI:

export SPSYNC_DATABRICKS_DSN='databricks://token:<PAT>@<workspace-host>/sql/1.0/warehouses/<warehouse-id>'
spsync export-config --output replication.yaml \
  --site-url https://contoso.sharepoint.com/sites/x --list Projects \
  --pipeline-name spsync_dev --prefix dev \
  --destination-kind databricks --connection-string-env SPSYNC_DATABRICKS_DSN \
  --databricks-catalog main --dataset-name sharepoint_sync \
  --client-id <entra-app-client-id>

AI assistant: with the admin MCP server connected, ask for the destination requirements, then let the assistant create the destination with the catalog name. You enter the string on the secure entry page.

What Test connection and doctor check

spsync doctor (destination-connectivity) opens a session on the SQL warehouse, runs SELECT 1, then SHOW SCHEMAS IN <catalog>. A pass proves the host, HTTP path, credential, a running warehouse and USE CATALOG on the catalog. It does not prove the schema grants above or the volume privileges; a missing one shows up on the first load.

SPSync’s live test against a real Databricks workspace is not complete, so run one small list first.

Common errors

What you see	What it means and the fix
`Invalid access token` or `401`	The token expired or was revoked, or the user name is not `token`. Create a new one.
`PERMISSION_DENIED` on the warehouse	The principal has no `CAN USE` on the SQL warehouse. Grant it.
`CATALOG_NOT_FOUND` or `USE CATALOG` denied	The catalog name is wrong, or the principal has no `USE CATALOG`.
`PERMISSION_DENIED ... CREATE VOLUME`	The volume privileges are missing on the schema. Run the grants above.
Test connection hangs for a minute	The warehouse was stopped and is starting. Run it again.
`Could not parse` the connection string	A character in the secret needs percent-encoding, or the HTTP path is missing its leading `/sql/`.