SharePoint to Databricks (Delta, Unity Catalog)

Last updated 23 October 2026

Status: Shipped · product: SPSync for Warehouses & Lakes

SPSync copies SharePoint lists one way into Databricks, and SharePoint stays the system of record. A lakehouse usually takes in a dozen sources through ingestion jobs that someone maintains. SPSync makes SharePoint one of them without a job to write: list data arrives as Delta tables in a Unity Catalog schema, governed and discoverable like everything else, and Spark, Databricks SQL and dbt read it straight away.

Does the Databricks Lakeflow SharePoint connector ingest lists?

Its main job is files. The SharePoint connector in Lakeflow Connect takes documents from a site as binary content, parses CSV, JSON, XML or Excel files into rows, or records file metadata only. List ingestion is a newer Beta feature: one list, or every list in a site, becomes a Delta table. The Beta limits exclude list attachments and lists in OneDrive, and the pipeline is defined through the API or a bundle because the UI does not offer lists. In the column mapping, a single lookup or person column is one string, and multi-value lookups and people are arrays of structs inside the row. We read Databricks’s documentation on 22 October 2026; the connector changes often, so check it again before you choose. Our post on what Openflow and Databricks Lakeflow do with SharePoint lists sets the Lakeflow Beta beside Snowflake’s file-only connector.

The native connector is built for files. SPSync replicates list items, as a shipped destination: lookups keep their key as a column you can join, and each multi-value column gets a child table. Items deleted in SharePoint stay in the table with sp_deleted_at set. If you also want the documents, Lakeflow’s file ingestion and SPSync’s list tables can sit side by side in the same catalog.

Connecting to a SQL warehouse

SPSync connects through a Databricks SQL warehouse, with a connection string held in an environment variable. It has two forms:

• Personal access token: databricks://token:<PAT>@<workspace-host>/<http-path>. The user name is literally token, the Databricks JDBC and ODBC convention.
• Service principal: databricks://<client-id>:<client-secret>@<workspace-host>/<http-path>. SPSync exchanges the pair for OAuth tokens and refreshes them itself.

The <http-path> is the SQL warehouse’s HTTP path from its connection details. You also set the catalog explicitly, since a workspace’s default catalog is an admin setting SPSync cannot see; the destination’s dataset name is the schema inside it.

Loading and merging

Databricks cannot insert a local file directly, so each run uploads its batch to a Unity Catalog volume SPSync creates in the target schema, then copies it into a staging table. No S3 or ADLS staging bucket is needed. Every list table is then updated with one atomic MERGE INTO per table, keyed on the SharePoint item ID.

A merge is atomic per table, not across tables. If a run fails half way, one list’s table may be current while another is not, and the next run completes the rest.

The tables

An expenses list with a Cost centre lookup, a Claimant person column and a multi-choice Receipt types column becomes fin_expenses, with cost_centre_id and cost_centre_value, a claimant_user_id that joins to sp_users, and a fin_expenses__receipt_types child table. Text is STRING, timestamps are UTC TIMESTAMP, and JSON-shaped values are STRING that Spark’s JSON functions read. Delta does not enforce the joins, so rows can load in any order. Delta’s time travel keeps earlier versions of each table queryable.

Who reads it

Data engineers add the tables to dbt models beside other sources. Data scientists read typed rows from a notebook when preparing training or evaluation data. Integrations that need reference data for an ERP or CRM read the governed tables. BI authors point Power BI or Tableau at a SQL warehouse.

Things to plan for

• A serverless or auto-stopping warehouse can take tens of seconds to start after idling. A very short sync interval against one will spend most of each run waiting for it.
• A principal granted table privileges but not the volume privileges fails on the first upload.
• A column renamed in SharePoint keeps its internal name, so the Delta column keeps its name too.

Steps

1. Sign in to SharePoint as yourself to start, then have an admin grant Sites.Selected on each site once the sync should run on its own.
2. Create or choose the catalog and schema, and grant the principal the privileges above.
3. Store the connection string in an environment variable and set the catalog.
4. Pick lists or a whole site and start a run.
5. Query the schema from a notebook, a SQL warehouse or dbt.

Databricks bills the warehouse compute each run uses and the storage, separately from SPSync.

spsync auth login --client-id 00000000-0000-0000-0000-000000000000
spsync export-config --output replication.yaml --pipeline-name databricks --prefix databricks --destination-kind databricks --connection-string-env SPSYNC_DB_CONNECTION --dataset-name databricks --client-id 00000000-0000-0000-0000-000000000000 --site-url https://contoso.sharepoint.com/sites/example --all-lists
spsync doctor replication.yaml
spsync run replication.yaml

pipeline_name: databricks
prefix: databricks
auth:
  mode: delegated
  client_id: 00000000-0000-0000-0000-000000000000
  tenant_id: common
destination:
  kind: databricks
  connection_string_env: SPSYNC_DB_CONNECTION
  dataset_name: databricks
scopes:
  - site_url: https://contoso.sharepoint.com/sites/example
    mode: entire_site
reconcile_interval_days: 7
create_reporting_views: false

# every night at 02:00
0 2 * * * cd /opt/spsync && SPSYNC_DB_CONNECTION="$(cat /etc/spsync/db.env)" uv run spsync run replication.yaml >> /var/log/spsync/run.log 2>&1

# .github/workflows/spsync-run.yml
name: SPSync
on:
  schedule:
    - cron: "0 2 * * *"
  workflow_dispatch: {}
jobs:
  run:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: astral-sh/setup-uv@v3
      - run: uv sync
      - run: uv run spsync run replication.yaml
        env:
          SPSYNC_DB_CONNECTION: ${{ secrets.SPSYNC_DB_CONNECTION }}
          SPSYNC_LICENCE_KEY: ${{ secrets.SPSYNC_LICENCE_KEY }}

# azure-pipelines.yml
schedules:
  - cron: "0 2 * * *"
    branches: { include: [main] }
    always: true
steps:
  - script: uv sync
  - script: uv run spsync run replication.yaml
    env:
      SPSYNC_DB_CONNECTION: $(SPSYNC_DB_CONNECTION)
      SPSYNC_LICENCE_KEY: $(SPSYNC_LICENCE_KEY)

# Windows Task Scheduler action (see /docs/cli/automation/)
$env:SPSYNC_DB_CONNECTION = (Get-Content C:\spsync\db.env)
uv run spsync run C:\spsync\replication.yaml
schtasks /Create /SC DAILY /ST 02:00 /TN "SPSync" /TR "powershell -File C:\spsync\run.ps1"

# ~/Library/LaunchAgents/io.spsync.run.plist, then: launchctl load ~/Library/LaunchAgents/io.spsync.run.plist