SharePoint to Delta Lake tables on blob

Last updated 23 October 2026

One way: SharePoint stays the system of record, and SPSync keeps a read-only copy of your lists current in Delta Lake tables on blob.

Status: Shipped · product: SPSync for Warehouses & Lakes

Delta Lake adds a transaction log to Parquet files, which gives a folder of files the merges, versioning and schema checks of a table. SPSync writes SharePoint lists into Delta tables directly, one transaction log per table, with no warehouse and no database server involved.

Storage and credentials

You give SPSync a bucket URL and, where needed, the name of an environment variable that holds the storage credentials as a JSON object:

Google Cloud Storage is not supported yet. For plain Parquet files rather than Delta tables in the same stores, see SharePoint to Amazon S3 and SharePoint to Azure Blob Storage. This destination needs no connection string and no local state file, because each table’s merge state lives in its own Delta log.

How a run writes

A maintenance log with an Asset lookup, a Technician person column and a multi-choice Parts used column becomes a maint_log Delta table with asset_id, asset_value and technician_user_id, a maint_log__parts_used table with one row per entry and part, and the shared sp_users table. Text is STRING, timestamps are UTC with microsecond precision, currency is DECIMAL(38,9) and JSON-shaped values are stored as text.

Each run is a Delta merge keyed on the SharePoint item ID, committed atomically to the log. A soft delete is a Delta update that sets sp_deleted_at. New SharePoint columns are added to the table’s schema on the next merge and never dropped. A retyped column is not forked into a _v2 column on Delta Lake today (SPSync does that on SQL Server, PostgreSQL, MySQL, ClickHouse, Oracle, SQLite and DuckDB).

Reading the tables

Spark and Databricks read the tables from the storage path. DuckDB reads them with delta_scan, so an analyst on a laptop can join maint_log to sp_users directly. Delta tables have no server to host a view, so SPSync’s optional reporting views are not available here; write the same join in the engine you read with.

Typical routes in: dbt models on a lakehouse, a Data Factory pipeline or flow that wrote extracts to blob storage by hand, and an audit that uses time travel to show a list as it stood on a given date.

Steps

1. Sign in with delegated access, or have an admin register an app with Sites.Selected on the sites you need.
2. Create the bucket or container, and a credential that can read and write it.
3. Put the credentials JSON in an environment variable and set the bucket URL.
4. Choose the lists or a whole site and run; the first run creates each Delta table.
5. Point Spark, Databricks or DuckDB at the path.

The Delta log grows with every run. On a short schedule against a large table, run compaction and vacuum from Spark or Databricks from time to time; SPSync does not do it for you. Storage is billed by your cloud provider.

spsync auth login --client-id 00000000-0000-0000-0000-000000000000
spsync export-config --output replication.yaml --pipeline-name delta_lake --prefix delta_lake --destination-kind deltalake --connection-string-env SPSYNC_DB_CONNECTION --dataset-name delta_lake --client-id 00000000-0000-0000-0000-000000000000 --site-url https://contoso.sharepoint.com/sites/example --all-lists
spsync doctor replication.yaml
spsync run replication.yaml

pipeline_name: delta_lake
prefix: delta_lake
auth:
  mode: delegated
  client_id: 00000000-0000-0000-0000-000000000000
  tenant_id: common
destination:
  kind: deltalake
  connection_string_env: SPSYNC_DB_CONNECTION
  dataset_name: delta_lake
scopes:
  - site_url: https://contoso.sharepoint.com/sites/example
    mode: entire_site
reconcile_interval_days: 7
create_reporting_views: false

# every night at 02:00
0 2 * * * cd /opt/spsync && SPSYNC_DB_CONNECTION="$(cat /etc/spsync/db.env)" uv run spsync run replication.yaml >> /var/log/spsync/run.log 2>&1

# .github/workflows/spsync-run.yml
name: SPSync
on:
  schedule:
    - cron: "0 2 * * *"
  workflow_dispatch: {}
jobs:
  run:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: astral-sh/setup-uv@v3
      - run: uv sync
      - run: uv run spsync run replication.yaml
        env:
          SPSYNC_DB_CONNECTION: ${{ secrets.SPSYNC_DB_CONNECTION }}
          SPSYNC_LICENCE_KEY: ${{ secrets.SPSYNC_LICENCE_KEY }}

# azure-pipelines.yml
schedules:
  - cron: "0 2 * * *"
    branches: { include: [main] }
    always: true
steps:
  - script: uv sync
  - script: uv run spsync run replication.yaml
    env:
      SPSYNC_DB_CONNECTION: $(SPSYNC_DB_CONNECTION)
      SPSYNC_LICENCE_KEY: $(SPSYNC_LICENCE_KEY)

# Windows Task Scheduler action (see /docs/cli/automation/)
$env:SPSYNC_DB_CONNECTION = (Get-Content C:\spsync\db.env)
uv run spsync run C:\spsync\replication.yaml
schtasks /Create /SC DAILY /ST 02:00 /TN "SPSync" /TR "powershell -File C:\spsync\run.ps1"

# ~/Library/LaunchAgents/io.spsync.run.plist, then: launchctl load ~/Library/LaunchAgents/io.spsync.run.plist