SharePoint to Azure Data Lake Storage Gen2

Last updated 23 October 2026

One way: SharePoint stays the system of record, and SPSync keeps a read-only copy of your lists current in Azure Data Lake Storage Gen2.

Status: Shipped · product: SPSync for Warehouses & Lakes

Sometimes SharePoint data only needs to exist as files in the lake: no database to run and no warehouse to pay for, just Parquet in a container that Spark, Fabric, Synapse, Power BI and DuckDB already know how to read. The ADLS Gen2 destination does that, and it often sits as the raw layer under a Databricks job or a Fabric Lakehouse shortcut.

Container and sign-in

Point SPSync at a container with an abfss:// URL: abfss://<container>@<account>.dfs.core.windows.net/<prefix>. Two ways to authenticate:

• SAS token. Put a token with read and write rights in an environment variable and name it in the destination settings. Nothing else on the sync host needs Azure access.
• Azure default credential chain. Leave the token out and SPSync signs in as whatever identity the host already has, such as the identity of an Azure VM or an az login session.

SPSync uploads through the account’s Blob endpoint, so a standard Blob account may also work; that has been tested against the Azurite emulator only, not a live account. SharePoint to Azure Blob Storage covers the DuckDB export route to a Blob container.

SPSync keeps its merged copy of each list in a local DuckDB state file on the sync host and uploads Parquet from it after each run. Give that file a stable path; losing it means the next run starts again with a full load.

Folder layout

Each table gets a folder in the container, and each run uploads two files into it:

• {table}/{run_id}.parquet, one file per run that is never changed afterwards, giving a history of every run;
• {table}/current.parquet, the latest snapshot, replaced every run.

A help desk list with a Category lookup, an Assignee person column and a multi-choice Affected systems column produces it_tickets/, it_tickets__affected_systems/ and the shared sp_users/. Lookups arrive as category_id and category_value, people as assignee_user_id. Every upload is a full snapshot of the table, deleted items included with sp_deleted_at set.

A new SharePoint column appears in later files, and earlier files keep the columns they had. A retyped column is not forked into a _v2 column by the Azure Data Lake Storage Gen2 destination today. Parquet exported from a DuckDB destination to Azure does get a _v2 column (SPSync forks retyped columns on SQL Server, PostgreSQL, MySQL, ClickHouse, Oracle, SQLite and DuckDB).

Who reads the files

Data engineers swap a Data Factory pipeline or flow that extracted lists by hand for a scheduled replication. A data lake across many sources gets SharePoint as one more set of folders. An analyst reads current.parquet straight into DuckDB. A Power BI author imports from the folders with the Azure Data Lake Storage connector.

Steps

1. Sign in with delegated access, or have an admin register an app with Sites.Selected on the sites you need.
2. Create or choose a storage account with hierarchical namespace enabled, and a container.
3. Issue a SAS token, or give the sync host’s identity rights on the container.
4. Pick the lists, or the whole site, and start the first run.
5. Point Spark, DuckDB, Fabric or Power BI at the folders.

Every run adds one history file per table. On a short schedule over many tables that adds up, so set a lifecycle rule on the container if you do not need every run kept. Storage capacity and transactions are billed by Microsoft.

spsync auth login --client-id 00000000-0000-0000-0000-000000000000
spsync export-config --output replication.yaml --pipeline-name azure_data_lake --prefix azure_data_lake --destination-kind adls-gen2 --connection-string-env SPSYNC_DB_CONNECTION --dataset-name azure_data_lake --client-id 00000000-0000-0000-0000-000000000000 --site-url https://contoso.sharepoint.com/sites/example --all-lists
spsync doctor replication.yaml
spsync run replication.yaml

pipeline_name: azure_data_lake
prefix: azure_data_lake
auth:
  mode: delegated
  client_id: 00000000-0000-0000-0000-000000000000
  tenant_id: common
destination:
  kind: adls-gen2
  connection_string_env: SPSYNC_DB_CONNECTION
  dataset_name: azure_data_lake
scopes:
  - site_url: https://contoso.sharepoint.com/sites/example
    mode: entire_site
reconcile_interval_days: 7
create_reporting_views: false

# every night at 02:00
0 2 * * * cd /opt/spsync && SPSYNC_DB_CONNECTION="$(cat /etc/spsync/db.env)" uv run spsync run replication.yaml >> /var/log/spsync/run.log 2>&1

# .github/workflows/spsync-run.yml
name: SPSync
on:
  schedule:
    - cron: "0 2 * * *"
  workflow_dispatch: {}
jobs:
  run:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: astral-sh/setup-uv@v3
      - run: uv sync
      - run: uv run spsync run replication.yaml
        env:
          SPSYNC_DB_CONNECTION: ${{ secrets.SPSYNC_DB_CONNECTION }}
          SPSYNC_LICENCE_KEY: ${{ secrets.SPSYNC_LICENCE_KEY }}

# azure-pipelines.yml
schedules:
  - cron: "0 2 * * *"
    branches: { include: [main] }
    always: true
steps:
  - script: uv sync
  - script: uv run spsync run replication.yaml
    env:
      SPSYNC_DB_CONNECTION: $(SPSYNC_DB_CONNECTION)
      SPSYNC_LICENCE_KEY: $(SPSYNC_LICENCE_KEY)

# Windows Task Scheduler action (see /docs/cli/automation/)
$env:SPSYNC_DB_CONNECTION = (Get-Content C:\spsync\db.env)
uv run spsync run C:\spsync\replication.yaml
schtasks /Create /SC DAILY /ST 02:00 /TN "SPSync" /TR "powershell -File C:\spsync\run.ps1"

# ~/Library/LaunchAgents/io.spsync.run.plist, then: launchctl load ~/Library/LaunchAgents/io.spsync.run.plist