SharePoint to Azure Blob Storage (Parquet)

Last updated 23 October 2026

Status: Shipped · product: SPSync for Warehouses & Lakes

Raw data for the reporting platform lives in a Blob container, and every source arrives there on a schedule except one: the SharePoint list the facilities team uses to log site inspections. It reaches the container as a CSV someone saves from Excel when they remember. SPSync takes that job over. It reads the list through Microsoft Graph, keeps a merged copy on the sync host, and uploads every table to your container as Parquet after each run. The copy goes one way, and SharePoint stays the system of record.

How do I copy a SharePoint list to Azure Blob Storage?

Pick one of two shipped destinations. Both write the same files in the same layout:

• The DuckDB destination’s Parquet export, with a bucket URL pointing at the container. It is set in the command-line tool’s YAML config today, and the same replication can also keep a local DuckDB file or a local export folder.
• The Azure Data Lake Storage Gen2 destination, which the desktop app and SPSync in your Azure offer as well as the command line. It also accepts a full storage connection string, intended for testing.

If you want tables with a transaction log rather than files, the Delta Lake destination merges each run into Delta tables in a Blob container. The Parquet page covers the export to a local folder and to Amazon S3.

Container URL and sign-in

Give SPSync the container as abfss://<container>@<account>.dfs.core.windows.net/<prefix>. SPSync takes the account name from that address and uploads through the account’s Blob endpoint, so a standard Blob account should work as well as one with hierarchical namespace switched on (Data Lake Storage hierarchical namespace), though the flat-namespace case has not been tried on a live account. SPSync’s automated tests run this upload against the Azurite emulator; it has not been run against a live Azure storage account, so try one table first.

Two ways to sign in:

• A SAS token with read, write and list rights, in an environment variable whose name goes in the config. Microsoft’s SAS overview covers how to scope one to a single container.
• No token at all, in which case the Azure default credential chain signs in as whatever identity the sync host has, such as an Azure VM’s managed identity or an az login session. Give that identity a data role on the container, for example Storage Blob Data Contributor (assign a role for blob data access).

spsync doctor checks that it can reach the container before the first run.

Files in the container

Each table gets a folder under the prefix, and every run uploads two blobs into it:

• <prefix>/<table>/<run_id>.parquet, the full table as it stood after that run, never changed afterwards;
• <prefix>/<table>/current.parquet, the latest snapshot, replaced every run.

Run IDs are random, so order the history by each blob’s last-modified time.

The site inspections list, with a Site lookup, an Inspector person column and a multi-choice Hazards found column, becomes site_inspections/, site_inspections__hazards_found/ and the shared sp_users/. The lookup arrives as site_id and site_value, the person as inspector_user_id, and each hazard as its own row with the parent item’s ID. When someone deletes an inspection in SharePoint, its row remains in later snapshots and sp_deleted_at records when it went.

A column added in SharePoint shows up in later files. On the DuckDB export, a retyped column also gets a _v2 column; the Azure Data Lake Storage Gen2 destination does not fork a retyped column today.

Which tools read Parquet from a Blob container?

Synapse serverless SQL reads a file in place: OPENROWSET with the URL of a table’s current.parquet returns the latest snapshot, with no load step (query Parquet files with serverless SQL pool). Spark and Databricks read the same path, and DuckDB’s azure extension reads it from a laptop. Power BI connects with the Azure Blob Storage connector, which lists the container’s files, and Power Query opens the Parquet file from there.

When an engine reads a whole table folder rather than one file, it gets the current snapshot and every history file together. Point it at current.parquet for the latest state, and at the history files only when you want to compare runs.

Steps

1. Sign in with delegated access, or have an admin grant Sites.Selected on the sites you replicate.
2. Create a storage account and container, or pick a prefix in an existing one.
3. Issue a SAS token, or give the sync host’s identity a data role on the container.
4. Set up the destination: the bucket URL on a DuckDB destination in replication.yaml, or an Azure Data Lake Storage Gen2 destination in the desktop app.
5. Choose the lists, run spsync doctor from the command line, then start the first run. It loads every item and writes the first snapshot.
6. Point Synapse, Spark, DuckDB or Power BI at the table folders.

Each run adds a history file per table. A lifecycle management rule on the container can delete blobs a set number of days after they were last modified (lifecycle policy structure). Every run rewrites current.parquet, so a rule like that removes old history files and leaves the current snapshot in place while runs continue. Storage capacity and transactions are billed by Microsoft.

# This destination is available now. `spsync export-config` does not take it as a
# --destination-kind value yet, so write replication.yaml by hand (see the YAML tab)
# and run it with `spsync run replication.yaml` like any other destination.

pipeline_name: azure_blob_storage
prefix: azure_blob_storage
auth:
  mode: delegated
  client_id: 00000000-0000-0000-0000-000000000000
  tenant_id: common
destination:
  kind: duckdb
  connection_string_env: SPSYNC_DB_CONNECTION
  dataset_name: azure_blob_storage
  parquet_export_bucket_url: abfss://<container>@<account>.dfs.core.windows.net/spsync
  # Leave this out to sign in with the Azure default credential chain instead.
  parquet_export_sas_token_env: AZURE_STORAGE_SAS_TOKEN
scopes:
  - site_url: https://contoso.sharepoint.com/sites/example
    mode: entire_site
reconcile_interval_days: 7
create_reporting_views: false

# every night at 02:00
0 2 * * * cd /opt/spsync && SPSYNC_DB_CONNECTION="$(cat /etc/spsync/db.env)" uv run spsync run replication.yaml >> /var/log/spsync/run.log 2>&1

# .github/workflows/spsync-run.yml
name: SPSync
on:
  schedule:
    - cron: "0 2 * * *"
  workflow_dispatch: {}
jobs:
  run:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: astral-sh/setup-uv@v3
      - run: uv sync
      - run: uv run spsync run replication.yaml
        env:
          SPSYNC_DB_CONNECTION: ${{ secrets.SPSYNC_DB_CONNECTION }}
          SPSYNC_LICENCE_KEY: ${{ secrets.SPSYNC_LICENCE_KEY }}

# azure-pipelines.yml
schedules:
  - cron: "0 2 * * *"
    branches: { include: [main] }
    always: true
steps:
  - script: uv sync
  - script: uv run spsync run replication.yaml
    env:
      SPSYNC_DB_CONNECTION: $(SPSYNC_DB_CONNECTION)
      SPSYNC_LICENCE_KEY: $(SPSYNC_LICENCE_KEY)

# Windows Task Scheduler action (see /docs/cli/automation/)
$env:SPSYNC_DB_CONNECTION = (Get-Content C:\spsync\db.env)
uv run spsync run C:\spsync\replication.yaml
schtasks /Create /SC DAILY /ST 02:00 /TN "SPSync" /TR "powershell -File C:\spsync\run.ps1"

# ~/Library/LaunchAgents/io.spsync.run.plist, then: launchctl load ~/Library/LaunchAgents/io.spsync.run.plist