SharePoint to Parquet files on Azure Blob, S3 or local disk

Last updated 26 October 2026

One way: SharePoint stays the system of record, and SPSync keeps a read-only copy of your lists current in Parquet files on Azure Blob, S3 or local disk.

Status: Shipped · product: SPSync for Warehouses & Lakes

Parquet files need no server and no running database, and nearly every analytics engine reads them. SPSync writes SharePoint lists as Parquet by keeping a DuckDB copy of the lists and exporting each table at the end of every run, so the files always match what the sync has seen.

Choosing where the files go

Parquet export is part of the DuckDB destination. Set it in the Parquet export fields of the Destination screen in the desktop app or server, in the command-line tool’s YAML config, or through an AI assistant using the admin MCP server, which sends you to a secure page for the keys and SAS token. There are two settings, and you can use either or both:

• A local folder. Each run writes {table}.parquet for every table, replacing the previous file.
• A bucket. Each run uploads a history file for that run plus a current.parquet per table that is overwritten each time. Supported targets:
  • Azure Blob Storage or ADLS Gen2, as an abfss://<container>@<account>.dfs.core.windows.net/<prefix> URL, with a SAS token, or with no token so the Azure default credential chain signs in;
  • Amazon S3 or an S3-compatible store such as MinIO, with an access key pair or the standard AWS credential chain, plus an optional region and endpoint.

Google Cloud Storage is not supported yet.

What each table looks like

A sales order list with a Customer lookup, a Rep person column and a multi-choice Channels column exports as sales_orders.parquet, with customer_id and customer_value columns and a rep_user_id column that lines up with the sp_users.parquet file. sales_orders__channels.parquet holds one row per order and channel. Parquet has no constraints, so the engine reading the files does the joining.

Every export is a full snapshot of each table’s current state, so the file is complete on its own. Deleted items stay in the snapshot with sp_deleted_at set. When SharePoint gains a column it appears in the next export, and Parquet’s schema merge lets older run files without that column read alongside newer ones.

Common uses

A data lake built from many sources gets SharePoint as one more set of files. dbt projects on Spark or DuckDB read them as a source. Teams decommissioning SharePoint on-premises keep the list data queryable after the servers go. After a merger, both tenants’ lists can sit side by side in one lake while a permanent home is chosen; see tenant migration and mergers.

Steps

1. Sign in with delegated access, or have an admin grant Sites.Selected on the site.
2. Set up a DuckDB destination and add the export folder, bucket URL or both, with the bucket’s credentials entered on the screen or held in environment variables for the YAML config.
3. Choose the lists or libraries.
4. Run. The first run loads every item and writes the first snapshot.
5. Point Spark, DuckDB, Athena, Trino or Power BI at the folder.

Every run writes a full snapshot, and a bucket keeps one history file per run. Set a retention or lifecycle rule on the bucket if you do not need every run kept. Storage and egress are billed by your cloud provider, and access control is whatever the bucket or folder already enforces.

# This destination is available now. `spsync export-config` does not take it as a
# --destination-kind value yet, so write replication.yaml by hand (see the YAML tab)
# and run it with `spsync run replication.yaml` like any other destination.

pipeline_name: parquet
prefix: parquet
auth:
  mode: delegated
  client_id: 00000000-0000-0000-0000-000000000000
  tenant_id: common
destination:
  kind: duckdb
  connection_string_env: SPSYNC_DB_CONNECTION
  dataset_name: parquet
  parquet_export_dir: ./export
  # parquet_export_bucket_url: s3://your-bucket/spsync
scopes:
  - site_url: https://contoso.sharepoint.com/sites/example
    mode: entire_site
reconcile_interval_days: 7
create_reporting_views: false

# every night at 02:00
0 2 * * * cd /opt/spsync && SPSYNC_DB_CONNECTION="$(cat /etc/spsync/db.env)" uv run spsync run replication.yaml >> /var/log/spsync/run.log 2>&1

# .github/workflows/spsync-run.yml
name: SPSync
on:
  schedule:
    - cron: "0 2 * * *"
  workflow_dispatch: {}
jobs:
  run:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: astral-sh/setup-uv@v3
      - run: uv sync
      - run: uv run spsync run replication.yaml
        env:
          SPSYNC_DB_CONNECTION: ${{ secrets.SPSYNC_DB_CONNECTION }}
          SPSYNC_LICENCE_KEY: ${{ secrets.SPSYNC_LICENCE_KEY }}

# azure-pipelines.yml
schedules:
  - cron: "0 2 * * *"
    branches: { include: [main] }
    always: true
steps:
  - script: uv sync
  - script: uv run spsync run replication.yaml
    env:
      SPSYNC_DB_CONNECTION: $(SPSYNC_DB_CONNECTION)
      SPSYNC_LICENCE_KEY: $(SPSYNC_LICENCE_KEY)

# Windows Task Scheduler action (see /docs/cli/automation/)
$env:SPSYNC_DB_CONNECTION = (Get-Content C:\spsync\db.env)
uv run spsync run C:\spsync\replication.yaml
schtasks /Create /SC DAILY /ST 02:00 /TN "SPSync" /TR "powershell -File C:\spsync\run.ps1"

# ~/Library/LaunchAgents/io.spsync.run.plist, then: launchctl load ~/Library/LaunchAgents/io.spsync.run.plist