Parquet, Iceberg and Delta in the lake, ready for Spark and DuckDB

Last updated 20 October 2026

SharePoint list data never makes it into the data lake because there is no straightforward path from Graph API rows to Parquet files a lake engine can read.

Lakehouses ingest the ERP, the web analytics events and the CRM exports, and SharePoint lists are usually the gap. There is no easy path from list rows behind the Graph API to files a lake engine can read, so project approvals and site reference lists never reach the lake, and every analyst who needs them falls back to a manual export, a spreadsheet and a join done by hand.

Lists straight into lake formats

SPSync writes SharePoint lists as Parquet files, Iceberg tables or Delta Lake tables, each with its own destination page, and DuckDB covers the single-file case. All of them ship today. The first run is a full load and later runs read only what changed in SharePoint, with no bespoke ingestion job to maintain. Delta tables are merged in place; Iceberg tables and Parquet exports are rewritten as a full snapshot each run, which keeps every file complete on its own. Spark, Trino, DuckDB and other lake engines query the tables with no separate loading step.

What the lake receives

Each list becomes a table with the source’s lookup, choice and person columns; managed metadata columns are not replicated yet. A lookup is a key column that points at the table for the target list. A multi-value column is its own child table, so a lake query joins on it and never parses a delimited string. Deleted items stay in the data with sp_deleted_at set, which fits lake formats built around auditable history.

Fitting the lake’s layout

Choose the format the lake already uses, and set the output location to the path its engine reads from. Run the first full load, check it from the engine you will query with, then set the schedule for later runs. If Airflow, Dagster or cron already orchestrates the lake, run SPSync from the command line as one scheduled step that writes into it. For local analysis with DuckDB, the desktop app can write the file to your own machine. SPSync never writes to SharePoint, so bringing list data into the lake puts the source lists at no risk.

spsync auth login --client-id 00000000-0000-0000-0000-000000000000
spsync export-config --output replication.yaml --pipeline-name sharepoint_data_lake --prefix sharepoint_data_lake --destination-kind duckdb --connection-string-env SPSYNC_DB_CONNECTION --dataset-name sharepoint_data_lake --client-id 00000000-0000-0000-0000-000000000000 --site-url https://contoso.sharepoint.com/sites/example --all-lists
spsync doctor replication.yaml
spsync run replication.yaml

pipeline_name: sharepoint_data_lake
prefix: sharepoint_data_lake
auth:
  mode: delegated
  client_id: 00000000-0000-0000-0000-000000000000
  tenant_id: common
destination:
  kind: duckdb
  connection_string_env: SPSYNC_DB_CONNECTION
  dataset_name: sharepoint_data_lake
scopes:
  - site_url: https://contoso.sharepoint.com/sites/example
    mode: entire_site
reconcile_interval_days: 7
create_reporting_views: false

# every night at 02:00
0 2 * * * cd /opt/spsync && SPSYNC_DB_CONNECTION="$(cat /etc/spsync/db.env)" uv run spsync run replication.yaml >> /var/log/spsync/run.log 2>&1

# .github/workflows/spsync-run.yml
name: SPSync
on:
  schedule:
    - cron: "0 2 * * *"
  workflow_dispatch: {}
jobs:
  run:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: astral-sh/setup-uv@v3
      - run: uv sync
      - run: uv run spsync run replication.yaml
        env:
          SPSYNC_DB_CONNECTION: ${{ secrets.SPSYNC_DB_CONNECTION }}
          SPSYNC_LICENCE_KEY: ${{ secrets.SPSYNC_LICENCE_KEY }}

# azure-pipelines.yml
schedules:
  - cron: "0 2 * * *"
    branches: { include: [main] }
    always: true
steps:
  - script: uv sync
  - script: uv run spsync run replication.yaml
    env:
      SPSYNC_DB_CONNECTION: $(SPSYNC_DB_CONNECTION)
      SPSYNC_LICENCE_KEY: $(SPSYNC_LICENCE_KEY)

# Windows Task Scheduler action (see /docs/cli/automation/)
$env:SPSYNC_DB_CONNECTION = (Get-Content C:\spsync\db.env)
uv run spsync run C:\spsync\replication.yaml
schtasks /Create /SC DAILY /ST 02:00 /TN "SPSync" /TR "powershell -File C:\spsync\run.ps1"

# ~/Library/LaunchAgents/io.spsync.run.plist, then: launchctl load ~/Library/LaunchAgents/io.spsync.run.plist