SharePoint to Apache Iceberg tables

Last updated 26 October 2026

One way: SharePoint stays the system of record, and SPSync keeps a read-only copy of your lists current in Apache Iceberg tables.

Status: Shipped · product: SPSync for Warehouses & Lakes

Iceberg is an open table format that many engines can read through one catalog. If your lakehouse already runs Spark, Trino, Athena or DuckDB against Iceberg tables, SPSync adds SharePoint lists to that catalog as more tables, with no separate copy for each engine.

The catalog connection

SPSync writes through an Iceberg REST catalog. It needs:

• the catalog URI, such as https://catalog.example.com;
• the warehouse location the catalog is configured with, such as s3://bucket/prefix;
• a namespace for the tables, created on first use if it is missing;
• optionally, a bearer token for the catalog, or OAuth2 client credentials (a client ID and secret, with the token endpoint and scope if the catalog needs them), but not both;
• optionally, S3 access keys plus an endpoint when the warehouse is MinIO or another S3-compatible store. With no keys, SPSync uses the AWS credential chain on the machine it runs on.

All of these can be set in the desktop app or server Destination screen, in the command-line tool’s YAML config, or through an AI assistant using the admin MCP server, which sends you to a secure page for the token, client secret and keys.

If the bucket only needs files, with no catalog, the Parquet export writes straight to Amazon S3 or Azure Blob Storage instead. Secrets go in environment variables. SPSync also keeps a local state file between runs, which needs a stable path on the sync host.

Full snapshots, every run

SPSync reads SharePoint with delta queries and keeps its own merged copy of each list. After every run it overwrites each Iceberg table with that copy’s full current state, as one new snapshot. Reading the latest snapshot always gives a complete, consistent table, and the snapshot history doubles as a record of each run. The cost is write volume: moderate lists on a schedule of minutes are fine, while a very large list on a very short interval rewrites a lot of data.

Table contents

A supplier audit list with a Supplier lookup, an Auditor person column and a multi-choice Findings column becomes qa_supplier_audits, with supplier_id, supplier_value and auditor_user_id, a qa_supplier_audits__findings table and the shared sp_users table. Text and IDs are string, whole numbers keep separate int and long types, currency is decimal(38,9) and timestamps are timestamptz.

Columns added in SharePoint are added to the table on the next run. Changing a column’s type on Iceberg is not supported yet, and a retyped column is not forked into a _v2 column (SPSync does that on SQL Server, PostgreSQL, MySQL, ClickHouse, Oracle, SQLite and DuckDB). If it happens, the run fails early with an error naming the list, the column and both types, and nothing is written to Iceberg.

Where it fits

dbt projects across a lakehouse read the tables from whichever engine they already use. A Data Factory or flow-based extract into the lake can be retired. A data lake built across many sources lets Trino serve ad hoc questions while Spark runs batch jobs on the same tables.

Steps

1. Sign in to SharePoint with delegated access; for unattended runs, an admin approves Sites.Selected on the sites you replicate.
2. Gather the catalog URI, warehouse and namespace, and any token or storage keys.
3. Choose the lists or a whole site and run; the first run creates each table.
4. Query the namespace from Spark, Trino, Athena or DuckDB.

Snapshots and metadata files accumulate with every run, so schedule snapshot expiry and compaction in your engine of choice; SPSync does not run them.

spsync auth login --client-id 00000000-0000-0000-0000-000000000000
spsync export-config --output replication.yaml --pipeline-name iceberg --prefix iceberg --destination-kind iceberg --connection-string-env SPSYNC_DB_CONNECTION --dataset-name iceberg --client-id 00000000-0000-0000-0000-000000000000 --site-url https://contoso.sharepoint.com/sites/example --all-lists
spsync doctor replication.yaml
spsync run replication.yaml

pipeline_name: iceberg
prefix: iceberg
auth:
  mode: delegated
  client_id: 00000000-0000-0000-0000-000000000000
  tenant_id: common
destination:
  kind: iceberg
  connection_string_env: SPSYNC_DB_CONNECTION
  dataset_name: iceberg
scopes:
  - site_url: https://contoso.sharepoint.com/sites/example
    mode: entire_site
reconcile_interval_days: 7
create_reporting_views: false

# every night at 02:00
0 2 * * * cd /opt/spsync && SPSYNC_DB_CONNECTION="$(cat /etc/spsync/db.env)" uv run spsync run replication.yaml >> /var/log/spsync/run.log 2>&1

# .github/workflows/spsync-run.yml
name: SPSync
on:
  schedule:
    - cron: "0 2 * * *"
  workflow_dispatch: {}
jobs:
  run:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: astral-sh/setup-uv@v3
      - run: uv sync
      - run: uv run spsync run replication.yaml
        env:
          SPSYNC_DB_CONNECTION: ${{ secrets.SPSYNC_DB_CONNECTION }}
          SPSYNC_LICENCE_KEY: ${{ secrets.SPSYNC_LICENCE_KEY }}

# azure-pipelines.yml
schedules:
  - cron: "0 2 * * *"
    branches: { include: [main] }
    always: true
steps:
  - script: uv sync
  - script: uv run spsync run replication.yaml
    env:
      SPSYNC_DB_CONNECTION: $(SPSYNC_DB_CONNECTION)
      SPSYNC_LICENCE_KEY: $(SPSYNC_LICENCE_KEY)

# Windows Task Scheduler action (see /docs/cli/automation/)
$env:SPSYNC_DB_CONNECTION = (Get-Content C:\spsync\db.env)
uv run spsync run C:\spsync\replication.yaml
schtasks /Create /SC DAILY /ST 02:00 /TN "SPSync" /TR "powershell -File C:\spsync\run.ps1"

# ~/Library/LaunchAgents/io.spsync.run.plist, then: launchctl load ~/Library/LaunchAgents/io.spsync.run.plist