SharePoint to Amazon S3 (Parquet)

Last updated 26 October 2026

Status: Shipped · product: SPSync for Warehouses & Lakes

The analytics team queries everything from Athena, and the one source missing from the bucket is the SharePoint list that tracks supplier onboarding. Someone exports it to Excel at month end, converts it, and uploads a file that is out of date by the next morning. SPSync replaces that chore: it reads the list through Microsoft Graph on a schedule and writes each table to your bucket as Parquet after every run. The copy goes one way, and SharePoint stays the system of record.

How do I get a SharePoint list into an S3 bucket?

S3 is a target of the Parquet export that comes with the DuckDB destination. SPSync keeps a DuckDB copy of the lists on the sync host, merges each run’s changes into it, and then uploads every table to the bucket. You set it in the Destination screen of the desktop app or server (the Parquet export fields), in the command-line tool’s YAML config, or through an AI assistant using the admin MCP server, which sends you to a secure page for the keys.

Two other destinations also write to S3, if you would rather have a table format than plain files:

• Delta Lake merges changes into Delta tables on s3://bucket/prefix, with a transaction log and time travel.
• Apache Iceberg writes through a REST catalog whose warehouse can sit on S3.

The rest of this page covers the Parquet route. The Parquet page covers the same export for a local folder.

Bucket, region and credentials

The bucket is a URL of the form s3://<bucket>/<prefix>. Everything SPSync writes goes under that prefix, so one bucket can hold several replications side by side.

Credentials come from one of two places:

• an access key ID and secret access key, plus a session token for temporary credentials, entered in the Destination screen or held in environment variables whose names go in the config;
• the standard AWS credential chain, when you leave those out. That covers the usual environment variables, a shared credentials file, and an EC2 instance or ECS task role.

Set the region when the bucket is outside the default region your AWS configuration resolves. For MinIO or another S3-compatible store, also set the endpoint URL. The identity needs to list the bucket and write objects under the prefix; spsync doctor lists the bucket to check the credentials before the first run.

SPSync’s automated tests run this upload against MinIO; it has not been run against a live AWS account, so try one table first.

Table folders and file names

Each table gets its own folder under the prefix, and each run writes two objects into it:

• <prefix>/<table>/<run_id>.parquet, a full snapshot for that run, never changed afterwards;
• <prefix>/<table>/current.parquet, the latest snapshot, replaced every run.

The run ID is a random identifier, so sort history files by their last-modified time rather than by name.

A supplier onboarding list with a Category lookup, an Owner person column and a multi-choice Documents received column produces supplier_onboarding/, supplier_onboarding__documents_received/ and the shared sp_users/. The lookup arrives as category_id and category_value, the person as owner_user_id, and the multi-choice column as one row per item and value in its own folder. An item deleted in SharePoint keeps its row, with sp_deleted_at filled in.

New SharePoint columns appear in the next run’s files, and older history files keep the columns they had. When a column changes type, the files get a _v2 column alongside the original, the same as the DuckDB copy behind them.

Can Athena and Redshift Spectrum query the files?

Yes, with one thing to watch. An external table in Athena or Redshift Spectrum points at a folder and reads every file in it, so a table over supplier_onboarding/ returns the current snapshot and every history file together. Both engines expose the file each row came from: Athena as the "$path" column (Athena SELECT reference) and Redshift Spectrum as the $path pseudocolumn (Redshift Spectrum external tables). Filter on paths ending /current.parquet for the latest state, and on the history files when you want to compare runs.

DuckDB, Spark and Trino can read s3://<bucket>/<prefix>/<table>/current.parquet as a single file, which avoids the question altogether. DuckDB does it with its httpfs extension, so an analyst on a laptop can join supplier_onboarding to sp_users in a few lines of SQL.

Steps

1. Sign in with delegated access, or have an admin grant Sites.Selected on the sites you replicate.
2. Create the bucket, or pick a prefix in an existing one, and an identity that can list it and write objects.
3. Write a replication.yaml with a DuckDB destination, the bucket URL, the region and the names of the environment variables that hold any keys.
4. Choose the lists, run spsync doctor, then run the first replication. It loads every item and writes the first snapshot.
5. Point Athena, Redshift Spectrum, DuckDB or Spark at the table folders.

Every run adds one history file per table, so a short schedule over many tables adds up. An S3 Lifecycle expiration rule deletes objects a set number of days after they were created, and replacing an object resets that date (lifecycle rules). Every run replaces current.parquet, so a rule like that removes old history files and leaves the current snapshot alone. In a bucket with versioning switched on, each run also leaves the previous current.parquet as a noncurrent version, so add a rule for those too. Storage, requests and egress are billed by AWS, and access is whatever the bucket policy already enforces.

# This destination is available now. `spsync export-config` does not take it as a
# --destination-kind value yet, so write replication.yaml by hand (see the YAML tab)
# and run it with `spsync run replication.yaml` like any other destination.

pipeline_name: amazon_s3
prefix: amazon_s3
auth:
  mode: delegated
  client_id: 00000000-0000-0000-0000-000000000000
  tenant_id: common
destination:
  kind: duckdb
  connection_string_env: SPSYNC_DB_CONNECTION
  dataset_name: amazon_s3
  parquet_export_bucket_url: s3://your-bucket/spsync
  parquet_export_s3_region: eu-west-2
  # Leave these out to use the standard AWS credential chain instead.
  parquet_export_s3_access_key_id_env: AWS_ACCESS_KEY_ID
  parquet_export_s3_secret_access_key_env: AWS_SECRET_ACCESS_KEY
scopes:
  - site_url: https://contoso.sharepoint.com/sites/example
    mode: entire_site
reconcile_interval_days: 7
create_reporting_views: false

# every night at 02:00
0 2 * * * cd /opt/spsync && SPSYNC_DB_CONNECTION="$(cat /etc/spsync/db.env)" uv run spsync run replication.yaml >> /var/log/spsync/run.log 2>&1

# .github/workflows/spsync-run.yml
name: SPSync
on:
  schedule:
    - cron: "0 2 * * *"
  workflow_dispatch: {}
jobs:
  run:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: astral-sh/setup-uv@v3
      - run: uv sync
      - run: uv run spsync run replication.yaml
        env:
          SPSYNC_DB_CONNECTION: ${{ secrets.SPSYNC_DB_CONNECTION }}
          SPSYNC_LICENCE_KEY: ${{ secrets.SPSYNC_LICENCE_KEY }}

# azure-pipelines.yml
schedules:
  - cron: "0 2 * * *"
    branches: { include: [main] }
    always: true
steps:
  - script: uv sync
  - script: uv run spsync run replication.yaml
    env:
      SPSYNC_DB_CONNECTION: $(SPSYNC_DB_CONNECTION)
      SPSYNC_LICENCE_KEY: $(SPSYNC_LICENCE_KEY)

# Windows Task Scheduler action (see /docs/cli/automation/)
$env:SPSYNC_DB_CONNECTION = (Get-Content C:\spsync\db.env)
uv run spsync run C:\spsync\replication.yaml
schtasks /Create /SC DAILY /ST 02:00 /TN "SPSync" /TR "powershell -File C:\spsync\run.ps1"

# ~/Library/LaunchAgents/io.spsync.run.plist, then: launchctl load ~/Library/LaunchAgents/io.spsync.run.plist