Connect SPSync to Delta Lake

This guide covers Delta Lake tables in object storage. SPSync uses one destination kind, deltalake. It writes one Delta table per SharePoint table, with a real transaction log, and merges changed rows into it. Unlike the Azure Data Lake destination it keeps no local working file.

Before you start

A location to write to: a folder on the machine that runs SPSync, an Amazon S3 bucket (or an S3-compatible store such as MinIO), or an Azure Blob Storage or Data Lake container.
Nothing to install on the machine that runs SPSync.
Rights to write to that location.
Something to read Delta tables with afterwards, such as Spark, DuckDB’s delta_scan or Microsoft Fabric. There are no reporting views: a Delta table has no server to hold a view. Build the join yourself.

Create the database and a least-privilege user

There is no database. Create the bucket or container and a credential that can read, write and list inside one prefix of it. The location, bucket_url, takes one of these forms:

Where	`bucket_url`
Local disk	`file:///path/to/folder`
Amazon S3 or compatible	`s3://bucket/prefix`
Azure Blob or Data Lake	`az://<container>/<prefix>` or `abfss://<container>@<account>.dfs.core.windows.net/<prefix>`

For S3, an IAM user whose policy allows s3:GetObject, s3:PutObject, s3:DeleteObject and s3:ListBucket on the bucket and prefix is enough. For Azure, assign Storage Blob Data Contributor on the container (see the Azure Data Lake Storage Gen2 guide). Both are from vendor documentation and have not been run here.

Google Cloud Storage (gs://) is not supported.

Allow SPSync to reach it

SPSync connects out over HTTPS to s3.<region>.amazonaws.com for S3, or to <account>.blob.core.windows.net or <account>.dfs.core.windows.net for Azure. Add the address SPSync connects from to any bucket policy, storage account firewall or VPC endpoint rule. A file:// location needs nothing.

Encrypt the connection

Remote locations use HTTPS. For a self-hosted S3-compatible store, a plain http:// AWS_ENDPOINT_URL is accepted, which is only sensible on a trusted network. Use https:// for anything else.

Authentication SPSync supports today

Set by storage options, a flat set of keys, per location:

Location	Keys
`file://`	None. Leave storage options empty.
`s3://`	`AWS_ACCESS_KEY_ID`, `AWS_SECRET_ACCESS_KEY`, optionally `AWS_SESSION_TOKEN`; `AWS_REGION` for real S3; `AWS_ENDPOINT_URL` for a compatible store.
`az://`	`AZURE_STORAGE_ACCOUNT_NAME` (required, because the URL names no account), plus `AZURE_STORAGE_ACCOUNT_KEY` or `AZURE_STORAGE_SAS_TOKEN`.
`abfss://`	`AZURE_STORAGE_ACCOUNT_KEY` or `AZURE_STORAGE_SAS_TOKEN`. The account name comes from the URL. Omit both to use Entra (`DefaultAzureCredential`) from the environment SPSync runs in.

An unrecognised key is an error, not ignored, so a typo shows up straight away. Lower-case spellings of these keys, AWS_DEFAULT_REGION and AZURE_STORAGE_CONNECTION_STRING are also accepted.

Connect from SPSync

CLI: --deltalake-storage-options-env names an environment variable holding the keys as a JSON object of string values. There is no connection_string_env for this kind.

export SPSYNC_DELTALAKE_STORAGE_OPTIONS='{"AWS_ACCESS_KEY_ID":"AKIA...","AWS_SECRET_ACCESS_KEY":"...","AWS_REGION":"eu-west-2"}'
spsync export-config --output replication.yaml \
  --site-url https://contoso.sharepoint.com/sites/x --list Projects \
  --pipeline-name spsync_dev --prefix dev \
  --destination-kind deltalake --dataset-name main \
  --bucket-url s3://spsync-bucket/ks \
  --deltalake-storage-options-env SPSYNC_DELTALAKE_STORAGE_OPTIONS \
  --client-id <entra-app-client-id>

Desktop app: open the Destination screen, choose Delta Lake, enter the location and choose Test connection. For a file:// location, or an abfss:// location that uses Entra, leave Storage options empty. This works end to end.
Storage keys in the desktop app: enter one KEY=value per line in Storage options. A JSON object of string values also works. The field is masked and write-only, and the server does not return it. Test connection and scheduled runs read it the same way.
AI assistant: the admin MCP server can set bucket_url for a file:// or Entra abfss:// location. Storage keys are entered through the secure entry page the assistant links to, never in the chat.

What Test connection and doctor check

spsync doctor and Test connection list the bucket or container at the location. A pass proves the location exists and the credentials can list it. For a file:// location the generic check proves nothing about the folder, so the list step is what matters. It does not prove the credential can write. Delta tables are created on the first run.

Common errors

What you see	What it means and the fix
`deltalake_storage_options is invalid: ...`	The Storage options text is neither `KEY=value` lines nor a JSON object of string values. The message names the problem. Fix the field.
`is not in key=value format`	A line is not `KEY=value`. Put one `KEY=value` per line, or paste a JSON object that starts with `{`.
An unrecognised storage option key	A typo or an unsupported key. Use the keys in the table above.
`403` or `AccessDenied` from S3	The IAM policy lacks list, get, put or delete on the bucket and prefix.
`AuthorizationPermissionMismatch` on Azure	The identity has no Storage Blob Data Contributor role, or the SAS lacks a permission.
`AWS_REGION` error against real S3	Set `AWS_REGION`. It is optional only when `AWS_ENDPOINT_URL` is set.