Connect SPSync to Azure Data Lake Storage Gen2

This guide covers Azure Data Lake Storage Gen2, a storage account with the hierarchical namespace turned on. SPSync uses one destination kind, adls-gen2. It writes Parquet files into a container; it does not create tables in a database.

Underneath, SPSync keeps a local DuckDB file as its working copy and uploads each round’s Parquet snapshot after it. That file must be at a stable path on the machine that runs SPSync. Losing it means the next round has to be a full load. The container holds only the exported snapshots.

Before you start

An Azure storage account with hierarchical namespace enabled, and a container in it.
Nothing to install on the machine that runs SPSync.
Rights to assign a role on the container, or to create a SAS token for it.
The container address in this form, which is the only form SPSync accepts: abfss://<container>@<account>.dfs.core.windows.net/<path>.

Create the database and a least-privilege user

There is no database. Create a container, and give SPSync write access to only that container. Both routes below are from the Azure documentation and have not been run here.

Role assignment (Entra identity): assign Storage Blob Data Contributor to the identity SPSync runs as, scoped to the container, not the storage account.

az role assignment create \
  --assignee <principal-object-id> \
  --role "Storage Blob Data Contributor" \
  --scope /subscriptions/<sub>/resourceGroups/<rg>/providers/Microsoft.Storage/storageAccounts/<account>/blobServices/default/containers/<container>

SAS token: create a container-level SAS with read, write, create, list and delete. Give it an expiry you will remember to renew.

az storage container generate-sas --account-name <account> --name <container> \
  --permissions rwcdl --expiry 2027-10-26T00:00Z --auth-mode login --as-user

Allow SPSync to reach it

SPSync connects to <account>.dfs.core.windows.net over HTTPS. If the storage account has a firewall, under Networking add the address SPSync connects from, or a virtual network rule or private endpoint if SPSync runs inside Azure. Allow-list changes can take a few minutes.

Encrypt the connection

Access to storage is over HTTPS, and the abfss scheme means TLS. There is no setting to change.

Authentication SPSync supports today

Three ways, in the order SPSync uses them:

Full connection string: DefaultEndpointsProtocol=...;AccountName=...;AccountKey=.... The desktop form offers this as an optional field aimed at Azurite and testing. The container address is still required and decides where data is written. If a connection string and a SAS token are both set, the connection string is used. The CLI has no connection-string option for this kind.
SAS token: a token for the container. No Entra identity is needed on the machine.
Entra, when neither of the above is given: SPSync uses DefaultAzureCredential, which picks up a managed identity, an az login session, or environment variables for a service principal. SPSync does not run its own Entra sign-in for storage. Whatever identity the process has is what gets used, so assign the role above to that identity.

Connect from SPSync

Desktop app: open the Destination screen, choose Azure Data Lake Storage Gen2, enter the Storage container URL, and either paste a SAS token or leave it blank to use the signed-in Entra identity, then choose Test connection.

CLI: connection_string_env names an environment variable holding the path of the local DuckDB state file, not a network string. --sas-token-env names the variable holding the SAS token; leave it out to use Entra.

export SPSYNC_ADLS_STATE_PATH=/var/lib/spsync/adls-state.duckdb
export SPSYNC_ADLS_SAS='<sas-token>'
spsync export-config --output replication.yaml \
  --site-url https://contoso.sharepoint.com/sites/x --list Projects \
  --pipeline-name spsync_dev --prefix dev \
  --destination-kind adls-gen2 --connection-string-env SPSYNC_ADLS_STATE_PATH \
  --dataset-name main \
  --bucket-url 'abfss://<container>@<account>.dfs.core.windows.net/ks' \
  --sas-token-env SPSYNC_ADLS_SAS --client-id <entra-app-client-id>

AI assistant: with the admin MCP server connected, ask for the destination requirements and give bucket_url. A SAS token goes through the secure entry page. Leave it unset to use Entra.

Each table gets a folder in the container. In it, <run-id>.parquet is one immutable file per round and current.parquet is this round’s snapshot, overwritten each round. Point reporting tools at current.parquet.

What Test connection and doctor check

Both build the target from your settings and list the container. A pass proves the account name resolves, the credential is accepted and the container exists and can be listed. It does not prove the identity can write, and a list-only SAS passes and then fails on the first upload.

Common errors

What you see	What it means and the fix
`AuthorizationPermissionMismatch` or `403 This request is not authorized`	The identity has no data-plane role, or the SAS lacks a permission. A Contributor or Owner role on the account is a management role and does not grant blob access. Assign Storage Blob Data Contributor.
`AuthenticationFailed` with a SAS	The token expired, or was made for another container or account. Create a new one.
`ContainerNotFound`	The container name in `bucket_url` is wrong.
`DefaultAzureCredential failed to retrieve a token`	No Entra identity is available to the SPSync process. Run `az login`, use a managed identity, or give a SAS token.
Refused `az://` URL	Use `abfss://<container>@<account>.dfs.core.windows.net/<path>`. An `az://` URL names no account.
A firewall error from `<account>.dfs.core.windows.net`	The storage account firewall does not list SPSync’s address.