Connect SPSync to Apache Iceberg

This guide covers Apache Iceberg tables written through a REST catalog. SPSync uses one destination kind, iceberg. It writes one Iceberg table per SharePoint table, with Parquet data files underneath.

Underneath, SPSync keeps a local DuckDB file as its working copy. After each round it overwrites the matching Iceberg table with the current full state, so each round rewrites whole tables rather than only the changed rows. Soft-deleted rows are kept, flagged by sp_deleted_at. The local file must stay at a stable path, or the next round has to be a full load.

Before you start

A running Iceberg REST catalog (for example Apache Polaris, Lakekeeper, Nessie’s REST endpoint or the open-source tabulario/iceberg-rest), and the object store its warehouse uses, such as S3 or MinIO.
Nothing to install on the machine that runs SPSync.
Three values: the catalog URI, the warehouse location and a namespace name.
Rights for your credential to create namespaces and tables in the catalog and to write files to the warehouse.

Create the database and a least-privilege user

The catalog is the database. Create a namespace for SPSync’s tables, or let SPSync create it on first use. Then give one principal the rights to create and write tables in that namespace and to read and write the warehouse path. How you do this is specific to the catalog, so use its own documentation. No catalog-specific script is given here because none has been run against SPSync.

Allow SPSync to reach it

SPSync makes two outbound connections: to the REST catalog’s URI, and to the object store that holds the warehouse files (the S3 endpoint, or s3.<region>.amazonaws.com for AWS). Allow both through any firewall or security group.

Encrypt the connection

Use an https:// catalog URI. A plain http:// URI such as http://localhost:8181 works and is only sensible on one machine or a trusted network. The same applies to an http:// S3-compatible endpoint.

Authentication SPSync supports today

No authentication: a catalog that needs none.
Bearer token for the catalog: set on the Destination screen, in YAML, or through the admin MCP server’s secure entry page.
OAuth2 client credentials for the catalog: a credential, a token endpoint and a scope, set on the Destination screen or in YAML (iceberg_oauth2_*). A bearer token and OAuth2 client credentials are mutually exclusive.
S3 endpoint and access keys for the warehouse: set on the Destination screen, in YAML, or through the secure entry page. If you leave them empty, SPSync uses the default AWS credential chain of the environment it runs in.

Not supported today: AWS Glue catalogs, and catalog types other than REST. The engine’s internal notes mention Glue, but only the REST client is wired in.

Connect from SPSync

CLI: connection_string_env names an environment variable holding the path of the local DuckDB state file, not a network string. The token and key variables are optional.

export SPSYNC_ICEBERG_STATE_PATH=/var/lib/spsync/iceberg-state.duckdb
export SPSYNC_ICEBERG_TOKEN='<catalog-bearer-token>'
export SPSYNC_ICEBERG_S3_ACCESS_KEY='<access-key-id>'
export SPSYNC_ICEBERG_S3_SECRET_KEY='<secret-access-key>'
spsync export-config --output replication.yaml \
  --site-url https://contoso.sharepoint.com/sites/x --list Projects \
  --pipeline-name spsync_dev --prefix dev \
  --destination-kind iceberg --connection-string-env SPSYNC_ICEBERG_STATE_PATH \
  --dataset-name main \
  --iceberg-catalog-uri https://catalog.example.com --iceberg-warehouse s3://warehouse/ \
  --iceberg-namespace ks \
  --iceberg-token-env SPSYNC_ICEBERG_TOKEN \
  --iceberg-s3-access-key-id-env SPSYNC_ICEBERG_S3_ACCESS_KEY \
  --iceberg-s3-secret-access-key-env SPSYNC_ICEBERG_S3_SECRET_KEY \
  --client-id <entra-app-client-id>

Add --iceberg-s3-endpoint http://localhost:9000 for MinIO or another S3-compatible store, and leave it out for AWS S3.

Desktop app: open the Destination screen, choose Apache Iceberg, fill in Catalog URI, Warehouse and Namespace, and choose Test connection. This is enough for an open catalog.
AI assistant: with the admin MCP server connected, ask for the destination requirements, then let the assistant create the destination with the same three fields. The assistant never sees a token or key: it returns a secure entry link where you type them.

What Test connection and doctor check

Both list the namespaces in the catalog. A pass proves the catalog URI answers and accepts the request. It does not prove the namespace exists, that the credential can create tables there, or that SPSync can write to the warehouse. Those show up on the first run.

Common errors

What you see	What it means and the fix
Connection refused or timed out on the catalog URI	The catalog is down, the URI is wrong, or a firewall blocks it.
`401` or `403` from the catalog	The catalog needs a bearer token or a role. Set the token or OAuth2 credentials on the Destination screen, in YAML, or through the secure entry page.
`NoSuchNamespaceError`	The namespace is missing and the catalog would not create it. Create it.
`AccessDenied` writing to the warehouse	The object-store credential cannot write to the warehouse path.
Column type change fails	Changing a column’s type is not supported yet on Iceberg. The run stops with an error rather than guessing.
Unsupported catalog type	Only REST catalogs work. Glue is not supported.