Connect SPSync to LanceDB

LanceDB is an embedded vector database: there is no server to run. SPSync writes one Lance table per table of your lists, a main table, each multi-value child table and the shared users table, into a folder on disk or a bucket in object storage. Rows are merged in by key, and a row deleted in SharePoint is deleted from the table. SPSync does not write a vector column today, so the tables hold the row’s columns only. The folder suits evaluation and local retrieval work on one machine.

Before you start

• A folder on the machine that runs SPSync, on a local disk. A network share is a poor home because the library relies on file locking.
• Or a bucket or container in S3, Google Cloud Storage or Azure Blob Storage, with a credential you can give SPSync (see below).
• No software to install. SPSync also keeps a small local state file (a DuckDB file) that records what it last sent, so it sends only the rows that changed. Keep that file on a disk that survives restarts, because losing it makes the next round a full resend.

Create the database and a least-privilege user

There is nothing to create. SPSync makes the folder and the tables the first time it runs.

• Local folder: the permission that matters is the operating system’s. The account that runs SPSync must be able to create, write and delete files in the folder. Give the folder to that account and nobody else.
• Object store: the credential needs to list, read, write and delete objects under the prefix SPSync uses, and nothing else. For S3 that is s3:ListBucket on the bucket plus s3:GetObject, s3:PutObject and s3:DeleteObject on the prefix. This is from the AWS documentation and has not been run here. The equivalent for Google Cloud is Storage Object Admin on the bucket, and for Azure is Storage Blob Data Contributor on the container.

Allow SPSync to reach it

• Local folder: not applicable. No network port is involved. If another machine needs the data, copy the folder, or point both machines at an object-store location. The Azure managed application cannot write to a folder on your computer.
• Object store: the machine that runs SPSync needs outbound HTTPS to the store. Allow it in the bucket policy or the storage account’s firewall.

Encrypt the connection

A local folder has no connection to encrypt; use disk encryption such as FileVault or BitLocker if the data is sensitive. Object-store locations (s3://, gs://, az://) are reached over HTTPS by the library.

Authentication SPSync supports today

A local folder needs none. For an object store, SPSync passes a set of storage options to LanceDB. Those hold the store’s credential and region, such as aws_access_key_id, aws_secret_access_key and region for S3. You can enter them in the Object-store options field of the Destination screen in the desktop app and the server, from the command line, or on the secure entry page through an AI assistant. If the environment the process runs in already provides the credential (for example an instance role on AWS), the library picks it up with no options.

Connect from SPSync

The location is a folder path or an object-store URI:

/data/lancedb
s3://my-bucket/spsync/lancedb

• Desktop app: open the Destination screen, choose LanceDB, type the folder or object-store URI into LanceDB URI, add the store’s credentials in Object-store options if it needs them (one key=value per line, or a JSON object), and choose Test connection. SPSync manages the local state file itself.

• CLI: the config names an environment variable for the state file and, for an object store, another that holds the storage options as a JSON object. SPSYNC_LANCEDB_STATE_PATH holds the path of the state file, not a network address.

  export SPSYNC_LANCEDB_STATE_PATH=/var/lib/spsync/lancedb-state.duckdb
  spsync export-config --output replication.yaml \
    --site-url https://contoso.sharepoint.com/sites/x --list Projects \
    --pipeline-name spsync_dev --prefix dev \
    --destination-kind lancedb --connection-string-env SPSYNC_LANCEDB_STATE_PATH \
    --dataset-name main --lancedb-uri /data/lancedb \
    --client-id <entra-app-client-id>

  For an object store, add --lancedb-uri s3://my-bucket/spsync/lancedb and --lancedb-storage-options-env SPSYNC_LANCEDB_STORAGE_OPTIONS, where that variable holds JSON such as {"aws_access_key_id": "...", "aws_secret_access_key": "...", "region": "eu-west-1"} (key=value lines work too).

• AI assistant: with the admin MCP server connected, ask for the destination requirements, then let the assistant create the destination with the folder or URI as lancedb_uri. Object-store credentials are entered on the secure entry page, never in the chat.

What Test connection and doctor check

Both open the location and list the tables in it, without writing anything. A pass means the folder exists and is readable, or the object-store credential works. It does not prove SPSync can write. spsync doctor reports the probe as destination-connectivity.

Common errors

What you see	What it means and the fix
No such file or directory or Permission denied	The folder does not exist or the account running SPSync cannot write to it. Create it and fix the permissions.
An object-store error such as AccessDenied or 403	The credential cannot list or write under the prefix, or the storage options are missing. Check the policy and the JSON.
Unable to load credentials or a region error	The storage options carry no region or key, and the environment provides none. Add them to the JSON.
The first sync works but later rounds resend everything	The state file moved or was deleted. Keep it at one persistent path.