SharePoint to LanceDB

Last updated 26 October 2026

One way: SharePoint stays the system of record, and SPSync keeps a read-only copy of your lists current in LanceDB.

Status: Shipped · product: SPSync for Search & Vector Stores

SPSync fills LanceDB tables from SharePoint lists, one way and one row per item, with the replicated columns and no vector column. To run a vector search over them, add your own step that embeds the text you choose and writes the vectors to a table of yours keyed on the item ID. Embeddings on sync for LanceDB are under consideration, not scheduled; SPSync generates them today for Postgres with pgvector and Azure AI Search.

LanceDB is a table format and library with no server. An application opens a Lance table from a folder or an object-store path and queries it in process, which makes it a natural fit for a desktop tool, a notebook or a small service.

Where the tables live

Set one location:

• A local folder, such as /data/lance/sharepoint. Nothing else to configure, and the desktop app can keep the tables fed on the same machine.
• An object-store URI, such as s3://bucket/prefix or az://container/prefix. Add the store’s credentials and region as storage options: in the Destination screen of the desktop app or server, in an environment variable as a JSON object for the command-line tool, or on a secure page when an AI assistant sets the destination up.

SPSync also keeps a local state file between runs, which needs a stable path.

The tables

A field survey list, env_samples, with a Site lookup, a Collector person column and a multi-choice Contaminants column produces three Lance tables: env_samples, env_samples__contaminants and the shared sp_users. The item table carries site_id, site_value and collector_user_id, so an application can show the site name and join to sp_users for the collector. LanceDB’s SQL-style filters work on any replicated column, for example site_value = 'North Quarry'.

Updates are merge_insert upserts keyed on the item ID, and a child table merges on the item ID and the value’s position together. When a SharePoint item is deleted, SPSync deletes its row from the item table and its rows from every child table. A table can only gain fields in place, so a SharePoint column that changes type sends writes to a new _v2 table.

What each run sends

Each run pushes only rows whose content changed since the last successful push, plus deletions. A reconcile, weekly by default, writes everything again, which repairs rows changed by hand. A push that fails is retried on the next run.

Uses

Developers building a desktop tool or small service over SharePoint content open the tables with the LanceDB client library. Agent frameworks running on one machine use them for a filtered lookup tool; see AI agents and tools over SharePoint. Retrieval with vectors needs pgvector or Azure AI Search; see RAG on SharePoint lists.

Steps

1. Sign in with your own SharePoint account, or use an app registration with Sites.Selected on the chosen sites if no one will be signed in when it runs.
2. Choose a local folder or object-store URI, with storage options if needed.
3. Choose the lists and run; the first run creates the tables and loads every item.
4. Open the tables from your application.

A local table should have one writer, so let SPSync’s schedule do the writing and read from other processes. Object storage adds request charges on top of storage, and SharePoint permissions do not travel with the data, so access to the folder or bucket is the boundary.

spsync auth login --client-id 00000000-0000-0000-0000-000000000000
spsync export-config --output replication.yaml --pipeline-name lancedb --prefix lancedb --destination-kind lancedb --connection-string-env SPSYNC_DB_CONNECTION --dataset-name lancedb --client-id 00000000-0000-0000-0000-000000000000 --site-url https://contoso.sharepoint.com/sites/example --all-lists
spsync doctor replication.yaml
spsync run replication.yaml

pipeline_name: lancedb
prefix: lancedb
auth:
  mode: delegated
  client_id: 00000000-0000-0000-0000-000000000000
  tenant_id: common
destination:
  kind: lancedb
  connection_string_env: SPSYNC_DB_CONNECTION
  dataset_name: lancedb
scopes:
  - site_url: https://contoso.sharepoint.com/sites/example
    mode: entire_site
reconcile_interval_days: 7
create_reporting_views: false

# every night at 02:00
0 2 * * * cd /opt/spsync && SPSYNC_DB_CONNECTION="$(cat /etc/spsync/db.env)" uv run spsync run replication.yaml >> /var/log/spsync/run.log 2>&1

# .github/workflows/spsync-run.yml
name: SPSync
on:
  schedule:
    - cron: "0 2 * * *"
  workflow_dispatch: {}
jobs:
  run:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: astral-sh/setup-uv@v3
      - run: uv sync
      - run: uv run spsync run replication.yaml
        env:
          SPSYNC_DB_CONNECTION: ${{ secrets.SPSYNC_DB_CONNECTION }}
          SPSYNC_LICENCE_KEY: ${{ secrets.SPSYNC_LICENCE_KEY }}

# azure-pipelines.yml
schedules:
  - cron: "0 2 * * *"
    branches: { include: [main] }
    always: true
steps:
  - script: uv sync
  - script: uv run spsync run replication.yaml
    env:
      SPSYNC_DB_CONNECTION: $(SPSYNC_DB_CONNECTION)
      SPSYNC_LICENCE_KEY: $(SPSYNC_LICENCE_KEY)

# Windows Task Scheduler action (see /docs/cli/automation/)
$env:SPSYNC_DB_CONNECTION = (Get-Content C:\spsync\db.env)
uv run spsync run C:\spsync\replication.yaml
schtasks /Create /SC DAILY /ST 02:00 /TN "SPSync" /TR "powershell -File C:\spsync\run.ps1"

# ~/Library/LaunchAgents/io.spsync.run.plist, then: launchctl load ~/Library/LaunchAgents/io.spsync.run.plist