Migrating list data off SharePoint into a database

Last updated 22 October 2026

The plan is to move off SharePoint lists entirely, but a one-off export loses history and cannot be re-run when the source changes before the migration finishes.

A one-off export is out of date the day after it runs. Migrations off SharePoint lists take weeks or months, and people keep editing the list throughout, because it is still where the work happens. A PowerShell script, a CSV or a Power Automate flow that dumps rows also turns lookup columns into raw IDs that mean nothing outside SharePoint, and running it a second time to catch up duplicates every row.

Take a contract register moving into a purpose-built application over three months. If you are still deciding whether the list needs to move at all, our post on where a list stops working as a database compares a full move with Dataverse and with keeping the list for editing beside a SQL copy.

A copy that keeps up until cutover

SPSync replicates the lists into the target database and keeps the copy current for as long as the migration takes. The first run loads everything; every later run is a delta, so on cutover day the database matches the live list as it is that morning. Running it twice is safe: each item has one row, keyed on its SharePoint item ID.

What the new application inherits

Each list becomes a table with its person, choice, calculated and lookup columns; managed metadata columns are not replicated yet. Lookups are key columns and multi-choice columns are child tables, so Contracts, Counterparties and Renewals arrive with working joins, with no flat CSVs to stitch back together. Items deleted during the migration window are kept with sp_deleted_at set, so everything that existed along the way is still on record. If the new application wants a flat read model, SPSync can generate a reporting view per list.

The migration plan, step by step

1. Choose the target: SQL Server, Postgres, MySQL or another supported destination.
2. Run the first full load, then check row counts and the schema against the list.
3. Leave the replication on a schedule while the application is built and tested.
4. Switch the application to the replicated tables.
5. Decide what happens next: keep the sync running while SharePoint is still edited, or stop it once SharePoint is no longer the source of truth.

Large lists, including ones far past the 5,000-item view threshold that breaks a naive export, load the same way as small ones. SPSync never writes to SharePoint, so the live list is untouched until you retire it. If the migration may need repeating in another environment, script the config with the command line and commit it, and the whole run can be reproduced.

spsync auth login --client-id 00000000-0000-0000-0000-000000000000
spsync export-config --output replication.yaml --pipeline-name migrate_sharepoint_lists --prefix migrate_sharepoint_lists --destination-kind mssql --connection-string-env SPSYNC_DB_CONNECTION --dataset-name migrate_sharepoint_lists --client-id 00000000-0000-0000-0000-000000000000 --site-url https://contoso.sharepoint.com/sites/example --all-lists
spsync doctor replication.yaml
spsync run replication.yaml

pipeline_name: migrate_sharepoint_lists
prefix: migrate_sharepoint_lists
auth:
  mode: delegated
  client_id: 00000000-0000-0000-0000-000000000000
  tenant_id: common
destination:
  kind: mssql
  connection_string_env: SPSYNC_DB_CONNECTION
  dataset_name: migrate_sharepoint_lists
scopes:
  - site_url: https://contoso.sharepoint.com/sites/example
    mode: entire_site
reconcile_interval_days: 7
create_reporting_views: false

# every night at 02:00
0 2 * * * cd /opt/spsync && SPSYNC_DB_CONNECTION="$(cat /etc/spsync/db.env)" uv run spsync run replication.yaml >> /var/log/spsync/run.log 2>&1

# .github/workflows/spsync-run.yml
name: SPSync
on:
  schedule:
    - cron: "0 2 * * *"
  workflow_dispatch: {}
jobs:
  run:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: astral-sh/setup-uv@v3
      - run: uv sync
      - run: uv run spsync run replication.yaml
        env:
          SPSYNC_DB_CONNECTION: ${{ secrets.SPSYNC_DB_CONNECTION }}
          SPSYNC_LICENCE_KEY: ${{ secrets.SPSYNC_LICENCE_KEY }}

# azure-pipelines.yml
schedules:
  - cron: "0 2 * * *"
    branches: { include: [main] }
    always: true
steps:
  - script: uv sync
  - script: uv run spsync run replication.yaml
    env:
      SPSYNC_DB_CONNECTION: $(SPSYNC_DB_CONNECTION)
      SPSYNC_LICENCE_KEY: $(SPSYNC_LICENCE_KEY)

# Windows Task Scheduler action (see /docs/cli/automation/)
$env:SPSYNC_DB_CONNECTION = (Get-Content C:\spsync\db.env)
uv run spsync run C:\spsync\replication.yaml
schtasks /Create /SC DAILY /ST 02:00 /TN "SPSync" /TR "powershell -File C:\spsync\run.ps1"

# ~/Library/LaunchAgents/io.spsync.run.plist, then: launchctl load ~/Library/LaunchAgents/io.spsync.run.plist