A script can get a SharePoint list into SQL Server in an afternoon. Keeping that copy correct every day, unattended, is a separate job that the script inherits without anyone deciding it should: schema drift, deletes, lookups, throttling and monitoring. This post prices each of the five and shows what SPSync does about it.
Schema drift and the mapping step
A hand-written script or pipeline maps each column once. When someone adds, renames or retypes a column in SharePoint, the mapping has no idea. Either the pipeline fails, or it carries on without the new column until a report is missing a field and someone asks why.
SPSync reads the list’s schema on every run. A new column appears in the destination table
on the next sync. Removed columns stay where they are, so nothing that reads them breaks. A
retyped column moves into a new _v2 column and the original keeps its old values.
Lookups resolved by hand
Through the SharePoint APIs, a lookup or multi-value column arrives as a flattened string or an internal id. Making it joinable is code, written once per column and kept up to date as the list changes.
SPSync does this as part of the schema mapping. A lookup becomes an id column that joins to the table for its list, and a multi-choice or multi-person column becomes a child table.
Throttling and retries
SharePoint and Microsoft Graph throttle under load. A script or flow needs its own
Retry-After handling and backoff, or it stops partway through a run on a busy tenant,
often on the day the numbers matter.
SPSync follows Microsoft’s guidance: it honours Retry-After and meters resource units per
tenant, so a sync does not make the throttling worse. It has no off-peak scheduler of its
own. When you run the command line, you decide when a heavy load starts with cron, a
scheduled task or a CI pipeline.
Code that outlives its author
A quick list copy grows: paging around the 5,000-item view threshold, retry logic, schema-drift handling, delete detection. Our post on what breaks past 5,000 items shows where the paging part goes wrong, and a SharePoint list into Postgres or SQL Server with Python is the delta-based script written out, with the gaps it leaves. Someone now owns that code indefinitely, and it is seldom the person who wrote the first version. Putting the replication in a product built for it takes that maintenance off any one person on your team.
Where Power Automate, Data Factory and PowerShell still fit
A scheduled flow, a copy pipeline or a PnP PowerShell script is still a reasonable answer for plenty of jobs. A flow that reacts to a single item change is one; a migration script that runs once is another. Automating the replication pays off once the script or flow runs unattended on a schedule that people depend on. If a script is the right answer for you, our PnP PowerShell export to SQL Server is a complete one to start from. The comparisons with Power Automate, Azure Data Factory and PnP PowerShell scripts go into more detail, and Replacing Data Factory pipelines and scheduled flows covers swapping an existing pipeline out.
SQL Server is one of many destinations, alongside other relational databases, warehouses, lakes, search indexes and vector stores. The same lists can go to several of them, one replication each. If the destination you need is missing, tell us.
Write the config once
What you see: One YAML replication config, written by a single CLI command, naming the site, the lists and the SQL Server destination.
Why it matters: A plain text config can be committed and reviewed in a pull request like any other change, with secrets held as environment variable references.
What it replaces: A script's hard-coded connection details, or a flow's settings buried in a designer that is hard to review line by line.
Point the desktop app at the site and the SQL Server destination. The same connection details it uses are exportable to a config file for unattended runs.
Write the replication config in one command. The connection string and the licence key are environment variable references, not literal secrets.
spsync export-config --output reporting.yaml --pipeline-name reporting --prefix sp --destination-kind mssql --connection-string-env REPORTING_DB_URL --dataset-name reporting --client-id $SPSYNC_CLIENT_ID --site-url https://contoso.sharepoint.com/sites/reporting --lists-by-name Projects,Tasks,Departments
Let schema drift handle itself
What you see: A column added in SharePoint appearing in the destination table on the next run, with no mapping step to update by hand.
Why it matters: A hand-written mapping goes out of date the moment a column is added, renamed or retyped. SPSync reads the list's own schema on every run.
What it replaces: A mapping file or a transform step someone has to remember to update every time the list's columns change.
Add a column to a list in SharePoint, then run the sync. The new column appears in the table with no config change.
Run the sync again after a schema change. The new or retyped column arrives without editing the config.
spsync run reporting.yaml
Keep deletes instead of losing them
What you see: A row deleted in SharePoint still present in the destination table, flagged as deleted.
Why it matters: A copy-only script keeps no record of a deleted item. With soft deletes on by default, a report keeps every row it used to show and can filter the deleted ones.
What it replaces: A separate delete-detection pass someone has to design and test themselves.
Delete an item in SharePoint, then run the sync. The row stays in the destination table, flagged.
Force a full reconcile any time you want to confirm every delete against SharePoint matches the destination.
spsync reconcile reporting.yaml
Run it unattended, on a schedule
What you see: A cron job, a scheduled task or a CI pipeline calling one command, with Retry-After and per-tenant throttling already handled.
Why it matters: Without backoff and retry logic, a script fails partway through a run on a busy tenant. SPSync follows Microsoft's own throttling guidance.
What it replaces: Retry and backoff code someone has to write, test and maintain alongside the script itself.
Set the schedule from the desktop app for a machine that stays on, or export the config for a server to run unattended.
Drive the schedule from a cron job, a scheduled task or a CI pipeline. The CLI does one run and exits with a clear code.
spsync run reporting.yaml
See what ran, and what broke
What you see: Rows read and written for every run in one place, with any error named by the list and the column it happened on.
Why it matters: A failed script or flow often goes unseen until a report looks stale. An error that names the list and column can be fixed in minutes.
What it replaces: Alerting and logging someone has to wire up separately around whatever script or flow is doing the copying.
Open the run history to see rows read and written, and any error named by list and column.
Check the doctor output before a scheduled run, and tail the log after one to confirm what happened.
spsync doctor reporting.yaml