Warehouses and lakes
Replicating SharePoint list data into Parquet, Delta Lake, Iceberg, BigQuery and Databricks, for analytics engines that read files and tables, not APIs.
A data engineer has been asked to bring three SharePoint lists into the lake so they can sit beside the finance and CRM data. The usual routes are a Data Factory pipeline or a scheduled CSV export. Both drop column types on the way, and both tend to break the first time someone adds a column to a list.
SPSync writes list data straight to the formats your engines already read. Parquet, Delta Lake and Iceberg are shipped, along with DuckDB, Azure Data Lake Storage Gen2, BigQuery and Databricks. Columns keep their types. Lookups become key columns pointing at the target list's table, and multi-value columns become child tables, each written as its own set of files, so a Spark job or a DuckDB query reads them with no extra loading step.
How each run writes depends on the format, so check it before you design partitions. After the first full load SPSync reads only the changes from SharePoint. Delta Lake tables are then merged in place, while Iceberg tables and Parquet exports are rewritten as a full snapshot of each table on every run, which keeps them simple to read but means the write grows with the list. A deleted item stays as a row with sp_deleted_at set, and your models decide whether to drop it.
Fabric and Snowflake are not destinations yet. Fabric can read the data today through a OneLake shortcut in a Lakehouse that points at the Azure Data Lake Storage Gen2 container, without a second copy. For modelling, the replica behaves like any other source. Column names follow the list's internal names, new columns appear on their own and removed columns stay, so dbt can declare the tables in sources.yml and run after each sync. The whole replication lives in a YAML config you can review in a pull request, which is what lets a data team treat SharePoint as a feed it can depend on month after month.
Start here
- SharePoint lists in Parquet, Delta Lake and Iceberg
The files come first, with their layout, the soft-delete column and a sample query.
- dbt models on SharePoint data: keys, deletes and views
Modelling in dbt builds on those files, with keys, soft deletes and tests for lookups that must resolve.
All posts on this topic
- Fabric SharePoint list mirroring, or a SQL replica?
Fabric mirrors a SharePoint list for free, but only on a running Fabric capacity. What that costs a SQL Server or Postgres team, and when a SQL replica fits.
- Openflow reads SharePoint files, Lakeflow adds lists in Beta
Snowflake Openflow's SharePoint connector ingests files and permissions. Databricks Lakeflow Connect adds list ingestion in Beta. What each does with a list.
- dbt models on SharePoint data: keys, deletes and views
Staging SharePoint lists in dbt from a replica: which key to use, how child tables and soft deletes behave, when the generated views help, and tests that catch broken lookups.
- SharePoint lists in Parquet, Delta Lake and Iceberg
SharePoint list data as Parquet files, Delta tables and Iceberg tables: folder layout, soft deletes, what each run writes, a Fabric route and sample queries.
Other topics on the blog
Frequently asked questions
Which lake formats does SPSync write to today?
Parquet files, Iceberg tables and Delta Lake tables, plus DuckDB as a single file. All are shipped; each has its own destination page with the details.
How does delta sync work against files?
SPSync reads only the changes from SharePoint after the first full load, as for a database. How the lake is written depends on the format: Delta Lake tables are merged in place, while Iceberg tables and Parquet exports are rewritten as a full snapshot of each table on every run.
Is the replicated schema stable enough for sources.yml?
It changes in predictable ways. Column names follow the list's internal names, new columns appear automatically, removed columns stay, and on SQL Server, PostgreSQL, MySQL, ClickHouse, Oracle, SQLite and DuckDB a retyped column moves into a new _v2 column, so the type under an existing model never shifts.