Are the Parquet files partitioned by date or by a column?

By run. In a bucket each table has a folder holding one file per run plus a current.parquet that is replaced every run. There is no partitioning on a column such as a date, so if you want that, write a partitioned copy in Spark from current.parquet.

Do I need to keep anything on the machine that runs SPSync?

For the DuckDB, Parquet and Iceberg routes, yes: the DuckDB file that holds the merged copy of each list, at a stable path. For Delta Lake, no. The merge state lives in each table's Delta log in the bucket.

Can Fabric read these files today?

Yes, through a OneLake shortcut. Replicate to the Azure Data Lake Storage Gen2 destination, add a shortcut in a Lakehouse that points at the container, and query current.parquet from a notebook or the SQL analytics endpoint.

Which format should a data engineer pick for SharePoint lists?

Delta Lake if your engines already read Delta and you want each run merged in place. Iceberg if you run a REST catalog. Parquet through DuckDB if you want plain files any tool can open, or a DuckDB file an analyst can query directly.

SharePoint lists in Parquet, Delta Lake and Iceberg

22 October 2026

Tags: Warehouses and lakes Desktop Command line

You have been asked to get the procurement team’s SharePoint lists into the lake. There is a Purchase requests list with a Supplier lookup, a Requester person column, an Amount, a Status choice and a multi-choice Categories column, plus the Suppliers list the lookup points at. Finance wants spend by supplier next to the ERP data you keep in the lake, and someone has already tried reading the list from a notebook through Microsoft Graph. It paged slowly, came back with SupplierLookupId and an integer where the requester’s name should be, and stopped at the 5,000-item view threshold the moment it filtered on an unindexed column.

What you want is a set of files, one table per list, in a format Spark, DuckDB or Trino reads without knowing SharePoint exists, kept current on a schedule. SPSync writes Parquet, Delta Lake and Iceberg, and the three differ more in what a run writes than in what the tables look like.

Getting SharePoint list data into a data lake

SPSync is the only process that talks to SharePoint. It reads each list through Microsoft Graph with delta queries, so the first run reads every item and every run after that reads only what changed since the last one. On our dev tenant a run read 10,303 items across four lists, one of them well past the view threshold, in 1m 52s. That run wrote to SQL Server, but the read side is the same whichever destination you choose, and a delta round with nothing to report costs 1 RU per list (resource units).

What happens after the read depends on the destination. You have three choices for a lake: the DuckDB destination with its Parquet export, the Delta Lake destination, and the Iceberg destination. The tables and columns are the same in all three, because they come from SPSync’s schema mapper and the destination never sees the raw Graph response.

What a SharePoint list looks like as tables

Every table name starts with a prefix you choose per site, followed by the list’s internal name in lower case. With proc as the prefix, the procurement site produces these tables:

proc_purchase_requests, one row per item, keyed on sp_item_id;
proc_purchase_requests__categories, one row per item and ticked category, with an ordinal column;
proc_suppliers, the list the lookup points at;
sp_users, one row per person or group, shared by every list in the replication.

The Supplier lookup becomes supplier_id, which joins to proc_suppliers.sp_item_id. The Requester column becomes requester_user_id, a key into sp_users.id, and sp_users carries login, display_name, email and is_group. Display names in SharePoint can change without renaming anything here, because column names come from internal names.

Every list table also gets SPSync’s metadata columns: sp_list_id, sp_site_id, sp_etag, sp_created, sp_modified, sp_author_id, sp_editor_id, sp_content_type, sp_deleted_at and sp_synced_at. Timestamps are stored in UTC. The schema reference has the full mapping, column type by column type.

SharePoint to Parquet with DuckDB

The DuckDB destination keeps a single DuckDB file on the sync host, merges each run into it by primary key, and then exports every table to Parquet with DuckDB’s own COPY ... TO ... (FORMAT PARQUET). You can export to a local folder, to a bucket, or to both in the same run. The export settings live in the YAML config the command-line tool reads.

The two targets lay the files out differently. A local folder gets one file per table, proc_purchase_requests.parquet, replaced on each run. A bucket gets a folder per table with two kinds of file in it:

s3://lake/procurement/
  proc_purchase_requests/
    current.parquet
    <run id>.parquet
    <run id>.parquet
  proc_purchase_requests__categories/
    current.parquet
    <run id>.parquet
  sp_users/
    current.parquet
    <run id>.parquet

current.parquet is the table as it stands after the latest run. Each run file is a full snapshot from that run and is never changed afterwards. That gives you a history of the list for free, and a storage bill that grows with every run, so put a lifecycle rule on the bucket if you do not need them all. Buckets can be Amazon S3, an S3-compatible store such as MinIO, or Azure Blob Storage and ADLS Gen2 through an abfss://<container>@<account>.dfs.core.windows.net/<prefix> URL.

So the files are partitioned by table and by run. Nothing is split by a date column or by site. If your downstream jobs want year=2026/month=10 folders, write them in Spark from current.parquet; SPSync hands you the clean table and leaves the layout of derived copies to you.

The DuckDB file is useful on its own. An analyst can open it, or a MotherDuck database through the same connection setting, and query the tables with SQL, including the reporting views SPSync can generate, which pre-join the lookups and people for you.

SharePoint list to Delta Lake tables

The Delta Lake destination works differently. There is no DuckDB file in the middle and nothing to keep on the sync host. SPSync writes one Delta table for each list, child table and sp_users, on local disk, S3 or MinIO, or Azure Blob Storage and ADLS Gen2. Each run is a Delta merge keyed on the table’s primary key, committed to the table’s transaction log. The tables sit under the bucket URL and dataset name you configure, so with a dataset of main you get s3://lake/procurement/main/proc_purchase_requests and its siblings.

Because every run is a commit, Delta’s time travel works out of the box. You can read proc_purchase_requests as it stood at an earlier version or timestamp, which is handy when finance asks why last month’s spend figure moved. Types follow Parquet’s vocabulary: text is STRING, timestamps are UTC with microsecond precision, and JSON-shaped values are stored as text, since Delta has no JSON type.

There are two trade-offs to know about. Delta tables have no server to host a view, so the reporting views are not available here and you write the joins in the engine you read with. The other is housekeeping. The log grows with every run, and SPSync does not run OPTIMIZE or VACUUM for you, so schedule those in Spark or Databricks if you sync a big list every few minutes.

SharePoint list to Apache Iceberg

The Iceberg destination writes through a REST catalog. You give it the catalog URI, the warehouse location and a namespace, plus a bearer token and S3 keys where the catalog needs them, and each SPSync table becomes an Iceberg table in that namespace with Parquet data files underneath. The namespace is created on first use.

Behind the catalog, SPSync uses the same DuckDB shadow as the Parquet route. SPSync merges each run into a local DuckDB file, then overwrites every Iceberg table with that file’s full current state as one new snapshot. A reader always gets a complete, consistent table, and the snapshot history gives you the same look back in time that Delta does.

Iceberg keeps 32-bit and 64-bit integers apart as int and long, stores timestamps as timestamptz and keeps GUIDs as strings. One limit matters for schema planning. Changing a column’s type in SharePoint is not supported for a list you replicate to Iceberg, so agree with the list owners that columns keep their type. New columns are fine and appear on the next run. Snapshots and metadata files accumulate as on Delta, so expire snapshots and compact from your engine on a schedule.

How are deleted SharePoint items stored in the lake?

They stay as rows, with a timestamp. When an item is deleted in SharePoint, the delta query reports it, and SPSync sets sp_deleted_at on that item’s row. The row stays in the main table, its child rows stay too, and every format keeps it: the Parquet snapshot includes it, the Delta destination writes it to the log as an update, and the Iceberg overwrite carries it across.

For a report, filter on sp_deleted_at IS NULL. For an audit, keep the filter off and you can see what was removed and when. A column added in SharePoint shows up as a new column on the next run, and columns removed in SharePoint stay in the lake with their old values, so a job that selects columns by name keeps working.

How incremental sync works for lake destinations

There are two halves to a run, and only one of them is about SharePoint. The read is always incremental after the first run. SPSync stores a delta token per list, asks Graph for what changed since that token, and gets back new items, edited items and deletions. A quiet list costs almost nothing to check, even on the desktop app’s default schedule of every five minutes.

The write is where the formats differ. Delta Lake merges only that round’s changed rows into each table. DuckDB merges them into its file the same way, but the Parquet export then writes each table’s full current state, and the Iceberg target overwrites each table with its full state as well. Both are simple and always consistent, and the price is write volume. A 200,000-row table with ten edits since the last run still gets a fresh current.parquet and a fresh Iceberg snapshot.

You may have read that SPSync now sends only changed rows downstream. That is true for the search indexes, the vector stores and Dataverse. For those, SPSync keeps a SHA-256 fingerprint of every row it has pushed in a _spsync_dest_row_state table inside the local DuckDB file, sends only rows whose fingerprint changed, and records the new fingerprints after the push succeeds. The Parquet export and Iceberg do not use those fingerprints; they publish whole tables. If you have a very large list on a short interval, Delta Lake is the format whose runs merge only the changed rows.

Reconcile sits beside all of this. It re-reads every item in the list and compares it with the copy, and it runs weekly by default, with the interval configurable. You can also start one with spsync reconcile from the command-line tool. Treat it as a QA check you can show an auditor, and as the way to bring the lake back in line after an outage on your side or Microsoft’s.

Getting SharePoint lists into Microsoft Fabric with a OneLake shortcut

SPSync has no Fabric destination of its own today, and you do not need one to get list data into a Lakehouse. Replicate to the Azure Data Lake Storage Gen2 destination, which writes the same folder-per-table layout described above, with a current.parquet replaced every run and a history file per run. In a Fabric Lakehouse, create a OneLake shortcut that points at that container. Fabric reads the files in place, with no copy.

The files show up in the Lakehouse’s Files area, not as Delta tables in the Tables area. To get managed tables, load current.parquet with a notebook or a pipeline on your own schedule. After that, the SQL analytics endpoint reads them, and so does a Power BI semantic model built on the Lakehouse. The Fabric page has the details and is the place to tell us how your workspace is set up.

Querying SharePoint data from Spark or DuckDB

Here is spend by supplier for live purchase requests, read straight from the Delta tables with DuckDB. It runs on a laptop with only DuckDB installed, and the credential_chain secret picks up the AWS credentials from the environment or an aws configure profile.

INSTALL delta;
LOAD delta;
CREATE SECRET (TYPE s3, PROVIDER credential_chain);

SELECT s.title          AS supplier,
       u.display_name   AS requester,
       count(*)         AS requests,
       sum(r.amount)    AS total_amount
FROM delta_scan('s3://lake/procurement/main/proc_purchase_requests') AS r
JOIN delta_scan('s3://lake/procurement/main/proc_suppliers') AS s
  ON s.sp_item_id = r.supplier_id
LEFT JOIN delta_scan('s3://lake/procurement/main/sp_users') AS u
  ON u.id = r.requester_user_id
WHERE r.sp_deleted_at IS NULL
  AND s.sp_deleted_at IS NULL
GROUP BY 1, 2
ORDER BY total_amount DESC;

In PySpark against the Parquet export, spend by category comes from the categories child table, which has one row per item and category. An item with two categories counts towards both.

base = "s3a://lake/procurement"
requests = spark.read.parquet(f"{base}/proc_purchase_requests/current.parquet")
categories = spark.read.parquet(f"{base}/proc_purchase_requests__categories/current.parquet")

live = requests.where("sp_deleted_at IS NULL")
(
    live.join(categories, "sp_item_id")
    .groupBy("value")
    .sum("amount")
    .orderBy("sum(amount)", ascending=False)
    .show()
)

For Iceberg, configure a Spark catalog against the same REST catalog and read spark.table("lake.procurement.proc_purchase_requests"), or attach the catalog in DuckDB with its iceberg extension. The SQL is the same as above once the tables are in scope.

Which lake format should you pick for SharePoint data?

Pick the one your engines already read. If the team runs Databricks or Spark on Delta, the Delta Lake destination merges in place, keeps its state in the bucket and writes only what changed. If you have an Iceberg REST catalog that Trino, Athena and Spark share, the Iceberg destination puts the lists in that catalog next to everything else. Plain Parquet through DuckDB is the lightest option: files any tool opens, a history file per run, and a DuckDB file on the side for the analyst who wants SQL and reporting views without a cluster. If Fabric is where the reports live, the ADLS Gen2 route and a shortcut get you there today.

You can also run more than one. The same lists can go to Delta for the data platform and to DuckDB for the finance analyst, one replication each, both reading SharePoint through the same delta queries. The data lake use case and dbt on SharePoint data go further into what teams build on top.