Library metadata as the catalogue that tells an AI which documents matter

Last updated 20 October 2026

Every AI project that touches our document libraries starts by re-scraping the same metadata from the Graph API, and it is slow and rate-limited.

Three weeks before an ISO 27001 surveillance audit, the auditor asks for an inventory of controlled documents. They want every current policy in the compliance Policies library with its owner and last review date, plus anything overdue for review. Someone usually builds that by exporting a library view to Excel and tidying it by hand, then does it again when the auditor sends a follow-up question.

An AI assistant can build the inventory, provided it can find out which documents exist and what state they are in before it opens a single file. Fetching that from the Graph API on every question is slow and rate-limited, and there is nowhere to keep it between runs, so the same metadata is fetched again and again. The Policies library holds a few thousand documents, and the inventory should list only the current, approved version of each.

Keep the catalogue in a table

SPSync replicates a document library’s metadata into a table and refreshes it on a schedule. Each row carries the item’s ID, content type, dates, author and editor, and the library’s custom columns, such as status, owner, department or review date. The files stay in SharePoint, and when a pipeline needs the content of one, it fetches that file from Graph using the IDs on the row.

For the audit, the assistant builds the inventory from reads against that table:

1. Filter policies to the content type your controlled documents use and to an approved status.
2. Drop rows marked superseded or withdrawn, and any row with sp_deleted_at set.
3. Join the owner column to sp_users for each owner’s name and email address.
4. Flag every row whose review date has passed.

The result is the inventory, a table the compliance lead reviews before it goes to the auditor. They check the flagged rows and correct anything out of date in SharePoint themselves before sending it. The assistant has no write path to SharePoint, since SPSync never writes to it, so a fix to an owner or a review date is always made by a person.

Two gaps to plan around today: managed metadata columns are not replicated yet, and on a live tenant the file name, URL, size, hash and folder path columns are not populated yet. Build the catalogue on the library’s own columns for now.

How the rows are shaped

There is one table per library, one row per file, and a stable primary key that survives renames and moves. A lookup column is a key column into the table it points at. A multi-choice column is a child table, so a policy with three tags is three rows you can join on. Multi-person columns, such as reviewers, are still being verified on a live tenant. SPSync can also generate a view per library that adds person names, so a catalogue query can skip the join to sp_users.

Deleted files keep their row with sp_deleted_at set, so the catalogue can tell a withdrawn policy from one that never existed. To have the table match SharePoint’s current state, hard delete is a per-replication setting in the command-line tool’s config.

Choosing where the catalogue lives

Postgres and SQL Server both work today. Sign in to the site with delegated access or an admin-granted Sites.Selected permission, pick the libraries, and set a schedule; every few minutes is common for a catalogue an assistant reads all day.

If the end goal is search, the metadata can go straight to an index. pgvector and Azure AI Search generate embeddings on sync, and Elasticsearch and OpenSearch ship as keyword search indexes.

The article on library metadata as the catalogue an AI needs works through the Policies library as a SQL catalogue, with the query the assistant runs to choose current, approved policies before it fetches and reads any file.

spsync auth login --client-id 00000000-0000-0000-0000-000000000000
spsync export-config --output replication.yaml --pipeline-name document_metadata_for_ai --prefix document_metadata_for_ai --destination-kind postgres --connection-string-env SPSYNC_DB_CONNECTION --dataset-name document_metadata_for_ai --client-id 00000000-0000-0000-0000-000000000000 --site-url https://contoso.sharepoint.com/sites/example --all-lists
spsync doctor replication.yaml
spsync run replication.yaml

pipeline_name: document_metadata_for_ai
prefix: document_metadata_for_ai
auth:
  mode: delegated
  client_id: 00000000-0000-0000-0000-000000000000
  tenant_id: common
destination:
  kind: postgres
  connection_string_env: SPSYNC_DB_CONNECTION
  dataset_name: document_metadata_for_ai
scopes:
  - site_url: https://contoso.sharepoint.com/sites/example
    mode: entire_site
reconcile_interval_days: 7
create_reporting_views: false

# every night at 02:00
0 2 * * * cd /opt/spsync && SPSYNC_DB_CONNECTION="$(cat /etc/spsync/db.env)" uv run spsync run replication.yaml >> /var/log/spsync/run.log 2>&1

# .github/workflows/spsync-run.yml
name: SPSync
on:
  schedule:
    - cron: "0 2 * * *"
  workflow_dispatch: {}
jobs:
  run:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: astral-sh/setup-uv@v3
      - run: uv sync
      - run: uv run spsync run replication.yaml
        env:
          SPSYNC_DB_CONNECTION: ${{ secrets.SPSYNC_DB_CONNECTION }}
          SPSYNC_LICENCE_KEY: ${{ secrets.SPSYNC_LICENCE_KEY }}

# azure-pipelines.yml
schedules:
  - cron: "0 2 * * *"
    branches: { include: [main] }
    always: true
steps:
  - script: uv sync
  - script: uv run spsync run replication.yaml
    env:
      SPSYNC_DB_CONNECTION: $(SPSYNC_DB_CONNECTION)
      SPSYNC_LICENCE_KEY: $(SPSYNC_LICENCE_KEY)

# Windows Task Scheduler action (see /docs/cli/automation/)
$env:SPSYNC_DB_CONNECTION = (Get-Content C:\spsync\db.env)
uv run spsync run C:\spsync\replication.yaml
schtasks /Create /SC DAILY /ST 02:00 /TN "SPSync" /TR "powershell -File C:\spsync\run.ps1"

# ~/Library/LaunchAgents/io.spsync.run.plist, then: launchctl load ~/Library/LaunchAgents/io.spsync.run.plist