Retrieval-augmented generation over list data with pgvector or Azure AI Search

Last updated 23 October 2026

We want to build RAG over our SharePoint lists, but there is no clean, typed, up-to-date source to build embeddings from.

A retrieval pipeline is only as current as the text it embeds. An HR assistant that answers questions from policy summaries and guidance notes kept in SharePoint lists will quote last month’s wording if the embedding job that feeds it reads the whole list through Graph each night, resolves lookups and guesses at deletes in its own code, and then fails or runs late.

Let the sync write the embeddings

SPSync replicates the lists into pgvector or Azure AI Search and generates embeddings for the text columns you configure on every sync. The RAG pipeline queries that table or index for retrieval, and there is no separate embedding job to write, run or keep in step with the source. Delta sync already knows what changed, so only changed rows are re-embedded, and the cost follows the volume of edits.

pgvector or Azure AI Search

Elasticsearch and OpenSearch replicate the same lists into a keyword search index, with no embedding field.

Wiring it into the pipeline

Pick the site and the lists to retrieve over, then the destination. Choose which text columns to embed and which embedding model to use, and set a schedule that matches how often the source changes. Point the RAG pipeline or agent framework at the same database or index. SharePoint to pgvector and SharePoint to Azure AI Search cover the setup and schema for each.

Further reading

The article on keeping a SharePoint vector index current shows how embeddings go out of date, and how re-embedding only changed rows keeps an HR policies assistant current.

The article on Graph throttling and AI pipelines explains why ingestion jobs and agent tools get 429 responses from SharePoint, and how reading once into a replica takes Graph out of query time.

The article on SharePoint lists in Foundry IQ after On Your Data retired moves an Azure OpenAI On Your Data app over a list onto Foundry IQ, with the index SPSync feeds kept in place.

spsync auth login --client-id 00000000-0000-0000-0000-000000000000
spsync export-config --output replication.yaml --pipeline-name rag_on_sharepoint_lists --prefix rag_on_sharepoint_lists --destination-kind postgres --connection-string-env SPSYNC_DB_CONNECTION --dataset-name rag_on_sharepoint_lists --client-id 00000000-0000-0000-0000-000000000000 --site-url https://contoso.sharepoint.com/sites/example --all-lists
spsync doctor replication.yaml
spsync run replication.yaml

pipeline_name: rag_on_sharepoint_lists
prefix: rag_on_sharepoint_lists
auth:
  mode: delegated
  client_id: 00000000-0000-0000-0000-000000000000
  tenant_id: common
destination:
  kind: postgres
  connection_string_env: SPSYNC_DB_CONNECTION
  dataset_name: rag_on_sharepoint_lists
scopes:
  - site_url: https://contoso.sharepoint.com/sites/example
    mode: entire_site
reconcile_interval_days: 7
create_reporting_views: false

# every night at 02:00
0 2 * * * cd /opt/spsync && SPSYNC_DB_CONNECTION="$(cat /etc/spsync/db.env)" uv run spsync run replication.yaml >> /var/log/spsync/run.log 2>&1

# .github/workflows/spsync-run.yml
name: SPSync
on:
  schedule:
    - cron: "0 2 * * *"
  workflow_dispatch: {}
jobs:
  run:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: astral-sh/setup-uv@v3
      - run: uv sync
      - run: uv run spsync run replication.yaml
        env:
          SPSYNC_DB_CONNECTION: ${{ secrets.SPSYNC_DB_CONNECTION }}
          SPSYNC_LICENCE_KEY: ${{ secrets.SPSYNC_LICENCE_KEY }}

# azure-pipelines.yml
schedules:
  - cron: "0 2 * * *"
    branches: { include: [main] }
    always: true
steps:
  - script: uv sync
  - script: uv run spsync run replication.yaml
    env:
      SPSYNC_DB_CONNECTION: $(SPSYNC_DB_CONNECTION)
      SPSYNC_LICENCE_KEY: $(SPSYNC_LICENCE_KEY)

# Windows Task Scheduler action (see /docs/cli/automation/)
$env:SPSYNC_DB_CONNECTION = (Get-Content C:\spsync\db.env)
uv run spsync run C:\spsync\replication.yaml
schtasks /Create /SC DAILY /ST 02:00 /TN "SPSync" /TR "powershell -File C:\spsync\run.ps1"

# ~/Library/LaunchAgents/io.spsync.run.plist, then: launchctl load ~/Library/LaunchAgents/io.spsync.run.plist