SharePoint to Postgres with pgvector, embeddings generated on the way

Last updated 23 October 2026

One way: SharePoint stays the system of record, and SPSync keeps a read-only copy of your lists current in Postgres with pgvector.

Status: Shipped · product: SPSync for Search & Vector Stores

SPSync replicates SharePoint lists one way into Postgres and, on every sync, writes the vectors itself: an embedding per row from the text columns you pick, made with OpenAI, Azure OpenAI or a local model. Only rows whose text changed are embedded again. pgvector puts similarity search inside Postgres, next to the columns you filter on, so one query can filter on status, owner or date and rank by meaning in the same statement, which is what most people mean by semantic search over SharePoint.

Requirements

• PostgreSQL 15 or newer. SPSync updates embedded tables with MERGE, which Postgres added in version 15. On version 13 or 14 the first run succeeds and the second fails, so spsync doctor checks the version up front.
• The vector extension, already created or creatable by the sync’s database role. spsync doctor checks that too.
• An embedding provider. OpenAI or Azure OpenAI, with an API key in an environment variable and, for Azure OpenAI, the endpoint and the deployment name as the model; or a local model that needs no key.

How are SharePoint embeddings built in Postgres?

You list the text columns to embed, for example title and summary. SPSync joins their values in that order into one piece of text per row and stores its embedding in a vector(n) column, sp_embedding by default, on the same table, with an HNSW index for fast similarity search. You can send vectors to a side table instead.

A companion sp_embedding_hash column records a hash of each row’s source text and the model name. Each run embeds only rows whose hash changed, so the embedding bill follows how much text changes, not the size of the list.

The relational side matches the Postgres destination: one table per list, {col}_id and {col}_value for lookups, {col}_user_id for people joined to sp_users, and a child table per multi-value column. One difference: a SharePoint column that changes type is not forked into a _v2 column on pgvector today, because embeddings are found by column name.

Uses

Teams building semantic search over free-text columns keep the source rows and the vectors in one database. A RAG pipeline uses pgvector as both the structured store and the vector store. Chat over SharePoint data and AI agents query it with a filter and a distance in one statement, with no separate vector service.

Steps

1. Give SPSync SharePoint access: delegated sign-in, or Sites.Selected app permissions on the sites, approved by an admin, for runs with no one signed in.
2. Store the Postgres connection string in an environment variable.
3. Choose the provider, model, dimensions and text columns to embed.
4. Run spsync doctor to check the server version and extension, then run the sync. The first run creates the tables, the vector column and the HNSW index.
5. Point your retrieval code at the same database.

Row-level security applies to the replicated tables as to any Postgres table, which helps when different users should see different rows in a search result. Hard delete cannot be combined with the side-table option, because the side table is only ever added to.

spsync auth login --client-id 00000000-0000-0000-0000-000000000000
spsync export-config --output replication.yaml --pipeline-name pgvector --prefix pgvector --destination-kind pgvector --connection-string-env SPSYNC_DB_CONNECTION --dataset-name pgvector --client-id 00000000-0000-0000-0000-000000000000 --site-url https://contoso.sharepoint.com/sites/example --all-lists
spsync doctor replication.yaml
spsync run replication.yaml

pipeline_name: pgvector
prefix: pgvector
auth:
  mode: delegated
  client_id: 00000000-0000-0000-0000-000000000000
  tenant_id: common
destination:
  kind: pgvector
  connection_string_env: SPSYNC_DB_CONNECTION
  dataset_name: pgvector
scopes:
  - site_url: https://contoso.sharepoint.com/sites/example
    mode: entire_site
reconcile_interval_days: 7
create_reporting_views: false

# every night at 02:00
0 2 * * * cd /opt/spsync && SPSYNC_DB_CONNECTION="$(cat /etc/spsync/db.env)" uv run spsync run replication.yaml >> /var/log/spsync/run.log 2>&1

# .github/workflows/spsync-run.yml
name: SPSync
on:
  schedule:
    - cron: "0 2 * * *"
  workflow_dispatch: {}
jobs:
  run:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: astral-sh/setup-uv@v3
      - run: uv sync
      - run: uv run spsync run replication.yaml
        env:
          SPSYNC_DB_CONNECTION: ${{ secrets.SPSYNC_DB_CONNECTION }}
          SPSYNC_LICENCE_KEY: ${{ secrets.SPSYNC_LICENCE_KEY }}

# azure-pipelines.yml
schedules:
  - cron: "0 2 * * *"
    branches: { include: [main] }
    always: true
steps:
  - script: uv sync
  - script: uv run spsync run replication.yaml
    env:
      SPSYNC_DB_CONNECTION: $(SPSYNC_DB_CONNECTION)
      SPSYNC_LICENCE_KEY: $(SPSYNC_LICENCE_KEY)

# Windows Task Scheduler action (see /docs/cli/automation/)
$env:SPSYNC_DB_CONNECTION = (Get-Content C:\spsync\db.env)
uv run spsync run C:\spsync\replication.yaml
schtasks /Create /SC DAILY /ST 02:00 /TN "SPSync" /TR "powershell -File C:\spsync\run.ps1"

# ~/Library/LaunchAgents/io.spsync.run.plist, then: launchctl load ~/Library/LaunchAgents/io.spsync.run.plist