SharePoint to Elasticsearch

Last updated 23 October 2026

One way: SharePoint stays the system of record, and SPSync keeps a read-only copy of your lists current in Elasticsearch.

Status: Shipped · product: SPSync for Search & Vector Stores

SPSync sends SharePoint list items one way into Elasticsearch indexes with explicit mappings, for keyword and filtered search, and writes no dense_vector field. Vector or hybrid search over these documents needs an embedding job of your own that reads the text, embeds it and keeps the vectors in an index you manage. Embeddings on sync for Elasticsearch are an idea under consideration, with no plan or date; SPSync writes them today only to Postgres with pgvector and Azure AI Search.

If your organisation already runs the Elastic stack, SharePoint lists become one more set of indexes on the cluster, searchable with the same query DSL and explorable in Kibana next to whatever else you index.

Connecting to the cluster

Give SPSync the cluster URL, for example https://my-cluster.example.com:9200 on your own servers or the endpoint Elastic Cloud shows, and one of two credentials held in environment variables:

• an Elasticsearch API key, or
• a username and password for HTTP basic auth.

SPSync keeps a local state file on the sync host to track what it has sent, so give that file a stable path.

One index per table

Every SPSync table becomes an index: one per list, one per multi-value column, and one shared sp_users index for people. A service catalogue list, it_services, gives an it_services index plus it_services__platforms for a multi-choice Platforms column. Documents carry a lookup’s ID and display value, and a person column’s user ID, which joins to the sp_users index for names and email.

The mapping is explicit. IDs are keyword, matched as whole values and never analysed. Text, choice values and lookup display values are analysed text, good for relevance search. Numbers, dates and Yes/No fields keep their types, so range filters and aggregations work on them. Elasticsearch cannot change a field’s type in place, so when a SharePoint column changes type SPSync creates a new {index}_v2 index with the fresh mapping and writes there from then on.

What each run sends

Only documents whose content changed since the last successful push are sent, plus deletions. A reconcile, weekly by default, re-sends everything, which repairs documents changed or removed in the cluster by hand. A failed push is retried on the next run.

How it gets used

A dedicated search page over SharePoint lists, keyword retrieval for chat over your SharePoint data, and a search tool for AI agents. Pipelines that need vector or hybrid retrieval use pgvector or Azure AI Search instead; see RAG on SharePoint lists.

Steps

1. Connect to SharePoint with delegated sign-in, or with an app that an admin has granted Sites.Selected on the sites to index.
2. Create an API key, or a user, that can create indexes and write documents.
3. Set the cluster URL and the credential variables.
4. Choose the lists and run; the first run creates the indexes and loads every item.
5. Point your search UI or Kibana at the indexes.

Shard counts and mapping choices affect cluster cost as lists grow. SharePoint permissions are not carried into the index, so cluster and index access is the boundary; scope read access accordingly.

Watch it run

Watch the full video

Chapters are in the video's description.

Watch the walkthrough: the Northwind Analytics demo lists in SharePoint, connecting to Microsoft 365 in the desktop app, picking lists, the run into an empty Elasticsearch cluster, and the replicated indices searched in Kibana.

spsync auth login --client-id 00000000-0000-0000-0000-000000000000
spsync export-config --output replication.yaml --pipeline-name elasticsearch --prefix elasticsearch --destination-kind elasticsearch --connection-string-env SPSYNC_DB_CONNECTION --dataset-name elasticsearch --client-id 00000000-0000-0000-0000-000000000000 --site-url https://contoso.sharepoint.com/sites/example --all-lists
spsync doctor replication.yaml
spsync run replication.yaml

pipeline_name: elasticsearch
prefix: elasticsearch
auth:
  mode: delegated
  client_id: 00000000-0000-0000-0000-000000000000
  tenant_id: common
destination:
  kind: elasticsearch
  connection_string_env: SPSYNC_DB_CONNECTION
  dataset_name: elasticsearch
scopes:
  - site_url: https://contoso.sharepoint.com/sites/example
    mode: entire_site
reconcile_interval_days: 7
create_reporting_views: false

# every night at 02:00
0 2 * * * cd /opt/spsync && SPSYNC_DB_CONNECTION="$(cat /etc/spsync/db.env)" uv run spsync run replication.yaml >> /var/log/spsync/run.log 2>&1

# .github/workflows/spsync-run.yml
name: SPSync
on:
  schedule:
    - cron: "0 2 * * *"
  workflow_dispatch: {}
jobs:
  run:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: astral-sh/setup-uv@v3
      - run: uv sync
      - run: uv run spsync run replication.yaml
        env:
          SPSYNC_DB_CONNECTION: ${{ secrets.SPSYNC_DB_CONNECTION }}
          SPSYNC_LICENCE_KEY: ${{ secrets.SPSYNC_LICENCE_KEY }}

# azure-pipelines.yml
schedules:
  - cron: "0 2 * * *"
    branches: { include: [main] }
    always: true
steps:
  - script: uv sync
  - script: uv run spsync run replication.yaml
    env:
      SPSYNC_DB_CONNECTION: $(SPSYNC_DB_CONNECTION)
      SPSYNC_LICENCE_KEY: $(SPSYNC_LICENCE_KEY)

# Windows Task Scheduler action (see /docs/cli/automation/)
$env:SPSYNC_DB_CONNECTION = (Get-Content C:\spsync\db.env)
uv run spsync run C:\spsync\replication.yaml
schtasks /Create /SC DAILY /ST 02:00 /TN "SPSync" /TR "powershell -File C:\spsync\run.ps1"

# ~/Library/LaunchAgents/io.spsync.run.plist, then: launchctl load ~/Library/LaunchAgents/io.spsync.run.plist