Connect SPSync to Elasticsearch

SPSync writes one Elasticsearch index per table: a list’s main table, each multi-value child table and the shared users table. Rows are indexed by key, and a row deleted in SharePoint is removed from the index rather than flagged. This guide covers a cluster you run yourself and Elastic Cloud.

Before you start

• An Elasticsearch cluster you can reach over HTTP(S), and its URL, for example https://my-cluster.es.example.com:9200.
• A user who can create roles, users or API keys on the cluster (the superuser role, or Kibana access with the right privileges).
• Nothing to install on the machine that runs SPSync. SPSync keeps a small local state file (a DuckDB file) that records what it last sent, so it sends only the rows that changed. Keep that file on a disk that survives restarts.

Create the database and a least-privilege user

SPSync needs to create indexes, add fields to them, write and delete documents and read them back. It needs nothing at cluster level beyond a reachable endpoint. This request is from the Elasticsearch documentation and has not been run here against a test cluster. In Kibana Dev Tools, or with curl, create a role:

PUT _security/role/spsync_loader
{
  "indices": [
    {
      "names": ["dev_*", "sp_users"],
      "privileges": ["create_index", "write", "read", "view_index_metadata", "manage"]
    }
  ]
}

dev_* is the --prefix you give SPSync: every index is named from that prefix and the list name. Use your own prefix. After the first sync, list the indexes (GET _cat/indices) and tighten the pattern if you want to. manage is what lets SPSync add a field to an index that already exists when a SharePoint column is added. When a column changes type, Elasticsearch cannot change the field in place, so SPSync starts a new index with _v2 on the end and writes there from then on. The pattern dev_* already covers it.

Then make an API key from that role (recommended) or a user:

POST _security/api_key
{
  "name": "spsync",
  "role_descriptors": {
    "spsync_loader": {
      "indices": [
        {
          "names": ["dev_*", "sp_users"],
          "privileges": ["create_index", "write", "read", "view_index_metadata", "manage"]
        }
      ]
    }
  }
}

The response carries an encoded value. That is the key SPSync needs. It is shown once.

Allow SPSync to reach it

Elasticsearch listens on TCP 9200 for HTTP unless you changed it. On the desktop app and a self-hosted server the connection starts from that machine, so allow that machine’s address. For the Azure managed application, allow the address it connects out from.

• Self-hosted: allow the SPSync address in the host firewall and in any load balancer in front of the cluster.
• Elastic Cloud: under Security, Traffic filters, add the address as an IP filter, or use a private link filter and run SPSync inside that network. Without a filter the deployment is open to the internet, protected only by the key.
• Elastic Cloud on Kubernetes and other managed hosts: allow the address on whichever ingress or security group fronts port 9200 or 443.

Encrypt the connection

Use an https:// URL. SPSync always verifies the server’s certificate, and there is no setting today to point it at a private certificate authority or to switch verification off. So the cluster’s certificate must chain to a root the machine already trusts, which Elastic Cloud’s does. A self-hosted cluster with a self-signed certificate or a private CA fails the check until you install a certificate from a public authority, or add your CA to the operating system’s trust store on the machine that runs SPSync. SPSync does not rewrite the URL, so an http:// address sends the key in clear text. Do not use one outside a closed test network.

Authentication SPSync supports today

Either an Elasticsearch API key, or a user name and password (HTTP basic). Use exactly one. Elasticsearch service account tokens, certificates and single sign-on are not supported. An API key is better: it can be limited to the role above and revoked without touching a user.

Connect from SPSync

• Desktop app: open the Destination screen, choose Elasticsearch, enter the Elasticsearch URL and either the API key or the username and password, then choose Test connection. SPSync manages the local state file itself.

• CLI: the config names environment variables, never the key itself. SPSYNC_ES_STATE_PATH holds the path of the local state file, not a network address.

  export SPSYNC_ES_STATE_PATH=/var/lib/spsync/elasticsearch-state.duckdb
  export SPSYNC_ES_API_KEY='<encoded key from the response above>'
  spsync export-config --output replication.yaml \
    --site-url https://contoso.sharepoint.com/sites/x --list Projects \
    --pipeline-name spsync_dev --prefix dev \
    --destination-kind elasticsearch --connection-string-env SPSYNC_ES_STATE_PATH \
    --dataset-name main \
    --elasticsearch-url https://my-cluster.es.example.com:9200 \
    --elasticsearch-api-key-env SPSYNC_ES_API_KEY \
    --client-id <entra-app-client-id>

  For basic auth, replace the last key flag with --elasticsearch-username spsync_loader --elasticsearch-password-env SPSYNC_ES_PASSWORD.

• AI assistant: with the admin MCP server connected, ask for the destination requirements, then let the assistant create the destination with the URL. You enter the API key (or the password, if you set a user name) on the secure entry page, never in the chat.

What Test connection and doctor check

Both ask the cluster to respond to a ping. A pass means the URL is reachable, the certificate is trusted and the cluster answered. It does not prove the key can create or write to an index, and some deployments answer a ping to any caller, so it does not prove the key is valid either. The first sync is the real test of the privileges. spsync doctor reports the probe as destination-connectivity.

Common errors

What you see	What it means and the fix
Connection refused or a timeout	A firewall, security group or traffic filter is blocking the port, or the URL is wrong.
SSL: CERTIFICATE_VERIFY_FAILED	The certificate is self-signed or from a private CA. Use a certificate from a public authority, or trust your CA on the machine that runs SPSync.
401 security_exception or AuthenticationException	The key is wrong, expired or revoked, or the password is wrong. Check that you used the encoded value.
403 security_exception ... is unauthorized for	The key’s role does not cover the index. Widen the names pattern to match your prefix.
could not ping Elasticsearch cluster	The cluster answered but is not healthy, or a proxy is in the way. Check the cluster from the same machine with curl.
mapper_parsing_exception	A document does not fit an existing field. SPSync starts a _v2 index after a column changes type; if you changed an index by hand, delete it and run a reconcile.