Avoiding and Correcting Hotspots: How Elasticsearch Serverless Balances Shards

Elasticsearch Serverless replaces the Elasticsearch node-weight based shard rebalancing algorithm with resource usage aware rebalancing that avoids index shard colocation, OOM events and write load hotspotting Free yourself from operations with Elastic Cloud Serverless . Scale automatically, handle load spikes, and focus on building—start a 14-day free trial to test it out yourself! You can follow these guides to build an AI-Powered search experience or search across business systems and software .

The Elasticsearch Serverless Balancer addresses write load hotspots, prevents data node out-of-memory (OOM) events and avoids index-level hotspots in Elasticsearch Serverless clusters: these are workload edge cases that in non-Serverless require manual intervention and custom tuning of cluster settings. Serverless shard balancing focuses on staying within the bounds of node-level resource constraints. Rebalancing moves are explainable, where moves are made explicitly to either avoid performance degradation or correct hotspots when they develop. Shard movements are generally found to be fewer, as well.

Elasticsearch uses a weights-based algorithm to create a Desired Balance, an assignment of shards to data nodes. The Balancer determines the target allocation of shards across a cluster of nodes using four key metrics weighted in a linear algorithm. A total weight is calculated per node, and the shard balancer aims to equalize the total weights across cluster nodes. A final Desired Balance shard allocation is precomputed based on the latest cluster state information, and then the elected master node initiates incremental shard moves to reach the desired shard allocation.

Write Load: the total write threadpool activity per node, using the sum of threadpool indexing activity per data-stream shard. Disk Usage: the total disk usage of shards per node, using the sum of disk space used per shard. Shard Count: the total number of shards assigned to a node. Index Balance (shard anti-affinity): per index, how many shards in the index are assigned to the node.