We are facing some issues with shard allocation (both primary and replica) after upgrading the ES cluster size. We had a three node elasticsearch cluster and we recently added two more nodes into this cluster.
The two new nodes are only data nodes (node.master: false, node.data: true).
The problem right now we're facing is, the newly created indexes are only allocated to the two new nodes (ES4 & ES5 - in our case). And the replica shards / primary shards are not getting created / allocated to old nodes (ES1, ES2 and ES3).
1. Is this an expected behaviour? if so, how we can split the shards across all nodes and the replica shard? 2. We are planning to upgrade the node size to 10 or more. So how we can distribute the shard allocation across all nodes?
Shards, which are created by a higher versions, cannot be allocated to lower version nodes (in some cases). We had the same issue with shards from version 6 which did not get allocated to ES 5 nodes :slight_smile:
I can see this error from _cluster/allocation/explain
"decider" : "throttling", "decision" : "THROTTLE", "explanation" : "reached the limit of incoming shard recoveries , cluster setting [cluster.routing.allocation.node_concurrent_incoming_recoveries=2] (can also be set via [cluster.routing.allocation.node_concurrent_recoveries])" }
It's not trying to allocate on old nodes. I tried to increase the above value, after that the replica shards also allocated only to new nodes with status "yellow."
Elasticsearch generally prefers to allocate shards to empty nodes, so if you add a node and then create some indices before the cluster has finished rebalancing then it will allocate more shards to the new nodes than you might expect.
Thanks @Christian_Dahlqvist All the specifications are same for all five nodes.
Thanks @DavidTurner for the explanation. I tried different combination of shard balancing heuristics value and the shards are only creating to new nodes. I will try the setting you suggested and test it again.
So eventually it will allocated to all nodes, correct?