We have a use-case where we have all the queries querying within a specific partition of data. For example, our document looks like
**PersonID**: Unique id of the person
**First Name**: First Name of Person
**Last Name**: Last Name of Person
**City**: Name of City
**PinCode**: (9 digit in general but can be transformed into string for optimization)
**Description**: Description of Person
**Hobbies**: List of Hobbies
**Income_range**: Value Between 1-5
**Score**: Sorting Score
We have around 1 billion records with Pincode in order of 50 million. Each record is on average 1 KB in size. There can be updates changing the pincode of each record.
All our queries always specify the Pincode and then optionally add other filters. For example,
* Find Persons in PinCode=123 and Is_Senior_Citizen = 1
* Find Persons in PinCode=456 and income range = 4 and description contains "good"
All our queries are always run on a specific partition of data which is partitioned by PinCode. Further we need the output to be always sorted by Score irrespective of any relevancy criteria. Is there is way to optimize the queries.
I know there are [alias](https://www.elastic.co/guide/en/elasticsearch/reference/current/indices-aliases.html
) and [_routing](https://www.elastic.co/guide/en/elasticsearch/reference/6.1/mapping-routing-field.html
We can't use one alias for each PinCode as pincodes change for a person very frequently. Further each alias create a different index and having more than 50 million index doesn't look scalable.
For routing, if we use PinCode as parameter to do custom routing, when we change the cluster size, does elastic search automatically re-indexes the records to the right shards/nodes. Further, do we know how much performance boost can be expect.
I am open to other suggestions on how can we setup the cluster and improve the performance. Our queries need to serve a live website so are looking for good querying latency.