Reading the examples for table design I've come across a question associated with the document partitioned index, specifically what is typically chosen as the BinId or maybe more appropriately what factors should influence what is chosen as the BinId and what impact do they have?
Often it'll be a hash of the document mod the number of bins you're using. The hash should be "good" in the sense that it uniquely identifies the document. It can be as simple as some unique field in the document or just a hash (like murmur) of the whole document.
On Saturday, February 6, 2016, Jamie Johnson <[EMAIL PROTECTED]> wrote:
You can get *really* fancy if you have lots of ingesters and lots of servers, include some attribute in the data you're hashing to control how many servers a given client will need to write to for some batch of documents. This is probably overkill for most setups though.
Guava provides a decent murmur3 implementation which will be much faster than your run-of-the-mill MD5 for generating the hash (which you'll mod by the max number of bins).
Thanks guys. I was also looking at some of the examples and saw the event store, I like the idea of including time as a prefix to the binning to limit the number of servers that need to be hit for time bound queries. Without something like this queries end up having to hit all tablets right? It's not always a full table scan since the iterators can bail on a row part way through but still needs to hit every row to some extent right?
I also was looking at the wiki example but wasn't able to find a good description of how all the tables are used, does anything more exist? On Feb 6, 2016 2:20 PM, "Josh Elser" <[EMAIL PROTECTED]> wrote:
For the wikisearch schemas, the general idea is that the inverted index tables can prune your row space for some terms. This way, you can know the exact rows you have to search in the sharded table to get good parallelism without a full-table scan.