I'm not that experienced with helix, so wait and see if anyone offers any

My understanding of helix is that it isn't trivial to dynamically add a
partition, and you in any case wouldn't want thousands of partitions or

For a problem like yours, I would define one task with say 100 partitions.
When the load balancer receives an entity id, you could use something like
consistent hashing to identify what partition that id belongs to.

That would also to some degree reduce your need for resource weights, as
averaged over thousand random entities, each partition should roughly be
the same. I'm not aware of any concept like resource/partition weights but
you can probably achieve the same effect by using custom rebalanced.

On Thu, 12 Jul 2018, 16:56 Diot Sébastien, <[EMAIL PROTECTED]> wrote: