On Jun 2, 2017 6:50 PM, "Subramanian Raghunathan" < [EMAIL PROTECTED]> wrote:
> Hi kishore, > > > > Trying to manage a cluster with a resource that has > Dynamic number of partitions please provide me your valuable thoughts on > same. > > > > Version(s) : > > Helix: 0.7.1 > > Zookeeper:3.3.4 > > > > -Zookeeper quorum (3 nodes) > > - State Model: OnlineOffline > > - Controller (leader elected from one of the cluster nodes) > > - Single resources with partitions. > > - Custom rebalancer > > > > Few of the approaches that I could think of ... > > > > Approach 1: *Buffer Partitions* > > Let’s say the current number of partitions are N. > > 1 ) Define N+X (Where x could be the > delta where it can expand maximum , expecting to be of few in number > (1-10)) > > 2 ) Use property store to identify which > are all valid partition(s) > > 3 ) Don’t act upon invalid ones by > looking up property store (or) Disable the unallocated partition(s) > > > > When a new partition to be added - > > 1. Update property store of new partitions (Delta merge) > 2. Enable the partition(s) > > > > --- The balancing algorithm should be able to balance valid & extra > partition(s) differently. Lots of customizations. > > > > Approach 2: *New Resource Addition* > > Let’s say the current number of partitions are N. > > > > Create Resource 1 with N partitions initially. > > When a new partition(s) to be added - > > 1. Drop the resource > 2. Create new resource with N+X (newly added partitions ) > > > > --- Caveat that I could see is Dropping the resource would stop the entire > workflow (It’s a bigger price to pay for us since we are real time). > > > > > > Thanks & Regards, > > Subramanian. > > > > Tel: +1 (650) 424 4655 <(650)%20424-4655> > > Mob:+1(650) 656 6006 <(650)%20656-6006> > > > > 850 Hansen Way, > > Palo Alto, CA 94304 > > www.integral.com > > [image: Logo_signature_block] > <http://www.integral.com/fxcloud_features/risk_management.html#ym> > > NOTICE: This e-mail message and any attachments, which may contain > confidential information, are to be viewed solely by the intended recipient > of Integral Development Corp. For further information, please visit > http://www.integral.com/about/disclaimer.html. > > > > > > >
NEW: Monitor These Apps!
Apache Lucene, Apache Solr and all other Apache Software Foundation project and their respective logos are trademarks of the Apache Software Foundation.
Elasticsearch, Kibana, Logstash, and Beats are trademarks of Elasticsearch BV, registered in the U.S. and in other countries. This site and Sematext Group is in no way affiliated with Elasticsearch BV.
Service operated by Sematext