Hi,

                Trying to follow up if there are any thoughts/suggestions over the approaches.

                Trying to manage a cluster with a resource that has Dynamic number of partitions please provide me your valuable thoughts on same.
 Version(s) :
                                Helix: 0.7.1
                                Zookeeper:3.3.4
 -Zookeeper quorum (3 nodes)
- State Model: OnlineOffline
- Controller (leader elected from one of the cluster nodes)
- Single resources with partitions.
- Custom rebalancer
 Few of the approaches that I could think of ...
 Approach 1: Buffer Partitions
                Let’s say the current number of partitions are N.
                                1 ) Define N+X (Where x could be the delta where it can expand maximum , expecting to be of few in number (1-10))
                                2 ) Use property store to identify which are all valid partition(s)
                                3 ) Don’t act upon invalid ones by looking up property store (or) Disable the unallocated partition(s)

                When a new partition to be added -
1.       Update property store of new partitions (Delta merge)
2.       Enable the partition(s)
 --- The balancing algorithm should be able to balance valid & extra partition(s) differently. Lots of customizations.
 Approach 2: New Resource Addition
                Let’s say the current number of partitions are N.
                Create Resource 1 with N partitions initially.
                When a new partition(s) to be added -
1.       Drop the resource
2.       Create new resource with N+X (newly added partitions )
 --- Caveat that I could see is Dropping the resource would stop the entire workflow (It’s a bigger price to pay for us since we are real time).

Thanks & Regards,
Subramanian.

From: kishore g [mailto:[EMAIL PROTECTED]]
Sent: Friday, June 02, 2017 9:52 PM
To: Subramanian Raghunathan <[EMAIL PROTECTED]>; [EMAIL PROTECTED]
Subject: Re: Dynamic partition management
+helix

On Jun 2, 2017 6:50 PM, "Subramanian Raghunathan" <[EMAIL PROTECTED]<mailto:[EMAIL PROTECTED]>> wrote:
Hi kishore,

                Trying to manage a cluster with a resource that has Dynamic number of partitions please provide me your valuable thoughts on same.

Version(s) :
                                Helix: 0.7.1
                                Zookeeper:3.3.4

-Zookeeper quorum (3 nodes)
- State Model: OnlineOffline
- Controller (leader elected from one of the cluster nodes)
- Single resources with partitions.
- Custom rebalancer

Few of the approaches that I could think of ...

Approach 1: Buffer Partitions
                Let’s say the current number of partitions are N.
                                1 ) Define N+X (Where x could be the delta where it can expand maximum , expecting to be of few in number (1-10))
                                2 ) Use property store to identify which are all valid partition(s)
                                3 ) Don’t act upon invalid ones by looking up property store (or) Disable the unallocated partition(s)

                When a new partition to be added -
3.       Update property store of new partitions (Delta merge)
4.       Enable the partition(s)

--- The balancing algorithm should be able to balance valid & extra partition(s) differently. Lots of customizations.

Approach 2: New Resource Addition
                Let’s say the current number of partitions are N.

                Create Resource 1 with N partitions initially.
                When a new partition(s) to be added -
3.       Drop the resource
4.       Create new resource with N+X (newly added partitions )

--- Caveat that I could see is Dropping the resource would stop the entire workflow (It’s a bigger price to pay for us since we are real time).
Thanks & Regards,
Subramanian.

Tel: +1 (650) 424 4655<tel:(650)%20424-4655>
Mob:+1(650) 656 6006<tel:(650)%20656-6006>

850 Hansen Way,
Palo Alto, CA 94304
www.integral.com<http://www.integral.com/>
[Logo_signature_block]<http://www.integral.com/fxcloud_features/risk_management.html#ym>

NOTICE: This e-mail message and any attachments, which may contain confidential information, are to be viewed solely by the intended recipient of Integral Development Corp. For further information, please visit http://www.integral.com/about/disclaimer.html.
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB