Perfect , Thanks Lei Xia .

From: Lei Xia [mailto:[EMAIL PROTECTED]]
Sent: Wednesday, June 07, 2017 10:43 PM
To: [EMAIL PROTECTED]
Cc: [EMAIL PROTECTED]
Subject: Re: Dynamic partition management

Hi, Subramanian

  Helix actually allows you to dynamically change the number of partitions in a resource.   If you are using your own customized rebalancer, i.e, your rebalance mode set in resource's IdealState is CUSTOMIZED, what you can do is to manipulate the IdealState's MapFields when adding or removing partitions.

  An example is given here, say initially, your resource has 3 partitions, your IS should looks like:
{
  "id":"myDB"
  ,"simpleFields":{
    ,"NUM_PARTITIONS":"3"
    ,"REBALANCE_MODE":"CUSTOMIZED"
    ,"REPLICAS":"2"
    ,"STATE_MODEL_DEF_REF":"OnlineOffline"
  }
  ,"listFields":{
  }
  ,"mapFields":{
    "myDB_0":{
      "node-1":"ONLINE"
      ,"node-2":"ONLINE"
    }
    ,"myDB_1":{
      "node-1":"ONLINE"
      ,"node-2":"ONLINE"
    }
    ,"myDB_2":{
      "node-1":"ONLINE"
      ,"node-2":"ONLINE"
    }
  }
}

   Now you would like to add 2 new partitions. You can update your IS with these new partitions, and Helix will bring all replicas to ONLINE for these two new partitions.
{
  "id":"myDB"
  ,"simpleFields":{
    ,"NUM_PARTITIONS":"5"
    ,"REBALANCE_MODE":"CUSTOMIZED"
    ,"REPLICAS":"2"
    ,"STATE_MODEL_DEF_REF":"OnlineOffline"
  }
  ,"listFields":{
  }
  ,"mapFields":{
    "myDB_0":{
      "node-1":"ONLINE"
      ,"node-2":"ONLINE"
    }
    ,"myDB_1":{
      "node-3":"ONLINE"
      ,"node-4":"ONLINE"
    }
    ,"myDB_2":{
      "node-5":"ONLINE"
      ,"node-6":"ONLINE"
    }
    ,"myDB_3":{
      "node-1":"ONLINE"
      ,"node-2":"ONLINE"
    }
    ,"myDB_4":{
      "node-3":"ONLINE"
      ,"node-4":"ONLINE"
    }
  }
}

    Removing an existing partition is a little tricky,  you can not simply remove it from IS.  You need to first set all replica for that partition to DROPPED state, for example, such as below.

{
  "id":"myDB"
  ,"simpleFields":{
    ,"NUM_PARTITIONS":"5"
    ,"REBALANCE_MODE":"CUSTOMIZED"
    ,"REPLICAS":"2"
    ,"STATE_MODEL_DEF_REF":"OnlineOffline"
  }
  ,"listFields":{
  }
  ,"mapFields":{
    "myDB_0":{
      "node-1":"DROPPED"
      ,"node-2":"DROPPED"
    }
    ,"myDB_1":{
      "node-3":"DROPPED"
      ,"node-4":"DROPPED"
    }
    ,"myDB_2":{
      "node-5":"ONLINE"
      ,"node-6":"ONLINE"
    }
    ,"myDB_3":{
      "node-1":"ONLINE"
      ,"node-2":"ONLINE"
    }
    ,"myDB_4":{
      "node-3":"ONLINE"
      ,"node-4":"ONLINE"
    }
  }

 Once you confirming the partition has been removed from resource's ExternalView, it is safe to remove it from IdealState now.
{
  "id":"myDB"
  ,"simpleFields":{
    ,"NUM_PARTITIONS":"3"
    ,"REBALANCE_MODE":"CUSTOMIZED"
    ,"REPLICAS":"2"
    ,"STATE_MODEL_DEF_REF":"OnlineOffline"
  }
  ,"listFields":{
  }
  ,"mapFields":{
    ,"myDB_2":{
      "node-5":"ONLINE"
      ,"node-6":"ONLINE"
    }
    ,"myDB_3":{
      "node-1":"ONLINE"
      ,"node-2":"ONLINE"
    }
    ,"myDB_4":{
      "node-3":"ONLINE"
      ,"node-4":"ONLINE"
    }
  }

  Help this helps on solving your problem.

Thanks
Lei
On Wed, Jun 7, 2017 at 2:58 PM, Subramanian Raghunathan <[EMAIL PROTECTED]<mailto:[EMAIL PROTECTED]>> wrote:
Hi,

                Trying to follow up if there are any thoughts/suggestions over the approaches.

                Trying to manage a cluster with a resource that has Dynamic number of partitions please provide me your valuable thoughts on same.
 Version(s) :
                                Helix: 0.7.1
                                Zookeeper:3.3.4
 -Zookeeper quorum (3 nodes)
- State Model: OnlineOffline
- Controller (leader elected from one of the cluster nodes)
- Single resources with partitions.
- Custom rebalancer
 Few of the approaches that I could think of ...
 Approach 1: Buffer Partitions
                Let’s say the current number of partitions are N.
                                1 ) Define N+X (Where x could be the delta where it can expand maximum , expecting to be of few in number (1-10))
                                2 ) Use property store to identify which are all valid partition(s)
                                3 ) Don’t act upon invalid ones by looking up property store (or) Disable the unallocated partition(s)

                When a new partition to be added -
1.       Update property store of new partitions (Delta merge)
2.       Enable the partition(s)
 --- The balancing algorithm should be able to balance valid & extra partition(s) differently. Lots of customizations.
 Approach 2: New Resource Addition
                Let’s say the current number of partitions are N.
                Create Resource 1 with N partitions initially.
                When a new partition(s) to be added -
1.       Drop the resource
2.       Create new resource with N+X (newly added partitions )
 --- Caveat that I could see is Dropping the resource would stop the entire workflow (It’s a bigger price to pay for us since we are real time).

Thanks & Regards,
Subramanian.

From: kishore g [mailto:[EMAIL PROTECTED]<mailto:[EMAIL PROTECTED]>]
Sent: Friday, June 02, 2017 9:52 PM
To: Subramanian Raghunathan <[EMAIL PROTECTED]<mailto:[EMAIL PROTECTED]>>; [EMAIL PROTECTED]<mailto:[EMAIL PROTECTED]>
Subject: Re: Dynamic partition management
+helix

On Jun 2, 2017 6:50 PM, "Subramanian Raghunathan" <[EMAIL PROTECTED]<mailto:[EMAIL PROTECTED]>> wrote:
Hi kishore,

                Trying to manage a cluster with a resource that has Dynamic number of partitions please provide me your valuable thoughts on same.

Version(s) :
                                Helix: 0.7.1
                                Zookeeper:3.3.4

-Zookeeper quorum (3 nodes)
- State Model: OnlineOffline
- Controller (leader elected from one of the cluster nodes)
- Single resources with partitions.
- C
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB