Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
MapReduce >> mail # user >> Re: are we able to decommission multi nodes at one time?


Copy link to this message
-
Re: are we able to decommission multi nodes at one time?
not at all.  so don't worry about that.
On Wed, Apr 3, 2013 at 2:04 PM, Yanbo Liang <[EMAIL PROTECTED]> wrote:

> It means that may be some replicas will be stay in under replica state?
>
>
> 2013/4/3 Azuryy Yu <[EMAIL PROTECTED]>
>
>> bq. then namenode start to copy block replicates on DN-2 to another DN,
>> supposed DN-2.
>>
>> sorry for typo.
>>
>> Correct for it:
>> then namenode start to copy block replicates on DN-1 to another DN,
>> supposed DN-2.
>>
>>
>> On Wed, Apr 3, 2013 at 9:51 AM, Azuryy Yu <[EMAIL PROTECTED]> wrote:
>>
>>> It's different.
>>> If you just want to stop DN-1 a short time, just kill the DataNode
>>> process on DN-1. then do what you want. during this time, Namenode  cannot
>>> receive the heart beat from DN-1, then namenode start to copy block
>>> replicates on DN-2 to another DN, supposed DN-2.
>>>
>>> But when you start DN-1 again, Namenode receive the DN-1 registration,
>>> then namenode stop to copy the DN-1's block replicates even if NN doesn't
>>> finish coping.
>>>
>>> Am I explain clearly?
>>>
>>>
>>>
>>> On Wed, Apr 3, 2013 at 9:43 AM, Henry Junyoung Kim <
>>> [EMAIL PROTECTED]> wrote:
>>>
>>>> @Harsh
>>>>
>>>> What's the reasons to make big gaps for removing nodes between
>>>> decommission and just down nodes?
>>>> In my understanding, both are necessary to copy un-replicated blocks to
>>>> another alive nodes.
>>>> If main costs of  them are this one, total elapsed time couldn't be big
>>>> different.
>>>>
>>>> Could you share some articles or documents to understand about
>>>> decommissioning procedures?
>>>> - explaining is always thanks ;)
>>>>
>>>>
>>>> 2013. 4. 2., 오후 5:37, Harsh J <[EMAIL PROTECTED]> 작성:
>>>>
>>>> > Yes, you can do the downtime work in steps of 2 DNs at a time,
>>>> > especially since you mentioned the total work would be only ~30mins at
>>>> > most.
>>>> >
>>>> > On Tue, Apr 2, 2013 at 1:46 PM, Henry Junyoung Kim
>>>> > <[EMAIL PROTECTED]> wrote:
>>>> >> the rest of nodes to be alive has enough size to store.
>>>> >>
>>>> >> for this one that you've mentioned.
>>>> >>> its easier to do so in a rolling manner without need of a
>>>> >>> decommission.
>>>> >>
>>>> >> to check my understanding, just shutting down 2 of them and then 2
>>>> more and then 2 more without decommissions.
>>>> >>
>>>> >> is this correct?
>>>> >>
>>>> >>
>>>> >> 2013. 4. 2., 오후 4:54, Harsh J <[EMAIL PROTECTED]> 작성:
>>>> >>
>>>> >>> Note though that its only possible to decommission 7 nodes at the
>>>> same
>>>> >>> time and expect it to finish iff the remaining 8 nodes have adequate
>>>> >>> free space for the excess replicas.
>>>> >>>
>>>> >>> If you're just going to take them down for a short while (few mins
>>>> >>> each), its easier to do so in a rolling manner without need of a
>>>> >>> decommission. You can take upto two down at a time on a replication
>>>> >>> average of 3 or 3+, and put it back in later without too much data
>>>> >>> movement impact.
>>>> >>>
>>>> >>> On Tue, Apr 2, 2013 at 1:06 PM, Yanbo Liang <[EMAIL PROTECTED]>
>>>> wrote:
>>>> >>>> It's reasonable to decommission 7 nodes at the same time.
>>>> >>>> But may be it also takes long time to finish it.
>>>> >>>> Because all the replicas in these 7 nodes need to be copied to
>>>> remaining 8
>>>> >>>> nodes.
>>>> >>>> The size of transfer from these nodes to the remaining nodes is
>>>> equal.
>>>> >>>>
>>>> >>>>
>>>> >>>> 2013/4/2 Henry Junyoung Kim <[EMAIL PROTECTED]>
>>>> >>>>>
>>>> >>>>> :)
>>>> >>>>>
>>>> >>>>> currently, I  have 15 data nodes.
>>>> >>>>> for some tests, I am trying to decommission until 8 nodes.
>>>> >>>>>
>>>> >>>>> Now, the total dfs used size is 52 TB which is including all
>>>> replicated
>>>> >>>>> blocks.
>>>> >>>>> from 15 to 8, total spent time is almost 4 days long. ;(
>>>> >>>>>
>>>> >>>>> someone mentioned that I don't need to decommission node by node.
>>>> >>>>> for this case, is there no problems if I decommissioned 7 nodes
>>>> at the
>>>> >>>>> same time?
>>>>
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB