Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
HBase, mail # user - Load balancer repeatedly close and open region in the same regionserver.


Copy link to this message
-
Re: Load balancer repeatedly close and open region in the same regionserver.
yuzhihong@... 2012-07-30, 07:50
Can you trace master log to see why there were two region servers on that ip with different start codes ?

Thanks

On Jul 29, 2012, at 10:46 PM, deanforwever2010 <[EMAIL PROTECTED]> wrote:

> hi Ted,I am in the same team of Howard's
> We didn't found two  region server processes running on
> 192.168.18.40
>
>
> 2012/7/27 Ted Yu <[EMAIL PROTECTED]>
>
>> bq. the region is move from regionserver 192.168.18.40 to 192.168.18.40
>>
>> Have you checked whether there were two region server processes running on
>> 192.168.18.40 ?
>>
>> Cheers
>>
>> On Fri, Jul 27, 2012 at 2:43 AM, Howard <[EMAIL PROTECTED]> wrote:
>>
>>> Thanks Suraj Varma,I have put the log into the pastebin.com.
>>>
>>> master log: http://pastebin.com/QWv3K9HQ
>>> regionserver log:http://pastebin.com/LM27ui72
>>>
>>> Because there is a lot of "region is not online" in the regionserver
>> log,so
>>> I have filter this in the regionserver log.
>>> The following is the count of "Region is not online:" log,start
>> 23:16,there
>>> is a lot of access fail because the region is not online.
>>> --------------------------d70285c1a12dec9289ce9290c9349a79
>>>     1 23:16
>>>    103 23:36
>>>    142 23:37
>>>    169 23:38
>>>     94 23:39
>>>    120 23:40
>>>     39 23:41
>>>    110 23:42
>>>    104 23:43
>>>    114 23:44
>>>     90 23:45
>>>    121 23:46
>>>    104 23:47
>>>     74 23:48
>>>     96 23:49
>>>    100 23:50
>>>    125 23:51
>>>     59 23:52
>>>    113 23:53
>>>    134 23:54
>>>    127 23:55
>>>    131 23:56
>>>    119 23:57
>>>     82 23:58
>>>    165 23:59
>>>
>>> and the region "d70285c1a12dec9289ce9290c9349a79" is move between two
>>> regionserver again and again by balancer.Start 23:36,the region is move
>>> from regionserver 192.168.18.40 to 192.168.18.40 and fail.
>>>
>>>
>>> 2012/7/19 Suraj Varma <[EMAIL PROTECTED]>
>>>
>>>> You can use pastebin.com or similar services to cut/paste your logs.
>>>> --S
>>>>
>>>> On Tue, Jul 17, 2012 at 7:11 PM, Howard <[EMAIL PROTECTED]> wrote:
>>>>> this problem just only once,Because it happens two day before,I
>>> remember
>>>> I
>>>>> check the master-status and only always see regions is "pending open"
>>> in
>>>>> Regions in Transition,not see there was two regionservers in the
>> same
>>>>> server.
>>>>>
>>>>> "Sent CLOSE to 192.168.0.2,60020,1342017399608",what
>>>>> does  "60020,1342017399608" mean?Is there some document can help to
>>> read
>>>>> the source code?
>>>>> If still need to upload the log,how to upload the log?
>>>>> sorry I am a freshman with HBase.
>>>>>
>>>>> 2012/7/17 Ted Yu <[EMAIL PROTECTED]>
>>>>>
>>>>>> Howard:
>>>>>> Before filing JIRA, can you verify with 0.94.1 RC that Lars sent out
>>>>>> yesterday ?
>>>>>> I guess you have noticed the following toward the end of log
>> snippet:
>>>>>>
>>>>>> 2012-07-16 00:17:50,774 DEBUG
>>>>>> org.apache.hadoop.hbase.
>>>>>> master.handler.OpenedRegionHandler: Handling OPENED
>>>>>> event for
>>>>>>
>>>>>>
>>>>
>>>
>> trackurl_status_list,zO6u4o8,1342291884831.93caf5147d40f5dd4625e160e1b7e956.
>>>>>> from 192.168.1.2,60020,1342017399608; deleting unassigned node
>>>>>>
>>>>>> As Ram pointed out, there might be two region server processes
>> running
>>>> on
>>>>>> 192.168.1.2
>>>>>>
>>>>>> Please verify whether that was the case.
>>>>>>
>>>>>> Cheers
>>>>>>
>>>>>> On Tue, Jul 17, 2012 at 7:30 AM, Ramkrishna.S.Vasudevan <
>>>>>> [EMAIL PROTECTED]> wrote:
>>>>>>
>>>>>>> From the logs I can see that though the server's are same their
>>> start
>>>>>> code
>>>>>>> is different.
>>>>>>> Need to analyse the previous logs also.  Pls file a JIRA, if
>>> possible
>>>>>>> attach
>>>>>>> the logs to that.
>>>>>>>
>>>>>>> Thanks Howard.
>>>>>>>
>>>>>>> Regards
>>>>>>> Ram
>>>>>>>
>>>>>>>> -----Original Message-----
>>>>>>>> From: Howard [mailto:[EMAIL PROTECTED]]
>>>>>>>> Sent: Tuesday, July 17, 2012 7:32 PM
>>>>>>>> To: [EMAIL PROTECTED]
>>>>>>>> Subject: Re: Load balancer repeatedly close and open region in