Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Plain View
HBase >> mail # user >> Load balancer repeatedly close and open region in the same regionserver.


+
Howard 2012-07-17, 12:09
+
Ted Yu 2012-07-17, 13:43
+
Howard 2012-07-17, 14:01
+
Ramkrishna.S.Vasudevan 2012-07-17, 14:30
+
Ted Yu 2012-07-17, 15:04
+
Howard 2012-07-18, 02:11
+
Suraj Varma 2012-07-18, 23:06
+
Howard 2012-07-27, 09:43
+
Ted Yu 2012-07-27, 13:49
+
deanforwever2010 2012-07-30, 05:46
Copy link to this message
-
Re: Load balancer repeatedly close and open region in the same regionserver.
Can you trace master log to see why there were two region servers on that ip with different start codes ?

Thanks

On Jul 29, 2012, at 10:46 PM, deanforwever2010 <[EMAIL PROTECTED]> wrote:

> hi Ted,I am in the same team of Howard's
> We didn't found two  region server processes running on
> 192.168.18.40
>
>
> 2012/7/27 Ted Yu <[EMAIL PROTECTED]>
>
>> bq. the region is move from regionserver 192.168.18.40 to 192.168.18.40
>>
>> Have you checked whether there were two region server processes running on
>> 192.168.18.40 ?
>>
>> Cheers
>>
>> On Fri, Jul 27, 2012 at 2:43 AM, Howard <[EMAIL PROTECTED]> wrote:
>>
>>> Thanks Suraj Varma,I have put the log into the pastebin.com.
>>>
>>> master log: http://pastebin.com/QWv3K9HQ
>>> regionserver log:http://pastebin.com/LM27ui72
>>>
>>> Because there is a lot of "region is not online" in the regionserver
>> log,so
>>> I have filter this in the regionserver log.
>>> The following is the count of "Region is not online:" log,start
>> 23:16,there
>>> is a lot of access fail because the region is not online.
>>> --------------------------d70285c1a12dec9289ce9290c9349a79
>>>     1 23:16
>>>    103 23:36
>>>    142 23:37
>>>    169 23:38
>>>     94 23:39
>>>    120 23:40
>>>     39 23:41
>>>    110 23:42
>>>    104 23:43
>>>    114 23:44
>>>     90 23:45
>>>    121 23:46
>>>    104 23:47
>>>     74 23:48
>>>     96 23:49
>>>    100 23:50
>>>    125 23:51
>>>     59 23:52
>>>    113 23:53
>>>    134 23:54
>>>    127 23:55
>>>    131 23:56
>>>    119 23:57
>>>     82 23:58
>>>    165 23:59
>>>
>>> and the region "d70285c1a12dec9289ce9290c9349a79" is move between two
>>> regionserver again and again by balancer.Start 23:36,the region is move
>>> from regionserver 192.168.18.40 to 192.168.18.40 and fail.
>>>
>>>
>>> 2012/7/19 Suraj Varma <[EMAIL PROTECTED]>
>>>
>>>> You can use pastebin.com or similar services to cut/paste your logs.
>>>> --S
>>>>
>>>> On Tue, Jul 17, 2012 at 7:11 PM, Howard <[EMAIL PROTECTED]> wrote:
>>>>> this problem just only once,Because it happens two day before,I
>>> remember
>>>> I
>>>>> check the master-status and only always see regions is "pending open"
>>> in
>>>>> Regions in Transition,not see there was two regionservers in the
>> same
>>>>> server.
>>>>>
>>>>> "Sent CLOSE to 192.168.0.2,60020,1342017399608",what
>>>>> does  "60020,1342017399608" mean?Is there some document can help to
>>> read
>>>>> the source code?
>>>>> If still need to upload the log,how to upload the log?
>>>>> sorry I am a freshman with HBase.
>>>>>
>>>>> 2012/7/17 Ted Yu <[EMAIL PROTECTED]>
>>>>>
>>>>>> Howard:
>>>>>> Before filing JIRA, can you verify with 0.94.1 RC that Lars sent out
>>>>>> yesterday ?
>>>>>> I guess you have noticed the following toward the end of log
>> snippet:
>>>>>>
>>>>>> 2012-07-16 00:17:50,774 DEBUG
>>>>>> org.apache.hadoop.hbase.
>>>>>> master.handler.OpenedRegionHandler: Handling OPENED
>>>>>> event for
>>>>>>
>>>>>>
>>>>
>>>
>> trackurl_status_list,zO6u4o8,1342291884831.93caf5147d40f5dd4625e160e1b7e956.
>>>>>> from 192.168.1.2,60020,1342017399608; deleting unassigned node
>>>>>>
>>>>>> As Ram pointed out, there might be two region server processes
>> running
>>>> on
>>>>>> 192.168.1.2
>>>>>>
>>>>>> Please verify whether that was the case.
>>>>>>
>>>>>> Cheers
>>>>>>
>>>>>> On Tue, Jul 17, 2012 at 7:30 AM, Ramkrishna.S.Vasudevan <
>>>>>> [EMAIL PROTECTED]> wrote:
>>>>>>
>>>>>>> From the logs I can see that though the server's are same their
>>> start
>>>>>> code
>>>>>>> is different.
>>>>>>> Need to analyse the previous logs also.  Pls file a JIRA, if
>>> possible
>>>>>>> attach
>>>>>>> the logs to that.
>>>>>>>
>>>>>>> Thanks Howard.
>>>>>>>
>>>>>>> Regards
>>>>>>> Ram
>>>>>>>
>>>>>>>> -----Original Message-----
>>>>>>>> From: Howard [mailto:[EMAIL PROTECTED]]
>>>>>>>> Sent: Tuesday, July 17, 2012 7:32 PM
>>>>>>>> To: [EMAIL PROTECTED]
>>>>>>>> Subject: Re: Load balancer repeatedly close and open region in
+
deanforwever2010 2012-07-30, 10:26
+
Howard 2012-07-30, 08:27
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB