Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
HBase >> mail # user >> Regionserver goes down while endpoint execution


Copy link to this message
-
Re: Regionserver goes down while endpoint execution
On Wed, Mar 13, 2013 at 8:19 AM, Kumar, Deepak8 <[EMAIL PROTECTED]> wrote:
> Thanks guys for assisting. I am getting OOM exception yet. I have one query about Endpoints. As endpoint executes in parallel, so if I have a table which is distributed at 101 regions across 5 regionserver. Would it be 101 threads of endpoint executing in parallel?

No and Yes.

The endpoints are not processed as separate threads, they are
processed as just another request (via regionserver handlers). Yes,
the execution will be in parallel in the sense that a separate client
side call will be used for each of the regions that are in the range
you specify.

>
> Regards,
> Deepak
>
> From: Gary Helmling [mailto:[EMAIL PROTECTED]]
> Sent: Tuesday, March 12, 2013 2:14 PM
> To: [EMAIL PROTECTED]
> Cc: lars hofhansl; Kumar, Deepak8 [CCC-OT_IT NE]
> Subject: Re: Regionserver goes down while endpoint execution
>
> To expand on what Himanshu said, your endpoint is doing an unbounded scan on the region, so with a region with a lot of rows it's taking more than 60 seconds to run to the region end, which is why the client side of the call is timing out.  In addition you're building up an in memory list of all the values for that qualifier in that region, which could cause you to bump into OOM issues, depending on how big your values are and how sparse the given column qualifier is.  If you trigger an OOMException, then the region server would abort.
>
> For this usage specifically, though -- scanning through a single column qualifier for all rows -- you would be better off just doing a normal client side scan, ie. HTable.getScanner().  Then you will avoid the client timeout and potential server-side memory issues.
>
> On Tue, Mar 12, 2013 at 9:29 AM, Ted Yu <[EMAIL PROTECTED]<mailto:[EMAIL PROTECTED]>> wrote:
> From region server log:
>
> 2013-03-12 03:07:22,605 DEBUG org.apache.hadoop.hdfs.DFSClient: Error
> making BlockReader. Closing stale
> Socket[addr=/10.42.105.112<http://10.42.105.112>,port=50010,localport=54114]
> java.io.EOFException: Premature EOF: no length prefix available
>         at org.apache.hadoop.hdfs.protocol.HdfsProtoUtil.vintPrefixed(HdfsProtoUtil.java:162)
>         at org.apache.hadoop.hdfs.RemoteBlockReader2.newBlockReader(RemoteBlockReader2.java:407)
>
> What version of HBase and hadoop are you using ?
> Do versions of hadoop on Eclipse machine and in your cluster match ?
>
> Cheers
>
> On Tue, Mar 12, 2013 at 4:46 AM, Kumar, Deepak8 <[EMAIL PROTECTED]<mailto:[EMAIL PROTECTED]>>wrote:
>>  Lars,****
>>
>> I am getting following errors at datanode & region servers.****
>>
>> ** **
>>
>> Regards,****
>>
>> Deepak****
>>
>> ** **
>>
>> *From:* Kumar, Deepak8 [CCC-OT_IT NE]
>> *Sent:* Tuesday, March 12, 2013 3:00 AM
>> *To:* Kumar, Deepak8 [CCC-OT_IT NE]; '[EMAIL PROTECTED]<mailto:[EMAIL PROTECTED]>'; 'lars
>> hofhansl'
>>
>> *Subject:* RE: Regionserver goes down while endpoint execution****
>>
>>  ** **
>>
>> Lars,****
>>
>> It is having following errors when I execute the Endpoint RPC client from
>> eclipse. It seems some of the regions at regionserver
>> vm-8aa9-fe74.nam.nsroot.net<http://vm-8aa9-fe74.nam.nsroot.net> is taking more time to reponse.****
>>
>> ** **
>>
>> Could you guide how to fix it. I don't find any option to set hbase.rpc.timeout
>> from hbase configuration menu in CDH4 CM server for hbase configuration.**
>> **
>>
>> ** **
>>
>> Regards,****
>>
>> Deepak****
>>
>> ** **
>>
>> 3/03/12 02:33:12 INFO zookeeper.ClientCnxn: Session establishment complete
>> on server vm-15c2-3bbf.nam.nsroot.net/10.96.172.44:2181<http://vm-15c2-3bbf.nam.nsroot.net/10.96.172.44:2181>, sessionid >> 0x53d591b77090026, negotiated timeout = 60000****
>>
>> Mar 12, 2013 2:33:13 AM org.apache.hadoop.conf.Configuration
>> warnOnceIfDeprecated****
>>
>> WARNING: hadoop.native.lib is deprecated. Instead, use
>> io.native.lib.available****
>>
>> Mar 12, 2013 2:44:00 AM
>> org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation