Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Plain View
HBase >> mail # user >> Regionserver goes down while endpoint execution


+
Kumar, Deepak8 2013-03-12, 05:51
+
lars hofhansl 2013-03-12, 06:01
+
Kumar, Deepak8 2013-03-12, 06:27
+
Kumar, Deepak8 2013-03-12, 06:59
+
Kumar, Deepak8 2013-03-12, 11:46
+
Ted Yu 2013-03-12, 16:29
Copy link to this message
-
Re: Regionserver goes down while endpoint execution
To expand on what Himanshu said, your endpoint is doing an unbounded scan
on the region, so with a region with a lot of rows it's taking more than 60
seconds to run to the region end, which is why the client side of the call
is timing out.  In addition you're building up an in memory list of all the
values for that qualifier in that region, which could cause you to bump
into OOM issues, depending on how big your values are and how sparse the
given column qualifier is.  If you trigger an OOMException, then the region
server would abort.

For this usage specifically, though -- scanning through a single column
qualifier for all rows -- you would be better off just doing a normal
client side scan, ie. HTable.getScanner().  Then you will avoid the client
timeout and potential server-side memory issues.
On Tue, Mar 12, 2013 at 9:29 AM, Ted Yu <[EMAIL PROTECTED]> wrote:

> From region server log:
>
> 2013-03-12 03:07:22,605 DEBUG org.apache.hadoop.hdfs.DFSClient: Error
> making BlockReader. Closing stale
> Socket[addr=/10.42.105.112,port=50010,localport=54114]
> java.io.EOFException: Premature EOF: no length prefix available
>         at
> org.apache.hadoop.hdfs.protocol.HdfsProtoUtil.vintPrefixed(HdfsProtoUtil.java:162)
>         at
> org.apache.hadoop.hdfs.RemoteBlockReader2.newBlockReader(RemoteBlockReader2.java:407)
>
> What version of HBase and hadoop are you using ?
> Do versions of hadoop on Eclipse machine and in your cluster match ?
>
> Cheers
>
> On Tue, Mar 12, 2013 at 4:46 AM, Kumar, Deepak8 <[EMAIL PROTECTED]
> >wrote:
>
> >  Lars,****
> >
> > I am getting following errors at datanode & region servers.****
> >
> > ** **
> >
> > Regards,****
> >
> > Deepak****
> >
> > ** **
> >
> > *From:* Kumar, Deepak8 [CCC-OT_IT NE]
> > *Sent:* Tuesday, March 12, 2013 3:00 AM
> > *To:* Kumar, Deepak8 [CCC-OT_IT NE]; '[EMAIL PROTECTED]'; 'lars
> > hofhansl'
> >
> > *Subject:* RE: Regionserver goes down while endpoint execution****
> >
> >  ** **
> >
> > Lars,****
> >
> > It is having following errors when I execute the Endpoint RPC client from
> > eclipse. It seems some of the regions at regionserver
> > vm-8aa9-fe74.nam.nsroot.net is taking more time to reponse.****
> >
> > ** **
> >
> > Could you guide how to fix it. I don’t find any option to set
> hbase.rpc.timeout
> > from hbase configuration menu in CDH4 CM server for hbase
> configuration.**
> > **
> >
> > ** **
> >
> > Regards,****
> >
> > Deepak****
> >
> > ** **
> >
> > 3/03/12 02:33:12 INFO zookeeper.ClientCnxn: Session establishment
> complete
> > on server vm-15c2-3bbf.nam.nsroot.net/10.96.172.44:2181, sessionid > > 0x53d591b77090026, negotiated timeout = 60000****
> >
> > Mar 12, 2013 2:33:13 AM org.apache.hadoop.conf.Configuration
> > warnOnceIfDeprecated****
> >
> > WARNING: hadoop.native.lib is deprecated. Instead, use
> > io.native.lib.available****
> >
> > Mar 12, 2013 2:44:00 AM
> >
> org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation
> > processExecs****
> >
> > WARNING: Error executing for row 153299:1362780381523:2932572079500658:
> > vm-ab1f-dd21.nam.nsroot.net:****
> >
> > *java.util.concurrent.ExecutionException*: *
> > org.apache.hadoop.hbase.client.RetriesExhaustedException*: Failed after
> > attempts=10, exceptions:****
> >
> > Tue Mar 12 02:34:15 EDT 2013,
> > org.apache.hadoop.hbase.ipc.ExecRPCInvoker$1@39443f, *
> > java.net.SocketTimeoutException*: Call to
> > vm-8aa9-fe74.nam.nsroot.net/10.42.105.91:60020 failed on socket timeout
> > exception: *java.net.SocketTimeoutException*: 60000 millis timeout while
> > waiting for channel to be ready for read. ch :
> > java.nio.channels.SocketChannel[connected local=/150.110.96.212:2271
> remote> > vm-8aa9-fe74.nam.nsroot.net/10.42.105.91:60020]****
> >
> > Tue Mar 12 02:35:16 EDT 2013,
> > org.apache.hadoop.hbase.ipc.ExecRPCInvoker$1@39443f, *
> > java.net.SocketTimeoutException*: Call to
> > vm-8aa9-fe74.nam.nsroot.net/10.42.105.91:60020 failed on socket timeout
> > exception: *java.net.SocketTimeoutException*: 60000 millis timeout while
+
Kumar, Deepak8 2013-03-13, 15:19
+
Ted Yu 2013-03-13, 16:01
+
Himanshu Vashishtha 2013-03-13, 16:08
+
Kumar, Deepak8 2013-03-14, 17:09
+
Ted Yu 2013-03-14, 17:15
+
Himanshu Vashishtha 2013-03-14, 17:45
+
Anoop Sam John 2013-03-15, 06:55
+
Kumar, Deepak8 2013-03-20, 07:41
+
ramkrishna vasudevan 2013-03-20, 08:01
+
Anoop Sam John 2013-03-20, 08:36
+
Kumar, Deepak8 2013-03-20, 12:44
+
Anoop Sam John 2013-03-20, 12:58
+
Kumar, Deepak8 2013-03-20, 13:18
+
Kumar, Deepak8 2013-03-25, 16:53
+
Anoop Sam John 2013-03-26, 06:20
+
Kumar, Deepak8 2013-03-26, 07:27
+
Adrien Mogenet 2013-03-26, 07:42
+
Kumar, Deepak8 2013-03-26, 08:27
+
Anoop John 2013-03-26, 17:17
+
Kumar, Deepak8 2013-03-28, 10:50
+
ramkrishna vasudevan 2013-03-28, 10:53
+
Agarwal, Saurabh 2013-03-28, 12:26
+
Anoop Sam John 2013-04-02, 06:51
+
Kumar, Deepak8 2013-03-28, 12:11
+
Himanshu Vashishtha 2013-03-12, 16:59
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB