Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Plain View
HBase, mail # user - Socket Timeout Exception when executing coprocessor


+
Nichole Treadway 2011-06-29, 22:24
+
Stack 2011-07-01, 06:11
+
Ted Yu 2011-06-29, 22:47
+
Nichole Treadway 2011-06-29, 22:58
Copy link to this message
-
Re: Socket Timeout Exception when executing coprocessor
Himanshu Vashishtha 2011-06-29, 23:08
Hey Nichole,
you are doing something compute intensive at the region level in your
coprocessors and it is not able to respond back with in the default client
timeout of 60 sec. Aggregate Coprocessor just do the row count and comes
back from each region. You can see corresponding exceptions in the region
server logs too.

Your use case is motivating me to restart work on the streaming result thing
in a cursor like fashion (3607) from coprocessor result sets; now I have got
one real user who is facing it! Can you take a look at it and comment if
that would be of some help in your case.
I got similar exceptions in my experiments when I did a row count on a 100
million  record dataset on a 3 RS node cluster. Applying patch of 2077 might
help, I haven't tried it yet.

Himanshu

On Wed, Jun 29, 2011 at 4:58 PM, Nichole Treadway <[EMAIL PROTECTED]>wrote:

> Yes...each region has roughly 300,000 rows, 60 million rows total. The rows
> are large, ~ 20,000 bytes.
>
> But in my tests, I'm setting filters such that only 1 KV should be returned
> to the client. And I'm seeing these issues even when I set start and stop
> rows on the scanner so that all regions won't be scanned. If i set start
> and
> stop rows to cover from 1 to 20 regions, it works fine. Any more regions
> than that and I see the socket exceptions.
>
>
>
> On Wed, Jun 29, 2011 at 6:47 PM, Ted Yu <[EMAIL PROTECTED]> wrote:
>
> > Can you tell us roughly how large the 200 regions are ?
> > You can run the rowcounter util.
> >
> > Without seeing your (GetListClient) code, I am not able to tell why
> > AggregationClient didn't have the following issue.
> >
> > On Wed, Jun 29, 2011 at 3:24 PM, Nichole Treadway <[EMAIL PROTECTED]
> > >wrote:
> >
> > > I've implemented my own coprocessor client, protocol and implementation
> > > that
> > > returns back to the user a List of KeyValues with values that match
> some
> > > criteria. I've tested this on a small table with just a few regions and
> > it
> > > works fine. I'm running into issues when I execute my code on a table
> > with
> > > 200 regions, and I'm not really sure how to resolve the issue. I'm
> > getting
> > > a
> > > SocketTimeoutException shown below.
> > >
> > > I'm able to run the AggregationClient coprocessor without seeing these
> > > issues. It might be something I'm doing in my code, but if anybody has
> > any
> > > ideas why the request seems to be timing out or what I can do about it,
> > I'd
> > > appreciate that.
> > >
> > >
> > > I'm running latest revision of hbase-0.92 and hadoop-0.20-append. My
> > > cluster
> > > has 15 regionservers. Running on RHEL 5.5, 64-bit.
> > >
> > >
> > > Some highlights of the errors are below...i've put the full thing in
> > > pastebin here: http://pastebin.com/rapYiNp3
> > >
> > >
> > > 11/06/29 17:35:12 INFO ipc.HBaseRPC: Using
> > > org.apache.hadoop.hbase.ipc.WritableRpcEngine for
> > > org.apache.hadoop.hbase.ipc.HRegionInterface
> > > 11/06/29 17:48:12 WARN
> > client.HConnectionManager$HConnectionImplementation:
> > > Error executing for row 00223199610B220970111:2:0:7524::
> > > java.util.concurrent.ExecutionException:
> > > org.apache.hadoop.hbase.client.RetriesExhaustedException: Trying to
> > contact
> > > region server mysite.com/myip:62091 for region
> > >
> > >
> >
> info_test,00223199610B220970111:2:0:7524::,1309363489443.c3192674341c8d10d84966e8e663a644.,
> > > row '00223199610B220970111:2:0:7524::', but failed after 10 attempts.
> > >
> > > Exceptions:
> > > java.net.SocketTimeoutException: Call to mysite.com/myip:62091 failed
> on
> > > socket timeout exception: java.net.SocketTimeoutException: 60000 millis
> > > timeout while waiting for channel to be ready for read. ch :
> > > java.nio.channels.SocketChannel[connected local=/myip:14738 remote> > > mysite.com/myip:62091]
> > >
> > > at
> > >
> > >
> >
> org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.getRegionServerWithRetries(HConnectionManager.java:1215)
> > >  at
> > >
+
Mingjie Lai 2011-06-30, 06:57