Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
HBase, mail # user - responseTooSlow


Copy link to this message
-
Re: responseTooSlow
Mohammad Tariq 2012-12-22, 00:16
It might be the RS which could not complete the operation in time. The
appropriate way to find out is to monitor that RS's metrics and see if
anything unusual is happening there. What type of keys are you using? It is
time-series data?You might be a victim of RS hotspotting in that case or
perhaps some other processes are eating up resources there. Try using
"put(List<Put> puts)" instead of "put(Put put)" and see if it makes any
difference.

I'm afraid, I can't say anything with 100% confidence as there could be 'n'
reasons which are not traceable from here. Some of the possible reasons
could be :
hotspotting region
too much I/O wait due to Swapping
overloaded disk
slowness due to high cpu consumption

Best Regards,
Tariq
+91-9741563634
https://mtariq.jux.com/
On Sat, Dec 22, 2012 at 5:23 AM, Mohit Anchlia <[EMAIL PROTECTED]>wrote:

> I am just doing a put. This operation generally takes 10ms but in this case
> it took more than 10sec. Nothing out of ordinary in the logs
>
> On Fri, Dec 21, 2012 at 3:26 PM, Mohammad Tariq <[EMAIL PROTECTED]>
> wrote:
>
> > what exactly is the operation your trying to do?how is your network's
> > health?is swapping too high at RS side?anything odd in your RS logs?
> >
> > Best Regards,
> > Tariq
> > +91-9741563634
> > https://mtariq.jux.com/
> >
> >
> > On Sat, Dec 22, 2012 at 4:36 AM, Mohit Anchlia <[EMAIL PROTECTED]
> > >wrote:
> >
> > > I looked at that link, but couldn't find anything useful. How do I
> check
> > if
> > > it was client who didn't write data within that time or if it was
> region
> > > server that didn't finish operation in time.
> > >
> > > On Fri, Dec 21, 2012 at 2:54 PM, Mohammad Tariq <[EMAIL PROTECTED]>
> > > wrote:
> > >
> > > > The socket through which your client is communicating is getting
> closed
> > > > before the operation could get finished. May be it is taking longer
> > than
> > > > usual or something.
> > > >
> > > > Best Regards,
> > > > Tariq
> > > > +91-9741563634
> > > > https://mtariq.jux.com/
> > > >
> > > >
> > > >  On Sat, Dec 22, 2012 at 4:08 AM, Mohammad Tariq <[EMAIL PROTECTED]
> >
> > > > wrote:
> > > >
> > > > > Hello Mohit,
> > > > >
> > > > >       You might this link<
> > > > http://hbase.apache.org/book/ops.monitoring.html>useful.
> > > > >
> > > > > Best Regards,
> > > > > Tariq
> > > > > +91-9741563634
> > > > > https://mtariq.jux.com/
> > > > >
> > > > >
> > > > > On Sat, Dec 22, 2012 at 2:09 AM, Mohit Anchlia <
> > [EMAIL PROTECTED]
> > > > >wrote:
> > > > >
> > > > >> Could someone help me understand what this really means. Is this
> the
> > > > >> network transfer taking long from client -> server or region
> server
> > > > taking
> > > > >> long time writing to the memory?
> > > > >>
> > > > >> 2012-12-21 10:54:21,980 WARN org.apache.hadoop.ipc.HBaseServer:
> > > > >> (responseTooSlow): {"processingtimems":135652,"call":"multi(
> > > > >> org.apache.hadoop.hbase.client.MultiAction@28338472), rpc
> > version=1,
> > > > >> client
> > > > >> version=29, methodsFingerPrint=54742778","client":"
> 10.18.3.80:48218
> > > > >>
> > > > >>
> > > >
> > >
> >
> ","starttimems":1356115926326,"queuetimems":0,"class":"HRegionServer","responsesize":0,"method":"multi"}
> > > > >> 2012-12-21 10:54:21,985 WARN org.apache.hadoop.ipc.HBaseServer:
> IPC
> > > > Server
> > > > >> handler 26 on 60020 caught:
> java.nio.channels.ClosedChannelException
> > > > >>         at
> > > > >>
> > >
> sun.nio.ch.SocketChannelImpl.ensureWriteOpen(SocketChannelImpl.java:133)
> > > > >>         at
> > > > sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:324)
> > > > >>         at
> > > > >>
> > > > >>
> > > >
> > >
> >
> org.apache.hadoop.hbase.ipc.HBaseServer.channelWrite(HBaseServer.java:1653)
> > > > >>         at
> > > > >>
> > > > >>
> > > >
> > >
> >
> org.apache.hadoop.hbase.ipc.HBaseServer$Responder.processResponse(HBaseServer.java:924)
> > > > >>         at
> > > > >>
> > > > >>
> > > >
> > >
> >
> org.apache.hadoop.hbase.ipc.HBaseServer$Responder.doRespond(HBaseServer.java:1003)