Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
HBase, mail # user - responseTooSlow


Copy link to this message
-
Re: responseTooSlow
Azuryy Yu 2012-12-23, 01:50
I am sure you have a long gc, please monitor your gc log.
On Dec 22, 2012 8:14 PM, "Mohammad Tariq" <[EMAIL PROTECTED]> wrote:

> yeah
>
> Best Regards,
> Tariq
> +91-9741563634
> https://mtariq.jux.com/
>
>
> On Sat, Dec 22, 2012 at 6:57 AM, Mohit Anchlia <[EMAIL PROTECTED]
> >wrote:
>
> > You mean batch multiple put?
> >
> > On Fri, Dec 21, 2012 at 4:16 PM, Mohammad Tariq <[EMAIL PROTECTED]>
> > wrote:
> >
> > > It might be the RS which could not complete the operation in time. The
> > > appropriate way to find out is to monitor that RS's metrics and see if
> > > anything unusual is happening there. What type of keys are you using?
> It
> > is
> > > time-series data?You might be a victim of RS hotspotting in that case
> or
> > > perhaps some other processes are eating up resources there. Try using
> > > "put(List<Put> puts)" instead of "put(Put put)" and see if it makes any
> > > difference.
> > >
> > > I'm afraid, I can't say anything with 100% confidence as there could be
> > 'n'
> > > reasons which are not traceable from here. Some of the possible reasons
> > > could be :
> > > hotspotting region
> > > too much I/O wait due to Swapping
> > > overloaded disk
> > > slowness due to high cpu consumption
> > >
> > > Best Regards,
> > > Tariq
> > > +91-9741563634
> > > https://mtariq.jux.com/
> > >
> > >
> > > On Sat, Dec 22, 2012 at 5:23 AM, Mohit Anchlia <[EMAIL PROTECTED]
> > > >wrote:
> > >
> > > > I am just doing a put. This operation generally takes 10ms but in
> this
> > > case
> > > > it took more than 10sec. Nothing out of ordinary in the logs
> > > >
> > > > On Fri, Dec 21, 2012 at 3:26 PM, Mohammad Tariq <[EMAIL PROTECTED]>
> > > > wrote:
> > > >
> > > > > what exactly is the operation your trying to do?how is your
> network's
> > > > > health?is swapping too high at RS side?anything odd in your RS
> logs?
> > > > >
> > > > > Best Regards,
> > > > > Tariq
> > > > > +91-9741563634
> > > > > https://mtariq.jux.com/
> > > > >
> > > > >
> > > > > On Sat, Dec 22, 2012 at 4:36 AM, Mohit Anchlia <
> > [EMAIL PROTECTED]
> > > > > >wrote:
> > > > >
> > > > > > I looked at that link, but couldn't find anything useful. How do
> I
> > > > check
> > > > > if
> > > > > > it was client who didn't write data within that time or if it was
> > > > region
> > > > > > server that didn't finish operation in time.
> > > > > >
> > > > > > On Fri, Dec 21, 2012 at 2:54 PM, Mohammad Tariq <
> > [EMAIL PROTECTED]>
> > > > > > wrote:
> > > > > >
> > > > > > > The socket through which your client is communicating is
> getting
> > > > closed
> > > > > > > before the operation could get finished. May be it is taking
> > longer
> > > > > than
> > > > > > > usual or something.
> > > > > > >
> > > > > > > Best Regards,
> > > > > > > Tariq
> > > > > > > +91-9741563634
> > > > > > > https://mtariq.jux.com/
> > > > > > >
> > > > > > >
> > > > > > >  On Sat, Dec 22, 2012 at 4:08 AM, Mohammad Tariq <
> > > [EMAIL PROTECTED]
> > > > >
> > > > > > > wrote:
> > > > > > >
> > > > > > > > Hello Mohit,
> > > > > > > >
> > > > > > > >       You might this link<
> > > > > > > http://hbase.apache.org/book/ops.monitoring.html>useful.
> > > > > > > >
> > > > > > > > Best Regards,
> > > > > > > > Tariq
> > > > > > > > +91-9741563634
> > > > > > > > https://mtariq.jux.com/
> > > > > > > >
> > > > > > > >
> > > > > > > > On Sat, Dec 22, 2012 at 2:09 AM, Mohit Anchlia <
> > > > > [EMAIL PROTECTED]
> > > > > > > >wrote:
> > > > > > > >
> > > > > > > >> Could someone help me understand what this really means. Is
> > this
> > > > the
> > > > > > > >> network transfer taking long from client -> server or region
> > > > server
> > > > > > > taking
> > > > > > > >> long time writing to the memory?
> > > > > > > >>
> > > > > > > >> 2012-12-21 10:54:21,980 WARN
> > org.apache.hadoop.ipc.HBaseServer:
> > > > > > > >> (responseTooSlow): {"processingtimems":135652,"call":"multi(
> > > > > > > >> org.apache.hadoop.hbase.client.MultiAction@28338472), rpc