Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
HBase, mail # user - High IPC Latency


Copy link to this message
-
RE: High IPC Latency
Pamecha, Abhishek 2012-10-19, 17:20
Also, I hope no coprocessors are in play.

Thanks,
Abhishek
-----Original Message-----
From: Yousuf Ahmad [mailto:[EMAIL PROTECTED]]
Sent: Friday, October 19, 2012 10:12 AM
To: [EMAIL PROTECTED]
Cc: Ivan Brondino; Ricardo Vilaça
Subject: Re: High IPC Latency

Hi Lars,

We are following your suggestion and testing against a single region server. We just ran a test against a remote region server and soon we will test against a local one as well. We will get back to you soon with the results.

It will take us a couple of days to port to and test our code with 0.94.2.
Once we have it working, we will run some experiments and update this thread.

Unfortunately, the nature of our project is such that we cannot easily translate the benchmark's workload into a program executing the equivalent HBase operations directly. For this reason, I attempted to roughly translate the workload in terms of HBase operations in my first email and I attached a portion of the logs to be a bit more concrete.

Your assistance is very much appreciated! Thank you! We'll keep you updated.

Best regards,
Yousuf
On Fri, Oct 19, 2012 at 1:25 AM, lars hofhansl <[EMAIL PROTECTED]> wrote:

> Can you reproduce this against a single, local region server?
> Any chance that you can try with the just released 0.94.2?
>
>
> I would love to debug this. If would be a tremendous help if you had a
> little test program that reproduces this against a single server, so
> that I can see what is going on.
>
> Thanks.
>
> -- Lars
>
>
>
> ----- Original Message -----
> From: Yousuf Ahmad <[EMAIL PROTECTED]>
> To: [EMAIL PROTECTED]; lars hofhansl <[EMAIL PROTECTED]>
> Cc: Ivan Brondino <[EMAIL PROTECTED]>; Ricardo Vilaça <
> [EMAIL PROTECTED]>
> Sent: Thursday, October 18, 2012 12:59 PM
> Subject: Re: High IPC Latency
>
> Hi,
>
> Thank you for your questions guys.
>
> We are using HBase 0.92 with HDFS 1.0.1.
>
> The experiment lasts 15 minutes. The measurements stabilize in the
> first two minutes of the run.
>
> The data is distributed almost evenly across the regionservers so each
> client hits most of them over the course of the experiment. However,
> for the data we have, any given multi-get or scan should touch only
> one or at most two regions.
>
> The client caches the locations of the regionservers, so after a
> couple of minutes of the experiment running, it wouldn't need to
> re-visit ZooKeeper, I believe. Correct me if I am wrong please.
>
> Regards,
> Yousuf
>
>
> On Thu, Oct 18, 2012 at 2:42 PM, lars hofhansl <[EMAIL PROTECTED]>
> wrote:
>
> > Also, what version of HBase/HDFS is this using?
> >
> >
> >
> >
> > ----- Original Message -----
> > From: "Pamecha, Abhishek" <[EMAIL PROTECTED]>
> > To: "[EMAIL PROTECTED]" <[EMAIL PROTECTED]>
> > Cc: Ivan Brondino <[EMAIL PROTECTED]>; Ricardo Vilaça <
> > [EMAIL PROTECTED]>
> > Sent: Thursday, October 18, 2012 11:38 AM
> > Subject: RE: High IPC Latency
> >
> > Is it sustained for the same client hitting the same region server
> > OR
> does
> > it get better for the same client-RS combination when run for longer
> > duration?  Trying to eliminate Zookeeper from this.
> >
> > Thanks,
> > Abhishek
> >
> > From: Yousuf Ahmad [mailto:[EMAIL PROTECTED]]
> > Sent: Thursday, October 18, 2012 11:26 AM
> > To: [EMAIL PROTECTED]
> > Cc: Ivan Brondino; Ricardo Vilaça
> > Subject: High IPC Latency
> >
> > Hello,
> >
> > We are seeing slow times for read operations in our experiments. We
> > are hoping that you guys can help us figure out what's going wrong.
> >
> > Here are some details:
> >
> >   *   We are running a read-only benchmark on our HBase cluster.
> >   *
> >   *   There are 10 regionservers, each co-located with a datanode. HDFS
> > replication is 3x.
> >   *   All the data read by the experiment is already in the block cache
> > and the hit ratio is 99%.
> >   *
> >   *   We have 10 clients, each with around 400 threads making a mix of
> > read-only requests involving multi-gets and scans.