Hi,

Could anybody please point to some article or paper which can give an
understanding of the transaction per second (both read and write) that is
supported or seen to be accomplished using HBase Map-Reduce.

We have written few HBase Map-reduces which are not giving us the
desired/expected performance .

--
Thanks & Regards
Himanish

See http://hbase.apache.org/book.html#performance
St.Ack

On Tue, May 24, 2011 at 11:31 AM, Himanish Kushary <[EMAIL PROTECTED]> wrote:
> Hi,
>
> Could anybody please point to some article or paper which can give an
> understanding of the transaction per second (both read and write) that is
> supported or seen to be accomplished using HBase Map-Reduce.
>
> We have written few HBase Map-reduces which are not giving us the
> desired/expected performance .
>
> --
> Thanks & Regards
> Himanish
>

Himanish,

Are we talking about an African or European Swallow?
(Sorry its a reference to the Monty Python movie scene where they cross the bridge after being asked 3 questions which they must answer correctly? [What's the forward air speed velocity of an unladen swallow?])

The point is that you're asking a question which has no valid answer because there are so many variables that will effect the timings.

Just to list a couple of the things that will impact your performance:
1) Hardware
2) Network topology
3) Table schema.
4) Tuning of Hadoop environment
5) Caching? Oh wait, that could be considered Tuning.

So while I can understand that you want to improve your performance, you really have to get down and look at a couple of larger issues.

-Mike

PS. Whatever expectations you have, you'll need to lower them. ;-)
 
----------------------------------------
> Date: Tue, 24 May 2011 14:31:21 -0400
> Subject: HBase Transaction per second in Map-Reduce
> From: [EMAIL PROTECTED]
> To: [EMAIL PROTECTED]
>
> Hi,
>
> Could anybody please point to some article or paper which can give an
> understanding of the transaction per second (both read and write) that is
> supported or seen to be accomplished using HBase Map-Reduce.
>
> We have written few HBase Map-reduces which are not giving us the
> desired/expected performance .
>
> --
> Thanks & Regards
> Himanish
     

Sorry,
Its been one of those days.

----------------------------------------
> From: [EMAIL PROTECTED]
> To: [EMAIL PROTECTED]
> Subject: RE: HBase Transaction per second in Map-Reduce
> Date: Tue, 24 May 2011 14:18:28 -0500
>
>
>
> Himanish,
>
> Are we talking about an African or European Swallow?
> (Sorry its a reference to the Monty Python movie scene where they cross the bridge after being asked 3 questions which they must answer correctly? [What's the forward air speed velocity of an unladen swallow?])
>
> The point is that you're asking a question which has no valid answer because there are so many variables that will effect the timings.
>
> Just to list a couple of the things that will impact your performance:
> 1) Hardware
> 2) Network topology
> 3) Table schema.
> 4) Tuning of Hadoop environment
> 5) Caching? Oh wait, that could be considered Tuning.
>
> So while I can understand that you want to improve your performance, you really have to get down and look at a couple of larger issues.
>
> -Mike
>
> PS. Whatever expectations you have, you'll need to lower them. ;-)
>
> ----------------------------------------
> > Date: Tue, 24 May 2011 14:31:21 -0400
> > Subject: HBase Transaction per second in Map-Reduce
> > From: [EMAIL PROTECTED]
> > To: [EMAIL PROTECTED]
> >
> > Hi,
> >
> > Could anybody please point to some article or paper which can give an
> > understanding of the transaction per second (both read and write) that is
> > supported or seen to be accomplished using HBase Map-Reduce.
> >
> > We have written few HBase Map-reduces which are not giving us the
> > desired/expected performance .
> >
> > --
> > Thanks & Regards
> > Himanish
>
     

Don't worry..its ok...i am going through one of those days for the last few
days :-)

Jokes apart, when you talk about Caching, could we put something like
ehCache in-front of hbase integrated like a level-2 cache on top of the
already provided block cache ?

- Himanish

On Tue, May 24, 2011 at 3:19 PM, Michael Segel <[EMAIL PROTECTED]>wrote:

>
> Sorry,
> Its been one of those days.
>
> ----------------------------------------
> > From: [EMAIL PROTECTED]
> > To: [EMAIL PROTECTED]
> > Subject: RE: HBase Transaction per second in Map-Reduce
> > Date: Tue, 24 May 2011 14:18:28 -0500
> >
> >
> >
> > Himanish,
> >
> > Are we talking about an African or European Swallow?
> > (Sorry its a reference to the Monty Python movie scene where they cross
> the bridge after being asked 3 questions which they must answer correctly?
> [What's the forward air speed velocity of an unladen swallow?])
> >
> > The point is that you're asking a question which has no valid answer
> because there are so many variables that will effect the timings.
> >
> > Just to list a couple of the things that will impact your performance:
> > 1) Hardware
> > 2) Network topology
> > 3) Table schema.
> > 4) Tuning of Hadoop environment
> > 5) Caching? Oh wait, that could be considered Tuning.
> >
> > So while I can understand that you want to improve your performance, you
> really have to get down and look at a couple of larger issues.
> >
> > -Mike
> >
> > PS. Whatever expectations you have, you'll need to lower them. ;-)
> >
> > ----------------------------------------
> > > Date: Tue, 24 May 2011 14:31:21 -0400
> > > Subject: HBase Transaction per second in Map-Reduce
> > > From: [EMAIL PROTECTED]
> > > To: [EMAIL PROTECTED]
> > >
> > > Hi,
> > >
> > > Could anybody please point to some article or paper which can give an
> > > understanding of the transaction per second (both read and write) that
> is
> > > supported or seen to be accomplished using HBase Map-Reduce.
> > >
> > > We have written few HBase Map-reduces which are not giving us the
> > > desired/expected performance .
> > >
> > > --
> > > Thanks & Regards
> > > Himanish
> >
>
>

--
Thanks & Regards
Himanish

Figure first what is slow before adding yet more stuff

On May 24, 2011, at 14:06, Himanish Kushary <[EMAIL PROTECTED]> wrote:

> Don't worry..its ok...i am going through one of those days for the last few
> days :-)
>
> Jokes apart, when you talk about Caching, could we put something like
> ehCache in-front of hbase integrated like a level-2 cache on top of the
> already provided block cache ?
>
> - Himanish
>
> On Tue, May 24, 2011 at 3:19 PM, Michael Segel <[EMAIL PROTECTED]>wrote:
>
>>
>> Sorry,
>> Its been one of those days.
>>
>> ----------------------------------------
>>> From: [EMAIL PROTECTED]
>>> To: [EMAIL PROTECTED]
>>> Subject: RE: HBase Transaction per second in Map-Reduce
>>> Date: Tue, 24 May 2011 14:18:28 -0500
>>>
>>>
>>>
>>> Himanish,
>>>
>>> Are we talking about an African or European Swallow?
>>> (Sorry its a reference to the Monty Python movie scene where they cross
>> the bridge after being asked 3 questions which they must answer correctly?
>> [What's the forward air speed velocity of an unladen swallow?])
>>>
>>> The point is that you're asking a question which has no valid answer
>> because there are so many variables that will effect the timings.
>>>
>>> Just to list a couple of the things that will impact your performance:
>>> 1) Hardware
>>> 2) Network topology
>>> 3) Table schema.
>>> 4) Tuning of Hadoop environment
>>> 5) Caching? Oh wait, that could be considered Tuning.
>>>
>>> So while I can understand that you want to improve your performance, you
>> really have to get down and look at a couple of larger issues.
>>>
>>> -Mike
>>>
>>> PS. Whatever expectations you have, you'll need to lower them. ;-)
>>>
>>> ----------------------------------------
>>>> Date: Tue, 24 May 2011 14:31:21 -0400
>>>> Subject: HBase Transaction per second in Map-Reduce
>>>> From: [EMAIL PROTECTED]
>>>> To: [EMAIL PROTECTED]
>>>>
>>>> Hi,
>>>>
>>>> Could anybody please point to some article or paper which can give an
>>>> understanding of the transaction per second (both read and write) that
>> is
>>>> supported or seen to be accomplished using HBase Map-Reduce.
>>>>
>>>> We have written few HBase Map-reduces which are not giving us the
>>>> desired/expected performance .
>>>>
>>>> --
>>>> Thanks & Regards
>>>> Himanish
>>>
>>
>>
>
>
>
> --
> Thanks & Regards
> Himanish

To understand that whether out HBase instance is slow or is at par with
industry standards/known implementations I was looking for some article,stats
or paper for HBase TPS and performance.So the initial question :-(

It seems to be slow as per our SLA of processing the data we get.

On Tue, May 24, 2011 at 5:16 PM, Stack <[EMAIL PROTECTED]> wrote:

> Figure first what is slow before adding yet more stuff
>
>
>
> On May 24, 2011, at 14:06, Himanish Kushary <[EMAIL PROTECTED]> wrote:
>
> > Don't worry..its ok...i am going through one of those days for the last
> few
> > days :-)
> >
> > Jokes apart, when you talk about Caching, could we put something like
> > ehCache in-front of hbase integrated like a level-2 cache on top of the
> > already provided block cache ?
> >
> > - Himanish
> >
> > On Tue, May 24, 2011 at 3:19 PM, Michael Segel <
> [EMAIL PROTECTED]>wrote:
> >
> >>
> >> Sorry,
> >> Its been one of those days.
> >>
> >> ----------------------------------------
> >>> From: [EMAIL PROTECTED]
> >>> To: [EMAIL PROTECTED]
> >>> Subject: RE: HBase Transaction per second in Map-Reduce
> >>> Date: Tue, 24 May 2011 14:18:28 -0500
> >>>
> >>>
> >>>
> >>> Himanish,
> >>>
> >>> Are we talking about an African or European Swallow?
> >>> (Sorry its a reference to the Monty Python movie scene where they cross
> >> the bridge after being asked 3 questions which they must answer
> correctly?
> >> [What's the forward air speed velocity of an unladen swallow?])
> >>>
> >>> The point is that you're asking a question which has no valid answer
> >> because there are so many variables that will effect the timings.
> >>>
> >>> Just to list a couple of the things that will impact your performance:
> >>> 1) Hardware
> >>> 2) Network topology
> >>> 3) Table schema.
> >>> 4) Tuning of Hadoop environment
> >>> 5) Caching? Oh wait, that could be considered Tuning.
> >>>
> >>> So while I can understand that you want to improve your performance,
> you
> >> really have to get down and look at a couple of larger issues.
> >>>
> >>> -Mike
> >>>
> >>> PS. Whatever expectations you have, you'll need to lower them. ;-)
> >>>
> >>> ----------------------------------------
> >>>> Date: Tue, 24 May 2011 14:31:21 -0400
> >>>> Subject: HBase Transaction per second in Map-Reduce
> >>>> From: [EMAIL PROTECTED]
> >>>> To: [EMAIL PROTECTED]
> >>>>
> >>>> Hi,
> >>>>
> >>>> Could anybody please point to some article or paper which can give an
> >>>> understanding of the transaction per second (both read and write) that
> >> is
> >>>> supported or seen to be accomplished using HBase Map-Reduce.
> >>>>
> >>>> We have written few HBase Map-reduces which are not giving us the
> >>>> desired/expected performance .
> >>>>
> >>>> --
> >>>> Thanks & Regards
> >>>> Himanish
> >>>
> >>
> >>
> >
> >
> >
> > --
> > Thanks & Regards
> > Himanish
>

--
Thanks & Regards
Himanish

You could, but you don't really need to do that.
Of course the size of your cache is tunable and based on how much memory you have.
On a side note... You said you're on CDH3, it doesn't have co-processor support, unless they snuck it in at the last minute.

What I. Think you need to do is to look at the doc in then link provided. GC tuning, good schema design will go a long way towards getting the performance you want.
Sent from a remote device. Please excuse any typos...

Mike Segel

On May 24, 2011, at 4:06 PM, Himanish Kushary <[EMAIL PROTECTED]> wrote:

> Don't worry..its ok...i am going through one of those days for the last few
> days :-)
>
> Jokes apart, when you talk about Caching, could we put something like
> ehCache in-front of hbase integrated like a level-2 cache on top of the
> already provided block cache ?
>
> - Himanish
>
> On Tue, May 24, 2011 at 3:19 PM, Michael Segel <[EMAIL PROTECTED]>wrote:
>
>>
>> Sorry,
>> Its been one of those days.
>>
>> ----------------------------------------
>>> From: [EMAIL PROTECTED]
>>> To: [EMAIL PROTECTED]
>>> Subject: RE: HBase Transaction per second in Map-Reduce
>>> Date: Tue, 24 May 2011 14:18:28 -0500
>>>
>>>
>>>
>>> Himanish,
>>>
>>> Are we talking about an African or European Swallow?
>>> (Sorry its a reference to the Monty Python movie scene where they cross
>> the bridge after being asked 3 questions which they must answer correctly?
>> [What's the forward air speed velocity of an unladen swallow?])
>>>
>>> The point is that you're asking a question which has no valid answer
>> because there are so many variables that will effect the timings.
>>>
>>> Just to list a couple of the things that will impact your performance:
>>> 1) Hardware
>>> 2) Network topology
>>> 3) Table schema.
>>> 4) Tuning of Hadoop environment
>>> 5) Caching? Oh wait, that could be considered Tuning.
>>>
>>> So while I can understand that you want to improve your performance, you
>> really have to get down and look at a couple of larger issues.
>>>
>>> -Mike
>>>
>>> PS. Whatever expectations you have, you'll need to lower them. ;-)
>>>
>>> ----------------------------------------
>>>> Date: Tue, 24 May 2011 14:31:21 -0400
>>>> Subject: HBase Transaction per second in Map-Reduce
>>>> From: [EMAIL PROTECTED]
>>>> To: [EMAIL PROTECTED]
>>>>
>>>> Hi,
>>>>
>>>> Could anybody please point to some article or paper which can give an
>>>> understanding of the transaction per second (both read and write) that
>> is
>>>> supported or seen to be accomplished using HBase Map-Reduce.
>>>>
>>>> We have written few HBase Map-reduces which are not giving us the
>>>> desired/expected performance .
>>>>
>>>> --
>>>> Thanks & Regards
>>>> Himanish
>>>
>>
>>
>
>
>
> --
> Thanks & Regards
> Himanish