Any of you out there who are running the 0.20+200+etc hadoop with 0.20
branch hbase want to try this patch? I think with it on, we can practically
turn off group commit and get quite a bit speedup.

--
Todd Lipcon
Software Engineer, Cloudera

Todd: Good stuff! Have you tried this out on a HBase test too?

> I think with it on, we can practically turn off group commit and get quite a bit speedup.

Will batching now happen underneath the DFSClient automatically now that syncs/appends are not serialized.

regards,
Kannan
-----Original Message-----
From: Todd Lipcon [mailto:[EMAIL PROTECTED]]
Sent: Thursday, April 15, 2010 1:59 AM
To: [EMAIL PROTECTED]
Subject: HDFS-895 hflush speedup

Any of you out there who are running the 0.20+200+etc hadoop with 0.20
branch hbase want to try this patch? I think with it on, we can practically
turn off group commit and get quite a bit speedup.

--
Todd Lipcon
Software Engineer, Cloudera

I'll try this out when I get a chance. Probably not until the weekend. This looks great.

  - Andy
> From: Kannan Muthukkaruppan <[EMAIL PROTECTED]>
> Subject: RE: HDFS-895 hflush speedup
> To: "[EMAIL PROTECTED]" <[EMAIL PROTECTED]>
> Date: Thursday, April 15, 2010, 10:51 AM
> Todd: Good stuff! Have you tried this
> out on a HBase test too?
>
> > I think with it on, we can practically turn off group
> > commit and get quite a bit speedup.
>
> Will batching now happen underneath the DFSClient
> automatically now that syncs/appends are not serialized.
>
> regards,
> Kannan
> -----Original Message-----
> From: Todd Lipcon [mailto:[EMAIL PROTECTED]]
> Sent: Thursday, April 15, 2010 1:59 AM
> To: [EMAIL PROTECTED]
> Subject: HDFS-895 hflush speedup
>
> Any of you out there who are running the 0.20+200+etc
> hadoop with 0.20 branch hbase want to try this patch?
> I think with it on, we can practically
> turn off group commit and get quite a bit speedup.
>
> --
> Todd Lipcon
> Software Engineer, Cloudera

On Thu, Apr 15, 2010 at 10:51 AM, Kannan Muthukkaruppan <[EMAIL PROTECTED]
> wrote:

> Todd: Good stuff! Have you tried this out on a HBase test too?
>
>
Not yet - I basically had a burst of inspiration last night around 11pm,
coded til I couldn't stay awake, and threw up a patch :) I hope to give it a
try today (and also update patch to trunk so I can get some reviews there)
> > I think with it on, we can practically turn off group commit and get
> quite a bit speedup.
>
> Will batching now happen underneath the DFSClient automatically now that
> syncs/appends are not serialized.
>
>
It's not quite batching, but essentially I think we get the same benefit.
While one thread is waiting for *its* packet/write to be synced, another
thread can still append to the stream and call sync for its own data.

-Todd

-----Original Message-----
> From: Todd Lipcon [mailto:[EMAIL PROTECTED]]
> Sent: Thursday, April 15, 2010 1:59 AM
> To: [EMAIL PROTECTED]
> Subject: HDFS-895 hflush speedup
>
> Any of you out there who are running the 0.20+200+etc hadoop with 0.20
> branch hbase want to try this patch? I think with it on, we can practically
> turn off group commit and get quite a bit speedup.
>
> --
> Todd Lipcon
> Software Engineer, Cloudera
>

--
Todd Lipcon
Software Engineer, Cloudera

I'm running whats on here:

http://github.com/ryanobjc/hadoop-cloudera

Which is basically the patches you sent me previously.

I have a cluster I would be willing to test with...

-ryan

On Thu, Apr 15, 2010 at 11:54 AM, Todd Lipcon <[EMAIL PROTECTED]> wrote:
> On Thu, Apr 15, 2010 at 10:51 AM, Kannan Muthukkaruppan <[EMAIL PROTECTED]
>> wrote:
>
>> Todd: Good stuff! Have you tried this out on a HBase test too?
>>
>>
> Not yet - I basically had a burst of inspiration last night around 11pm,
> coded til I couldn't stay awake, and threw up a patch :) I hope to give it a
> try today (and also update patch to trunk so I can get some reviews there)
>
>
>> > I think with it on, we can practically turn off group commit and get
>> quite a bit speedup.
>>
>> Will batching now happen underneath the DFSClient automatically now that
>> syncs/appends are not serialized.
>>
>>
> It's not quite batching, but essentially I think we get the same benefit.
> While one thread is waiting for *its* packet/write to be synced, another
> thread can still append to the stream and call sync for its own data.
>
> -Todd
>
> -----Original Message-----
>> From: Todd Lipcon [mailto:[EMAIL PROTECTED]]
>> Sent: Thursday, April 15, 2010 1:59 AM
>> To: [EMAIL PROTECTED]
>> Subject: HDFS-895 hflush speedup
>>
>> Any of you out there who are running the 0.20+200+etc hadoop with 0.20
>> branch hbase want to try this patch? I think with it on, we can practically
>> turn off group commit and get quite a bit speedup.
>>
>> --
>> Todd Lipcon
>> Software Engineer, Cloudera
>>
>
>
>
> --
> Todd Lipcon
> Software Engineer, Cloudera
>