I have noticed some strange performance numbers on EC2.  If someone can give
me some hints to improve performance that would be greatly appreciated.
 Here are the details:

I have a process that runs a series of Jobs under Hadoop 0.20.1 & Hbase
0.20.2  I ran the *exact* same process with following configurations:

1) 1 Master & 4 Workers (*c1.xlarge* instances) & 1 Zookeeper (*c1.medium*)
with *8 Reducers *for every Reduce task.  The process completed in *849*
 seconds.

2) 1 Master, 4 Workers & 1 Zookeeper  *ALL m1.small* instances with *8
Reducers *for every Reduce task.  The process completed in *906* seconds.

3) 1 Master, *11* Workers & *3* Zookeepers  *ALL m1.small* instances with *20
Reducers *for every Reduce task.  The process completed in *984* seconds!
Two main questions:

1)  It's totally surprising that when I have 11 workers with 20 Reducers it
runs slower than when I have exactly same type of fewer machines with fewer
reducers..
2)  As expected it runs faster on c1.xlarge, but the performance improvement
doesn't justify the high cost difference.  I must not be utilizing the
machine power, but I don't know how to do that.

Here are some of the performance improvements tricks that I have learnt from
this mailing list in the past that I am using:

1)  conf.set("hbase.client.scanner.caching", "30");   I have this for all
jobs.

2)  Using the following code every time I open a HTable:
        this.table = new HTable(new HBaseConfiguration(), "tablenameXYZ");
        table.setAutoFlush(false);
        table.setWriteBufferSize(1024 * 1024 * 12);

3)  For every Put I do this:
          Put put = new Put(Bytes.toBytes(out));
          put.setWriteToWAL(false);

4)  Change the No. of Reducers as per the No. of Workers.  I believe the
formula is:  # of workers * 1.75.

Any other hints?  As always, greatly appreciate the help.  Thanks.

On Tue, Jan 26, 2010 at 8:47 AM, Something Something
<[EMAIL PROTECTED]> wrote:
> I have noticed some strange performance numbers on EC2.  If someone can give
> me some hints to improve performance that would be greatly appreciated.
>  Here are the details:
>
> I have a process that runs a series of Jobs under Hadoop 0.20.1 & Hbase
> 0.20.2  I ran the *exact* same process with following configurations:
>
> 1) 1 Master & 4 Workers (*c1.xlarge* instances) & 1 Zookeeper (*c1.medium*)
> with *8 Reducers *for every Reduce task.  The process completed in *849*
>  seconds.

How many concurrent reducers run on each node?  Default two?

>
> 2) 1 Master, 4 Workers & 1 Zookeeper  *ALL m1.small* instances with *8
> Reducers *for every Reduce task.  The process completed in *906* seconds.
>
> 3) 1 Master, *11* Workers & *3* Zookeepers  *ALL m1.small* instances with *20
> Reducers *for every Reduce task.  The process completed in *984* seconds!
>

How much of this overall time is spent in reduce phase, in particular
the time spent inserting into hbase? (Starts at 66% IIRC)

>
> Two main questions:
>
> 1)  It's totally surprising that when I have 11 workers with 20 Reducers it
> runs slower than when I have exactly same type of fewer machines with fewer
> reducers..
Yes.  My guess is that on the small instances, that if you ran the job
multiple times that there would be large variance in how long it takes
to complete.
> 2)  As expected it runs faster on c1.xlarge, but the performance improvement
> doesn't justify the high cost difference.  I must not be utilizing the
> machine power, but I don't know how to do that.
>

The main reason for xlarge is that the platform is more predictable in
its performance profile than small sized instances.  I'm a little
surprised that all worked on the small instances, that your jobs
completed.

I'd suggest you spend a bit of time figuring where your MR jobs are
spending their time?  Is it all doing hbase inserts?  Are inserts to a
new table?
> Here are some of the performance improvements tricks that I have learnt from
> this mailing list in the past that I am using:
>
> 1)  conf.set("hbase.client.scanner.caching", "30");   I have this for all
> jobs
FYI, you can set this on the Scan instance rather than globally in the
conf.  Just FYI.
>
> 2)  Using the following code every time I open a HTable:
>        this.table = new HTable(new HBaseConfiguration(), "tablenameXYZ");
>        table.setAutoFlush(false);
>        table.setWriteBufferSize(1024 * 1024 * 12);

Are you opening a new table inside each task or once up in the config?

>
> 4)  Change the No. of Reducers as per the No. of Workers.  I believe the
> formula is:  # of workers * 1.75.

You have to temper the above general rule with the fact that
tasktrackers and datanodes running on the same node can impinge upon
each other, often to the regionservers detriment.

Thats enough for now.  I'm sure others on list have opinions on the above.

St.Ack

How big is your dataset?

J-D

On Tue, Jan 26, 2010 at 8:47 AM, Something Something
<[EMAIL PROTECTED]> wrote:
> I have noticed some strange performance numbers on EC2.  If someone can give
> me some hints to improve performance that would be greatly appreciated.
>  Here are the details:
>
> I have a process that runs a series of Jobs under Hadoop 0.20.1 & Hbase
> 0.20.2  I ran the *exact* same process with following configurations:
>
> 1) 1 Master & 4 Workers (*c1.xlarge* instances) & 1 Zookeeper (*c1.medium*)
> with *8 Reducers *for every Reduce task.  The process completed in *849*
>  seconds.
>
> 2) 1 Master, 4 Workers & 1 Zookeeper  *ALL m1.small* instances with *8
> Reducers *for every Reduce task.  The process completed in *906* seconds.
>
> 3) 1 Master, *11* Workers & *3* Zookeepers  *ALL m1.small* instances with *20
> Reducers *for every Reduce task.  The process completed in *984* seconds!
>
>
> Two main questions:
>
> 1)  It's totally surprising that when I have 11 workers with 20 Reducers it
> runs slower than when I have exactly same type of fewer machines with fewer
> reducers..
> 2)  As expected it runs faster on c1.xlarge, but the performance improvement
> doesn't justify the high cost difference.  I must not be utilizing the
> machine power, but I don't know how to do that.
>
> Here are some of the performance improvements tricks that I have learnt from
> this mailing list in the past that I am using:
>
> 1)  conf.set("hbase.client.scanner.caching", "30");   I have this for all
> jobs.
>
> 2)  Using the following code every time I open a HTable:
>        this.table = new HTable(new HBaseConfiguration(), "tablenameXYZ");
>        table.setAutoFlush(false);
>        table.setWriteBufferSize(1024 * 1024 * 12);
>
> 3)  For every Put I do this:
>          Put put = new Put(Bytes.toBytes(out));
>          put.setWriteToWAL(false);
>
> 4)  Change the No. of Reducers as per the No. of Workers.  I believe the
> formula is:  # of workers * 1.75.
>
> Any other hints?  As always, greatly appreciate the help.  Thanks.
>

Here are some of the answers:

>>  How many concurrent reducers run on each node?  Default two?
I was assuming 2 on each node would be the default.  If not, this could be a
problem.  Please let me know.

>>'d suggest you spend a bit of time figuring where your MR jobs
are spending their time?
I agree.  Will do some more research :)

>>How much of this overall time is spent in reduce phase?
Mostly time is spent in the Reduce phases, because that's where most of the
critical code is.

>>Are inserts to a new table?
Yes, all inserts will always be in a new table.  In fact, I disable/drop
HTables during this process.  Not using any special indexes, should I be?

>>I'm a little surprised that all worked on the small instances, that your
jobs completed.
But, really, shouldn't Amazon guarantee predictability :)  After all I am
paying for these instances.. albeit a small amount!

>>Are you opening a new table inside each task or once up in the config?
I open HTable in the 'setup' method for each mapper/reducer, and close table
in the 'cleanup' method.

>>You have to temper the above general rule with the fact that...
I will try a few combinations.

>>How big is your dataset?
This one in particular is not big, but the real production ones will be.
 Here's approximately how many rows get processed:
Phase 1:  300 rows
Phase 2 thru 8:  100 rows.
(Note:  Each phase does complex calculations on the row.)

Thanks for the help.
On Tue, Jan 26, 2010 at 10:36 AM, Jean-Daniel Cryans <[EMAIL PROTECTED]>wrote:

> How big is your dataset?
>
> J-D
>
> On Tue, Jan 26, 2010 at 8:47 AM, Something Something
> <[EMAIL PROTECTED]> wrote:
> > I have noticed some strange performance numbers on EC2.  If someone can
> give
> > me some hints to improve performance that would be greatly appreciated.
> >  Here are the details:
> >
> > I have a process that runs a series of Jobs under Hadoop 0.20.1 & Hbase
> > 0.20.2  I ran the *exact* same process with following configurations:
> >
> > 1) 1 Master & 4 Workers (*c1.xlarge* instances) & 1 Zookeeper
> (*c1.medium*)
> > with *8 Reducers *for every Reduce task.  The process completed in *849*
> >  seconds.
> >
> > 2) 1 Master, 4 Workers & 1 Zookeeper  *ALL m1.small* instances with *8
> > Reducers *for every Reduce task.  The process completed in *906* seconds.
> >
> > 3) 1 Master, *11* Workers & *3* Zookeepers  *ALL m1.small* instances with
> *20
> > Reducers *for every Reduce task.  The process completed in *984* seconds!
> >
> >
> > Two main questions:
> >
> > 1)  It's totally surprising that when I have 11 workers with 20 Reducers
> it
> > runs slower than when I have exactly same type of fewer machines with
> fewer
> > reducers..
> > 2)  As expected it runs faster on c1.xlarge, but the performance
> improvement
> > doesn't justify the high cost difference.  I must not be utilizing the
> > machine power, but I don't know how to do that.
> >
> > Here are some of the performance improvements tricks that I have learnt
> from
> > this mailing list in the past that I am using:
> >
> > 1)  conf.set("hbase.client.scanner.caching", "30");   I have this for all
> > jobs.
> >
> > 2)  Using the following code every time I open a HTable:
> >        this.table = new HTable(new HBaseConfiguration(), "tablenameXYZ");
> >        table.setAutoFlush(false);
> >        table.setWriteBufferSize(1024 * 1024 * 12);
> >
> > 3)  For every Put I do this:
> >          Put put = new Put(Bytes.toBytes(out));
> >          put.setWriteToWAL(false);
> >
> > 4)  Change the No. of Reducers as per the No. of Workers.  I believe the
> > formula is:  # of workers * 1.75.
> >
> > Any other hints?  As always, greatly appreciate the help.  Thanks.
> >
>

Re "Amazon predictability", did you guys see this recent paper:
http://people.csail.mit.edu/tromer/cloudsec/

Also some addl background on "noisy neighbor effects":
http://bit.ly/4O7dHx
http://bit.ly/8zPvQd

Some interesting bits of information in there.

Patrick

Something Something wrote:
> Here are some of the answers:
>
>>>  How many concurrent reducers run on each node?  Default two?
> I was assuming 2 on each node would be the default.  If not, this could be a
> problem.  Please let me know.
>
>>> 'd suggest you spend a bit of time figuring where your MR jobs
> are spending their time?
> I agree.  Will do some more research :)
>
>>> How much of this overall time is spent in reduce phase?
> Mostly time is spent in the Reduce phases, because that's where most of the
> critical code is.
>
>>> Are inserts to a new table?
> Yes, all inserts will always be in a new table.  In fact, I disable/drop
> HTables during this process.  Not using any special indexes, should I be?
>
>>> I'm a little surprised that all worked on the small instances, that your
> jobs completed.
> But, really, shouldn't Amazon guarantee predictability :)  After all I am
> paying for these instances.. albeit a small amount!
>
>>> Are you opening a new table inside each task or once up in the config?
> I open HTable in the 'setup' method for each mapper/reducer, and close table
> in the 'cleanup' method.
>
>>> You have to temper the above general rule with the fact that...
> I will try a few combinations.
>
>>> How big is your dataset?
> This one in particular is not big, but the real production ones will be.
>  Here's approximately how many rows get processed:
> Phase 1:  300 rows
> Phase 2 thru 8:  100 rows.
> (Note:  Each phase does complex calculations on the row.)
>
> Thanks for the help.
>
>
> On Tue, Jan 26, 2010 at 10:36 AM, Jean-Daniel Cryans <[EMAIL PROTECTED]>wrote:
>
>> How big is your dataset?
>>
>> J-D
>>
>> On Tue, Jan 26, 2010 at 8:47 AM, Something Something
>> <[EMAIL PROTECTED]> wrote:
>>> I have noticed some strange performance numbers on EC2.  If someone can
>> give
>>> me some hints to improve performance that would be greatly appreciated.
>>>  Here are the details:
>>>
>>> I have a process that runs a series of Jobs under Hadoop 0.20.1 & Hbase
>>> 0.20.2  I ran the *exact* same process with following configurations:
>>>
>>> 1) 1 Master & 4 Workers (*c1.xlarge* instances) & 1 Zookeeper
>> (*c1.medium*)
>>> with *8 Reducers *for every Reduce task.  The process completed in *849*
>>>  seconds.
>>>
>>> 2) 1 Master, 4 Workers & 1 Zookeeper  *ALL m1.small* instances with *8
>>> Reducers *for every Reduce task.  The process completed in *906* seconds.
>>>
>>> 3) 1 Master, *11* Workers & *3* Zookeepers  *ALL m1.small* instances with
>> *20
>>> Reducers *for every Reduce task.  The process completed in *984* seconds!
>>>
>>>
>>> Two main questions:
>>>
>>> 1)  It's totally surprising that when I have 11 workers with 20 Reducers
>> it
>>> runs slower than when I have exactly same type of fewer machines with
>> fewer
>>> reducers..
>>> 2)  As expected it runs faster on c1.xlarge, but the performance
>> improvement
>>> doesn't justify the high cost difference.  I must not be utilizing the
>>> machine power, but I don't know how to do that.
>>>
>>> Here are some of the performance improvements tricks that I have learnt
>> from
>>> this mailing list in the past that I am using:
>>>
>>> 1)  conf.set("hbase.client.scanner.caching", "30");   I have this for all
>>> jobs.
>>>
>>> 2)  Using the following code every time I open a HTable:
>>>        this.table = new HTable(new HBaseConfiguration(), "tablenameXYZ");
>>>        table.setAutoFlush(false);
>>>        table.setWriteBufferSize(1024 * 1024 * 12);
>>>
>>> 3)  For every Put I do this:
>>>          Put put = new Put(Bytes.toBytes(out));
>>>          put.setWriteToWAL(false);
>>>
>>> 4)  Change the No. of Reducers as per the No. of Workers.  I believe the
>>> formula is:  # of workers * 1.75.

Wow.. how naive I am to think that I could trust Amazon.  Thanks for
forwarding the links, Patrick.  Seems like Amazon's reliability has gone
down considerably over the past few months.  (Occasionally my instances fail
on startup or die in the middle for no apparent reason, and I used to think
I was doing something dumb!)

But what I don't understand is this... if I *reserve* an instance then I
wouldn't be sharing its CPU with anyone, right?  The blog seems to indicate
otherwise.

I guess, I will have to look for alternatives to Amazon EC2.  Any one has
any recommendations?  Thanks again.
On Tue, Jan 26, 2010 at 11:44 AM, Patrick Hunt <[EMAIL PROTECTED]> wrote:

> Re "Amazon predictability", did you guys see this recent paper:
> http://people.csail.mit.edu/tromer/cloudsec/
>
> Also some addl background on "noisy neighbor effects":
> http://bit.ly/4O7dHx
> http://bit.ly/8zPvQd
>
> Some interesting bits of information in there.
>
> Patrick
>
>
> Something Something wrote:
>
>> Here are some of the answers:
>>
>>   How many concurrent reducers run on each node?  Default two?
>>>>
>>> I was assuming 2 on each node would be the default.  If not, this could
>> be a
>> problem.  Please let me know.
>>
>>  'd suggest you spend a bit of time figuring where your MR jobs
>>>>
>>> are spending their time?
>> I agree.  Will do some more research :)
>>
>>  How much of this overall time is spent in reduce phase?
>>>>
>>> Mostly time is spent in the Reduce phases, because that's where most of
>> the
>> critical code is.
>>
>>  Are inserts to a new table?
>>>>
>>> Yes, all inserts will always be in a new table.  In fact, I disable/drop
>> HTables during this process.  Not using any special indexes, should I be?
>>
>>  I'm a little surprised that all worked on the small instances, that your
>>>>
>>> jobs completed.
>> But, really, shouldn't Amazon guarantee predictability :)  After all I am
>> paying for these instances.. albeit a small amount!
>>
>>  Are you opening a new table inside each task or once up in the config?
>>>>
>>> I open HTable in the 'setup' method for each mapper/reducer, and close
>> table
>> in the 'cleanup' method.
>>
>>  You have to temper the above general rule with the fact that...
>>>>
>>> I will try a few combinations.
>>
>>  How big is your dataset?
>>>>
>>> This one in particular is not big, but the real production ones will be.
>>  Here's approximately how many rows get processed:
>> Phase 1:  300 rows
>> Phase 2 thru 8:  100 rows.
>> (Note:  Each phase does complex calculations on the row.)
>>
>> Thanks for the help.
>>
>>
>> On Tue, Jan 26, 2010 at 10:36 AM, Jean-Daniel Cryans <[EMAIL PROTECTED]
>> >wrote:
>>
>>  How big is your dataset?
>>>
>>> J-D
>>>
>>> On Tue, Jan 26, 2010 at 8:47 AM, Something Something
>>> <[EMAIL PROTECTED]> wrote:
>>>
>>>> I have noticed some strange performance numbers on EC2.  If someone can
>>>>
>>> give
>>>
>>>> me some hints to improve performance that would be greatly appreciated.
>>>>  Here are the details:
>>>>
>>>> I have a process that runs a series of Jobs under Hadoop 0.20.1 & Hbase
>>>> 0.20.2  I ran the *exact* same process with following configurations:
>>>>
>>>> 1) 1 Master & 4 Workers (*c1.xlarge* instances) & 1 Zookeeper
>>>>
>>> (*c1.medium*)
>>>
>>>> with *8 Reducers *for every Reduce task.  The process completed in *849*
>>>>  seconds.
>>>>
>>>> 2) 1 Master, 4 Workers & 1 Zookeeper  *ALL m1.small* instances with *8
>>>> Reducers *for every Reduce task.  The process completed in *906*
>>>> seconds.
>>>>
>>>> 3) 1 Master, *11* Workers & *3* Zookeepers  *ALL m1.small* instances
>>>> with
>>>>
>>> *20
>>>
>>>> Reducers *for every Reduce task.  The process completed in *984*
>>>> seconds!
>>>>
>>>>
>>>> Two main questions:
>>>>
>>>> 1)  It's totally surprising that when I have 11 workers with 20 Reducers
>>>>
>>> it
>>>
>>>> runs slower than when I have exactly same type of fewer machines with
>>>>
>>> fewer
>>>
>>>> reducers..
>>>> 2)  As expected it runs faster on c1.xlarge, but the performance

FYI, just noticed this one:

Rackspace Cloud Servers versus Amazon EC2: Performance Analysis
http://bit.ly/bkG1AB

Patrick

Something Something wrote:
> Wow.. how naive I am to think that I could trust Amazon.  Thanks for
> forwarding the links, Patrick.  Seems like Amazon's reliability has gone
> down considerably over the past few months.  (Occasionally my instances fail
> on startup or die in the middle for no apparent reason, and I used to think
> I was doing something dumb!)
>
> But what I don't understand is this... if I *reserve* an instance then I
> wouldn't be sharing its CPU with anyone, right?  The blog seems to indicate
> otherwise.
>
> I guess, I will have to look for alternatives to Amazon EC2.  Any one has
> any recommendations?  Thanks again.
>
>
> On Tue, Jan 26, 2010 at 11:44 AM, Patrick Hunt <[EMAIL PROTECTED]> wrote:
>
>> Re "Amazon predictability", did you guys see this recent paper:
>> http://people.csail.mit.edu/tromer/cloudsec/
>>
>> Also some addl background on "noisy neighbor effects":
>> http://bit.ly/4O7dHx
>> http://bit.ly/8zPvQd
>>
>> Some interesting bits of information in there.
>>
>> Patrick
>>
>>
>> Something Something wrote:
>>
>>> Here are some of the answers:
>>>
>>>   How many concurrent reducers run on each node?  Default two?
>>>> I was assuming 2 on each node would be the default.  If not, this could
>>> be a
>>> problem.  Please let me know.
>>>
>>>  'd suggest you spend a bit of time figuring where your MR jobs
>>>> are spending their time?
>>> I agree.  Will do some more research :)
>>>
>>>  How much of this overall time is spent in reduce phase?
>>>> Mostly time is spent in the Reduce phases, because that's where most of
>>> the
>>> critical code is.
>>>
>>>  Are inserts to a new table?
>>>> Yes, all inserts will always be in a new table.  In fact, I disable/drop
>>> HTables during this process.  Not using any special indexes, should I be?
>>>
>>>  I'm a little surprised that all worked on the small instances, that your
>>>> jobs completed.
>>> But, really, shouldn't Amazon guarantee predictability :)  After all I am
>>> paying for these instances.. albeit a small amount!
>>>
>>>  Are you opening a new table inside each task or once up in the config?
>>>> I open HTable in the 'setup' method for each mapper/reducer, and close
>>> table
>>> in the 'cleanup' method.
>>>
>>>  You have to temper the above general rule with the fact that...
>>>> I will try a few combinations.
>>>  How big is your dataset?
>>>> This one in particular is not big, but the real production ones will be.
>>>  Here's approximately how many rows get processed:
>>> Phase 1:  300 rows
>>> Phase 2 thru 8:  100 rows.
>>> (Note:  Each phase does complex calculations on the row.)
>>>
>>> Thanks for the help.
>>>
>>>
>>> On Tue, Jan 26, 2010 at 10:36 AM, Jean-Daniel Cryans <[EMAIL PROTECTED]
>>>> wrote:
>>>  How big is your dataset?
>>>> J-D
>>>>
>>>> On Tue, Jan 26, 2010 at 8:47 AM, Something Something
>>>> <[EMAIL PROTECTED]> wrote:
>>>>
>>>>> I have noticed some strange performance numbers on EC2.  If someone can
>>>>>
>>>> give
>>>>
>>>>> me some hints to improve performance that would be greatly appreciated.
>>>>>  Here are the details:
>>>>>
>>>>> I have a process that runs a series of Jobs under Hadoop 0.20.1 & Hbase
>>>>> 0.20.2  I ran the *exact* same process with following configurations:
>>>>>
>>>>> 1) 1 Master & 4 Workers (*c1.xlarge* instances) & 1 Zookeeper
>>>>>
>>>> (*c1.medium*)
>>>>
>>>>> with *8 Reducers *for every Reduce task.  The process completed in *849*
>>>>>  seconds.
>>>>>
>>>>> 2) 1 Master, 4 Workers & 1 Zookeeper  *ALL m1.small* instances with *8
>>>>> Reducers *for every Reduce task.  The process completed in *906*
>>>>> seconds.
>>>>>
>>>>> 3) 1 Master, *11* Workers & *3* Zookeepers  *ALL m1.small* instances
>>>>> with
>>>>>
>>>> *20
>>>>
>>>>> Reducers *for every Reduce task.  The process completed in *984*
>>>>> seconds!
>>>>>
>>>>>
>>>>> Two main questions:
>>>>>
>>>>> 1)  It's totally surprising that when I have 11 workers with 20 Reducers

> > But what I don't understand is this... if I *reserve* an instance then I
> > wouldn't be sharing its CPU with anyone, right?  The blog seems to indicate
> > otherwise.

No, that is not how EC2 and similar cloud infrastructure platforms work. You are
given a virtual machine. The hypervisor multiplexes many virtual machines on a
single physical server. The instance types purport to guarantee a certain level
of performance measured in virtual CPU units. The EC2 website describes in detail
how Amazon defines a virtual CPU unit.
> > (Occasionally my instances fail
> > on startup or die in the middle for no apparent reason, and I used to think
> > I was doing something dumb!)

That is also a misunderstanding of how to use cloud infrastructure. No single
instance is reliable. The platform is a commodity, cheap, with basic usability.
Durable services are built on top of these building blocks with service
architecture that does not rely on the stability of any single instance.

   - Andy

I think the reserved EC2 instances just give you a better deal price-wise in exchange for an advanced payment and, essentially, a contract.  I didn't see any mentions of reserved instances mean no sharing.  If AWS did that, they'd be nothing more than a regular hosting service.

 Otis
----
Sematext :: http://sematext.com/ :: Solr - Lucene - Nutch
Hadoop ecosystem search :: http://search-hadoop.com/

----- Original Message ----
> From: Something Something <[EMAIL PROTECTED]>
> To: [EMAIL PROTECTED]
> Cc: [EMAIL PROTECTED]
> Sent: Tue, January 26, 2010 3:49:31 PM
> Subject: Re: Performance of EC2
>
> Wow.. how naive I am to think that I could trust Amazon.  Thanks for
> forwarding the links, Patrick.  Seems like Amazon's reliability has gone
> down considerably over the past few months.  (Occasionally my instances fail
> on startup or die in the middle for no apparent reason, and I used to think
> I was doing something dumb!)
>
> But what I don't understand is this... if I *reserve* an instance then I
> wouldn't be sharing its CPU with anyone, right?  The blog seems to indicate
> otherwise.
>
> I guess, I will have to look for alternatives to Amazon EC2.  Any one has
> any recommendations?  Thanks again.
>
>
> On Tue, Jan 26, 2010 at 11:44 AM, Patrick Hunt wrote:
>
> > Re "Amazon predictability", did you guys see this recent paper:
> > http://people.csail.mit.edu/tromer/cloudsec/
> >
> > Also some addl background on "noisy neighbor effects":
> > http://bit.ly/4O7dHx
> > http://bit.ly/8zPvQd
> >
> > Some interesting bits of information in there.
> >
> > Patrick
> >
> >
> > Something Something wrote:
> >
> >> Here are some of the answers:
> >>
> >>   How many concurrent reducers run on each node?  Default two?
> >>>>
> >>> I was assuming 2 on each node would be the default.  If not, this could
> >> be a
> >> problem.  Please let me know.
> >>
> >>  'd suggest you spend a bit of time figuring where your MR jobs
> >>>>
> >>> are spending their time?
> >> I agree.  Will do some more research :)
> >>
> >>  How much of this overall time is spent in reduce phase?
> >>>>
> >>> Mostly time is spent in the Reduce phases, because that's where most of
> >> the
> >> critical code is.
> >>
> >>  Are inserts to a new table?
> >>>>
> >>> Yes, all inserts will always be in a new table.  In fact, I disable/drop
> >> HTables during this process.  Not using any special indexes, should I be?
> >>
> >>  I'm a little surprised that all worked on the small instances, that your
> >>>>
> >>> jobs completed.
> >> But, really, shouldn't Amazon guarantee predictability :)  After all I am
> >> paying for these instances.. albeit a small amount!
> >>
> >>  Are you opening a new table inside each task or once up in the config?
> >>>>
> >>> I open HTable in the 'setup' method for each mapper/reducer, and close
> >> table
> >> in the 'cleanup' method.
> >>
> >>  You have to temper the above general rule with the fact that...
> >>>>
> >>> I will try a few combinations.
> >>
> >>  How big is your dataset?
> >>>>
> >>> This one in particular is not big, but the real production ones will be.
> >>  Here's approximately how many rows get processed:
> >> Phase 1:  300 rows
> >> Phase 2 thru 8:  100 rows.
> >> (Note:  Each phase does complex calculations on the row.)
> >>
> >> Thanks for the help.
> >>
> >>
> >> On Tue, Jan 26, 2010 at 10:36 AM, Jean-Daniel Cryans
> >> >wrote:
> >>
> >>  How big is your dataset?
> >>>
> >>> J-D
> >>>
> >>> On Tue, Jan 26, 2010 at 8:47 AM, Something Something
> >>> wrote:
> >>>
> >>>> I have noticed some strange performance numbers on EC2.  If someone can
> >>>>
> >>> give
> >>>
> >>>> me some hints to improve performance that would be greatly appreciated.
> >>>>  Here are the details:
> >>>>
> >>>> I have a process that runs a series of Jobs under Hadoop 0.20.1 & Hbase
> >>>> 0.20.2  I ran the *exact* same process with following configurations:
> >>>>
> >>>> 1) 1 Master & 4 Workers (*c1.xlarge* instances) & 1 Zookeeper

Something Something wrote:
> Wow.. how naive I am to think that I could trust Amazon.  Thanks for
> forwarding the links, Patrick.  Seems like Amazon's reliability has gone
> down considerably over the past few months.  (Occasionally my instances fail
> on startup or die in the middle for no apparent reason, and I used to think
> I was doing something dumb!)

That's unfair. Large datacentres are inherently unreliable, because we
build out them out of "normal availability" stuff rather than HA
hardware. This then pushes the problem of availability down to the
applications, to you.
* Most of the problems people have been discussing are bandwidth issues;
it may be that AWS is coming under some massive DDoS attack and you are
seeing the fringes of it. It could be that your neighbours are noisy
-but if you are running big Hadoop jobs, you are the noisy neighbour.

* A more likely problem for you is where your machines are placed. If
they all share a single switch, very high bandwidth. But if they are on
different racks, the network becomes the bottleneck.
> But what I don't understand is this... if I *reserve* an instance then I
> wouldn't be sharing its CPU with anyone, right?  The blog seems to indicate
> otherwise.

I think you only get exclusive use of a CPU when you rent an XL node.
Reservations are a form of capacity planning, may or may not help with
scheduling at all.

Thanks everyone for the replies.  I agree I was being a bit unfair to
Amazon.  I apologize.

On Fri, Jan 29, 2010 at 4:00 AM, Steve Loughran <[EMAIL PROTECTED]> wrote:

> Something Something wrote:
>
>> Wow.. how naive I am to think that I could trust Amazon.  Thanks for
>> forwarding the links, Patrick.  Seems like Amazon's reliability has gone
>> down considerably over the past few months.  (Occasionally my instances
>> fail
>> on startup or die in the middle for no apparent reason, and I used to
>> think
>> I was doing something dumb!)
>>
>
> That's unfair. Large datacentres are inherently unreliable, because we
> build out them out of "normal availability" stuff rather than HA hardware.
> This then pushes the problem of availability down to the applications, to
> you.
>
>
> * Most of the problems people have been discussing are bandwidth issues; it
> may be that AWS is coming under some massive DDoS attack and you are seeing
> the fringes of it. It could be that your neighbours are noisy -but if you
> are running big Hadoop jobs, you are the noisy neighbour.
>
> * A more likely problem for you is where your machines are placed. If they
> all share a single switch, very high bandwidth. But if they are on different
> racks, the network becomes the bottleneck.
>
>
>
>  But what I don't understand is this... if I *reserve* an instance then I
>> wouldn't be sharing its CPU with anyone, right?  The blog seems to
>> indicate
>> otherwise.
>>
>
> I think you only get exclusive use of a CPU when you rent an XL node.
> Reservations are a form of capacity planning, may or may not help with
> scheduling at all.
>
>
>

I have observed "noisy neighbor" effects.

If you are using HBase EC2 scripts, which run HBase region servers on all of
the slaves colocated with tasktrackers and user tasks, I do not recommend using
other than c1.xlarge instances.

Our scripts use c1.medium instances for the separate Zookeeper quorum ensemble
as they need fewer resources in terms of RAM but are still sensitive to io and
cpu scheduling latencies.

   - Andy

----- Original Message ----
> From: Patrick Hunt <[EMAIL PROTECTED]>
> To: [EMAIL PROTECTED]
> Cc: [EMAIL PROTECTED]
> Sent: Wed, January 27, 2010 3:44:16 AM
> Subject: Re: Performance of EC2
>
> Re "Amazon predictability", did you guys see this recent paper:
> http://people.csail.mit.edu/tromer/cloudsec/
>
> Also some addl background on "noisy neighbor effects":
> http://bit.ly/4O7dHx
> http://bit.ly/8zPvQd
>
> Some interesting bits of information in there.
>
> Patrick
>
> Something Something wrote:
> > Here are some of the answers:
> >
> >>>  How many concurrent reducers run on each node?  Default two?
> > I was assuming 2 on each node would be the default.  If not, this could be a
> > problem.  Please let me know.
> >
> >>> 'd suggest you spend a bit of time figuring where your MR jobs
> > are spending their time?
> > I agree.  Will do some more research :)
> >
> >>> How much of this overall time is spent in reduce phase?
> > Mostly time is spent in the Reduce phases, because that's where most of the
> > critical code is.
> >
> >>> Are inserts to a new table?
> > Yes, all inserts will always be in a new table.  In fact, I disable/drop
> > HTables during this process.  Not using any special indexes, should I be?
> >
> >>> I'm a little surprised that all worked on the small instances, that your
> > jobs completed.
> > But, really, shouldn't Amazon guarantee predictability :)  After all I am
> > paying for these instances.. albeit a small amount!
> >
> >>> Are you opening a new table inside each task or once up in the config?
> > I open HTable in the 'setup' method for each mapper/reducer, and close table
> > in the 'cleanup' method.
> >
> >>> You have to temper the above general rule with the fact that...
> > I will try a few combinations.
> >
> >>> How big is your dataset?
> > This one in particular is not big, but the real production ones will be.
> >  Here's approximately how many rows get processed:
> > Phase 1:  300 rows
> > Phase 2 thru 8:  100 rows.
> > (Note:  Each phase does complex calculations on the row.)
> >
> > Thanks for the help.
> >
> >
> > On Tue, Jan 26, 2010 at 10:36 AM, Jean-Daniel Cryans
> wrote:
> >
> >> How big is your dataset?
> >>
> >> J-D
> >>
> >> On Tue, Jan 26, 2010 at 8:47 AM, Something Something
> >> wrote:
> >>> I have noticed some strange performance numbers on EC2.  If someone can
> >> give
> >>> me some hints to improve performance that would be greatly appreciated.
> >>>  Here are the details:
> >>>
> >>> I have a process that runs a series of Jobs under Hadoop 0.20.1 & Hbase
> >>> 0.20.2  I ran the *exact* same process with following configurations:
> >>>
> >>> 1) 1 Master & 4 Workers (*c1.xlarge* instances) & 1 Zookeeper
> >> (*c1.medium*)
> >>> with *8 Reducers *for every Reduce task.  The process completed in *849*
> >>>  seconds.
> >>>
> >>> 2) 1 Master, 4 Workers & 1 Zookeeper  *ALL m1.small* instances with *8
> >>> Reducers *for every Reduce task.  The process completed in *906* seconds.
> >>>
> >>> 3) 1 Master, *11* Workers & *3* Zookeepers  *ALL m1.small* instances with
> >> *20
> >>> Reducers *for every Reduce task.  The process completed in *984* seconds!
> >>>
> >>>
> >>> Two main questions:
> >>>
> >>> 1)  It's totally surprising that when I have 11 workers with 20 Reducers
> >> it
> >>> runs slower than when I have exactly same type of fewer machines with
> >> fewer
> >>> reducers..
> >>> 2)  As expected it runs faster on c1.xlarge, but the performance
> >> improvement
> >>> doesn't justify the high cost difference.  I must not be utilizing the