Hello,

We have a Hadoop cluster running on EC2 with Cloudera's
hadoop-0.20.2-cdh3u2 distribution.  We are now ready to install HBase on
it.  Trying to figure out what's the best way to accomplish this.

We have quite a few machines in the cluster, so installing HBase on each
machine would be time consuming.  But if that's the only way, we can do it
by creating our own RPMs.  Is this document the best resource:
https://ccp.cloudera.com/display/CDHDOC/HBase+Installation#HBaseInstallation-InstallingHBase

Are there ec2 scripts that work with Cloudera's distribution to make this
process easier?

Please help.  Thanks.

We use ec2 and cdh as well and have around 80 Hadoop/hbase nodes deployed across a few different clusters. We use a combination of puppet for package management and fabric scripts for pushing configs and managing services.

Our base AMI is a pretty bare centos6 install and puppet handles most of the rest after spinning up. Puppet also worked fine for managing configs, until we started having many clusters with different setups. That's the point we moved to fabric for that.

There is certainly an investment required for setting this stuff up initially, but it pays off as you continually need to spin up replacements or new nodes. We can do that with only a couple minutes of work at this point.
Sent from iPhone.

On Apr 26, 2012, at 1:12 AM, Something Something <[EMAIL PROTECTED]> wrote:

> Hello,
>
> We have a Hadoop cluster running on EC2 with Cloudera's
> hadoop-0.20.2-cdh3u2 distribution.  We are now ready to install HBase on
> it.  Trying to figure out what's the best way to accomplish this.
>
> We have quite a few machines in the cluster, so installing HBase on each
> machine would be time consuming.  But if that's the only way, we can do it
> by creating our own RPMs.  Is this document the best resource:
> https://ccp.cloudera.com/display/CDHDOC/HBase+Installation#HBaseInstallation-InstallingHBase
>
> Are there ec2 scripts that work with Cloudera's distribution to make this
> process easier?
>
> Please help.  Thanks.

Regards, Bryan.
Can you share with us a blog post or simple a message how do you use
this combination of Puppet and Fabric to manage your clusters on EC2?
Best wishes

On 04/26/2012 01:27 AM, Bryan Beaudreault wrote:
> We use ec2 and cdh as well and have around 80 Hadoop/hbase nodes deployed across a few different clusters. We use a combination of puppet for package management and fabric scripts for pushing configs and managing services.
>
> Our base AMI is a pretty bare centos6 install and puppet handles most of the rest after spinning up. Puppet also worked fine for managing configs, until we started having many clusters with different setups. That's the point we moved to fabric for that.
>
> There is certainly an investment required for setting this stuff up initially, but it pays off as you continually need to spin up replacements or new nodes. We can do that with only a couple minutes of work at this point.
>
>
> Sent from iPhone.
>
> On Apr 26, 2012, at 1:12 AM, Something Something<[EMAIL PROTECTED]>  wrote:
>
>> Hello,
>>
>> We have a Hadoop cluster running on EC2 with Cloudera's
>> hadoop-0.20.2-cdh3u2 distribution.  We are now ready to install HBase on
>> it.  Trying to figure out what's the best way to accomplish this.
>>
>> We have quite a few machines in the cluster, so installing HBase on each
>> machine would be time consuming.  But if that's the only way, we can do it
>> by creating our own RPMs.  Is this document the best resource:
>> https://ccp.cloudera.com/display/CDHDOC/HBase+Installation#HBaseInstallation-InstallingHBase
>>
>> Are there ec2 scripts that work with Cloudera's distribution to make this
>> process easier?
>>
>> Please help.  Thanks.
> 10mo. ANIVERSARIO DE LA CREACION DE LA UNIVERSIDAD DE LAS CIENCIAS INFORMATICAS...
> CONECTADOS AL FUTURO, CONECTADOS A LA REVOLUCION
>
> http://www.uci.cu
> http://www.facebook.com/universidad.uci
> http://www.flickr.com/photos/universidad_uci

--
Marcos Luis Ort�z Valmaseda (@marcosluis2186)
  Data Engineer at UCI
  http://marcosluis2186.posterous.com

10mo. ANIVERSARIO DE LA CREACION DE LA UNIVERSIDAD DE LAS CIENCIAS INFORMATICAS...
CONECTADOS AL FUTURO, CONECTADOS A LA REVOLUCION

http://www.uci.cu
http://www.facebook.com/universidad.uci
http://www.flickr.com/photos/universidad_uci

Our initial implementation was based a lot on
http://hstack.org/hstack-automated-deployment-using-puppet/.  However,
since then we have made many customizations and moved the whole
configuration stack to fabric.  We've been planning a blog post on this,
but we haven't gotten around to writing one yet.  Using puppet for
everything wasn't bad, but we didn't like the way it was evolving as more
and more different clusters were added to our fleet.

On Thu, Apr 26, 2012 at 9:24 AM, Marcos Ortiz <[EMAIL PROTECTED]> wrote:

> Regards, Bryan.
> Can you share with us a blog post or simple a message how do you use this
> combination of Puppet and Fabric to manage your clusters on EC2?
> Best wishes
>
>
> On 04/26/2012 01:27 AM, Bryan Beaudreault wrote:
>
>> We use ec2 and cdh as well and have around 80 Hadoop/hbase nodes deployed
>> across a few different clusters. We use a combination of puppet for package
>> management and fabric scripts for pushing configs and managing services.
>>
>> Our base AMI is a pretty bare centos6 install and puppet handles most of
>> the rest after spinning up. Puppet also worked fine for managing configs,
>> until we started having many clusters with different setups. That's the
>> point we moved to fabric for that.
>>
>> There is certainly an investment required for setting this stuff up
>> initially, but it pays off as you continually need to spin up replacements
>> or new nodes. We can do that with only a couple minutes of work at this
>> point.
>>
>>
>> Sent from iPhone.
>>
>> On Apr 26, 2012, at 1:12 AM, Something Something<mailinglists19@**
>> gmail.com <[EMAIL PROTECTED]>>  wrote:
>>
>>  Hello,
>>>
>>> We have a Hadoop cluster running on EC2 with Cloudera's
>>> hadoop-0.20.2-cdh3u2 distribution.  We are now ready to install HBase on
>>> it.  Trying to figure out what's the best way to accomplish this.
>>>
>>> We have quite a few machines in the cluster, so installing HBase on each
>>> machine would be time consuming.  But if that's the only way, we can do
>>> it
>>> by creating our own RPMs.  Is this document the best resource:
>>> https://ccp.cloudera.com/**display/CDHDOC/HBase+**Installation#**
>>> HBaseInstallation-**InstallingHBase<https://ccp.cloudera.com/display/CDHDOC/HBase+Installation#HBaseInstallation-InstallingHBase>
>>>
>>> Are there ec2 scripts that work with Cloudera's distribution to make this
>>> process easier?
>>>
>>> Please help.  Thanks.
>>>
>> 10mo. ANIVERSARIO DE LA CREACION DE LA UNIVERSIDAD DE LAS CIENCIAS
>> INFORMATICAS...
>> CONECTADOS AL FUTURO, CONECTADOS A LA REVOLUCION
>>
>> http://www.uci.cu
>> http://www.facebook.com/**universidad.uci<http://www.facebook.com/universidad.uci>
>> http://www.flickr.com/photos/**universidad_uci<http://www.flickr.com/photos/universidad_uci>
>>
>
> --
> Marcos Luis Ortíz Valmaseda (@marcosluis2186)
>  Data Engineer at UCI
>  http://marcosluis2186.**posterous.com<http://marcosluis2186.posterous.com>
>
>
>
> 10mo. ANIVERSARIO DE LA CREACION DE LA UNIVERSIDAD DE LAS CIENCIAS
> INFORMATICAS...
> CONECTADOS AL FUTURO, CONECTADOS A LA REVOLUCION
>
> http://www.uci.cu
> http://www.facebook.com/**universidad.uci<http://www.facebook.com/universidad.uci>
> http://www.flickr.com/photos/**universidad_uci<http://www.flickr.com/photos/universidad_uci>