Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
HBase >> mail # user >> Puppetized HBASE + Passwordless SSH examples


Copy link to this message
-
Re: Puppetized HBASE + Passwordless SSH examples
A couple years ago we started our stack with this as a base:
http://hstack.org/hstack-automated-deployment-using-puppet/

However, we ended up simplifying it quite a bit.  In the end, for us,
puppet just sets up the directories (i.e. data directories for hdfs,
namenode, etc), and installs the services from the right yum repo
(cloudera's or our own).

We then use fabric to push configs to the servers and for managing services.

We chose this approach for a few reasons:

1) We have multiple hbase and hadoop clusters with different
configurations, and managing those differences in puppet configs was a
headache.
2) Using manual pushing of configs over an automated approach (puppet)
gives you much greater control over the rollout of new configurations.  We
found it annoying to have to stop puppet or otherwise gate certain changes
so we could test, for fear of puppet pushing it out everywhere.
3) Similar to how it is best practice to keep markup and presentation (html
and css) separate, we found it nice to keep configuration separate from
service installation/server setup.

We haven't opensourced our fabric-based configuration setup yet
unfortunately, but hopefully the above article and the idea of separating
the two may help.
On Mon, Aug 5, 2013 at 10:52 AM, Jay Vyas <[EMAIL PROTECTED]> wrote:

> Hi everyone .
>
> tl;dr? Any hints on where to start with puppetization of my hbase stack
> would be much appreciated.
>
> Here goes..
>
> This is an open quesiton about puppet and hbase - any deployment stories or
> suggested modules would be appreciated.
>
> Have been looking around at some puppet modules for HBASE, and am what
> experiences people might have had with some of them.  In particular, for
> HBase, you need modules for
>
> - passwordless ssh
> - hbase itself
> - zookeeper
> - hadoop mapred
> - some HCFS file system (HDFS, S3FileSystem, ...).  Obviously, most of the
> time this is HDFS.
> - firewall
>
> So - I'm assuming since HBASE clusters are often dedicated, that somewhere
> out there a mature end-to-end configuration template for puppetizing HBASE
> against brand new servers is in existence, :)
>
>
>
> --
> Jay Vyas
> http://jayunit100.blogspot.com
>