Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Bigtop, mail # user - Getting Started Guide? (and some installation issues)


Copy link to this message
-
Re: Getting Started Guide? (and some installation issues)
Sean Mackrory 2013-11-21, 02:10
>> Is there a ‘getting started’ guide?

Beyond just installation, most of our documentation is very
developer-centric, I'm afraid. What there is can be found on our wiki:
https://cwiki.apache.org/confluence/display/BIGTOP/Index

>> Something that will describe the filesystem and configuration file
conventions?

Bigtop is a distribution of other open-source projects, so there is no
single configuration system. The file conventions will vary from project to
project, however Bigtop does not modify much about how the configuration
files work, so I would refer you to the upstream projects for details of
their configuration files (eg. http://hadoop.apache.org,
http://hbase.apache.org)

>> In particular the existence of these conf.empty directories is confusing.

The conf.dist and conf.empty directories provide some default or template
configuration files. You should create a directory at the same level for
your own configuration. Perhaps "conf.steven". There is a symlink for each
component at /etc/<component>/conf. This symlink, through a system called
"alternatives", eventually points to the currently active configuration for
that component. Once you have modified the configuration to suit your
needs, you can make it the active configuration using the alternatives
command. See here for it's documentation:
http://linux.about.com/library/cmd/blcmdl8_alternatives.htm. For example,
if you look at the /etc/hadoop/conf symlink, you will probably find that it
points to /etc/alternatives/hadoop-conf. You can see how the alternatives
are configured and point the configuration to your new folder like this:

alternatives --display hadoop-conf
alternatives --set hadoop-conf /etc/hadoop/conf.steven

>> Is Hue supposed to be configured separately, or is BigTop supposed to do
that?

As I recall, the misconfigurations that are reported at startup are things
like services not running (like Oozie, etc.) Once you configure and start
those services, these warnings should disappear. For other warnings, post
them here and we'll see if we can help you.

>> What is the target time to set-up a Hadoop installation via BigTop?

Not sure what to tell you here. I regularly set up pseudo-distributed
Hadoop installations in minutes with little more than "yum install
hadoop-conf-pseudo", "sudo service hadoop-hdfs-namenode init" and a reboot.
If you're using a bunch of other services on a fully-distributed cluster
and you're completely new to this, I would expect it take hours / days to
get everything running. Bigtop also maintains puppet code that will
configure everything with a pretty good default configuration and have your
cluster working pretty much out-of-the-box. Maybe this is a good option for
you?

>> Can you send me your private email and I will be able to send you my
configuration up to now.

As I mentioned, our documentation is very developer-centric, and as Steven
is showing, some user-centric documentation would be a huge help to the
community. Could I persuade you to share what you've learned on the mailing
list, or perhaps on the wiki so others can benefit?
On Wed, Nov 20, 2013 at 11:04 AM, ivaylo frankov
<[EMAIL PROTECTED]>wrote:

> Hi Steve,
>
> Can you send me your private email and I will be able to send you my
> configuration up to now.
>
> HBase , partly Hive , partly Pig (not much but still something ;))
> I think that experienced specialist needs some hours to configure bigtop
> (at max) Newbee like me needs some days at least ;).
> I want to start pseudo distributed node with hbase pig giraph solr mahout
> and hue. Flume is also interesting.
> What is your desire ?
>
> Best regards,
> Ivo
> P.S: if you want spark be very carefully by installing it : You have to
> give repository or you may be receive some Spark language at least that was
> the case in ubuntu.
>
> Am Mittwoch, 20. November 2013 schrieb i.frankov :
>
> Hi Stive,
>> I am not at home
>>  I will send you my status tonight
>>
>> Cheers,
>> Ivo
>>
>>
>> Von Samsung Mobile gesendet