Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
MapReduce >> mail # user >> Best way to collect Hadoop logs across cluster


Copy link to this message
-
Re: Best way to collect Hadoop logs across cluster
Thank you for all the advice, it was indeed very useful.

Mark
On Thu, Apr 18, 2013 at 11:44 PM, Roman Shaposhnik <[EMAIL PROTECTED]> wrote:

> On Thu, Apr 18, 2013 at 9:23 PM, Mark Kerzner <[EMAIL PROTECTED]>
> wrote:
> > Hi,
> >
> > my clusters are on EC2, and they disappear after the cluster's instances
> are
> > destroyed. What is the best practice to collect the logs for later
> storage?
> >
> > EC2 does exactly that with their EMR, how do they do it?
>
> Apache Flume could be extremely useful for this purpose. You
> can even configure it to deposit log data in realtime into
> S3.
>
> Thanks,
> Roman.
>
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB